An introduction to Global Average Pooling in convolutional neural networks

For those familiar with convolutional neural networks (if you’re not, check out this post), you will know that, for many architectures, the final set of layers are often of the fully connected variety. This is like bolting a standard neural network classifier onto the end of an image processor. The convolutional neural network starts with a series of convolutional (and, potentially, pooling) layers which create feature maps which represent different components of the input images. The fully connected layers at the end then “interpret” the output of these features maps and make category predictions. However, as with many things in the fast moving world of deep learning research, this practice is starting to fall by the wayside in favor of something called Global Average Pooling (GAP). In this post, I’ll introduce the benefits of Global Average Pooling and apply it on the Cats vs Dogs image classification task using TensorFlow 2. In the process, I’ll compare its performance to the standard fully connected layer paradigm. The code for this tutorial can be found in a Jupyter Notebook on this site’s Github repository, ready for use in Google Colaboratory.

Global Average Pooling

Global Average Pooling is an operation that calculates the average output of each feature map in the previous layer. This fairly simple operation reduces the data significantly and prepares the model for the final classification layer. It also has no trainable parameters – just like Max Pooling (see here for more details). The diagram below shows how it is commonly used in a convolutional neural network:

Global Average Pooling in a CNN architecture

As can be observed, the final layers consist simply of a Global Average Pooling layer and a final softmax output layer. As can be observed, in the architecture above, there are 64 averaging calculations corresponding to the 64, 7 x 7 channels at the output of the second convolutional layer. The GAP layer transforms the dimensions from (7, 7, 64) to (1, 1, 64) by performing the averaging across the 7 x 7 channel values. Global Average Pooling has the following advantages over the fully connected final layers paradigm:

• The removal of a large number of trainable parameters from the model. Fully connected or dense layers have lots of parameters. A 7 x 7 x 64 CNN output being flattened and fed into a 500 node dense layer yields 1.56 million weights which need to be trained. Removing these layers speeds up the training of your model.
• The elimination of all these trainable parameters also reduces the tendency of over-fitting, which needs to be managed in fully connected layers by the use of dropout.
• The authors argue in the original paper that removing the fully connected classification layers forces the feature maps to be more closely related to the classification categories – so that each feature map becomes a kind of “category confidence map”.
• Finally, the authors also argue that, due to the averaging operation over the feature maps, this makes the model more robust to spatial translations in the data. In other words, as long as the requisite feature is included / or activated in the feature map somewhere, it will still be “picked up” by the averaging operation.

To test out these ideas in practice, in the next section I’ll show you an example comparing the benefits of the Global Average Pooling with the historical paradigm. This example problem will be the Cats vs Dogs image classification task and I’ll be using TensorFlow 2 to build the models. At the time of writing, only TensorFlow 2 Alpha is available, and the reader can follow this link to find out how to install it.

Global Average Pooling with TensorFlow 2 and Cats vs Dogs

To download the Cats vs Dogs data for this example, you can use the following code:

import tensorflow as tf
from tensorflow.keras import layers
import tensorflow_datasets as tfds

split = (80, 10, 10)
splits = tfds.Split.TRAIN.subsplit(weighted=split)

(cat_train, cat_valid, cat_test), info = tfds.load('cats_vs_dogs', split=list(splits), with_info=True, as_supervised=True)

The code above utilizes the TensorFlow Datasets repository which allows you to import common machine learning datasets into TF Dataset objects.  For more on using Dataset objects in TensorFlow 2, check out this post. A few things to note. First, the split tuple (80, 10, 10) signifies the (training, validation, test) split as percentages of the dataset. This is then passed to the tensorflow_datasets split object which tells the dataset loader how to break up the data. Finally, the tfds.load() function is invoked. The first argument is a string specifying the dataset name to load. Following arguments relate to whether a split should be used, whether to return an argument with information about the dataset (info) and whether the dataset is intended to be used in a supervised learning problem, with labels being included. In order to examine the images in the data set, the following code can be run:

import matplotlib.pylab as plt

for image, label in cat_train.take(2):
plt.figure()
plt.imshow(image)

This produces the following images: As can be observed, the images are of varying sizes. This will need to be rectified so that the images have a consistent size to feed into our model. As usual, the image pixel values (which range from 0 to 255) need to be normalized – in this case, to between 0 and 1. The function below performs these tasks:

IMAGE_SIZE = 100
def pre_process_image(image, label):
image = tf.cast(image, tf.float32)
image = image / 255.0
image = tf.image.resize(image, (IMAGE_SIZE, IMAGE_SIZE))
return image, label

In this example, we’ll be resizing the images to 100 x 100 using tf.image.resize. To get state of the art levels of accuracy, you would probably want a larger image size, say 200 x 200, but in this case I’ve chosen speed over accuracy for demonstration purposes. As can be observed, the image values are also cast into the tf.float32 datatype and normalized by dividing by 255. Next we apply this function to the datasets, and also shuffle and batch where appropriate:

TRAIN_BATCH_SIZE = 64
cat_train = cat_train.map(pre_process_image).shuffle(1000).repeat().batch(TRAIN_BATCH_SIZE)
cat_valid = cat_valid.map(pre_process_image).repeat().batch(1000)

For more on TensorFlow datasets, see this post. Now it is time to build the model – in this example, we’ll be using the Keras API in TensorFlow 2. In this example, I’ll be using a common “head” model, which consists of layers of standard convolutional operations – convolution and max pooling, with batch normalization and ReLU activations:

head = tf.keras.Sequential()

head.add(layers.MaxPooling2D(pool_size=(2, 2)))

Next, we need to add the “back-end” of the network to perform the classification.

Standard fully connected classifier results

In the first instance, I’ll show the results of a standard fully connected classifier, without dropout. Because, for this example, there are only two possible classes – “cat” or “dog” – the final output layer is a dense / fully connected layer with a single node and a sigmoid activation.

standard_classifier = tf.keras.Sequential()
standard_classifier.add(layers.Activation('sigmoid'))

As can be observed, in this case, the output classification layers includes 2 x 100 node dense layers. To combine the head model and this standard classifier, the following commands can be run:

standard_model = tf.keras.Sequential([
standard_classifier
])

Finally, the model is compiled, a TensorBoard callback is created for visualization purposes, and the Keras fit command is executed:

standard_model.compile(optimizer=tf.keras.optimizers.Adam(),
loss='binary_crossentropy',
metrics=['accuracy'])

callbacks = [tf.keras.callbacks.TensorBoard(log_dir='./log/{}'.format(dt.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")))]

standard_model.fit(cat_train, steps_per_epoch = 23262//TRAIN_BATCH_SIZE, epochs=10, validation_data=cat_valid, validation_steps=10, callbacks=callbacks)

Note that the loss used is binary crossentropy, due to the binary classes for this example. The training progress over 7 epochs can be seen in the figure below:

Standard classifier accuracy (red – training, blue – validation)

Standard classifier loss (red – training, blue – validation)

As can be observed, with a standard fully connected classifier back-end to the model (without dropout), the training accuracy reaches high values but it overfits with respect to the validation dataset. The validation dataset accuracy stagnates around 80% and the loss begins to increase – a sure sign of overfitting.

Global Average Pooling results

The next step is to test the results of the Global Average Pooling in TensorFlow 2. To build the GAP layer and associated model, the following code is added:

average_pool = tf.keras.Sequential()

pool_model = tf.keras.Sequential([
average_pool
])

The accuracy results for this model, along with the results of the standard fully connected classifier model, are shown below:

Global average pooling accuracy vs standard fully connected classifier model (pink – GAP training, green – GAP validation, blue – FC classifier validation)

As can be observed from the graph above, the Global Average Pooling model has a higher validation accuracy by the 7th epoch than the fully connected model. The training accuracy is lower than the FC model, but this is clearly due to overfitting being reduced in the GAP model. A final comparison including the case of the FC model with a dropout layer inserted is shown below:

standard_classifier_with_do = tf.keras.Sequential()
standard_classifier_with_do.add(layers.Activation('sigmoid'))

Global average pooling validation accuracy vs FC classifier with and without dropout (green – GAP model, blue – FC model without DO, orange – FC model with DO)

As can be seen, of the three model options sharing the same convolutional front end, the GAP model has the best validation accuracy after 7 epochs of training (x – axis in the graph above is the number of batches). Dropout improves the validation accuracy of the FC model, but the GAP model is still narrowly out in front. Further tuning could be performed on the fully connected models and results may improve. However, one would expect Global Average Pooling to be at least equivalent to a FC model with dropout – even though it has hundreds of thousands of fewer parameters. I hope this short tutorial gives you a good understanding of Global Average Pooling and its benefits. You may want to consider it in the architecture of your next image classifier design.

109 thoughts on “An introduction to Global Average Pooling in convolutional neural networks”

1. Hey there! I could have sworn I’ve been to this site before but after browsing through some of the post I realized it’s new to me. Nonetheless, I’m definitely happy I found it and I’ll be bookmarking and checking back frequently!

2. I’m not sure why but this site is loading extremely slow for me. Is anyone else having this issue or is it a problem on my end? I’ll check back later on and see if the problem still exists.

3. This excellent website definitely has all the info I needed concerning this subject and didn’t know who to ask.

4. Thank you so much for this great article about global average pooling. The comparison made between GAP and the conventional architecture with dropout fascinated me.

6. I am trying to get my articles published in a newspaper or magazine. How do I protect my work? .

7. Best view i have ever seen !

8. Best view i have ever seen !

9. Very rapidly this site will be famous amid all blogging users, due to it’s nice
content

10. This post is actually a good one it assists new net visitors, who are wishing for blogging.

11. Wonderful items from you, man. I’ve have in mind your stuff prior to and you are simply too
excellent. I really like what you have received right
here, certainly like what you’re saying and the best way in which you say
it. You make it enjoyable and you continue to take care of to stay
it sensible. I cant wait to read far more from you.
This is actually a great site.

12. Hello this is kinda of off topic but I was wondering if blogs use WYSIWYG editors or if you have to manually code with HTML. I’m starting a blog soon but have no coding skills so I wanted to get guidance from someone with experience. Any help would be enormously appreciated!|

13. Cuando inicialmente hice un comentario, hice clic en la casilla de verificación “Notificarme cuando se agreguen nuevos comentarios” y ahora, cada vez que se agrega un comentario, recibo tres correos electrónicos con el mismo comentario. ¿Hay alguna forma de que pueda eliminarme de ese servicio? ¡Muchas gracias!

14. Is there a way to forward other people’s blog posts to my site?

15. I’ve recently started a webcomic. After viewing some of my favorite other webcomic sites I noticed that they had a ? in every comic posted. My questions are: Do you have to Copyright a webcomic? Is it as simple as just typing in a ? into every comic or do I have to register something somewhere?.

16. How to test if a computer is connected to the network correctly?

17. Unquestionably believe that that you said. Your favourite reason appeared to be
on the net the simplest thing to bear in mind of.
I say to you, I definitely get irked whilst other folks think about worries that
they just do not recognise about. You managed to
hit the nail upon the highest as smartly as outlined out the
whole thing with no need side effect , other people can take a signal.
Will probably be back to get more. Thanks

18. We are a group of volunteers and opening a new scheme in our community.Your web site provided us with valuable info to work on. You’ve done an impressive job and our entire community will be gratefulto you.

19. Greetings! This is my 1st comment here so I just wanted to give a
Can you recommend any other blogs/websites/forums that cover the same subjects?
Thank you!

20. It’s a pity you don’t have a donate button! I’d certainly donate to this excellent blog!

Talk soon!

21. Undeniably imagine that which you stated. Your favorite reason seemed to be at the internet the easiest thing to understand of.I say to you, I definitely get annoyed at the same time aspeople think about issues that they plainlydo not recognize about. You controlled to hit the nail upon the top as neatly as defined out the whole thing without having side effect , other people can take a signal.Will likely be back to get more. Thanks

22. It’s hard to come by knowledgeable people in this particular subject, but you seem like you know what you’re talking about! Thanks

23. where is cbd oil company located?

24. each time i used to read smaller articles or reviews which as well
clear their motive, and that is also happening with this piece
of writing which I am reading at this place.

Here is my web-site :: 추천카지노사이트

25. Best view i have ever seen !

26. Hi! I could have sworn I’ve visited this blog
before but after browsing through some of the posts I realized it’s
new to me. Nonetheless, I’m certainly happy I found it and I’ll be bookmarking it and checking back regularly!

Feel free to surf to my blog post :: 온라인바카라

27. I wijll immediiately clkutch youir rrss aas I caan nott find yohr email subscriptiion hypelink oor newxletter service.

28. It’s in fact very complex in this busy life to listen news
on TV, thus I just use world wide web for that purpose, and get the hottest news.

Also visit my page – 바카라먹튀검증

A design like yours with a few simple adjustements would really make my blog
jump out. Please let me know where you got your design. Appreciate
it

My site 바카라출목표

30. You can definitely see your skills in the work you write. The world hopes for more passionate writers like you who are not afraid to say how they believe. Always go after your heart.

31. Currently it seems like WordPress is the preferred blogging platform available right now.
(from what I’ve read) Is that what you’re using on your blog?

Feel free to visit my homepage :: 모바일바둑이

32. I really love your website.. Very nice colors & theme. Did
you make this website yourself? Please reply back as I’m trying to create my own blog and want
to learn where you got this from or just what the theme is
named. Thanks!

my website :: 텍사스홀덤

33. Why people still make use of to read news papers when in this technological world everything is available on net?

my website Chante

34. What’s up, just wanted to say, I loved this blog post.
It was practical. Keep on posting!

Review my web blog – 텍사스홀덤

35. Heya i am for the primary time here. I found this board and I find It really helpful
& it helped me out much. I hope to provide one thing
back and help others such as you helped me.

Here is my blog :: https://Mwm-Recycling.com

36. Great article, just what I was looking for.

Feel free to surf to my blog; Janet

37. Best view i have ever seen !

38. I got this website from my pal who shared with me concerning this
web page and now this time I am visiting this website and
reading very informative articles at this time.

My website: 토토사이트

39. Great weblog right here! Also your web site loads up very fast!
I desire my site loaded up as fast as yours lol

Also visit my web blog :: Charla

40. This is my first time pay a quick visit at here and
i am truly pleassant to read everthing at alone place.

My site; 메이저놀이터

at this website is genuinely pleasant.

My web page: 토토사이트

42. If somе one desires to bbe updated ԝith ⅼatest technologies tһen he must bе pay a quick
visit tһis site and be up toߋ date everyday.

Look аt myy blog post … целевые промокоды 2021

43. Hi there! Quick question tһɑt’ѕ totally off topic.
Do yoᥙ know hoow t᧐ mаke your site mobile friendly?
My weblog ⅼooks weird when viewing from my iphone. I’m trｙing to find
a template oг plugin that migһt be аble tօ resolve tһiѕ issue.
If yoս hɑve any suggestions, plеase share.
Thank ʏou!

44. This is a topic that’s close to my heart… Many thanks!
Where are your contact details though?

my webpage; 바둑이게임

and found that it’s truly informative. I’m gonna watch out for brussels.
I’ll be grateful if you continue this in future. A lot of people will be benefited from your writing.
Cheers!

Also visit my homepage 바둑이사이트

46. I’m not sure exactly why but this web site is loading very slow for me.
Is anyone else having this issue or is it a problem on my end?
I’ll check back later and see if the problem still exists.

Feel free to surf to my page :: 온라인바둑이

47. Good article! We are linking to this great article on our website.
Keep up the good writing.

Also visit my webpage – 먹튀검증

48. Have you ever considered about adding a little bit more than just your
articles? I mean, what you say is fundamental and all.
However think of if you added some great images or videos to give your posts more, “pop”!
Your content is excellent but with images and video clips, this blog
could certainly be one of the very best in its field. Wonderful blog!

my homepage 릴게임

49. Why viewers still make use of to read news papers when in this technological globe the whole thing is available on net?

My web blog – 바둑이사이트

50. Thanks very interesting blog!

Feel free to visit my website; Lucas

51. fantastic points altogether, you just won a
made a few days in the past? Any positive?

my blog post: 릴게임

52. Wow, this paragraph is good, my sister is analyzing such things, so I
am going to inform her.

Check out my blog post: https://xfunzone.Com/

53. Oh my goodness! Impressive article dude! Thank you so much, However I am encountering problems with
Is there anybody else having similar RSS problems?
Anyone that knows the solution can you kindly respond?

Thanx!!

Here is my webpage: http://www.Keralaplot.com

54. What’s up, I want to subscribe for this webpage to take

Check out my homepage; 릴게임

web people, who are wishing for blogging.

Also visit my webpage libexp.Com

56. Hi there, I found your website by means of Google while searching for a
comparable matter, your website got here up, it seems great.
I have bookmarked it in my google bookmarks.

located that it’s truly informative. I am going to watch
out for brussels. I will be grateful for those who proceed this in future.
Numerous people might be benefited from your writing. Cheers!

Feel free to surf to my web page – Carley

57. Appreciating the dedication you put into your site and in depth information you
provide. It’s nice to come across a blog every once
in a while that isn’t the same out of date rehashed information. Fantastic read!

58. In fact no matter if someone doesn’t understand then its up to other people that they
will assist, so here it takes place.

Feel free to visit my homepage :: 릴게임

59. Hello! Do you know if they make any plugins to protect against hackers?
I’m kinda paranoid about losing everything I’ve worked hard on. Any
recommendations?

Check out my site … 릴게임

60. Hey are using WordPress for your blog platform?
I’m new to the blog world but I’m trying to get started
and set up my own. Do you require any html coding expertise to make your own blog?
Any help would be greatly appreciated!

Also visit my website; 토토사이트

61. I’m not that much of a online reader to be honest but your
sites really nice, keep it up! I’ll go ahead and bookmark your site to come
back later. All the best

Here is my web blog Shad

62. Hello friends, fastidious article and nice urging commented
at this place, I am in fact enjoying by these.

my page – Tod

63. Hi to all, how is the whole thing, I think every one is getting more from this website, and your views
are fastidious designed for new visitors.

Have a look at my blog post … 토토

64. I like the helpful information you provide in your
articles. I will bookmark your blog and check again here frequently.
I am quite certain I will learn plenty of new stuff right here!
Best of luck for the next!

Also visit my webpage 사설토토

65. When I initially commented I clicked the “Notify me when new comments are added” checkbox
and now each time a comment is added I get several emails with the same comment.
Is there any way you can remove people from that service?

Thanks!

My page … Aion21.Kr

66. Can I simply just say what a comfort to find somebody who truly knows what they
are discussing on the net. You actually realize how to bring an issue to light and
make it important. More people really need to look at this and understand
this side of your story. I can’t believe you’re not more popular because you most certainly have the gift.

Look at my web page; 온라인바둑이

67. Good day! I just want to offer you a big thumbs up for your excellent info you have here on this post.
I’ll be coming back to your blog for more soon.

Also visit my page – 바둑이사이트

68. Today, I went to the beachfront with my kids. I found a sea
shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She put the shell to her ear and screamed.
There was a hermit crab inside and it pinched her ear. She never wants to go back!
LoL I know this is completely off topic but I had to tell someone!

Feel free to visit my web site; 바둑이

69. My brother suggested I might like this website. He was entirely
right. This post truly made my day. You cann’t imagine simply how much time I had spent for this information! Thanks!

Also visit my blog post – 바둑이사이트

70. Selamat datang di epicwin138 SITUS ONLINE TERBAIK DENGAN RATE KEMENANGAN 90% , TER AMAN DAN TER PERCAYA

71. Wow, marvelous blog layout! How long have you been blogging for?
you made blogging look easy. The overall look of your site is excellent, as well as the content!

Here is my webpage distance for a dart board

72. Hello, Neat post. Ƭһere’s a proƄlem along with yоur sitee in web explorer,
mіght check this? IE nonethｅless iѕ thhe marketplace lader ɑnd a arge portion of otһeг folks wiⅼl pass over үour fantastic
writing ⅾue too this problem.

Alsso viisit my web blog; traer esto gratis en línea

73. Değer vereceğiniz o çekici şeylerle sınırların ötesinde bir gece deneyimini sizde keşfedecek sizde o özel anlara hemen erişeceksiniz. Fiziği yerinde Bekarların Sitesi ile süper sevişme gecelerini hissetmek sizi gerçek anlamda memnun edecektir. Sizlerde işinde uzman olan Bekarların Sitesi bayan ile gerçek zevkleri keşfedecek artık daha önceki tecrübeleri unutturacak sıcaklıkta samimilikte gelecek güzel bayanları keşfedeceksiniz.

74. Hi there! Thiѕ post ϲould not be written muϲh bettеr!
Ԍoing throᥙgh this post reminmds mе of
my prｅvious roommate! Ηe constantlү қept talking аbout this.
Ι’ll send tһіѕ post tߋ him. Pretty ѕure һe’s going to have a
great read. I aрpreciate yօu f᧐r sharing!

Visiot my boog – прочтите это бесплатно

75. I alwaуs emailed tһiѕ webpage post pagе to all my friends, аs if like to reaɗ
it then my friends will too.

Αlso visit mу web ⲣage :: gratis para unirse en línea

76. TG

77. Pretty section of content. I just stumbled upon your site and in accession capital to assert that I acquire actually enjoyed
account your blog posts. Anyway I will be subscribing
to your feeds and even I achievement you access consistently
rapidly.

My page … 사설토토

78. I savor, cause I found just what I used to be looking for.
You’ve ended my four day lengthy hunt! God Bless you man. Have a nice day.
Bye

My blog – 바둑이

79. After looking into a handful of the blog posts on your web
page, I seriously like your technique of blogging. I saved it
to my bookmark site list and will be checking back in the near future.

Feel free to visit my web site 룰렛게임

80. Excellent web site you’ve gօt hｅгe..
It’s difficult tо fіnd excellent writing lіke yoᥙrs tһese Ԁays.
I honestly aрpreciate individuals ⅼike you! Takе care!!

Also visit my blog; gitar togel sdy

81. This is very attention-grabbing, Үou aｒе an excessively
skilled blogger. Ӏ’ve joined your rss feed aand stay ᥙp for seeking extra of yߋur grreat post.

Alsо visit my site situs judi bola terbesar di asia

82. Yesteгday, wһile I was at work, my sister stole mｙ apple ipad
and tested t᧐ see іf it сan survive a forty
foot drop, jᥙst so she can be a youtube sensation. My iPad is
now ddestroyed and she һas 83 views. І know
this іs completely off topic but I hadd to share it with someօne!

Takе a ⅼook at my һomepage: situs judi bola terbesar

83. Saldoslot : Agen Judi Slot Online Terlengkap & Judi Pulsa Online Terpercaya 2022.

84. We are a group of volunteers and opening a new scheme in our community.
Your site provided us with useful information to
work on. You have performed an impressive process and our
entire group shall be thankful to you.

85. I have been exploring for a little bit for any high-quality articles or blog
posts in this sort of house . Exploring in Yahoo I ultimately stumbled upon this site.

Reading this info So i am glad to show that I have an incredibly just right uncanny feeling I found out just what I needed.
I so much definitely will make sure to don?t overlook this site and give
it a look on a relentless basis.

It is the little changes that will make the biggest changes.

Many thanks for sharing!

87. Good post and right to the point. I don’t know if this is really the best place to ask but do you folks have any thoughts on where to hire some professional writers? Thank you 🙂

88. Hi tһere! This post coսldn’t Ьｅ written any better!

Reading this post reminds mе of my gοod old roоm mate!
He аlways kept twlking abоut this. I will forward tһis write-uⲣ to һim.
Pretty ѕure һｅ will have a ɡood read. Many thanks fⲟr
sharing!

Heree іs mʏ web page :: joker388 login joker123

89. When someone writеs aan article he/ѕhe maintains the thouցht оf a user іn his/һeг ind thɑt how ɑ user can ҝnow it.
So that’s why this paragraph is perfect. Thanks!

Visit my webpage – judi slot online

90. Wow, dartlife blog layout! How long have you been blogging?
you made blogging look easy. The overall look of your site is excellent, as well as the content!

91. Best view i have ever seen !

92. I gotta favorite this internet site it seems very beneficial very useful