{"id":926,"date":"2022-10-08T16:48:45","date_gmt":"2022-10-08T16:48:45","guid":{"rendered":"https:\/\/galaxyinferno.com\/?p=926"},"modified":"2023-08-09T12:52:53","modified_gmt":"2023-08-09T12:52:53","slug":"3-lessons-from-the-paper-attention-is-all-you-need-as-a-beginner","status":"publish","type":"post","link":"https:\/\/galaxyinferno.com\/3-lessons-from-the-paper-attention-is-all-you-need-as-a-beginner\/","title":{"rendered":"3 Lessons from the paper “Attention Is All You Need” as a Beginner"},"content":{"rendered":"\n

With the recent surge of news about image, or even art, generating AI<\/strong>, I’m sure I’m not the only one who was interested in how this works under the hood. Most of these results are achieved by so-called “transformer” models and even though it seems like they are coming out just now with the huge media coverage, the idea of these actually started over 5 years ago.<\/p>\n\n\n

\n
\"\"
Current news about DALL-E<\/a>, one of the image generating models developed by OpenAI<\/a> and based on GPT-3<\/a>, a transformer network<\/figcaption><\/figure><\/div>\n\n\n

I wanted to get a glimpse into the transformer-world. So in my personal September study goals<\/a> I decided to read the paper that kicked off the developments in the area of transformers that we have seen in the years since its publications in 2017: “Attention Is All You Need” by Vaswani et al.<\/a>, working for Google at the time. I knew that I wouldn’t understand the whole paper (research papers can be quite overwhelming for people not actively researching in the field), but my goal was to find out where my knowledge gaps might be<\/strong>, so I could start filling them. <\/p>\n\n\n

What does it mean to be a “beginner” in this context?<\/h2>\n\n\n

I studied mathematics and computer science, and in my master’s thesis I focused on theoretical properties of stochastic gradient descent for deep learning. So I do have a good understanding of the general theory of machine learning, but I have limited experience with language translation models or deep learning architectures that process sequences, so I’d call myself a “beginner” in the area of transformers<\/strong>. <\/p>\n\n\n\n

If you are completely new to machine learning<\/strong>, but want to learn how it works and possibly apply it yourself, this post about a scientific research article, might be confusing. In this case, I would redirect you to a beginner-oriented course like Andrew Ng’s Machine Learning<\/a> on Coursera, followed by a course like the Deep Learning Specialization<\/a> for the basics. <\/p>\n\n\n

Lesson 1: Previous architectures not only included Recurrent Neural Networks, but also Convolutional Networks<\/h2>\n\n\n

The attention-based transformer architectures didn’t just replace recurrent neural networks. Another type of architecture that was used for sequence modeling were convolutional neural networks. This surprised me quite a bit. <\/p>\n\n\n\n

During my studies I’ve learned about recurrent network architectures, like LSTMs,<\/strong> in the context of image descriptions or text generation in a specific style. However, I’d only encountered convolutional networks in the context of processing images<\/strong>. When looking at images, the advantage of a convolution is that it looks at a small square of pixels and can discover spatial information in multiple dimensions at the same time. <\/p>\n\n\n\n

This is definitely something I want to read more about, because I currently don’t see why convolutional layers would work well on one dimensional sequence data. <\/p>\n\n\n

Lesson 2: Attention comes in many flavors<\/h2>\n\n\n

In my naive state, I thought attention was like a convolutional layer: it has a specific form and only varies in the size and parameters. <\/p>\n\n\n\n

While reading the paper, I learned instead that there is also a method called self-attention and another called multi-head attention<\/strong>. I won’t pretend to have understood the details, but in essence multi-head attention<\/strong> seems to be a parallelization of attention, each with its own parameters. This reminds me of using multiple neurons per layer in a fully connected neural network, or creating multiple channels from an image in a convolutional layer. Each of these parallel calculations can focus on a specific aspect of the input – in images, this could be vertical lines and horizontal lines for example. <\/p>\n\n\n

\n
\"img_0254\"
Diagram from Figure 2 of “Attention Is All You Need”<\/a> illustrating the parallel nature of multi-head attention<\/figcaption><\/figure><\/div>\n\n\n

Self-attention<\/strong>, on the other hand, is looking for connections between words in the same sentence, like in the following example between “dog” and “his”: <\/p>\n\n\n

\n
\"img_0253\"
Example for a dependency between “dog” and “his” – two words in a single sentence<\/figcaption><\/figure><\/div>\n\n

Lesson 3: Attention itself is just a weighted sum<\/h2>\n\n\n

This insight is very similar to the above lesson, because it turns out that even the “simple” attention can differ in the specific attention function used. The two most commonly used attention functions, however, are both surprisingly simple<\/strong>: <\/p>\n\n\n\n