Sign in

Transformer architecture

2017Turning point

Google researchers publish "Attention Is All You Need," the design behind modern AI language models.

The story

In June 2017 eight researchers at Google posted a paper titled 'Attention Is All You Need'. It introduced a new design for neural networks, the Transformer, built to translate languages. On standard English to German and English to French tests it beat the best existing systems while taking far less time to train. Within a few years, the Transformer became the foundation of nearly all modern AI language models, including the GPT in ChatGPT, which stands for generative pre-trained transformer.

Before 2017, the leading language networks were recurrent neural networks, which read a sentence one word at a time, passing a summary along like a message in a game of telephone. That made them slow to train, because each step had to wait for the one before, and they often forgot words from early in a long sentence. In 2014 Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio added a mechanism called attention to translation networks, letting them look back at any input word when producing each output word. The deep learning boom that began in 2012 had supplied the GPUs, data and training methods. The Google team's bold idea was to drop recurrence entirely and build the whole network out of attention.

In self-attention, every word in a passage looks at every other word and decides how much each matters to its meaning. In the sentence 'The animal didn't cross the street because it was too tired', attention lets the model link 'it' to 'animal'. The Transformer runs many of these attention 'heads' in parallel, each free to track a different kind of relationship, and stacks layers of them. Because all words are processed at once rather than in sequence, the work suits GPUs and Google's TPU chips perfectly, so models could be trained on far more text.

That scalability turned out to matter most. In 2018 Google's BERT and OpenAI's first GPT showed that Transformers trained on huge amounts of text could be adapted to many tasks, and performance kept improving as models and data grew. The design spread to images, audio, computer code and protein structures. The paper became one of the most cited scientific papers of the century.

Why it mattered

  • OpenAI's GPT-3 in 2020 showed that a giant Transformer could write, summarize and answer questions from a few examples.
  • ChatGPT, launched in 2022, brought Transformer-based AI to hundreds of millions of people.
  • AI image generators combined Transformer-style language understanding with image models to turn text prompts into pictures.
  • Machine translation, search and code completion tools were rebuilt around Transformers.

Drafted by AI (Claude) from the sources above and checked against them by AI; no historian has reviewed this page. Spot a mistake? Suggest a correction. How pages are made

Community notes

Loading notes…

Learn more ranked by votes

BookParmy Olson
Supremacy
Search Open Library
VideoSearch YouTube
3Blue1Brown transformers attention
Search YouTube