The story
In September 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio posted a paper titled 'Neural Machine Translation by Jointly Learning to Align and Translate'. It showed a way for a translation network to look back at the specific words of the source sentence that mattered most for each word it wrote. That technique, later called attention, became the central building block of modern language models, including ChatGPT and its successors.
The problem they were solving was this. The first neural translation systems, built from recurrent networks such as LSTMs, followed an encoder-decoder design: one network read the whole sentence and squeezed it into a single fixed-length list of numbers, and a second network produced the translation from that summary. A long sentence loses details in so small a container, and translation quality fell sharply as sentences grew. The idea of deep learning, training large layered networks on big datasets, had already made these systems possible, but the bottleneck limited them. Bahdanau, Cho and Bengio changed it by keeping the encoder's output at every word position and letting the decoder compute, at each step, a set of weights saying how much to rely on each one.
The weights are learned during training and behave like a soft, flexible alignment between words, so the model can learn that an adjective in French follows its noun in a way that is reversed in English. The paper's plots of these alignments looked sensible to human readers, which made the method both effective and interpretable. Improvements followed quickly, including a simpler version by Minh-Thang Luong in 2015, and Google's neural translation system adopted attention in 2016. In 2017 a Google team showed that attention alone, with the recurrent layers removed, was enough, which gave rise to the Transformer.
Why it mattered
- The Transformer architecture of 2017 was built entirely around attention and removed the recurrent networks that came before it.
- Large language models such as GPT and Claude rely on attention to relate every word in a passage to every other word.
- Machine translation quality jumped, so tools like Google Translate became far more fluent after the switch to neural models in 2016.
- Attention spread beyond text to images, speech, protein structure prediction and other fields.
Sources
- Neural Machine Translation by Jointly Learning to Align and Translate arXiv
- Effective Approaches to Attention-based Neural Machine Translation arXiv
- Attention Model Statistical Machine Translation survey
Drafted by AI (Claude) from the sources above and checked against them by AI; no historian has reviewed this page. Spot a mistake? Suggest a correction. How pages are made
Community notes
Loading notes…