Sign in

Long short-term memory (LSTM)

1997Big moment

Sepp Hochreiter and Jürgen Schmidhuber design a recurrent network that can keep information over long sequences. LSTMs later drive speech recognition and translation.

The story

In 1997 two German computer scientists, Sepp Hochreiter and Jürgen Schmidhuber, published a paper in the journal Neural Computation describing long short-term memory, or LSTM. It is a kind of recurrent neural network, a system that reads a sequence one step at a time, such as words in a sentence or sounds in speech, and carries a memory forward from step to step. LSTM added a memory cell protected by gates, which are small learned switches that decide what information to write, keep or read, and the design let the network remember relevant information across hundreds of steps.

The need came from a mathematical obstacle. Neural networks learn by backpropagation, which adjusts connection weights by passing error signals backward through the network. In a recurrent network the error signal has to travel back through many time steps, and it tends to shrink toward zero (or blow up) on the way. This is the vanishing gradient problem, which Hochreiter analyzed in his 1991 diploma thesis. Ordinary recurrent networks therefore could not link a word at the start of a sentence to one far later. LSTM's cell lets the error flow back almost unchanged, which Hochreiter and Schmidhuber called the constant error carousel. A forget gate was added in 1999 by Felix Gers and colleagues.

For years LSTM was admired but not widely used because computers were slow and data was scarce. That changed in the 2010s, when GPUs and large datasets arrived. LSTMs powered handwriting recognition, then speech recognition at Google around 2015, and later Apple's Siri and Amazon's Alexa. Google Translate switched to a neural system using LSTMs in 2016, which reduced translation errors considerably. In 2017 the Transformer architecture dropped recurrence altogether and took over language modeling, but LSTM remains the best-known solution to the problem of remembering over time.

Why it mattered

  • LSTM-based speech recognition made voice assistants on phones accurate enough for everyday use.
  • Neural machine translation, with LSTMs and attention, replaced phrase-based statistical translation at major companies around 2016.
  • The attention mechanism, first added to recurrent translation networks in 2014, led to the Transformer of 2017.
  • The success of sequence models helped lead to today's large language models.

Drafted by AI (Claude) from the sources above and checked against them by AI; no historian has reviewed this page. Spot a mistake? Suggest a correction. How pages are made

Community notes

Loading notes…

Learn more ranked by votes

BookIan Goodfellow, Yoshua Bengio and Aaron Courville
Deep Learning
Search Open Library
VideoSearch YouTube
LSTM explained
Search YouTube
VideoSearch YouTube
long short-term memory networks
Search YouTube