The story
In October 1986 the journal Nature published a short paper by David Rumelhart, Geoffrey Hinton and Ronald Williams titled 'Learning representations by back-propagating errors.' It described a practical method for training neural networks with several layers, in which the network adjusts its internal connection strengths so that its outputs better match desired answers. Backpropagation, as the method is called, is still the basic learning procedure of nearly all modern artificial intelligence systems.
Early neural networks, such as Frank Rosenblatt's perceptron of 1958, could learn only simple patterns. In 1969 Marvin Minsky and Seymour Papert showed in a book that single-layer perceptrons could not solve certain basic problems, and funding for neural network research fell. Networks with hidden layers could in principle do much more, but no one had a good way of training them. The mathematical idea of running the chain rule of calculus backward through a computation had been found several times, by Seppo Linnainmaa in 1970 and by Paul Werbos, who proposed using it for neural networks in his 1974 thesis, without drawing much attention.
The method works in two passes. In the forward pass, an input is fed through the layers to produce an output, and the error against the correct answer is measured. In the backward pass the error is passed back layer by layer, working out how much each connection contributed to the mistake, and each weight is nudged slightly in the direction that reduces it, a process called gradient descent. Repeated over many examples, this makes the network learn useful internal features by itself. The 1986 paper, together with the book Parallel Distributed Processing edited by Rumelhart and James McClelland, showed networks learning language and pattern tasks and revived the field. Computing power and data were still limited, and it took until the 2010s for the approach to deliver spectacular results.
Why it mattered
- Convolutional neural networks trained with backpropagation learned to read handwritten digits from 1989.
- Long short-term memory networks, designed to be trained by the same method, became a standard tool for speech and text.
- The deep learning breakthrough of 2012, when a large network won the ImageNet image-recognition contest, relied on backpropagation running on fast graphics processors.
- Hinton shared the 2024 Nobel Prize in Physics with John Hopfield for foundational work on artificial neural networks.
Sources
Drafted by AI (Claude) from the sources above and checked against them by AI; no historian has reviewed this page. Spot a mistake? Suggest a correction. How pages are made
Community notes
Loading notes…