The story
In the autumn of 2012 a team from the University of Toronto, graduate students Alex Krizhevsky and Ilya Sutskever and their professor Geoffrey Hinton, won the ImageNet image-recognition challenge by a margin that stunned the field. Their program, later called AlexNet, had to label photos with one of 1,000 categories, from 'container ship' to 'leopard'. Allowed five guesses per image, it was wrong 15.3 percent of the time; the next best entry was wrong 26.2 percent of the time. AlexNet was a deep neural network, a program loosely inspired by the brain that learns patterns from examples rather than following rules written by people.
The ingredients had been gathering for decades. Neural networks dated back to the 1950s but had fallen out of favor more than once. In 1986 Hinton, David Rumelhart and Ronald Williams popularized backpropagation, a way for a network to adjust millions of internal settings, called weights, by measuring its error and nudging each weight to reduce it. In 1989 Yann LeCun built convolutional neural networks, which scan an image in small patches the way a magnifying glass would, to read handwritten digits. What had been missing was data and computing power. The web supplied the data: Fei-Fei Li's ImageNet project, launched in 2009, gathered over 14 million labeled images. Graphics processing units (GPUs), designed for video games, supplied the power, because they can do thousands of simple calculations at once.
AlexNet had eight learning layers and about 60 million weights, and it was trained for several days on two consumer Nvidia graphics cards. It used tricks that became standard, such as a simple activation function called ReLU that trained faster and 'dropout', which randomly switches off parts of the network during training so it cannot just memorize examples.
The result convinced researchers and companies almost overnight. Within a few years nearly every competitor in the challenge used deep learning, and error rates fell below human performance on the task. Google bought Hinton's small company in 2013, and tech giants raced to hire neural network experts. Hinton, LeCun and Yoshua Bengio shared the 2018 Turing Award.
Why it mattered
- Speech recognition, translation and photo search improved rapidly as deep learning spread across tech products.
- Word2vec in 2013 and neural attention for translation in 2014 carried deep learning into language.
- AlphaGo defeated top Go champion Lee Sedol in 2016, a feat many experts had expected to be a decade away.
- The Transformer architecture of 2017 and later AI image generators built directly on the deep learning boom.
- AlphaFold applied the same approach to predicting protein structures, solving a 50-year problem in biology.
Sources
Drafted by AI (Claude) from the sources above and checked against them by AI; no historian has reviewed this page. Spot a mistake? Suggest a correction. How pages are made
Community notes
Loading notes…