Sign in

GPT-3

2020

OpenAI's 175-billion-parameter language model shows that scaling lets one model perform many tasks from a few examples.

The story

In May 2020 OpenAI published a paper, 'Language Models are Few-Shot Learners', describing GPT-3, a language model with 175 billion parameters (adjustable numbers that store what a model has learned), more than 100 times larger than GPT-2. It was trained to predict the next word across hundreds of billions of words from the internet, books and Wikipedia. The surprise was what it could do without retraining: given a few examples in its prompt, or just an instruction, it could translate, answer questions, write code and draft essays. In OpenAI's tests, people could barely tell its news-style articles from human-written ones.

GPT-3 was a scaled-up version of the Transformer architecture of 2017, following earlier GPT models, and a January 2020 study of 'scaling laws' had suggested that bigger models trained on more data would improve in predictable ways. GPT-3 provided the demonstration. OpenAI offered it through a paid API, with Microsoft acquiring an exclusive license to the underlying model in September 2020. It also produced convincing falsehoods, repeated biases from its training data, and was hard to control, so researchers developed instruction tuning and human feedback methods to make successors more reliable. Those led to InstructGPT and then ChatGPT in 2022.

Why it mattered

  • ChatGPT and the generative AI boom of 2022 and 2023 grew from refining GPT-3 era models with human feedback.
  • GitHub Copilot, introduced in 2021, descended from a GPT-3 variant trained on code.
  • The 'scaling' strategy of ever larger models and datasets drove huge investment in computing power.
  • Concerns about misinformation, bias and access to powerful models moved to the center of AI policy.

Drafted by AI (Claude) from the sources above and checked against them by AI; no historian has reviewed this page. Spot a mistake? Suggest a correction. How pages are made

Community notes

Loading notes…

Learn more ranked by votes

BookKaren Hao
Empire of AI
Search Open Library
VideoSearch YouTube
GPT-3 language model explained
Search YouTube