Sign in

Word2vec word embeddings

2013

Tomas Mikolov's team at Google trains fast models that map words to vectors capturing meaning. Learned embeddings become standard in language AI.

The story

In 2013 Tomas Mikolov and colleagues at Google published two papers describing word2vec, a pair of small neural network models, called continuous bag-of-words and skip-gram, that learn to represent each word as a list of a few hundred numbers (a vector). The models are trained on a simple guessing game: predict a word from its neighbors, or neighbors from a word. Words used in similar contexts end up with similar vectors, so 'Paris' lands near 'Rome' and far from 'carburetor'.

The idea of learned word vectors was not new; Yoshua Bengio's group had trained a neural language model in 2003. What word2vec added was speed. By dropping expensive layers and using shortcuts such as negative sampling, it could learn from billions of words in hours on an ordinary server, and Google released the code and pretrained vectors in 2013. The vectors also showed surprising structure: subtracting the vector for 'man' from 'king' and adding 'woman' lands close to 'queen', so directions in the space encode relationships. Word2vec soon became a default first step for language software, and rivals such as GloVe (Stanford, 2014) followed. It built on the deep learning breakthrough that made neural networks practical, and its approach of learning meaning from raw text paved the way for the contextual models that came later.

Why it mattered

  • Pretrained embeddings became a standard first layer in language software for search, translation and sentiment analysis through the mid-2010s.
  • The same idea was extended to sentences, documents, products and graphs, powering recommendation and retrieval systems.
  • Contextual models such as ELMo, BERT and the Transformer-based language models behind ChatGPT replaced fixed word vectors but kept the core idea of learning meaning from patterns of use.
  • Researchers found that embeddings absorb social biases from their training text, which opened the study of fairness in language AI.

Drafted by AI (Claude) from the sources above and checked against them by AI; no historian has reviewed this page. Spot a mistake? Suggest a correction. How pages are made

Community notes

Loading notes…

Learn more ranked by votes

BookDaniel Jurafsky and James H. Martin
Speech and Language Processing
Search Open Library
VideoSearch YouTube
word2vec explained
Search YouTube
VideoSearch YouTube
word embeddings explained
Search YouTube