What are embeddings?

What it means at work

Ordinary search matches words. Embeddings let a system match ideas: a search for ‘cancel my subscription’ can find a document titled ‘ending your plan.’ The system converts each document, and your question, into numbers and looks for the closest matches. It is a building block behind document search, recommendations and RAG tools.

What the sources say

• What they measure: OpenAI’s guide says its text embeddings measure the relatedness of text strings.
• How closeness works: each embedding is a list of numbers, and the distance between two vectors measures their relatedness. Small distance means closely related text.
• What they are used for: OpenAI’s guide lists search, where results are ranked by relevance to a query string.
• How they are billed: the OpenAI embeddings guide says requests are billed based on the number of tokens in the input.
• Google’s model: the Gemini API docs say gemini-embedding-2 has default 3072-dimension embeddings that are always normalized. The Gemini API docs add that this ensures semantic similarity is calculated via vector direction rather than magnitude.
• Where they fit in RAG: the original retrieval-augmented generation paper pairs a language model with a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. The abstract notes that for plain pre-trained models, providing provenance for their decisions and updating their world knowledge remain open research problems.

An example

A law firm indexes its past memos. An associate searches ‘clauses limiting damages for late delivery’ and gets memos that discuss liability caps and delay penalties, even though none use her exact words.

A common misconception

You will rarely handle embeddings yourself; they are a behind-the-scenes part. The misconception is that they store the documents’ facts. They store a kind of meaning fingerprint used to find things, and the original text still has to be retrieved and read. Another way to picture it is a map: each document gets a location based on its meaning, and documents about similar things land near each other. A search then asks which locations sit closest to the question.

Sources

• OpenAI, Vector embeddings guide: “text embeddings measure the relatedness of text strings”
• OpenAI, Vector embeddings guide: “The distance between two vectors measures their relatedness”
• OpenAI, Vector embeddings guide: “results are ranked by relevance to a query string”
• OpenAI, Vector embeddings guide: “Requests are billed based on the number of tokens in the input”
• Google, Gemini API, Embeddings: “the default 3072-dimension embeddings are always normalized”
• Google, Gemini API, Embeddings: “This ensures semantic similarity is calculated via vector direction rather than magnitude”
• arXiv, Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: “the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever”
• arXiv, Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: “providing provenance for their decisions and updating their world knowledge remain open research problems”