Embeddings Explained: How AI Turns Words, Images, and Anything Into Numbers

Every modern AI system, from semantic search to retrieval-augmented generation (RAG), starts with the same trick: convert messy human data into lists of numbers where meaning becomes distance. These lists are called embeddings. Once text, images, or audio live in the same geometric space, a machine can measure similarity with a single line of math. This guide explains how embeddings are built, which models to choose, and how to ship working code.

Quick Takeaways

• An embedding is a fixed-length vector (typically 384 to 3,072 dimensions) that encodes semantic meaning. Similar items sit close together in vector space.
• Similarity is usually measured with cosine similarity, which ranges from -1 to 1.
• Embedding models are separate from chat LLMs. They are smaller, faster, and priced far lower, often $0.02 to $0.13 per 1M tokens for hosted APIs.
• Embeddings power semantic search, RAG, recommendations, clustering, deduplication, and anomaly detection.

Pricing and limits change often. Verify against each provider’s current documentation before budgeting.

What Is an Embedding?

An embedding maps a discrete object (a word, sentence, image, or product) to a point in a continuous vector space. A sentence like “How do I reset my password?” becomes something like [0.021, -0.113, 0.874, ...] with hundreds or thousands of values.

No single number carries a human-readable meaning. Meaning emerges from relative position. The vectors for “reset my password” and “recover account access” land near each other. The vector for “banana bread recipe” lands far away.

This is why embeddings solve a problem keyword search cannot. Keyword matching fails when a user types “car” and your document says “automobile.” Embeddings capture that both refer to the same concept.

Sparse vs. Dense Representations

Older methods represented text as sparse vectors. Embeddings are dense.

In production, teams often combine both in hybrid search to get exact-match precision and semantic recall.

How Embedding Models Are Built

Static Word Embeddings (Word2Vec, GloVe)

Word2Vec (2013) trained a shallow neural network to predict a word from its neighbors (CBOW) or neighbors from a word (Skip-gram). The result was one fixed vector per word, usually 300 dimensions.

The famous result: vector("king") - vector("man") + vector("woman") ≈ vector("queen").

The limitation is that “bank” has one vector whether you mean a river bank or a financial bank.

Contextual Embeddings (Transformers)

Transformer encoders such as BERT produce a different vector for each token depending on its surrounding words. To get a single vector for a whole sentence, models apply pooling, usually mean pooling across token vectors.

Modern sentence embedding models are trained with contrastive learning. The training objective pulls matching pairs together (a question and its answer) and pushes mismatched pairs apart. The common loss is InfoNCE, with in-batch negatives and large batch sizes (often 1,024 or more).

Matryoshka Representation Learning

Newer models, including OpenAI’s text-embedding-3 family, support Matryoshka embeddings. The most important information is packed into the first dimensions, so you can truncate a 3,072-dimension vector to 1,024 or 256 with modest accuracy loss. This cuts storage and search latency substantially.

Measuring Similarity: The Core Math

Three metrics dominate.

Most providers return unit-normalized vectors, so cosine similarity and dot product give identical rankings. Dot product is faster to compute, so use it when you know vectors are normalized.

Step-by-Step: Build Semantic Search in Python

Step 1: Install Dependencies

pip install openai numpy

Step 2: Generate Embeddings with the OpenAI API

from openai import OpenAI
import numpy as np

client = OpenAI()  # reads OPENAI_API_KEY from environment

documents = [
    "How to reset your account password",
    "Best practices for training neural networks",
    "Quarterly revenue report for Q3",
    "Steps to recover access to a locked account",
]

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=documents,
    dimensions=512,  # Matryoshka truncation: 1536 -> 512
)

doc_vectors = np.array([item.embedding for item in response.data])
print(doc_vectors.shape)  # (4, 512)

Step 3: Query and Rank by Cosine Similarity

def embed(text: str) -> np.ndarray:
    r = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
        dimensions=512,
    )
    return np.array(r.data[0].embedding)

query_vec = embed("I forgot my login credentials")

# Vectors are unit-normalized, so dot product == cosine similarity
scores = doc_vectors @ query_vec
ranked = np.argsort(scores)[::-1]

for i in ranked:
    print(f"{scores[i]:.3f}  {documents[i]}")

The two account-related documents rank first even though the query shares no keywords with them.

Step 4: Run Locally with an Open-Source Model

pip install sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-large-en-v1.5")  # 1,024 dims

vectors = model.encode(
    documents,
    normalize_embeddings=True,  # unit vectors for dot-product search
    batch_size=32,
)
print(vectors.shape)  # (4, 1024)

For BGE models, prepend the instruction "Represent this sentence for searching relevant passages: " to queries only, not documents.

Embeddings Beyond Text

The same principle applies to every data type. The key is multimodal alignment, where different modalities share one vector space.

CLIP, for example, trains an image encoder and a text encoder jointly so a photo of a dog and the caption “a dog on a beach” land near each other. You can then search a photo library with plain English.

Vector Databases and Indexing

Comparing a query against every stored vector (brute force) works up to roughly 100,000 vectors. Beyond that, use approximate nearest neighbor (ANN) indexes.

Storage Math

Storage per vector = dimensions × bytes per value. A 1,536-dimension float32 vector uses 6,144 bytes. One million vectors need about 6.1 GB. Switching to int8 quantization cuts that to roughly 1.5 GB, and truncating to 512 dimensions shrinks it further.

Real-World Use Cases and Workflows

RAG Pipeline for Enterprise Knowledge Bases

Enterprises use embeddings to reduce hallucinations by grounding LLM answers in company documents.

1. Chunk documents into 300-500 token segments with 10-15% overlap.
2. Embed each chunk and store it with metadata (source, date, access level).
3. Embed the user query with the same model.
4. Retrieve the top k = 5 to 10 chunks by similarity.
5. Rerank with a cross-encoder to boost precision.
6. Generate an answer with the chunks injected into the prompt, using temperature 0 to 0.2 for factual output.

The query and documents must use the same embedding model. Mixing models produces meaningless similarity scores.

Other High-Value Applications

• Customer support routing: Match incoming tickets to the closest solved tickets or help articles.
• Deduplication: Flag near-identical records with a cosine threshold around 0.95.
• Recommendations: Embed users and items together, then retrieve nearest items to a user’s vector.
• Clustering and topic discovery: Run k-means or HDBSCAN on embeddings to find themes in thousands of survey responses.
• Anomaly detection: Flag logs or transactions far from any cluster centroid.

Common Mistakes and How to Fix Them

Evaluate models on your own data. Public leaderboards such as MTEB (Massive Text Embedding Benchmark) are a useful starting point, but domain-specific performance often differs.

Choosing the Right Embedding Model

Frequently Asked Questions

What is an embedding in simple terms?

An embedding is a list of numbers that represents the meaning of a piece of data. Similar items get similar numbers, so a computer can compare meaning by measuring the distance between vectors.

What is the difference between embeddings and tokens?

Tokens are the text pieces a model reads, such as words or subwords, each mapped to an integer ID. Embeddings are the dense vectors those tokens (or whole passages) are converted into. Tokenization comes first, and embedding comes after.

How many dimensions should an embedding have?

It depends on your accuracy and cost needs. 384 to 768 dimensions suit lightweight search. 1,024 to 1,536 is the common production range. 3,072 offers the highest accuracy but costs more to store and search. Matryoshka models let you truncate without retraining.

Are embeddings the same as LLMs?

No. Embedding models are typically encoder-style networks optimized to output a single vector per input. LLMs are decoder-style models that generate text token by token. They share Transformer foundations but serve different purposes, and embedding models are far cheaper to run.