Embedding Models Explained: From Word2Vec to text-embedding-3

What Are Embeddings?

An embedding is a numerical representation of data — text, images, audio — as a vector of floating-point numbers. Similar items have vectors that are close together in this high-dimensional space. Dissimilar items are far apart.

This simple idea powers modern search, recommendations, anomaly detection, and retrieval-augmented generation (RAG). If you are building any AI-powered feature, you need to understand embeddings.

The Evolution of Text Embeddings

2013: Word2Vec

Google's Word2Vec was the breakthrough that started it all. It learned word relationships from raw text:

king - man + woman = queen
paris - france + germany = berlin

Each word became a 300-dimensional vector. Words used in similar contexts had similar vectors.

Limitation: One vector per word. "Bank" (river) and "bank" (financial) had the same embedding.

2018: BERT Embeddings

BERT introduced contextual embeddings. The same word got different vectors depending on surrounding text:

from transformers import AutoModel, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

text = "The bank of the river was steep"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# "bank" here gets a river-related embedding
embedding = outputs.last_hidden_state.mean(dim=1)

768 dimensions. Much better at understanding meaning, but slow and not designed for similarity search.

2022: Sentence Transformers

Models specifically trained for similarity search. The key innovation: they were trained with contrastive learning — push similar sentences together, push dissimilar ones apart.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

sentences = [
    "How to deploy Docker containers",
    "Docker container deployment guide",
    "Best pizza recipe in New York"
]

embeddings = model.encode(sentences)

# First two will be close together
# Third will be far from both

384 dimensions. Fast, accurate, and open-source.

2024-2025: text-embedding-3 and Beyond

OpenAI's text-embedding-3 family represents the current state of the art for API-based embeddings:

• text-embedding-3-small: 1536 dimensions, $0.02/1M tokens
• text-embedding-3-large: 3072 dimensions, $0.13/1M tokens

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Deploy a PostgreSQL database with automated backups"
)

vector = response.data[0].embedding  # 1536 floats

A unique feature: you can reduce dimensions while preserving quality:

response = client.embeddings.create(
    model="text-embedding-3-large",
    input="Your text here",
    dimensions=256  # Reduce from 3072 to 256
)

How Similarity Search Works

Two vectors are compared using cosine similarity — the cosine of the angle between them:

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Returns 0.0 to 1.0 (for normalized vectors)
# 1.0 = identical meaning
# 0.0 = completely unrelated

For efficient search over millions of vectors, use approximate nearest neighbor (ANN) algorithms implemented in vector databases:

• pgvector: PostgreSQL extension (great if you already use Postgres)
• ChromaDB: Simple, developer-friendly
• Qdrant: High performance, Rust-based
• Pinecone: Managed service (cloud only)

Choosing the Right Model

For self-hosted deployments (which we recommend at TechSaaS), nomic-embed-text via Ollama gives excellent results without external API calls:

ollama pull nomic-embed-text

curl http://localhost:11434/api/embeddings \
  -d '{"model": "nomic-embed-text", "prompt": "Your text here"}'

Practical Tips

1. Chunk your text: Embeddings work best on 256-512 token chunks, not entire documents
2. Normalize vectors: Pre-normalize for faster cosine similarity (just dot product)
3. Batch requests: Embed multiple texts in one API call to reduce latency
4. Cache aggressively: Store embeddings in your database — never re-embed unchanged text
5. Test with your data: Benchmark models on YOUR domain, not generic benchmarks

Embeddings are the foundation of modern AI applications. Whether you are building search, RAG, or recommendations, understanding how they work gives you a massive advantage.