Embeddings Explained: The Math-Free Guide Every AI Engineer Needs

What Is an Embedding? (No Math Required)

Imagine you could place every English sentence somewhere in a vast library — not alphabetically, but by meaning. Sentences about Kubernetes networking would cluster in one section. Sentences about Python exception handling in another. Sentences about database indexing in a third. The distance between any two sentences in this library represents their semantic similarity: closer = more similar meaning.

An embedding is the address of a piece of text in that library, expressed as a list of numbers. all-MiniLM-L6-v2 gives every text a 384-number address. text-embedding-3-small gives every text a 1,536-number address. The address itself is meaningless — what matters is that similar texts have similar addresses.

When you ask a RAG system "How do I configure a Kubernetes readiness probe?", the system:

1. Converts your question to its embedding (address)
2. Searches the vector store for document chunks whose addresses are closest to your question's address
3. Retrieves the top-K closest chunks
4. Passes those chunks as context to the LLM
5. The LLM generates an answer grounded in that context

The entire system's quality depends on step 2 — whether the retrieved chunks actually contain the information needed to answer the question. That retrieval quality depends on two things: the embedding model's ability to capture semantic meaning, and the chunking strategy's ability to pack meaningful information into each chunk.

Choosing an Embedding Model: The Decision Framework

Decision rule: Start with all-MiniLM-L6-v2 for technical documentation (DevOps runbooks, API docs, code). It runs locally (no API dependency, no per-token cost), has 384 dimensions (fast similarity search), and consistently outperforms ada-002 on technical content in head-to-head benchmarks. Only switch to text-embedding-3-small if you need cloud-managed infrastructure and cannot run local models. Only switch to bge-large-en-v1.5 if you need maximum accuracy and have the RAM budget (2GB for model weights).

Chunking Strategies: Why They Matter More Than Model Choice

Chunking is the process of splitting your source documents into smaller pieces before embedding them. Each chunk becomes one entry in your vector store. When a query arrives, the top-K most similar chunks are retrieved.

The chunking decision determines whether retrieved chunks are actually useful for answering questions. Too small: each chunk lacks enough context to stand alone (a single sentence about a Kubernetes concept without the surrounding explanation is nearly useless). Too large: each chunk contains too much information, diluting the semantic signal and causing the embedding to represent the average of many concepts rather than one clear topic.

The Four Chunking Strategies

Fixed-size chunking: Split on token count (e.g., every 512 tokens) with overlap (e.g., 51 tokens = 10%). Simple, consistent, works well for homogeneous prose. The overlap ensures that context at chunk boundaries is not lost. Best for: general documentation, blog posts, articles.

Semantic chunking: Split on topic shifts — when the embedding similarity between adjacent sentences drops below a threshold, start a new chunk. Produces chunks that each cover one coherent concept. Slower to compute but produces significantly better retrieval for heterogeneous documents. Best for: long documents with many distinct sections.

Document structure chunking: Split on document structure markers (H1, H2, H3, table boundaries, code block boundaries). Each chunk corresponds to one structural unit. Best for: structured documentation with clear headings.

Recursive chunking: Attempt to split on paragraphs first; if a paragraph exceeds the target size, split on sentences; if a sentence exceeds the target, split on tokens. LangChain's RecursiveCharacterTextSplitter implements this. Best for: mixed-format documents (prose + code + tables).

The 3 Production Mistakes That Kill RAG Quality

Mistake 1: Wrong Chunk Size (the Most Common)

Default chunk sizes (LangChain's default is 1000 characters ≈ 250 tokens) are not optimised for any specific content type. For technical documentation where each paragraph covers one concept, 250 tokens is often too small — the chunk ends mid-explanation. For dense API reference documentation, 250 tokens might be too large — you retrieve an entire method's documentation when the user asked about one parameter.

Fix: Measure retrieval accuracy (hit rate and mean reciprocal rank) on a sample of 50 questions with known correct chunks. Test chunk sizes of 128, 256, 512, and 1024 tokens. Choose the size that maximises retrieval accuracy for your specific content.

Mistake 2: Mixing Embedding Models

Vectors from different models cannot be compared. If you embed 80% of your documents with all-MiniLM-L6-v2 and then switch to text-embedding-3-small for new documents (even temporarily), your vector store contains numerically incomparable vectors. Similarity searches will return incorrect results for queries that happen to be compared against the mixed vectors.

Fix: One vector store = one embedding model. If you switch models, delete and re-create the vector store from scratch. Track your embedding model version in metadata alongside each document.

Mistake 3: Not Re-Embedding When Documents Change

A document's embedding represents its content at the time of embedding. When the document is updated — a configuration parameter changes, a new API endpoint is added, a deprecation notice is added — the embedding remains stale. Queries about the new content return the old chunk, and the LLM answers based on outdated information.

Fix: Track document modification timestamps in metadata. On each document update (detected via git commit, CMS webhook, or file watcher), re-embed the changed chunks and update the vector store. This is a standard LLMOps pipeline task, not a manual process.

Python Code: Model Comparison and Production Chunking

# pip install sentence-transformers langchain-text-splitters chromadb

from sentence_transformers import SentenceTransformer
import numpy as np
import time

# Load three models for comparison
# WHY compare: benchmarks on generic datasets don't predict performance
# on YOUR specific documents. Always measure on your own content.
models = {
    "all-MiniLM-L6-v2": SentenceTransformer("all-MiniLM-L6-v2"),       # 384 dims
    "bge-large-en-v1.5": SentenceTransformer("BAAI/bge-large-en-v1.5"),# 1024 dims
}

# Sample technical documentation from our workshop runbook
query = "How do I configure resource limits for a Kubernetes pod?"

candidate_chunks = [
    "To set resource limits for a pod, add a resources section to the container spec with limits and requests for CPU and memory.",
    "Kubernetes uses namespaces to isolate workloads. Each namespace can have resource quotas applied.",
    "A Deployment manages a set of identical pods, ensuring the specified number of replicas are running.",
    "Pod resource limits define the maximum CPU and memory a container can consume. If exceeded, the container is OOMKilled.",
]

print("Model Comparison on Technical Query")
print("="*60)

for model_name, model in models.items():
    start_time = time.time()
    query_embedding = model.encode([query])
    chunk_embeddings = model.encode(candidate_chunks)
    latency_ms = (time.time() - start_time) * 1000

    # Cosine similarity
    similarities = np.dot(query_embedding, chunk_embeddings.T)[0] / (
        np.linalg.norm(query_embedding) * np.linalg.norm(chunk_embeddings, axis=1)
    )

    ranked = sorted(zip(similarities, candidate_chunks), reverse=True)
    print(f"\n{model_name} (dims={model.get_sentence_embedding_dimension()}, {latency_ms:.0f}ms)")
    print(f"Top result: [{ranked[0][0]:.3f}] {ranked[0][1][:60]}...")

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from datetime import datetime

# Production chunking strategy for technical documentation
# WHY RecursiveCharacterTextSplitter: tries paragraph splits first,
# falls back to sentence, then token. Preserves logical units.
# chunk_size=512 tokens (~2048 chars), overlap=10% = 204 chars
splitter = RecursiveCharacterTextSplitter(
    chunk_size=2048,           # ~512 tokens for technical docs
    chunk_overlap=204,         # 10% overlap preserves context at boundaries
    separators=[
        "\n\n",    # paragraph break — preferred split point
        "\n",      # line break
        ". ",      # sentence end
        ", ",      # clause break
        " ",       # word break
        "",        # character (last resort)
    ],
    length_function=len,
)

# Sample workshop runbook document
document = """
## Kubernetes Resource Management

### Setting Resource Requests and Limits

Resource requests and limits control how Kubernetes allocates compute
resources to containers. Requests are used for scheduling decisions:
the scheduler places pods on nodes with sufficient unreserved resources.
Limits enforce the maximum resources a container can consume.

```yaml
resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"
    cpu: "500m"
```

If a container exceeds its memory limit, the OOM killer terminates it
(OOMKilled). If it exceeds its CPU limit, it is throttled — not killed.

### Resource Quotas

ResourceQuota objects limit aggregate resource consumption per namespace.
Teams sharing a cluster should have namespace quotas to prevent one workload
from consuming all cluster resources.
"""

chunks = splitter.split_text(document)

# Embed with local model — no API key, no per-token cost
embedding_model = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

# Store in ChromaDB with version metadata
# WHY metadata: enables re-embedding detection when documents change
vectorstore = Chroma.from_texts(
    texts=chunks,
    embedding=embedding_model,
    metadatas=[{
        "source": "workshop-runbook-k8s.md",
        "embedding_model": "all-MiniLM-L6-v2",   # CRITICAL: track model version
        "embedded_at": datetime.now().isoformat(),
        "doc_version": "v2.1.0"                   # track document version
    }] * len(chunks)
)

# Retrieve
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
results = retriever.invoke("What happens when a pod exceeds memory limit?")
for i, doc in enumerate(results):
    print(f"Result {i+1}: {doc.page_content[:100]}...")

Conclusion: Embeddings Are Infrastructure, Not Magic

The teams that build the most reliable RAG systems are not the ones using the most expensive embedding models. They are the ones that treat embeddings as infrastructure: they choose a model appropriate for their content type, they design their chunking strategy based on measured retrieval accuracy, they version-track their embedding models, and they build re-embedding pipelines that run automatically when documents change.

In Day 2 of the Agentic AI Workshop, we build a complete RAG pipeline from scratch — document loading, chunking strategy comparison, vector store creation, and retrieval evaluation. Every participant measures retrieval accuracy at three different chunk sizes on their own sample data before choosing a configuration. By end of day, the difference between a 41% retrieval hit rate (default settings) and a 78% hit rate (tuned chunking) is not theoretical — it is visible on their own laptop, in their own ChromaDB instance.