Build a Production RAG Pipeline on Kubernetes (2026 Guide)

In Rajesh Gheware's 25 years building enterprise systems at JPMorgan Chase, Deutsche Bank, and Morgan Stanley, one pattern repeats: teams build a beautiful proof-of-concept, show it to leadership, and watch it collapse in production.

This guide skips the toy tutorials. It covers exactly what production RAG on Kubernetes requires in 2026 — from vector database selection criteria that actually matter, to retrieval quality that does not degrade after week two, to the deployment patterns that survive Monday morning traffic spikes.

Why RAG Pipelines Stall in Production

The typical enterprise RAG failure does not look like a crash. It looks like slow decline: week one, 92% answer accuracy. Week four, 67%. Month three, users stop asking questions and go back to Google.

Three root causes drive nearly every RAG production failure:

1. Retrieval quality was not measured — only generation quality was. Teams optimize for beautiful LLM outputs while the retrieved context degrades silently. The LLM confabulates confidently around poor retrieved chunks.

2. The vector database was not chosen for production scale. Milvus in demo mode handles 100K vectors beautifully. At 50 million vectors with 200 QPS, the same Milvus deployment becomes a latency floor you cannot tune your way out of.

3. Kubernetes was treated as a deployment target, not an architecture. A RAG pipeline that is not designed for Kubernetes-native scaling — horizontal pod autoscaling, PDBs, proper resource limits — will fall over at the exact moment it is most visible.

The fix is not more prompt engineering. It is architectural decisions made before the first chunk is indexed.

The Four-Layer RAG Architecture on Kubernetes

A production RAG pipeline decomposes into four independently scalable layers. Each layer has distinct scaling behavior and failure modes.

Let me walk through each layer with specific Kubernetes configurations that work in enterprise environments.

Layer 1 — Ingestion: Document Processing at Scale

The ingestion layer is batch-oriented. It does not need to run 24/7 — it needs to process large document volumes on demand without overwhelming the vector database. The Kubernetes primitive is a Kubernetes Job triggered by either a CronJob schedule or an event (file upload, webhook, message queue).

Chunking Strategy: The Variable Nobody Tunes

The single highest-impact ingestion decision is chunking strategy. Most tutorials use fixed-size chunks (512 tokens, 50% overlap) and call it done. In Rajesh Gheware's experience training 5,000+ engineers at Fortune 500 companies, this is the most common and most expensive oversight.

Fixed chunking ignores document structure. A 300-page PDF is not a uniform text blob — it has sections, tables, code blocks, and headers. Chunking should respect structure:

• Semantic chunking: Split on header boundaries. Each chunk is a coherent topic.
• Recursive chunking: Start large, recursively split on paragraph then sentence then token until target size.
• Table-aware chunking: Tables need special handling — splitting a table mid-row destroys meaning. Use table-specific chunkers.

For enterprise documents (contracts, regulatory filings, technical manuals), semantic chunking with a 512–1024 token target consistently outperforms fixed chunking by 15–25% on retrieval precision metrics.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document

def semantic_chunk(document: Document, chunk_size: int = 1024,
                   chunk_overlap: int = 128) -> list[Document]:
    """
    Chunk a document respecting semantic boundaries.
    Targets 1024 tokens with 128-token overlap for enterprise docs.
    """
    splitter = RecursiveCharacterTextSplitter(
        separators=["\n\n## ", "\n\n# ", "\n## ", "\n# ", "\n\n", "\n", ". "],
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        length_function=len,
    )
    return splitter.split_documents([document])

def table_aware_chunk(docs: list[Document]) -> list[Document]:
    """Detect and protect table structures during chunking."""
    from langchain.document_transformers import HtmlTableTransformer
    transformer = HtmlTableTransformer()
    transformed = transformer.transform_documents(docs)
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=512, chunk_overlap=64,
        separators=["\n", ". ", " ", ""]
    )
    return splitter.split_documents(transformed)

Embedding Model Selection: 2026 Reality Check

The embedding model market has consolidated significantly in 2026. For enterprise Kubernetes deployments, three models dominate production usage:

Matryoshka representation learning is a critical feature for production: it allows you to truncate embeddings from 3072 to 256 dimensions without retraining, dramatically reducing vector database storage and query latency.

Layer 2 — Vector Store: Choosing and Deploying at Production Scale

The vector database is the load-bearing wall of your RAG pipeline. Pick it wrong and you will be migrating under production load within six months.

Vector DB Selection Criteria for 2026

Not all vector databases handle enterprise scale the same way. Five criteria determine which vector DB survives production in financial services and telecommunications deployments:

1. Filtering performance: Can you filter by metadata (date, document type, region) without scanning the entire index?
2. Horizontal scalability: Does it scale by adding nodes or only by scaling up one node?
3. Consistency model: Strongly consistent (for financial docs) vs. eventual consistent (for chatbots) — the difference matters.
4. Managed vs. self-hosted TCO: At 100M vectors, managed Pinecone costs $6K–$18K/month. Qdrant on Kubernetes costs $2–4K/month infrastructure.
5. ANNS algorithm options: HNSW for highest recall, IVF-Flat for memory efficiency, DiskANN for billion-scale with limited RAM.

Qdrant on Kubernetes: Production Configuration

For enterprise teams choosing self-hosted, Qdrant on Kubernetes offers the best balance of performance, consistency, and cost. Here is the production deployment that Rajesh Gheware's teams use at financial services clients:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: qdrant
  namespace: rag-system
spec:
  serviceName: qdrant
  replicas: 3
  selector:
    matchLabels:
      app: qdrant
  template:
    metadata:
      labels:
        app: qdrant
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant:v1.7.4
          ports:
            - containerPort: 6333  # REST API
            - containerPort: 6334  # gRPC
          resources:
            requests:
              memory: "32Gi"
              cpu: "4"
              nvidia.com/gpu: "1"
            limits:
              memory: "64Gi"
              cpu: "8"
          env:
            - name: QDRANT__SERVICE__GRPC_PORT
              value: "6334"
            - name: QDRANT__STORAGE__IN_MEMORY
              value: "false"
            - name: QDRANT__STORAGE__WAL__ENABLED
              value: "true"
          volumeMounts:
            - name: qdrant-storage
              mountPath: /qdrant/storage
          readinessProbe:
            httpGet:
              path: /readyz
              port: 6333
            initialDelaySeconds: 10
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: 6333
            initialDelaySeconds: 30
      volumes:
        - name: qdrant-storage
          persistentVolumeClaim:
            claimName: qdrant-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: qdrant
  namespace: rag-system
spec:
  clusterIP: None
  selector:
    app: qdrant
  ports:
    - port: 6333
      name: rest
    - port: 6334
      name: grpc

Layer 3 — Retrieval: Getting Relevant Chunks Every Time

Retrieval is where most RAG pipelines quietly fail. The vector search works — but the wrong documents come back. Users get plausible but incorrect answers.

Hybrid Search: The Production Standard

Pure dense vector search (embedding then ANN then top-k) misses exact keyword matches and document-specific terminology. A medical RAG system searching for "hypertension" needs both "high blood pressure" (semantic) and "hypertension" (keyword).

Hybrid search combines dense vector retrieval with sparse retrieval (BM25 or SPLADE). This is not optional for enterprise — it is the baseline. In Rajesh Gheware's evaluation across 12 enterprise RAG deployments, hybrid search outperformed pure vector search in 87% of cases.

import weaviate
from weaviate.auth import AuthApiKey

def hybrid_retrieval(query: str, namespace: str = "default",
                     top_k: int = 10, alpha: float = 0.7) -> list[dict]:
    """
    Hybrid search: alpha=0.7 means 70% vector, 30% keyword (BM25).
    Tune alpha based on your use case:
      - alpha > 0.7: more semantic (legal, research)
      - alpha < 0.5: more keyword-matching (support docs, policies)
    """
    client = weaviate.Client(
        url="http://weaviate.rag-system:8080",
        auth_client_secret=AuthApiKey("WEAVIATE_API_KEY")
    )

    result = client.query.get(
        class_name="Document",
        properties=["chunk_text", "source_doc", "page", "section"]
    ).with_hybrid(
        query=query,
        alpha=alpha,
        limit=top_k
    ).with_additional(["score", "rerank"]).do()

    chunks = []
    for item in result["data"]["Get"]["Document"]:
        chunks.append({
            "text": item["chunk_text"],
            "source": item["source_doc"],
            "page": item["page"],
            "section": item.get("section", ""),
            "score": item["_additional"]["score"]
        })
    return chunks

def rerank_chunks(query: str, chunks: list[dict], top_n: int = 5) -> list[dict]:
    """Cross-encoder reranking for precision-critical applications."""
    from sentence_transformers import CrossEncoder
    model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")
    pairs = [(query, chunk["text"]) for chunk in chunks]
    scores = model.predict(pairs)
    scored = sorted(zip(chunks, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in scored[:top_n]]

Query Routing: The Pattern That Scales Retrieval Quality

Not every query needs the same retrieval strategy. A customer asking "What is my account balance?" needs exact numeric retrieval. A researcher asking about risk factors needs semantic retrieval.

For Kubernetes deployments, implement query routing as a separate microservice that classifies the incoming query type and routes accordingly. This adds approximately 5ms latency but eliminates the single biggest source of retrieval failure.

Layer 4 — Generation: Assembling Answers That Do Not Confabulate

Once retrieval is solid, generation is straightforward. The Kubernetes pattern is a stateless inference service (vLLM or TGI) behind a Service, with HPA scaling on CPU or GPU utilization.

Context Assembly: The Prompt Engineering That Matters

The generation prompt is not where you should get creative. Use a structured template that states the task explicitly, includes source attribution, sets the refusal boundary, and provides response format instructions.

def build_rag_prompt(user_query: str, retrieved_chunks: list[dict]) -> str:
    """
    Build a production-grade RAG prompt with source attribution.
    Citation requirement cuts confabulation rates by 40% in enterprise testing.
    """
    context_parts = []
    for i, chunk in enumerate(retrieved_chunks, 1):
        context_parts.append(
            f"[Source {i}] {chunk['source']} (Page {chunk['page']}):\n{chunk['text']}"
        )
    context = "\n\n".join(context_parts)

    prompt = f"""You are an enterprise AI assistant. Answer the user's question using ONLY the context provided below.

INSTRUCTIONS:
- Answer based strictly on the provided context
- For each major claim, cite the source using [Source N] notation
- If the context does not contain sufficient information, say: "Based on the available documents, I cannot fully answer this question."
- Do NOT use knowledge outside the provided context
- Be precise and concise

CONTEXT:
{context}

USER QUESTION: {user_query}

ANSWER:"""
    return prompt

def build_citation(chunks: list[dict]) -> str:
    """Generate source citations block for the response footer."""
    citations = []
    for i, chunk in enumerate(chunks, 1):
        citations.append(f"[{i}] {chunk['source']} — Page {chunk['page']}")
    return "\nSources:\n" + "\n".join(citations)

Production Kubernetes Deployment: End-to-End YAML

Putting it together, here is the complete Kubernetes deployment for a production RAG pipeline. This includes all four layers with proper HPA, PDB, resource limits, and network policies:

apiVersion: v1
kind: Namespace
metadata:
  name: rag-system
  labels:
    app.kubernetes.io/part-of: rag-pipeline
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: rag-retrieval-hpa
  namespace: rag-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rag-retrieval
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Pods
      pods:
        metric:
          name: qps
        target:
          type: AverageValue
          averageValue: "50"
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: rag-inference-hpa
  namespace: rag-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rag-inference
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: nvidia.com/gpu
        target:
          type: Utilization
          averageUtilization: 75
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: rag-network-policy
  namespace: rag-system
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: qdrant
        - podSelector:
            matchLabels:
              app: rag-inference
    - to:
        - namespaceSelector: {}

The 7 Production Patterns That Separate Lab from Reality

After deploying RAG pipelines at scale across financial services, telecom, and healthcare clients, seven patterns consistently separate systems that survive production from systems that get quietly decommissioned after three months:

1. Measure retrieval quality continuously. Track recall@k and MRR@k in production. If retrieval precision drops, generation quality drops with it — invisibly.
2. Index incrementally, not in batches. Full re-indexing causes retrieval gaps. Use incremental indexing with change data capture (CDC) from your document store.
3. Separate embedding generation from ingestion. Embedding APIs have rate limits. Decouple with a message queue (Kafka, RabbitMQ) so bursts do not overwhelm API quotas.
4. Use namespaces for multi-tenant isolation. Isolate vector DB namespaces at the database level — not just at the query level.
5. Set retrieval latency SLOs, not just generation SLOs. If your vector DB p99 exceeds 200ms, no prompt engineering fixes the user experience.
6. Implement stale retrieval detection. Track when documents were last updated. Flag retrieved chunks from documents older than threshold (30 days for policies, 7 days for support docs).
7. Design for graceful degradation. If vector DB is unavailable, fall back to keyword-only search. Never let the entire RAG pipeline go dark because one dependency fails.

Getting Your Team Production-Ready

Building a production RAG pipeline requires a specific set of skills that most teams are missing: Kubernetes-native deployment, vector database tuning, retrieval evaluation, and LLM prompt engineering. These are not theoretical skills — they require hands-on labs with production-scale datasets.

The Agentic AI Workshop by gheWARE covers end-to-end RAG pipeline architecture, from document ingestion through production Kubernetes deployment, with 119 hands-on labs. Teams at Oracle scored it 4.91 out of 5.0 for its practical, production-focused approach. The curriculum includes dedicated modules on vector database selection, hybrid search implementation, and retrieval evaluation.

Conclusion

The gap between a RAG proof-of-concept and a production RAG system is not about better prompts or smarter LLMs. It is about treating retrieval as a first-class engineering discipline, choosing infrastructure that survives production scale, and measuring quality at every layer.

The Kubernetes architecture described here — four independent scaling layers, hybrid search, production-grade vector DB configuration, and continuous retrieval monitoring — is the foundation that Rajesh Gheware's teams have deployed at financial services and enterprise clients. It works because every architectural decision was made for production failure modes, not for demo aesthetics.

If your team is building a RAG pipeline and wants a structured, hands-on path to production without the expensive trial-and-error, the Agentic AI Workshop provides the curriculum, labs, and implementation roadmap that gets you there in five days.