Build Your Enterprise AI Control Plane

The Cost of No Control Plane

In Rajesh Gheware's 25 years building systems at JPMorgan Chase, Deutsche Bank, and Morgan Stanley, Rajesh Gheware has seen the same pattern repeat across every new technology wave: teams rush to production without governance, then spend 18 months retrofitting what they should have built first.

AI agents are no different — except the blast radius is larger. When a payment gateway microservice misbehaves, it affects one transaction. When an AI agent in your OTC derivatives workflow makes unauthorized decisions, you have a regulatory incident on your hands.

Consider what "no control plane" actually means in practice:

• Runaway costs: Agents that loop or re-call LLM APIs can generate $50,000+ in API charges in a single weekend. gheWARE has seen this pattern: a code-review agent that gets into a recursive self-correction loop, calling an LLM API hundreds of times per pull request.
• Compliance violations: EU AI Act Article 14 requires that AI systems be accurate, robust, and compliant. Without audit trails generated at the control plane layer, you cannot prove compliance.
• Security breaches: Agents with tool access to internal APIs, databases, or file systems are prime targets for prompt injection. An ungoverned agent is an attacker who has already compromised your infrastructure from the inside.
• Agent chaos: At one data-platform company, teams discovered that eight different AI agents were duplicating each other's work — creating conflicting data transformations because no orchestration layer existed.

The fix is not a policy document. It is an agentic AI control plane — a production-grade management layer that treats AI agents the way Kubernetes treats containers: with orchestration, resource limits, networking rules, and observability baked in from day one.

What Is an Agentic Control Plane?

Think of an agentic control plane as three things your platform team already knows how to build, combined into one system:

1. A Kubernetes Operator that manages the lifecycle of AI agents — create, update, scale, terminate — with custom resource definitions (CRDs) for agent configurations
2. An admission webhook layer that validates agent requests before they are dispatched — checking prompts for PII, tool requests against scope, cost budgets against limits
3. An observability sidecar that traces every agent action, LLM call, and tool invocation using OpenTelemetry — feeding data to Langfuse for cost analysis and latency percentiles

The inspiration is the Kubernetes control plane itself: API server, etcd, scheduler, controller manager. The agentic AI control plane uses the same patterns because they already work at enterprise scale for the most critical workloads on the planet.

What it is NOT: a monolithic agent framework lock-in. The control plane orchestrates agents built with LangGraph, AutoGen, CrewAI, or custom frameworks. It governs HOW they run, not WHAT they compute.

Model Context Protocol (MCP) servers are the "kubectl" of the agentic world — a standardized interface for agents to connect to tools and data sources. MCP hit 97 million installs in 16 months, becoming the USB standard for AI agent connectivity.

The Five Pillars of an Enterprise Agent Control Plane

Pillar 1: Agent Orchestration and Lifecycle Management

The control plane must own agent creation, versioning, scaling, and termination. This goes beyond starting an agent process:

• CRD-based agent definitions — store agent configs (model, temperature, tool permissions, memory settings) as Kubernetes custom resources
• Controller-driven reconciliation — the operator continuously reconciles desired state with actual state, automatically restarting failed agents
• HPA integration — scale agent replicas based on queue depth, not CPU (agent workloads are I/O bound on LLM API calls)

Pillar 2: Guardrails and Execution Policies

Guardrails are the most underinvested pillar in enterprise agent deployments. In my experience at Deutsche Bank designing OTC derivatives workflows, every autonomous action needs a pre-check. For AI agents, this means:

• Prompt injection detection — scan incoming prompts for injection patterns before they reach the agent
• Tool access policies — define which agents can call which APIs, databases, or file system paths. Use Open Policy Agent (OPA) with Gatekeeper for policy enforcement
• Kill switches — every agent must have a hard-stop mechanism the control plane can trigger immediately
• PII redaction — automatically redact sensitive data from prompts before LLM submission using a validation webhook

Pillar 3: Cost Governance and Budget Enforcement

The single biggest surprise for enterprises in 2025-2026: AI agents are expensive and costs scale non-linearly with agent autonomy. Without cost governance:

• Agents making unlimited LLM API calls can consume your entire AI budget before month-end
• No one can explain why the bill went from $12,000 to $180,000 in 30 days
• Finance teams lose trust in AI initiatives entirely after one surprise invoice

The control plane enforces cost budgets at the agent level. When projected monthly cost exceeds budget, the control plane queues requests, scales down replicas, or routes to a cheaper model — all without human intervention.

Pillar 4: Observability with OpenTelemetry

Agentic AI systems generate two types of telemetry that traditional APM tools cannot capture: LLM spans (individual model calls with token counts and latency) and agent state transitions (when an agent moves from planning to tool use to response). Both require OpenTelemetry instrumentation with LLM-specific semantic conventions.

Every agent gets an OpenTelemetry sidecar that emits traces for:

• llm.request — model name, prompt tokens, completion tokens, latency
• agent.tool_call — tool name, arguments, result, duration
• agent.state_transition — previous state, new state, trigger event
• agent.guardrail_triggered — policy name, severity, action taken

Metrics flow to Prometheus, traces to Jaeger or Tempo, and both feed Langfuse for LLM cost analysis and latency percentiles. This stack answers the CFO question.

Pillar 5: Audit Trails and Compliance Reporting

For regulated industries (financial services, healthcare, government), every agent decision that touches a business process must be traceable. The control plane generates immutable audit logs stored in S3/GCS with object-level versioning:

• Agent ID, version, model, and tool permissions at execution time
• Full prompt + context (token-bounded for storage efficiency)
• All tool calls made, with arguments and responses
• Guardrail triggers and actions taken
• Cost attribution by team, project, and agent type

The output feeds directly into your GRC tooling — and is the evidence package your auditors will request when EU AI Act enforcement begins August 2026.

Kubernetes-Native Control Plane Architecture

The production architecture below uses the Kubernetes Operator pattern — your platform team can maintain it using skills they already have.

┌─────────────────────────────────────────────────────────────┐
│                    AGENT CONTROL PLANE                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │  Agent       │  │  Guardrail  │  │  Cost Governor   │  │
│  │  Controller  │  │  Webhook    │  │  Sidecar         │  │
│  │  (Operator)  │  │  (OPA)      │  │  (Budget Enf.)   │  │
│  └──────┬───────┘  └──────┬───────┘  └────────┬─────────┘  │
│         │                 │                    │             │
│  ┌──────▼─────────────────▼─────────────────────▼────────┐  │
│  │            OBSERVABILITY SIDCAR (otel-agent)           │  │
│  │   Traces → Jaeger  Metrics → Prometheus  Logs → Loki  │  │
│  └───────────────────────────────────────────────────────┘  │
└────────────────────────┬────────────────────────────────────┘
                         │
          ┌──────────────▼──────────────┐
          │   AGENT RUNTIME (per agent) │
          │  ┌────────────────────────┐  │
          │  │  Agent Process        │  │
          │  │  (LangGraph/AutoGen) │  │
          │  └──────────┬────────────┘  │
          │             │                 │
          │  ┌─────────▼───────────┐    │
          │  │  MCP Server        │    │
          │  │  (Tool Interface)  │    │
          │  └────────────────────┘    │
          └────────────────────────────┘

The key insight: agents run as Kubernetes pods, managed by a custom controller that handles reconciliation. The guardrail webhook intercepts every agent dispatch request and validates it against OPA policies before the agent receives it. Zero runtime overhead — validation happens at admission time, not in the hot path.

CRD for Agent Definition

Agents are defined as Kubernetes custom resources. Here is the CRD schema for an agent:

apiVersion: agentc.gheware.com/v1
kind: AIAgent
metadata:
  name: derivatives-risk-agent
  namespace: ai-agents
spec:
  model: gpt-4-turbo
  temperature: 0.1
  maxTokens: 2048
  tools:
    - name: risk_calculator
      enabled: true
      rateLimit: 10  # per minute
    - name: db_query
      enabled: true
      requireApproval: true  # human-in-the-loop for DB writes
  budget:
    monthlyLimitUSD: 5000
    alertThreshold: 0.75
  guardrails:
    blockPII: true
    maxPromptTokens: 8192
    toolWhitelist: ["risk_calculator", "db_query"]
  replicas: 2
  observability:
    otelEnabled: true
    langfusePublicKey: "$(LANGFUSE_PUBLIC_KEY)"

With this CRD, your entire agent fleet becomes declarative — version-controlled, auditable, and deployable via kubectl apply. The Agent Controller reconciles the desired state against actual pod status, automatically healing failed agents.

Implementation: Building the Control Plane Operator

A full implementation guide exceeds this post's scope, but here is the architectural skeleton your platform team needs. I have taught this pattern in our 5-Day Agentic AI Workshop — rated 4.91/5.0 by Oracle engineers in February 2026, with 119 hands-on labs and real production agent deployments.

Step 1: Install the Agent Controller (Kubebuilder)

Use Kubebuilder to scaffold the operator. The controller watches AIAgent resources and manages the underlying pods:

# Scaffold the operator project
kubebuilder init --domain agentc.gheware.com
kubebuilder create api --group agentc --version v1 --kind AIAgent

# Implement reconciliation loop
func (r *AIAgentReconciler) Reconcile(
    ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

    aiAgent := &agentcv1.AIAgent{}
    if err := r.Get(ctx, req.NamespacedName, aiAgent); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Reconcile agent pods to match Spec
    if err := r.reconcileAgentDeployment(ctx, aiAgent); err != nil {
        return ctrl.Result{}, err
    }

    // Enforce budget via sidecar annotation
    if err := r.enforceCostBudget(ctx, aiAgent); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

Step 2: Deploy the Guardrail Admission Webhook

Use OPA Gatekeeper with a Constraining mutation to intercept agent dispatch requests:

# Gatekeeper ConstraintTemplate for PII detection
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: NoPIIInAgentPrompts
metadata:
  name: no-pii-agent-prompts
spec:
  match:
    kinds:
      - apiGroups: ["agentc.gheware.com"]
        kinds: ["AgentDispatch"]
  parameters:
    piiPatterns:
      - type: email
      - type: phone
      - type: credit_card
  enforcementAction: deny

Step 3: Instrument OpenTelemetry Sidecar

Add the OTEL sidecar to every agent pod:

# agent-sidecar.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: derivatives-risk-agent
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9464"
    spec:
      containers:
        - name: otel-agent
          image: otel/opentelemetry-collector:0.109.0
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "otel-collector.observability:4317"
            - name: OTEL_SERVICE_NAME
              value: "derivatives-risk-agent"
          ports:
            - containerPort: 9464

        - name: agent
          image: gheware/agent-derivatives:v2.3.1
          env:
            - name: OTEL_SERVICE_NAME
              value: "derivatives-risk-agent"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "localhost:4317"

Observability data flows to Prometheus for metrics, Jaeger/Tempo for distributed traces, and Langfuse for LLM-specific cost and latency analysis.

Step 4: Integrate MCP for Tool Standardization

MCP servers provide the standardized tool interface the control plane governs:

# MCP server registry in the control plane CRD
apiVersion: v1
kind: Service
metadata:
  name: risk-mcp-server
  annotations:
    agentc.mcp/enabled: "true"
    agentc.mcp/tools: "risk_calculator,scenario_simulator"
spec:
  ports:
    - port: 3000
      name: mcp
  selector:
    app: risk-mcp-server

# Agent spec references MCP servers by name
spec:
  mcpServers:
    - name: risk-mcp-server
      allowedTools: ["risk_calculator"]

New tools are added to MCP servers — existing agents pick them up automatically without redeployment. The control plane enforces tool access policies, so even if an MCP server advertises 50 tools, your risk agent only gets access to the 2 it needs.

Conclusion: Control Plane First, Agents Second

The question I get from enterprise CTOs: "Can we deploy a few agents first and add the control plane later?" My answer, backed by 25 years of enterprise architecture at JPMorgan, Deutsche Bank, and Morgan Stanley: you can — but the retrofit will cost three times more than building it right the first time, and you will have had 18 months of ungoverned risk in production.

The agentic AI control plane is not optional infrastructure. It is the difference between AI agents that create value and AI agents that create liability. The good news: the architecture is proven, the Kubernetes patterns are well-understood, and our 5-day Agentic AI Workshop covers every component with 119 hands-on labs.

Start with the control plane. Your compliance team, your CFO, and your security team will thank you.

Ready to build it right? Explore our Agentic AI Workshop — Production-Ready AI Agents in Five Days. 119 hands-on labs. 4.91/5.0 at Oracle.