OpenTelemetry for AI Agents: Production Observability Guide 2026

Why OpenTelemetry for AI Agents?

After architecting payment systems at JPMorgan Chase and derivatives platforms at Deutsche Bank, I learned one truth the hard way: you cannot debug what you cannot see. AI agents are fundamentally different from traditional microservices — they involve non-deterministic reasoning, tool call chains, and LLM interactions where the same input can produce different outputs.

Without observability, you are flying blind. When your agent enters a loop, calls the wrong tool, or degrades in quality over time, you need to see the full trace — from user prompt through reasoning steps to final response. That is exactly what OpenTelemetry gives you.

Here is why OpenTelemetry specifically for AI agents in 2026:

• Vendor neutrality: OTel is the CNCF standard. Ship traces to any backend — Grafana Tempo, Jaeger, Honeycomb, Datadog — without changing your instrumentation code.
• Cross-stack correlation: Your AI agent runs on Kubernetes, talks to PostgreSQL, calls external APIs, and uses an LLM. OpenTelemetry lets you correlate an LLM token anomaly with a Pod OOM kill in the same trace.
• Semantic conventions: OTel's AI semantic conventions provide standard attribute names for model names, token counts, prompt hashes, and tool call metadata.
• Zero vendor lock-in: Unlike LangSmith or Langfuse (which are excellent but proprietary), OTel ensures your observability data is portable.

In my Agentic AI workshop at Oracle — rated 4.91/5.0 by participants — the first module we cover is production-grade observability. Teams that instrument their agents from day one spend 60% less time debugging production incidents.

LLM Call Instrumentation

At minimum, you need to trace three types of operations in an AI agent: LLM calls, tool executions, and agent reasoning steps. Let me show you how to instrument each.

Wrapping LLM API Calls

Wrap every LLM API call in an OpenTelemetry span. Use span kind CLIENT and add semantic attributes:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode

# Initialize tracer provider
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://tempo:4317", insecure=True)
))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

def call_llm(prompt: str, model: str = "gpt-4-turbo", temperature: float = 0.7) -> dict:
    with tracer.start_as_current_span(
        "llm.generation",
        kind=SpanKind.CLIENT,
        attributes={
            "llm.system": model,
            "llm.temperature": temperature,
            "llm.request.type": "chat"
        }
    ) as span:
        start = time.time()
        try:
            response = openai.ChatCompletion.create(
                model=model,
                messages=[{"role": "user", "content": prompt}],
                temperature=temperature
            )
            span.set_attribute("llm.completion.tokens", response.usage.completion_tokens)
            span.set_attribute("llm.total.tokens", response.usage.total_tokens)
            span.set_status(Status(StatusCode.OK))
            return response
        except Exception as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.record_exception(e)
            raise
        finally:
            span.set_attribute("llm.latency.ms", (time.time() - start) * 1000)

Tool Call Instrumentation

AI agents are defined by their tool use. Trace every tool call with full context:

def trace_tool_call(tool_name: str, tool_args: dict, result: dict):
    with tracer.start_as_current_span(
        f"tool.{tool_name}",
        kind=SpanKind.INTERNAL,
        attributes={
            "tool.name": tool_name,
            "tool.input": str(tool_args),
            "tool.success": not isinstance(result, Exception)
        }
    ) as span:
        if isinstance(result, Exception):
            span.set_status(Status(StatusCode.ERROR, str(result)))
            span.record_exception(result)
        else:
            span.set_status(Status(StatusCode.OK))
        return result

The key insight from my years at Morgan Stanley: instrument at the boundary, not deep inside. Wrap the API calls, capture inputs and outputs, and let the distributed tracing system handle correlation.

Tracing Multi-Agent Workflows

Multi-agent systems introduce a new challenge: a single user request fans out to multiple agents, each calling different tools and LLMs, with results flowing back through a supervisor. Without context propagation, you see disconnected spans. With proper trace context, you see the full conversation tree.

The pattern used in production is context injection into agent state. Each agent node receives the OpenTelemetry trace context as part of its input state, propagates it through tool calls:

import json
from opentelemetry import trace
from opentelemetry.propagate import inject, extract
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

propagator = TraceContextTextMapPropagator()

def create_agent_span(agent_name: str, trace_context: dict = None):
    ctx = extract(trace_context or {}) if trace_context else None
    with tracer.start_as_current_span(
        f"agent.{agent_name}",
        context=ctx,
        kind=SpanKind.INTERNAL,
        attributes={"agent.name": agent_name, "agent.type": "reasoning"}
    ) as span:
        carrier = {}
        inject(carrier)
        span.set_attribute("trace.context", json.dumps(carrier))
        return carrier

This pattern works in LangGraph supervisor chains, AutoGen group chats, and CrewAI crew hierarchies. The trace context flows through the graph state, ensuring every span links to the root span.

LangGraph + OpenTelemetry: Step-by-Step

LangGraph is the most popular framework for building stateful, cyclic agent workflows. Here is how to instrument it with OpenTelemetry.

Step 1: Install Dependencies

pip install opentelemetry-api opentelemetry-sdk \
  opentelemetry-exporter-otlp-proto-grpc \
  opentelemetry-instrumentation-langchain \
  opentelemetry-instrumentation-openai

Step 2: Initialize the OTel Provider

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME

resource = Resource(attributes={SERVICE_NAME: "langgraph-agent"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://tempo:4317", insecure=True)
))
trace.set_tracer_provider(provider)

Step 3: Instrument LangGraph Nodes

from opentelemetry import trace
tracer = trace.get_tracer("langgraph-agent")

def traced_node(node_fn, node_name: str):
    def wrapper(state: dict):
        with tracer.start_as_current_span(
            f"node.{node_name}",
            attributes={
                "langgraph.node": node_name,
                "langgraph.thread_id": state.get("thread_id", "unknown")
            }
        ) as span:
            result = node_fn(state)
            return result
    return wrapper

supervisor_node = traced_node(supervisor_logic, "supervisor")
reasoning_node = traced_node(reasoning_logic, "reasoning")
tool_execution_node = traced_node(tool_execution_logic, "tool_execution")

Step 4: Handle Cycles with Span Links

LangGraph allows cycles. Use span links to connect back-edges without creating circular trace structures:

from opentelemetry.trace import Link

def node_with_cycle_support(node_fn, node_name: str, previous_span_context=None):
    kwargs = {"attributes": {"langgraph.node": node_name, "langgraph.cycle": True}}
    if previous_span_context:
        kwargs["links"] = [Link(previous_span_context)]
    with tracer.start_as_current_span(f"node.{node_name}", **kwargs) as span:
        return node_fn(span.context)

This approach gives full visibility into each reasoning step while maintaining correct trace tree semantics even when your agent loops.

Grafana Tempo Backend Setup

For most enterprise teams, the most cost-effective production stack for AI agent observability is Prometheus + Grafana Tempo + Grafana.

Kubernetes Deployment

# tempo-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tempo-config
  namespace: observability
data:
  tempo.yaml: |
    server:
      http_listen_port: 3100
    distributor:
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318
    storage:
      trace:
        backend: s3
        s3:
          bucket: your-tempo-bucket
          region: us-east-1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tempo
  namespace: observability
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: tempo
        image: grafana/tempo:2.4.1
        args: ["-config.file=/etc/tempo.yaml"]
        ports:
        - containerPort: 3100
        - containerPort: 4317
        resources:
          requests:
            cpu: 500m
            memory: 2Gi
          limits:
            cpu: 2000m
            memory: 8Gi

OTel Collector Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlp/tempo:
    endpoint: tempo.observability.svc:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]

This stack handles millions of spans per day at a fraction of the cost of commercial observability platforms. At JPMorgan, we processed 50M+ spans daily through a similar Tempo-based architecture.

Production Patterns and Sampling

At scale, every trace is too much data. The key production decision is your sampling strategy:

Sampling Strategies

• Tail-based sampling: Collect 100% of traces that contain errors, then sample a percentage of successful traces. Most effective for debugging production issues.
• Probabilistic sampling: Sample 10–50% of all traces uniformly. Simple but may miss rare error conditions.
• Rule-based sampling: Sample 100% for high-value endpoints (payment processing, trade execution), lower for informational endpoints.

Critical Attributes for AI Agent Spans

# AI Agent span attributes
span.set_attribute("ai.agent.session_id", session_id)
span.set_attribute("ai.llm.model", model_name)
span.set_attribute("ai.llm.prompt_tokens", prompt_tokens)
span.set_attribute("ai.llm.completion_tokens", completion_tokens)
span.set_attribute("ai.llm.latency_ms", llm_latency)
span.set_attribute("ai.tool.calls", json.dumps(tool_calls))
span.set_attribute("ai.retrieval.chunks_retrieved", len(context_chunks))

These attributes let you build dashboards for token cost tracking, response quality over time, tool call frequency, and retrieval effectiveness.

Alerting on Anomaly Traces

• LLM latency > 30 seconds (model provider degradation)
• Error rate > 5% on agent spans within a 5-minute window
• Token usage spike > 50% above baseline (prompt injection or loop)
• Agent reaches > 20 reasoning steps (potential loop condition)

Conclusion

OpenTelemetry is the observability backbone that production AI agent systems need. It is vendor-neutral, scales to millions of spans per day, and integrates with the Grafana stack your SRE team already knows.

In Rajesh Gheware's 25 years building enterprise systems at JPMorgan Chase, Deutsche Bank, and Morgan Stanley, Rajesh Gheware has never seen a production system that could not benefit from better observability. AI agents need it more — given the non-deterministic nature of LLM reasoning.

If your team is building production AI agents and wants hands-on practice with LangGraph, OpenTelemetry, and enterprise observability patterns, our Agentic AI Workshop covers this end-to-end with 119 hands-on labs. We guarantee 40% faster agent development cycles within 90 days — or your money back plus $1,000.

Start with one span. Instrument your first LLM call. Ship it to Tempo. Once you see that trace, you will understand why observability is not optional for AI agents in production.