LangFuse in Production: Tracing, Evaluation, and Cost Tracking for AI Agents

Why AI Agent Observability Is Non-Negotiable

Traditional software observability (Prometheus, Datadog, OpenTelemetry) captures metrics, logs, and traces for deterministic systems. AI agents introduce a new observability challenge: the output is probabilistic. The same input can produce different outputs on different runs. A change to the system prompt, model version, temperature, or context content can silently shift output quality without causing any error, exception, or metric spike.

This is why AI observability requires a fourth dimension that traditional monitoring lacks: semantic quality evaluation. You need to know not just that the agent responded (uptime), how fast it responded (latency), and how much it cost (token usage) — you need to know whether the response was good.

LangFuse provides all four dimensions:

• Traces: complete execution records for every agent invocation
• Spans: individual steps within traces (LLM calls, tool calls, retrieval operations)
• Cost tracking: token usage and estimated API cost per span, trace, user, and workflow
• Scores: quality evaluations attached to traces — automated, LLM-as-judge, or human-annotated

Instrumenting a LangChain Agent in 10 Minutes

LangFuse integrates with LangChain via a callback handler. Every LLM call, tool invocation, and chain step is automatically captured as a structured trace — no manual span creation required for standard LangChain components.

# pip install langfuse langchain langchain-openai

import os
from langfuse.callback import CallbackHandler
from langchain_openai import ChatOpenAI
from langchain.agents import create_react_agent, AgentExecutor
from langchain_core.prompts import PromptTemplate
from langchain_core.tools import tool

# LangFuse configuration — set in environment, never hardcode in production
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."       # from LangFuse project settings
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."       # from LangFuse project settings
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"  # or your self-hosted URL

# Create the LangFuse callback handler
# WHY named traces: session_id groups all traces for a user session,
# making it easy to see the full conversation history in the LangFuse UI.
langfuse_handler = CallbackHandler(
    session_id="workshop-demo-session-001",   # groups traces by session
    user_id="student-rajiv-001",              # enables per-user cost tracking
    metadata={
        "environment": "production",
        "workshop_batch": "April-2026",
        "agent_version": "v2.1.0"
    }
)

@tool
def search_knowledge_base(query: str) -> str:
    """Search the workshop knowledge base for information."""
    # In production: ChromaDB or Pinecone retrieval here
    return f"Knowledge base result for: {query}"

@tool
def calculate_deployment_cost(nodes: int, hours: int) -> str:
    """Calculate Kubernetes cluster cost for given nodes and hours."""
    cost = nodes * hours * 0.12  # $0.12/node/hour estimate
    return f"Estimated cost: ${cost:.2f} for {nodes} nodes × {hours} hours"

llm = ChatOpenAI(model="gpt-4o", temperature=0)
tools = [search_knowledge_base, calculate_deployment_cost]

# Standard ReAct agent setup
prompt = PromptTemplate.from_template("""
You are a DevOps and AI training assistant.
Answer the user's question using available tools.

Tools: {tools}
Tool names: {tool_names}
Question: {input}
Scratchpad: {agent_scratchpad}
""")

agent = create_react_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=False)

# Pass langfuse_handler as a callback — all steps are automatically traced
result = agent_executor.invoke(
    {"input": "What does a 10-node Kubernetes cluster cost for 8 hours?"},
    config={"callbacks": [langfuse_handler]}   # ← this is all that's needed
)

print(result["output"])
# LangFuse now contains a complete trace with:
# - The agent's reasoning steps
# - Each tool call (name, input, output, latency)
# - Each LLM call (model, prompt tokens, completion tokens, cost)
# - Total trace duration and cost

Creating Named Traces for Multi-Step Workflows

For workflows with multiple distinct phases, use the LangFuse SDK directly to create named parent traces. This gives you clear hierarchy in the dashboard — you can see that the "retrieval" phase cost 800 tokens while "synthesis" cost 3,200 tokens:

from langfuse import Langfuse

langfuse = Langfuse()

# Create a named parent trace for the entire workflow
trace = langfuse.trace(
    name="document-summarisation-workflow",
    user_id="user-deepti-001",
    metadata={"document_type": "legal_contract", "pages": 24}
)

# Create child spans for each phase — gives cost breakdown per phase
retrieval_span = trace.span(name="retrieval-phase")
# ... run retrieval
retrieval_span.end(
    output={"chunks_retrieved": 8, "retrieval_latency_ms": 145}
)

# LLM generation span — automatically captures token costs
generation = trace.generation(
    name="summary-generation",
    model="gpt-4o",
    input=[{"role": "user", "content": "Summarise the following contract..."}]
)
response = llm.invoke("Summarise the following contract...")
generation.end(
    output=response.content,
    usage={"input": 2400, "output": 380}  # LangFuse calculates cost automatically
)

# Flush before process exit in short-lived scripts
langfuse.flush()

Automated Evaluations with langfuse.score()

Tracing tells you what happened. Evaluations tell you whether what happened was good. LangFuse's scoring API attaches quality scores to individual traces — enabling you to track quality trends over time, compare agent versions, and trigger alerts when scores drop below threshold.

from langfuse import Langfuse
from langchain_openai import ChatOpenAI

langfuse = Langfuse()
judge_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)  # cheaper model for eval

def evaluate_summary_quality(
    trace_id: str,
    original_document: str,
    generated_summary: str,
    reference_summary: str  # ground truth from your eval dataset
) -> dict:
    """
    Three-dimension evaluation using LLM-as-judge.
    WHY three dimensions: a summary can be accurate but incomplete,
    or complete but factually wrong. Single-score evals miss this.
    """

    # Dimension 1: Accuracy (does it contradict the source?)
    accuracy_prompt = f"""
Rate the factual accuracy of this summary vs the original document.
Score: 0.0 (completely wrong) to 1.0 (perfectly accurate).
Respond with ONLY a number between 0.0 and 1.0.

Original: {original_document[:2000]}
Summary: {generated_summary}
"""
    accuracy_score = float(judge_llm.invoke(accuracy_prompt).content.strip())

    # Dimension 2: Completeness (does it cover key points?)
    completeness_prompt = f"""
Rate how completely this summary covers the key points of the original.
Score: 0.0 (misses everything important) to 1.0 (covers all key points).
Respond with ONLY a number between 0.0 and 1.0.

Reference summary (key points): {reference_summary}
Generated summary: {generated_summary}
"""
    completeness_score = float(judge_llm.invoke(completeness_prompt).content.strip())

    # Dimension 3: Conciseness (is it appropriately brief?)
    conciseness_score = max(0.0, 1.0 - (len(generated_summary) / len(original_document)))

    # Attach scores to the trace — visible in LangFuse dashboard
    langfuse.score(trace_id=trace_id, name="accuracy", value=accuracy_score)
    langfuse.score(trace_id=trace_id, name="completeness", value=completeness_score)
    langfuse.score(trace_id=trace_id, name="conciseness", value=conciseness_score)

    overall = (accuracy_score + completeness_score + conciseness_score) / 3
    langfuse.score(trace_id=trace_id, name="overall_quality", value=overall)

    # Alert if quality drops below threshold
    if overall < 0.70:
        print(f"⚠️  QUALITY ALERT: trace {trace_id} scored {overall:.2f} — below 0.70 threshold")
        # In production: PagerDuty / Slack alert here

    return {
        "accuracy": accuracy_score,
        "completeness": completeness_score,
        "conciseness": conciseness_score,
        "overall": overall
    }

# Run evaluation on the most recent 20 traces
traces = langfuse.fetch_traces(limit=20).data
for trace in traces:
    if trace.name == "document-summarisation-workflow":
        scores = evaluate_summary_quality(
            trace_id=trace.id,
            original_document=trace.input.get("document", ""),
            generated_summary=trace.output.get("summary", ""),
            reference_summary=eval_dataset.get(trace.input.get("doc_id"))
        )
        print(f"Trace {trace.id}: overall={scores['overall']:.2f}")

Self-Hosted LangFuse: Docker Compose Setup

For enterprise teams where LLM inputs and outputs — which may contain confidential data — cannot leave the on-premises network, self-hosted LangFuse is the only viable option. The Docker Compose configuration below deploys LangFuse with PostgreSQL on a single VM (minimum: 2 CPU, 4 GB RAM):

# docker-compose.yml — LangFuse self-hosted (production-grade)
# WHY self-host: enterprise data never leaves your network.
# All LLM inputs/outputs are stored in your PostgreSQL instance.

version: "3.9"

services:
  langfuse-server:
    image: langfuse/langfuse:latest
    container_name: langfuse
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      # Database — use a dedicated PostgreSQL instance in production
      - DATABASE_URL=postgresql://langfuse:${POSTGRES_PASSWORD}@postgres:5432/langfuse

      # Auth — generate secure random strings for these
      - NEXTAUTH_SECRET=${NEXTAUTH_SECRET}
      - NEXTAUTH_URL=https://langfuse.internal.yourcompany.com

      # Salt for API key hashing
      - SALT=${SALT}

      # Encryption for secrets stored in DB
      - ENCRYPTION_KEY=${ENCRYPTION_KEY}

      # Optional: disable signup for internal-only deployments
      - AUTH_DISABLE_SIGNUP=true

      # Optional: restrict to company email domain
      - AUTH_ALLOWED_EMAIL_DOMAINS=yourcompany.com

    depends_on:
      postgres:
        condition: service_healthy

  postgres:
    image: postgres:16-alpine
    container_name: langfuse-postgres
    restart: unless-stopped
    environment:
      - POSTGRES_USER=langfuse
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=langfuse
    volumes:
      - langfuse-postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U langfuse"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  langfuse-postgres-data:

Start with: POSTGRES_PASSWORD=$(openssl rand -hex 32) NEXTAUTH_SECRET=$(openssl rand -hex 32) SALT=$(openssl rand -hex 16) ENCRYPTION_KEY=$(openssl rand -hex 32) docker-compose up -d

Then point your LangFuse SDK to your internal URL: LANGFUSE_HOST=https://langfuse.internal.yourcompany.com

LangFuse vs LangSmith vs Arize: Which Should You Use?

LangFuse is the open-source, self-hostable choice. It gives you full control over data residency, unlimited traces at zero per-trace cost on self-hosted, and a clean UI for trace exploration and evaluation. The trade-off: you manage the infrastructure. Best for: enterprise teams with data privacy requirements, high-volume deployments where managed pricing would be prohibitive, and teams that want to contribute to or customise the platform.

LangSmith is Anthropic/LangChain's managed observability platform. It has the tightest integration with LangChain — auto-tracing with a single environment variable (LANGCHAIN_TRACING_V2=true). The UI is polished and the playground for testing chains is excellent. The trade-off: data leaves your infrastructure, pricing at $10/1M traces on the paid tier. Best for: teams already deep in the LangChain ecosystem who want zero-friction setup and don't have strict data residency requirements.

Arize AI is the enterprise MLOps platform with AI agent observability capabilities added in 2025. It has the strongest integrations with traditional ML model monitoring, A/B testing, and model comparison. The trade-off: higher cost, steeper learning curve, more complexity than most teams need for pure LLM observability. Best for: organisations that already run traditional ML models and want a single platform for all model types.

Conclusion: Observe First, Optimise Second

The document summarisation incident I opened with is not an edge case. It is the default state of unobserved AI systems. Models change. Prompts drift. Config parameters get updated without documentation. Without observability, these changes are invisible until users notice — and by then, the damage is done.

The LangFuse setup I've described takes one afternoon: instrument your agents, deploy self-hosted LangFuse on Docker, write three evaluation functions that cover your key quality dimensions, and set an alert threshold. From that point forward, every deployment is automatically evaluated. Every regression is caught in staging, not production. Your on-call rotation stops getting 2 AM pages about "the AI giving weird answers."

In Day 4 of the Agentic AI Workshop, we build the complete LangFuse observability stack as a live lab — from raw agent to fully instrumented, evaluated, cost-tracked production system. By end of day, every participant has LangFuse running on their machine, tracing their agents, and running LLM-as-judge evaluations on real outputs.