The Cardinality Trap: Why Your Bill Is Growing

In my twenty-five years building infrastructure at firms like JPMorgan Chase and Deutsche Bank, I have seen one consistent trend: observability costs grow faster than the number of servers. In 2026, this is no longer just about storage; it is about cardinality. When you allow unbounded dimensions in your metrics or logs, you create a combinatorial explosion of time series that your backend cannot efficiently compress or query.

The problem is rarely the volume of data alone; it is the uniqueness of the attributes attached to that data. A single misconfigured label in a Kubernetes pod metric can turn a manageable dataset into a multi-terabyte monthly bill. We stopped treating cardinality as a technical afterthought and started treating it as a financial constraint. If you do not define the boundaries of your telemetry, your vendor will define them for you, usually in the most expensive way possible.

Strategic Sampling: From Static to Dynamic

The era of static sampling rates is over. In a monolithic application, taking a 1% sample of traces might have been sufficient. In a microservices architecture with hundreds of endpoints, that approach throws away critical context for high-value transactions while retaining noise from low-value health checks.

We implemented dynamic sampling based on system health and transaction value. The logic is simple but effective:

  • High error rates trigger a 100% sampling rate for the affected service to ensure full visibility into the failure.
  • Normal operation maintains a 10% sample rate for standard requests.
  • Internal health checks and ping endpoints are sampled at 0.1% or dropped entirely.

This approach reduced our tracing volume by approximately 60% without impacting our ability to debug production incidents. By preserving the context only when it matters, we maintain signal integrity while drastically reducing egress and storage costs. You can read more about how we handle these complex flows in our post on AI agent observability in production, where similar dynamic principles apply to non-deterministic workloads.

Retention Policies: Tiering Your Data

Not all telemetry is created equal. Storing raw logs for ninety days is a financial mistake. We adopted a tiered retention strategy that aligns with the "hot," "warm," and "cold" data patterns of enterprise operations.

The first tier is hot data, stored in high-performance indexes for immediate debugging. This is typically kept for 7 to 14 days. The second tier is warm data, aggregated or summarized, kept for 30 days for trend analysis. The third tier is cold data, archived for compliance, kept for 90 days or longer but accessible only for rare audits.

By moving data out of the primary hot storage bucket after just two weeks, we cut our primary storage costs by nearly 40%. This requires a disciplined approach to data lifecycle management. You must define clear schemas for what constitutes "aggregated" data. For example, instead of storing every individual log line, we store summary metrics for specific error codes and service names. This reduces the cardinality of the data significantly while preserving the ability to answer high-level business questions.

Enforcing Cardinality Limits at the Edge

Waiting until data reaches your backend to enforce cardinality limits is too late. The processing cost has already been incurred. We implemented cardinality guards at the edge, within the telemetry agents deployed on our nodes.

These agents perform two critical functions:

  1. They drop metrics that exceed predefined cardinality thresholds, such as unique user IDs or dynamic request IDs that are not correlated to a session.
  2. They hash or anonymize high-cardinality fields before transmission, ensuring that the backend receives structured data without unique identifiers that drive up costs.

This shift-left approach to observability governance ensures that only necessary data enters the pipeline. It also reduces the load on our central collectors, improving overall system stability. For those dealing with AI-native workloads, the challenges are even more pronounced. You can explore our deep dive on AI-native Kubernetes observability to understand how these principles scale in containerized environments.

Frequently Asked Questions

How do I identify high-cardinality fields in my current setup?

You can identify high-cardinality fields by querying your metrics backend for series counts. Look for any label with more than 10,000 unique values. In many cases, fields like trace_id, span_id, or user-specific identifiers are the culprits. These should be sampled or hashed, not stored as distinct time series.

Is sampling safe for debugging distributed systems?

Sampling is safe if done dynamically. Static sampling risks missing rare but critical errors. Dynamic sampling, which increases the sample rate during error bursts, ensures that you capture the full context of incidents when they happen, while reducing noise during normal operations.

What is the impact of cardinality on query performance?

High cardinality degrades query performance because the database must scan more unique series. By enforcing cardinality limits, you reduce the number of series the query engine must process, leading to faster response times and lower compute costs for analytical queries.

Can I apply these strategies to log data as well?

Yes. While metrics are the most common target for cardinality control, logs also benefit from sampling and retention policies. Implementing field-level sampling for logs, such as dropping sensitive or high-entropy fields, can significantly reduce storage costs without losing the ability to debug specific issues.