A service records a request metric.
http.server.requests
The metric includes useful dimensions:
service:
catalog-service
environment:
production
method:
GET
route:
/products/{product_id}
status_class:
2xx
These attributes allow the platform to answer:
- Which service is failing?
- Which environment is affected?
- Which route is slow?
- Which status classes are increasing?
Then another attribute is added:
product_id:
6f7a8e42
Every product may now create a distinct metric series.
Another engineer adds:
request_id:
4f0b2c11-a15a-4ae1-b965-6ba2e8a17861
Every request may create another one.
The metric still looks informative.
The backend now needs to process an enormous number of unique combinations:
service
× environment
× method
× route
× status_class
× product_id
× request_id
The telemetry has become extremely precise.
It may also have become operationally unusable.
High-cardinality context is useful when it answers a specific question. It becomes expensive when unique values are promoted into metric identity, indexed indiscriminately or retained beyond their diagnostic value.
The problem is not that detailed context is bad.
The problem is placing every kind of context into every kind of telemetry signal.
Metrics, logs and traces carry information differently.
Cardinality should follow the signal and the question being asked.
Cardinality has several forms
Cardinality is often discussed as though it were one number.
In practice, several related properties determine telemetry cost.
Attribute cardinality
How many distinct values
can one attribute produce?
An environment field may have three values:
development
staging
production
A request ID may have millions.
Series cardinality
A metric series is usually identified by the metric name and the complete set of attribute values.
Suppose a metric has:
10 services
3 environments
5 routes
4 status classes
The possible combinations are:
10 × 3 × 5 × 4
= 600 series
Now add:
100,000 customer IDs
The theoretical space becomes:
60,000,000 series
Not every combination will appear.
Enough may appear to overwhelm ingestion, memory, storage, indexes, dashboards and alert evaluation.
Cardinality is multiplicative.
One dangerous label can turn an ordinary metric into a telemetry confetti cannon.
Series churn
Current series count is not the whole story.
A platform may show only 20,000 active series while creating millions of short-lived series each day.
Common sources include:
- Pod names
- container IDs
- job IDs
- build numbers
- autoscaled instances
- temporary resource names
OLD SERIES
become inactive
NEW SERIES
continue appearing
Churn creates work for ingestion, compaction, caches, indexes, storage and garbage collection.
A stable active count can hide a backend furiously cataloguing telemetry mayflies.
Index cardinality
Logs and traces do not create metric series in the same way.
Their fields may still be indexed.
A request ID, customer ID or full URL can create millions of distinct index values, increasing memory, storage, ingestion latency and query cost.
STORED FIELD
available in the event
INDEXED FIELD
optimised for filtering,
searching and grouping
Not every stored field needs to be indexed.
Event volume and query fan-out
Cardinality is not the same as volume.
A low-cardinality schema can still be expensive when it produces billions of events.
A high-cardinality field may be tolerable when only a small number of events contain it and the backend does not index it broadly.
Queries add another cost.
A dashboard that groups by customer across thirty days may scan huge amounts of data even when ingestion remains within limits.
The platform therefore needs to reason about:
distinct values
series combinations
new-series creation
event volume
indexed fields
retention
query fan-out
“High cardinality” is useful shorthand.
The engineering decision depends on which of these costs is actually growing.
Put detail in the signal designed to carry it
High-cardinality information is not automatically wrong.
During an incident, an engineer may need to find:
request_id:
4f0b2c11-a15a-4ae1-b965-6ba2e8a17861
or:
order_id:
bf-order-184027
Those values can connect:
- support reports
- business records
- logs
- traces
- deployment events
The mistake is not recording them anywhere.
The mistake is storing them as metric dimensions when the metric system is designed to aggregate repeated observations.
METRICS
How often?
How much?
How fast?
How many?
LOGS
Which event occurred,
and what did it report?
TRACES
What happened during
this individual request?
Metrics need useful bounded dimensions
Metrics should usually describe stable operational categories.
For an HTTP service, a reasonable metric identity might include:
service.name
deployment.environment
HTTP request method
route template
status class
This preserves questions such as:
Which production route is failing?
Are POST requests slower than GET requests?
Are 5xx responses increasing?
The goal is not minimum cardinality.
It is useful bounded cardinality.
Logs can retain detail without indexing everything
A structured log may contain:
service:
catalog-service
request_id:
4f0b2c11-a15a-4ae1-b965-6ba2e8a17861
product_id:
product-1847
message:
product lookup completed
The platform may index service and request_id because they support common investigations.
It may store product_id without building a broad index if direct product searches are rare.
Indexing policy should follow real query patterns rather than the hopeful belief that every field may become fascinating one Tuesday.
Traces can carry request-level context more naturally
A span may reasonably include:
trace_id
span_id
order_id
product_id
peer address
deployment.version
Trace backends still pay for event volume, attribute storage, indexing, retention and search.
The platform may index only selected attributes:
indexed:
service.name
deployment.environment
deployment.version
error.type
http.route
stored but not broadly indexed:
order_id
product_id
peer address
Tracing tolerates high-cardinality context better than metrics.
It does not provide infinite free detail.
Exemplars connect aggregate signals to individual requests
An aggregate metric may show:
checkout latency increased
An exemplar can attach a representative trace identifier to a metric observation.
METRIC SPIKE
│
▼
TRACE EXEMPLAR
│
▼
INDIVIDUAL SLOW REQUEST
Metrics detect the pattern.
Traces explain an example.
The metric does not need request_id as a permanent label.
The detail has not disappeared.
It has been placed where it can be used.
Common sources of uncontrolled cardinality
Certain instrumentation choices create recurring problems.
Raw paths and identifiers
Consider two HTTP attributes.
Raw path:
/products/1847
Route template:
/products/{product_id}
The raw path creates one value for every product.
The route template groups requests by handler behaviour.
For latency and error metrics, the route template is usually more useful.
Instrumentation should emit the framework’s route template at the source, where the application router knows which pattern matched.
A Collector can remove raw paths or normalise known patterns as a safety layer.
It should not be expected to infer every application route reliably from arbitrary URLs.
The product ID can remain available in a trace or log when an individual request needs investigation.
Dynamic error values
An application may record:
error.message:
connection timed out after 1001 ms
The next error says:
error.message:
connection timed out after 1002 ms
Dynamic messages can produce many unique values and may contain customer data, file paths, identifiers, hostnames, SQL fragments or timestamps.
A safer metric dimension is a stable category:
error.type:
database_timeout
That category still needs governance.
Renaming an arbitrary message field to error.type does not make its values bounded.
The taxonomy should remain stable enough to support alerts, trends and ownership.
The detailed message belongs in the associated log or trace.
Ephemeral infrastructure
Kubernetes telemetry often includes:
- cluster
- Namespace
- workload
- Pod
- container
- node
A Deployment may replace Pods frequently:
catalog-service-7bbfc99c65-k8p2m
catalog-service-7bbfc99c65-r49xz
catalog-service-86d49bf8c7-2f5wq
Pod-level dimensions can help with short-lived infrastructure diagnosis.
They also create churn as old Pods disappear and new names arrive.
For service-level dashboards, more stable dimensions may be:
cluster
Namespace
Deployment
service
Pod-level metrics can use shorter retention, narrower collection or dedicated troubleshooting views.
The platform does not need to pretend Pods are permanent citizens merely because they left behind several gigabytes of historical paperwork.
User-controlled values and schemas
Telemetry attributes may be derived from:
- URL parameters
- request headers
- usernames
- search terms
- product names
- tenant identifiers
These values may be unbounded, sensitive, unexpectedly large or intentionally malicious.
An attacker could send a different header value on every request.
If that value becomes a metric label, they can create cardinality pressure remotely.
UNTRUSTED INPUT
│
▼
METRIC ATTRIBUTE
│
▼
SERIES EXPLOSION
Attribute names must also come from a bounded schema.
User input should not determine either the key or the value used for telemetry identity.
Instrumentation should not promote arbitrary input into indexed dimensions without:
- validation
- normalisation
- allow-listing
- truncation
- redaction
- aggregation
Hashing may obscure the original value.
It does not reduce cardinality, because one million unique values usually remain one million unique hashes.
It also does not automatically remove data-protection obligations.
Telemetry is another data-processing boundary.
It should not accept unlimited schema design from the nearest HTTP client.
Tenant identifiers
In a multi-tenant service, the platform may want to answer:
Which tenant is affected?
Adding tenant_id to every metric enables precise tenant-level views.
It may also create enormous series cardinality and alert fan-out.
Alternatives include:
- metrics for bounded service tiers
- traces and logs containing tenant ID
- separate tenant-health events
- top-N aggregation
- temporary diagnostic metrics
- per-tenant metrics for selected critical journeys
For example:
METRIC
tenant_tier:
standard
TRACE
tenant_id:
tenant-18492
The correct design depends on the number of tenants, contractual reporting, isolation, support requirements, privacy and backend capacity.
Cardinality limits can influence product capability.
That decision should be explicit rather than discovered when the observability invoice acquires executive visibility.
Control the complete cost lifecycle
Telemetry cost does not end when data is accepted.
It continues through aggregation, sampling, querying, alerting and retention.
Sampling reduces volume, not unsafe identity
Tracing systems often use sampling.
Sampling can reduce the number of traces stored.
It does not repair a metric schema that creates one new series for each request.
1 request
with a unique request_id label
creates 1 new series
Likewise, sampling logs after expensive indexing or upstream processing may leave much of the earlier cost intact.
Controls need to operate at the correct point:
PRODUCE ATTRIBUTE
│
▼
NORMALISE OR REMOVE
│
▼
AGGREGATE
│
▼
SAMPLE OR STORE
Sampling is valuable.
It does not repair an unsafe metric schema.
Dashboards can multiply query cost
A dashboard may ask:
show request rate
by customer_id
for the last 30 days
Repeated refreshes multiply the work.
Platform teams should observe:
- slow queries
- expensive aggregations
- broad time windows
- unbounded grouping
- refresh frequency
- abandoned dashboards
Useful controls may include recording rules, pre-aggregation, query limits, shorter default windows and review of persistent high-cost queries.
A dashboard can be technically read-only while still enthusiastically attacking the backend with mathematics.
Alerts should group by distinct action
Suppose an alert groups by:
customer_id
A widespread incident may produce thousands of alert instances.
ONE SERVICE FAILURE
│
▼
10,000 TENANT-SPECIFIC ALERTS
The incident system becomes part of the incident.
Alerts should usually group by actionable ownership and failure domain:
service
environment
region
route
A useful test is:
Will each alert instance
lead to a distinct action?
If not, the grouping is probably too granular.
Retention should follow signal value
High-cardinality telemetry becomes more expensive as retention grows.
A useful model may keep:
raw high-cardinality traces:
short retention
selected error traces:
longer retention
aggregated service metrics:
long retention
audit events:
policy-defined retention
Retention should follow incident-investigation windows, compliance, cost, query performance and sensitivity.
Keeping every unique event indefinitely is not observability maturity.
It is storage accumulation with excellent search ambitions.
Treat cardinality as a platform product concern
Service teams should produce sensible telemetry at the source.
The platform should provide shared controls when they do not.
Instrumentation standards
The platform can publish and embed guidance such as:
use route templates
use stable error categories
never use request IDs as metric attributes
keep attribute names bounded
separate indexed and stored fields
OpenTelemetry semantic conventions help standardise names and meanings.
They reduce inconsistent labels such as:
service
service_name
application
app
Standardisation does not guarantee safe cardinality.
A conventionally named attribute can still contain unsafe values.
The distinction between a raw path and a route template remains important even when both attributes use standard names.
Collector safeguards
The OpenTelemetry Collector can apply shared policy before telemetry reaches the backend.
Possible actions include:
- removing prohibited metric attributes
- renaming fields
- redacting sensitive values
- filtering unwanted metrics
- aggregating observations
- routing high-volume signals
- enforcing required service metadata
APPLICATION TELEMETRY
│
▼
COLLECTOR
│
├── reject or remove unsafe metric labels
├── preserve request context in traces
├── redact prohibited data
└── route signals by policy
│
▼
BACKEND
Central removal is not always lossless.
Deleting an attribute can collapse previously distinct series or change how observations are combined.
The correct metric identity should be produced at the source.
Collector removal should be a tested safety mechanism, not the primary schema-design strategy.
Teaching hundreds of services to emit dangerous telemetry and relying on one emergency filter is a less convincing platform model.
Backend limits must fail visibly
A backend may enforce:
- maximum active series
- maximum attributes per series
- label-length limits
- ingestion quotas
- index-field limits
When limits are exceeded, telemetry may be rejected, dropped, delayed or partially stored.
The platform needs signals for:
series rejected
attributes removed
logs not indexed
traces dropped
service quota exceeded
Silent telemetry loss converts a cost problem into an incident-diagnosis problem.
A cardinality guardrail should fail visibly.
The platform should measure cardinality and ownership
Useful signals include:
- active metric series
- series-creation rate
- highest-cardinality metrics
- highest-cardinality attributes
- ingestion volume by service
- indexed-field growth
- trace-attribute growth
- rejected telemetry
- storage and query cost by owner
For example:
service:
recommendation-service
metric:
recommendation.requests
attribute:
candidate_id
unique values:
2,400,000
This creates an actionable correction.
A generic alert saying:
observability is expensive
is accurate but somewhat under-equipped for engineering action.
Cost visibility can change behaviour.
The purpose is not to invoice every developer for each span.
It is to support a product conversation about whether to remove attributes, reduce retention, sample more aggressively, preserve only error cases or create a better aggregate.
FinOps and observability meet inside the telemetry schema.
A bfstore catalog path
Suppose the catalog service records:
catalog.product.requests
Unsafe metric attributes include:
product_id
request_id
search_query
customer_id
full_url
A safer metric uses:
service.name:
catalog-service
deployment.environment:
production
operation:
get_product
result:
found
source:
product_page
The trace can include:
product_id:
product-1847
request_id:
4f0b2c11-a15a-4ae1-b965-6ba2e8a17861
customer_id:
customer-9021
subject to data-protection policy.
The log can record:
message:
product lookup completed
product_id:
product-1847
duration_ms:
42
cache_result:
hit
The platform then applies controls across the path.
At instrumentation:
emit route templates
use stable error categories
keep unique request values out of metric identity
At the Collector:
remove prohibited metric attributes
normalise service metadata
route high-volume traces
redact sensitive fields
At the backend:
enforce visible series limits
apply retention profiles
observe query and index cost
report rejected telemetry
At the platform-product layer:
publish instrumentation guidance
provide approved dashboards
show cost and cardinality by service
review recurring violations
SERVICE
│
▼
SDK STANDARDS
│
▼
COLLECTOR SAFEGUARDS
│
▼
BACKEND LIMITS
│
▼
OWNER-VISIBLE EVIDENCE
The signals now have different responsibilities:
METRIC
Is catalog lookup healthy?
TRACE
Why was this request slow?
LOG
What happened during this operation?
No single layer carries the entire control.
Together they reduce the chance that one innocent-looking attribute damages the shared observability service.
Questions I now ask
Purpose
Which operational question
does this attribute answer?
Scale
How many values can it produce,
and how quickly will they change?
Shape
Is it a stable category
or an individual-event identifier?
Signal
Which telemetry signal
should carry it?
Indexing
Must it be indexed,
or only stored?
Trust
Can users control it,
and is it sensitive?
Lifetime
How long will this detail
remain useful?
Control
Which limit, owner and visible
failure behaviour apply?
These questions make telemetry design part of engineering rather than something discovered later in a backend capacity graph.
The mental model I am keeping
My earlier model was:
MORE TELEMETRY ATTRIBUTES
equals
MORE OBSERVABILITY
The stronger model is:
OPERATIONAL QUESTION
│
▼
CHOOSE SIGNAL
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
METRIC LOG TRACE
│ │ │
▼ ▼ ▼
BOUNDED LABELS STRUCTURED EVENT REQUEST CONTEXT
│ │ │
└───────────────┼───────────────┘
▼
CARDINALITY POLICY
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
NORMALISE SAMPLE RETAIN
│ │ │
└───────────────────┼───────────────────┘
▼
USEFUL, AFFORDABLE
TELEMETRY
High-cardinality telemetry is not inherently wasteful.
Unique identifiers and detailed attributes can make a difficult incident understandable.
They become dangerous when they are attached indiscriminately to metrics, indexed everywhere and retained without a clear operational purpose.
The goal is not to remove detail. It is to keep aggregate signals bounded while preserving request-level detail in the telemetry designed to carry it.
A metric does not need to remember every customer, request and product to describe whether the service is healthy.
A trace can carry the individual story.
A log can preserve the event.
The observability platform becomes more useful when each signal is allowed to do its own job, rather than every telemetry type attempting to become a searchable autobiography of the entire system.