The alert says:
Checkout error rate is above 5%.
That tells me something is wrong.
It does not tell me:
- which requests failed
- which service first became unhealthy
- whether every customer is affected
- whether the problem began after a deployment
- whether payment timed out
- whether inventory rejected the order
- whether the database became slow
- whether one availability zone is responsible
- whether the operation eventually completed
I open a dashboard.
The error rate increased at 14:07.
Payment latency rose at almost the same time.
The order service restarted twice.
That gives me a pattern.
I still do not know what happened to one failing checkout.
I search the logs for an order ID.
The order service recorded:
payment authorisation failed
The payment service recorded:
provider request timed out
The payment provider may still have completed the charge.
Now I need to reconstruct the observed path of that request across several services and decide whether the customer’s operation failed, succeeded or became uncertain.
One incident has produced three different questions:
METRICS
When did behaviour change,
and how widespread is it?
LOGS
What discrete events and details
did components record?
TRACES
Which observed path did this request take,
and where was its time spent?
I had previously treated all three as variations of “application monitoring”.
That flattened their strengths.
A metric is not merely a compact log.
A log is not a verbose metric.
A trace is not a collection of logs arranged in a pleasing waterfall.
They represent different views of a running system.
Metrics reveal patterns across populations. Logs preserve detailed events. Traces reconstruct the observed paths of individual operations across instrumented boundaries.
Observability becomes useful when those signals can be connected during an investigation.
Monitoring, observability and incomplete evidence
Monitoring turns known reliability concerns into measurements, expectations and alerts.
Is the service available?
Is error rate increasing?
Is the queue growing?
Is latency above the accepted threshold?
These questions can become dashboards and alert conditions.
KNOWN CONDITION
│
▼
MEASUREMENT
│
▼
EXPECTATION
│
▼
ALERT
Observability is the broader ability to investigate system behaviour using the telemetry and context available, including questions that were not predicted completely when the dashboards were designed.
KNOWN QUESTION
Is checkout error rate high?
UNEXPECTED QUESTION
Are failures limited to orders containing
items stored in one inventory partition
after version 1.8.0 was deployed?
Monitoring and observability are not opponents.
Monitoring tells me when a known condition requires attention.
Observability helps me explore why that condition exists, which population it affects and what happened to representative operations.
Both depend on telemetry, and telemetry is evidence rather than complete truth.
REAL SYSTEM BEHAVIOUR
│
▼
INSTRUMENTATION
│
▼
TELEMETRY
│
▼
OPERATOR INTERPRETATION
Every stage can lose or distort information.
A service may fail before writing its final log.
A metric may arrive late.
A trace may be sampled out or lose context at an uninstrumented boundary.
A clock may be wrong.
Instrumentation may contain a bug.
Missing evidence should not automatically become evidence that nothing happened.
In a distributed request:
No response
does not mean
no effect.
In observability:
No log line
does not always mean
no event.
A useful telemetry design exposes its own limitations so operators can distinguish healthy behaviour from missing evidence.
Logs preserve detailed event context
A log records something a component decided was worth describing.
Examples include:
order accepted
payment request started
inventory reservation rejected
database migration completed
consumer joined group
configuration reload failed
A structured log entry can preserve detailed context:
{
"timestamp": "2023-11-12T14:07:31.482Z",
"level": "error",
"service": "payment",
"environment": "prod",
"event": "payment_provider_timeout",
"message": "Payment provider did not respond before the operation deadline.",
"order_id": "ord_72f91",
"payment_id": "pay_91a2c",
"provider": "example-payments",
"operation": "authorise",
"timeout_ms": 2000,
"trace_id": "4f918c...",
"span_id": "f20a7e..."
}
This can answer:
- which service recorded the event
- which environment was affected
- which order and payment were involved
- which provider operation timed out
- which timeout applied
- which distributed trace contains the request
Logs are particularly useful for:
- detailed errors
- state transitions
- security and audit events
- deployment activity
- configuration changes
- rare conditions
- diagnostic context
- business workflow milestones
They preserve individual events rather than summarising a population.
Structured fields make those events searchable without requiring a logging system to recover meaning from prose.
{
"event": "payment_authorisation_failed",
"order_id": "ord_72f91",
"reason": "provider_timeout"
}
The message remains useful for people.
The fields make the event useful to machines.
A strong event usually contains stable context such as:
timestamp
severity
service and environment
event name
operation and result
relevant resource identity
trace or business-operation identity
The exact fields depend on the event.
Not every log needs every identifier available to the application.
Include enough context to investigate the event without turning the log into a replica of the application database.
Stable event names matter because human-readable messages change.
Payment request timed out.
Payment provider did not respond in time.
Provider deadline exceeded.
These may all describe the same operational event.
A stable field such as:
{
"event": "payment_provider_timeout"
}
can support dashboards and queries even while the wording evolves.
The event name becomes part of the telemetry contract. It needs clear meaning, consistent field types, documented ownership and controlled changes.
Severity also needs agreed meaning.
A practical model is:
DEBUG
Detailed diagnostic information,
normally disabled or sampled in production.
INFO
Expected lifecycle or business events
useful for understanding operation.
WARN
Unexpected or degraded condition
from which the component can continue.
ERROR
An operation failed or produced
an outcome requiring attention.
The same failure should not be logged as an error at every layer.
Suppose a database error passes through a repository, service, transport handler and request interceptor. Four nearly identical log records create noise without adding four times the evidence.
A better design identifies which boundary owns the useful failure record. Lower layers can enrich the returned error. The request boundary can emit one complete event with the relevant context.
Logs require data and retention discipline
Logs are event records, not an accidental application database.
It is tempting to record:
every request
every database query
every queue poll
every retry
complete request and response bodies
That volume creates storage, indexing and network cost. It also increases privacy and security exposure.
Payloads may contain:
- credentials
- personal information
- payment data
- authentication tokens
- confidential business fields
The right question is not:
Can this value be logged?
It is:
Which investigation requires this value,
and what risk is created by retaining it?
High-volume population facts such as request counts are usually better represented as metrics.
Logs need deliberate redaction, access control and retention. Security audit events may require longer protected retention than routine diagnostic records. High-volume debug logs may need a much shorter lifecycle.
Metrics reveal behaviour across populations
A metric records a numerical observation associated with a name and selected attributes.
Examples include:
request count
request latency
error count
queue depth
CPU utilisation
memory use
active connections
consumer lag
outbox backlog age
Metrics are designed for aggregation across instances and time.
They answer questions such as:
How many requests are failing?
How has latency changed?
Is resource use approaching a limit?
Which service has the highest error rate?
Is the queue growing faster than it drains?
A dashboard can show:
CHECKOUT ERROR RATE
14:00 0.4%
14:05 0.5%
14:10 6.7%
14:15 8.1%
14:20 7.4%
That pattern is visible immediately.
Finding the same pattern by counting millions of log lines would usually be slower and more expensive.
Metrics trade individual detail for efficient aggregation.
Three common metric forms answer different numerical questions:
| Type | Best suited to | Example |
|---|---|---|
| Counter | Cumulative occurrences | Requests, failures or retries |
| Gauge | Current measured state | Queue depth or active connections |
| Histogram | Distribution of observations | Request latency or job duration |
A counter normally increases during one process lifetime. Monitoring systems derive rates from the change over time.
A gauge can rise and fall as the measured state changes.
A histogram groups observations into buckets so the system can analyse a distribution rather than one average.
This matters because an average can hide tail behaviour. Most requests may complete quickly while a small but important population stalls.
Histogram usefulness still depends on sensible bucket design and aggregation. Recording a latency histogram does not automatically make every percentile meaningful.
Labels make aggregation useful
A request counter may include attributes such as:
service = catalog
operation = GetProduct
status = error
environment = prod
This supports queries such as:
catalog error rate in production
grouped by operation
Labels should describe bounded dimensions that support population-level questions.
Useful examples include:
- service
- operation
- status class
- environment
- region
- deployment version
- dependency name
An identifier such as:
order_id
may create one time series for every order.
A value such as:
customer_email
can create unbounded cardinality while leaking sensitive information.
BOUNDED LABEL
status = success | error
UNBOUNDED LABEL
order_id = millions of possible values
Individual identifiers belong more naturally in logs and traces.
METRIC LABEL
payment_provider = provider-a
LOG OR SPAN FIELD
payment_id = pay_91a2c
The first supports aggregation.
The second supports investigation of one operation.
Metrics support alerts and reliability decisions
Alerts work best when based on sustained symptoms and service impact.
Examples include:
error rate exceeds an accepted threshold
latency breaches a service objective
queue age continues increasing
available replicas remain below desired count
A single error event may not require an alert. Distributed systems produce occasional failures during normal operation.
A metric can express impact over time:
failed requests
divided by
total requests
This distinguishes one isolated failure from service-level degradation.
Service-level indicators then connect telemetry to reliability.
99.9% of eligible catalog requests
succeed during the measurement window
The metric supplies evidence.
The objective supplies meaning.
Resource metrics remain valuable for diagnosis, but user-facing service indicators should drive the most important reliability decisions.
Traces reconstruct observed execution paths
A trace represents one observed operation as it passes through instrumented components.
A bfstore checkout might travel through:
client
API gateway
order service
inventory service
payment service
database
external payment provider
The trace contains spans.
Each span represents a unit of work.
TRACE: checkout
gateway request
└── order CreateOrder
├── inventory ReserveStock
│ └── inventory database query
└── payment Authorise
└── payment provider request
A span can record:
- operation name
- start and end time
- parent relationship
- service
- status
- selected attributes
- events
- links to related work
The trace can answer:
Which instrumented components participated?
In which order?
How long did each operation take?
Where did the failure appear?
Which downstream call consumed the deadline?
This is especially valuable once one customer request crosses several network boundaries.
Context propagation connects spans
For separate services to contribute to one trace, trace context must cross the boundary.
BASKET SERVICE
│
│ trace context
▼
CATALOG SERVICE
│
│ trace context
▼
DATABASE CLIENT SPAN
Without propagation, the services produce unrelated fragments.
Context may travel through HTTP headers, gRPC metadata, message headers or job metadata.
The propagation format should be standardised. OpenTelemetry commonly uses W3C Trace Context for compatible HTTP-style propagation.
Asynchronous work does not always form one tree
Synchronous request chains often form parent-child trees.
Asynchronous workflows may not.
Suppose the order service publishes:
OrderConfirmed
Several consumers react independently:
shipping
notification
analytics
The producer request may finish before those consumers begin.
Span links can represent causal relationships between separate traces or operations:
ORDER TRACE
│
└── publishes OrderConfirmed
│
├── linked shipping trace
├── linked notification trace
└── linked analytics trace
Trying to force every asynchronous workflow into one enormous trace can create traces that remain open for hours or days.
The trace model should reflect the actual lifecycle of the operation.
Sampling and attributes determine what remains observable
Recording every span for every request can be expensive.
Tracing systems therefore sample.
Head sampling decides near the beginning of the trace.
Tail sampling waits until enough spans have arrived to make a decision using later information.
That enables policies such as:
retain errors
retain slow traces
sample routine successful traces
Tail sampling requires more collection infrastructure because the decision is delayed.
Sampling policy should follow investigation needs and cost constraints. It should also be documented so operators understand which conclusions the retained population can support.
Trace attributes can contain richer per-request context than metric labels, but they still require privacy and cardinality discipline.
Useful attributes may include:
service operation
RPC method
response status
dependency name
deployment version
message topic
partition
retry attempt
Dangerous attributes include credentials, complete request bodies, payment data and large database results.
The trace should support investigation without becoming an ungoverned copy of application data.
Correlation turns telemetry into an investigation
The three signals become much more useful when they share context.
Suppose an alert identifies a latency spike for:
service = payment
operation = Authorise
environment = prod
A dashboard can link to traces matching those dimensions during the affected period.
A selected trace contains:
trace_id = 4f918c...
The payment span links to logs carrying the same trace ID.
METRIC
payment latency increased
│
▼
TRACE
one slow payment path
│
▼
LOGS
provider timeout and retry details
This investigation moves from population to representative operation to detailed events.
METRICS
find the pattern
TRACES
find an example path
LOGS
inspect detailed events
The order can change depending on the incident.
A log may reveal a business identifier first.
A trace may reveal which service deserves a metric query.
The important capability is movement between signals.
Shared context can include:
- service and environment
- operation name
- deployment version
- image digest
- region or cluster
- trace ID
- stable business-operation ID
Deployment metadata is particularly valuable because a release is a likely cause of behavioural change.
14:06
payment-service 1.6.0 deployment begins
14:07
checkout error rate rises
This does not prove causation.
It creates a precise hypothesis that can be tested against traces, logs and comparison populations.
Business identity may outlive a trace
Applications often create a durable identifier for a business operation.
For example:
checkout operation ID:
checkout_72f91
That identity may persist across:
- the original API request
- asynchronous payment confirmation
- shipping creation
- reconciliation
The initial request trace may finish much earlier.
BUSINESS OPERATION ID
follows the logical workflow
TRACE ID
follows one observed execution path
Both identifiers may be recorded together, but a trace ID should not replace a durable business-operation ID when the workflow can outlive one execution trace.
The business identifier supports investigation of the complete workflow.
The trace identifier connects spans and telemetry from one observed execution.
This distinction matters most when outcomes become uncertain. A payment request can time out while the provider completes the charge. The original trace describes the timed-out attempt. The payment operation ID allows later reconciliation to find the eventual business outcome.
Instrumentation and telemetry ownership
Automatic instrumentation can create spans and metrics for:
- HTTP requests
- gRPC calls
- database clients
- message brokers
- runtime behaviour
It may know:
POST /checkout returned 500
It may not know:
payment outcome became uncertain
or:
stock reservation expired before confirmation
Those are domain meanings.
Custom instrumentation should add the business context and state transitions that platform libraries cannot infer.
TECHNICAL SIGNAL
gRPC call failed with DeadlineExceeded
DOMAIN SIGNAL
payment authorisation outcome unknown
Each service should expose telemetry about the work it owns.
ORDER SERVICE
order state transitions
checkout orchestration
workflow duration
INVENTORY SERVICE
reservation attempts
reservation expiry
stock conflicts
PAYMENT SERVICE
authorisation attempts
provider latency
unknown outcomes
A caller can record that an inventory RPC failed.
The inventory service should record the reason it owns.
Clear telemetry ownership mirrors service and data ownership.
Platform and application signals also answer different questions.
PLATFORM TELEMETRY
Is the runtime environment healthy?
APPLICATION TELEMETRY
Is useful work completing correctly?
A service can have healthy Pods while every request fails because a dependency rejects authentication.
A Pod can restart while customers see no impact because other replicas absorb the failure.
Platform telemetry should preserve workload boundaries such as service, Pod, container and node. Host-level capacity alone may hide cgroup throttling or workload-level memory limits.
OpenTelemetry provides a common instrumentation layer
OpenTelemetry provides vendor-neutral APIs, SDKs, semantic conventions and protocols for traces, metrics and logs.
A simplified arrangement is:
APPLICATION
│
▼
OPENTELEMETRY SDK
│
▼
OPENTELEMETRY COLLECTOR
│
├── logs backend
├── metrics backend
└── traces backend
The collector can receive, batch, enrich, filter, sample, transform and export telemetry.
This reduces coupling between application instrumentation and a particular backend.
It does not make observability vendor-free by magic. Backends still differ in storage, querying, retention, cost, alerting and user experience.
The standard reduces coupling at the instrumentation and transport layer.
Telemetry should support operational decisions
Dashboards, alerts and service objectives serve different operational purposes.
DASHBOARD
explore behaviour
ALERT
request action
SERVICE OBJECTIVE
define whether behaviour is acceptable
A dashboard is not improved merely by adding more charts.
Each panel should help answer a question such as:
How much demand is arriving?
How much work is failing?
How long does work take?
Which resource is constrained?
Which dependency changed?
What changed recently?
An alert should identify a condition that requires action.
Weak alert:
CPU is 81%.
Stronger alert:
Catalog request latency has exceeded its objective
for ten minutes while CPU throttling is increasing.
The first reports a number.
The second connects service impact to a plausible resource constraint.
Not every alert needs a complete diagnosis, but an alert for a problem nobody intends to act upon is a scheduled interruption rather than an operational control.
Service-level objectives give the most important telemetry organisational meaning.
METRIC
0.4% request failures
OBJECTIVE
no more than 0.1% failures allowed
INTERPRETATION
reliability budget is being consumed too quickly
The observability system must also be operable
Telemetry consumes application CPU, memory, network bandwidth, collector capacity, storage, indexing and engineering attention.
Volume grows through:
more services
more replicas
more requests
more labels
longer retention
richer trace sampling
more detailed logs
Cost, reliability and capacity are connected.
A collector can become unavailable.
An exporter can time out.
A backend can reject writes.
A queue can fill.
Labels can exceed limits.
Clock skew can distort timestamps.
The telemetry pipeline needs its own telemetry:
records received
records exported
records dropped
export failures
queue size
batch delay
backend ingestion latency
Otherwise the system measuring failures can fail silently.
Instrumentation should also avoid making business operations depend directly on observability storage.
APPLICATION REQUEST
│
├── complete business work
│
└── emit telemetry asynchronously
Buffers need limits.
When those limits are reached, dropping routine telemetry may be safer than exhausting application memory. Security audit records may require stronger delivery guarantees than debug traces.
The trade-off should be explicit.
Retention and sampling should follow operational value.
high-volume debug logs:
short retention
security audit logs:
longer protected retention
routine successful traces:
sampled
error traces:
retained more aggressively
aggregated service metrics:
longer retention
A self-managed observability platform does not remove cost. It changes where that cost appears.
Returning to the checkout incident
The opening alert said:
Checkout error rate is above 5%.
The investigation now has a clearer path.
Metrics show:
checkout request rate:
normal
checkout error rate:
increased from 0.3% to 7.2%
inventory latency:
normal
payment latency:
increased sharply
payment errors:
concentrated on provider-a
deployment:
payment-service 1.6.0 released at 14:05
The working hypothesis becomes:
Failures are associated with provider-a
after the payment-service release.
That is not yet proof.
A representative trace shows:
gateway:
18 ms
order CreateOrder:
2.4 s
inventory ReserveStock:
72 ms
payment Authorise:
2.1 s
provider-a request:
deadline exceeded
Another trace using provider-b completes normally.
The payment logs for the affected trace record:
{
"event": "payment_outcome_unknown",
"payment_id": "pay_91a2c",
"provider": "provider-a",
"reason": "client_deadline_exceeded",
"trace_id": "4f918c..."
}
The log does not claim that the payment failed.
It records the business state the service actually knows:
outcome unknown
A reconciliation process later queries the provider using the stable payment operation ID.
Its backlog metric shows how many unresolved payments remain.
METRICS
show provider-a failures are widespread
TRACES
show time is spent in the provider request
LOGS
show individual payments entered
an unknown-outcome state
BUSINESS METRIC
shows the unresolved reconciliation backlog
No one signal carries the complete investigation.
Together, they support detection, diagnosis and recovery.
A compact signal-design review
| Area | Review question |
|---|---|
| Logs | Which discrete events need durable, structured context? |
| Metrics | Which population-level behaviours need aggregation and alerting? |
| Traces | Which execution paths and boundaries need reconstruction? |
| Correlation | Can operators move between symptoms, representative operations and detailed events? |
| Identity | Can telemetry identify the workload, release and durable business operation? |
| Reliability | Can missing, delayed or dropped telemetry be detected? |
| Governance | Are cost, retention, cardinality and sensitive data controlled? |
| Ownership | Who acts when a dashboard or alert reveals a problem? |
Telemetry should be designed with the same care as any other production interface.
The mental model I am keeping
The useful model is not:
OBSERVABILITY
collect lots of application data
put it in dashboards
It is:
RUNNING SYSTEM
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
LOGS METRICS TRACES
│ │ │
▼ ▼ ▼
detailed events population observed paths
and trends
│ │ │
└────────────────┼────────────────┘
▼
CORRELATED EVIDENCE
│
▼
DETECTION, DIAGNOSIS, RECOVERY
Metrics show the weather moving across the city.
Traces follow one observed journey through its streets.
Logs preserve what happened at particular doors along the way.
A useful observability system does not ask one lens to do every job.
It aligns them closely enough that an operator can begin with:
Something changed.
and eventually reach:
This operation followed this observed path,
failed here for this recorded reason,
affected this population,
and requires this recovery action.
That is the difference between collecting telemetry and being able to use it.
References and further reading
OpenTelemetry documentation Introduces vendor-neutral instrumentation, context propagation, telemetry SDKs and the OpenTelemetry Collector.
OpenTelemetry signals Explains the logs, metrics, traces and baggage models supported by OpenTelemetry.
W3C Trace Context Defines standard HTTP headers for propagating trace identity across distributed systems.
Prometheus data model Documents time-series metrics, labels and the Prometheus metric model.
Prometheus metric types Introduces counters, gauges, histograms and summaries.
Google SRE: Monitoring Distributed Systems Discusses monitoring signals, alerting and the operational value of latency, traffic, errors and saturation.
Google SRE Workbook: Implementing SLOs Explores service-level indicators, objectives and error-budget-based reliability management.
Distributed Systems Observability, Cindy Sridharan Examines observability practices for understanding complex distributed systems.