An application exports a trace.
The simplest path is direct:
APPLICATION
│
▼
TRACE BACKEND
That can work perfectly well.
The application knows the backend endpoint, configures the appropriate exporter and sends its spans there.
Then the platform changes.
Metrics need to go to one storage system.
Traces need to go somewhere else.
Security wants sensitive attributes removed before export.
Production telemetry needs a different retention destination from development telemetry.
A second tracing backend must receive copies during a migration.
The application now needs to understand:
- several destinations
- several credentials
- backend-specific exporters
- routing rules
- retry behaviour
- filtering policy
- migration configuration
The observability decision has leaked into every workload.
A collector changes the boundary:
APPLICATION
│
▼
OPENTELEMETRY COLLECTOR
│
├──► LOG BACKEND
├──► METRICS BACKEND
└──► TRACE BACKEND
The application exports telemetry through a common protocol.
The Collector decides what happens next.
The OpenTelemetry Collector is a telemetry router: it receives signals, applies platform policy and delivers them to one or more destinations.
Calling it a router is useful because routing is one of its clearest responsibilities.
But the Collector does more than forward packets.
It can reshape the telemetry stream while keeping that platform logic outside the application.
The Collector has three basic stages
A Collector pipeline is built from:
RECEIVERS
│
▼
PROCESSORS
│
▼
EXPORTERS
A receiver accepts telemetry.
A processor changes, groups, filters or controls it.
An exporter sends it onwards.
A simplified traces pipeline might look like:
service:
pipelines:
traces:
receivers:
- otlp
processors:
- memory_limiter
- batch
exporters:
- traces_backend
The flow is:
OTLP TELEMETRY
│
▼
OTLP RECEIVER
│
▼
MEMORY LIMITER
│
▼
BATCH PROCESSOR
│
▼
TRACE EXPORTER
The same Collector can have separate pipelines for logs, metrics and traces:
service:
pipelines:
logs:
receivers:
- otlp
processors:
- memory_limiter
- batch
exporters:
- logs_backend
metrics:
receivers:
- otlp
processors:
- memory_limiter
- batch
exporters:
- metrics_backend
traces:
receivers:
- otlp
processors:
- memory_limiter
- batch
exporters:
- traces_backend
Each signal can follow a different route.
The applications do not need to know the details of those routes.
Receivers define how telemetry enters
The Collector can receive telemetry through several protocols and integrations.
For OpenTelemetry-instrumented applications, OTLP is the natural boundary:
APPLICATION SDK
│
│ OTLP
▼
COLLECTOR
Other receivers may ingest telemetry from existing systems or infrastructure.
This allows the Collector to sit between several kinds of producer and a common platform pipeline:
OTLP APPLICATIONS ────┐
│
HOST SIGNALS ─────────┼──► COLLECTOR
│
OTHER TELEMETRY ──────┘
The receiver normalises incoming data into the Collector’s internal telemetry model.
That makes the Collector useful during gradual adoption.
A platform does not need every workload to change instrumentation on the same afternoon.
Existing telemetry sources can continue entering through suitable receivers while newer applications use OpenTelemetry directly.
Exporters define where telemetry leaves
An exporter sends processed telemetry to another system.
Possible destinations include:
- a storage backend
- another Collector
- a message or buffering layer
- a temporary migration destination
- a debugging output
COLLECTOR
│
├──► PRIMARY BACKEND
├──► SECONDARY BACKEND
└──► DEBUG OUTPUT
A pipeline can use more than one exporter:
service:
pipelines:
traces:
receivers:
- otlp
processors:
- batch
exporters:
- current_backend
- candidate_backend
The same trace population can now be delivered to two backends.
This is useful for:
- backend evaluation
- migration
- disaster-recovery testing
- temporary validation
- specialised analysis
The applications remain unchanged.
The Collector becomes the place where the destination decision lives.
Processors turn forwarding into policy
Without processors, the Collector could behave mainly as a relay.
Processors make it a policy boundary.
A processor may:
- batch records
- limit memory use
- add attributes
- remove attributes
- filter telemetry
- transform fields
- sample traces
- detect and group related spans
RAW TELEMETRY
│
▼
PROCESSING POLICY
│
▼
EXPORTABLE TELEMETRY
This creates a useful separation:
APPLICATION
describes what happened
COLLECTOR
applies platform-wide handling rules
The service owner should still instrument responsibly.
The Collector can then enforce or supplement common platform behaviour.
Batching makes export more efficient
Sending every telemetry record as a separate network request would be expensive.
A batch processor groups records before export:
SPAN
SPAN
SPAN
SPAN
│
▼
BATCH
│
▼
EXPORT REQUEST
This can reduce:
- connection overhead
- request volume
- exporter work
- backend ingestion overhead
Batching introduces delay.
A span may wait briefly in memory before being exported.
That is usually acceptable for diagnostic telemetry.
The important point is that export behaviour can be tuned centrally rather than implemented separately in every service.
Memory needs an explicit boundary
The backend may slow down or become unavailable.
Telemetry may continue arriving.
Without limits, the Collector could keep buffering until it exhausts memory.
APPLICATIONS
continue sending
│
▼
COLLECTOR BUFFER
grows
│
▼
MEMORY EXHAUSTION
A memory-limiting processor helps protect the Collector itself.
When capacity is exhausted, the Collector may refuse or drop telemetry rather than allowing unbounded growth.
That is an uncomfortable trade-off.
It is generally better than allowing the telemetry pipeline to consume the node and perhaps affect unrelated workloads.
BACKEND FAILURE
│
▼
BOUNDED RETRY
│
▼
CAPACITY EXHAUSTED
│
▼
TELEMETRY LOST
The loss must be observable.
A bounded failure that produces clear signals is safer than invisible buffering followed by process collapse.
Routing can use telemetry attributes
Suppose one Collector receives telemetry from several environments:
development
staging
production
Resource attributes identify the source:
deployment.environment:
prod
The Collector can route production telemetry differently from development telemetry:
┌──► DEV BACKEND
TELEMETRY ──► ROUTER ───┼──► STAGING BACKEND
└──► PROD BACKEND
Routing could also consider:
- service name
- cloud account
- cluster
- Namespace
- signal type
- tenant
- data classification
This creates powerful possibilities.
It also creates risk.
A routing error could send production telemetry to the wrong environment or cause one tenant’s data to enter another tenant’s destination.
Routing policy should therefore be:
- explicit
- tested
- version controlled
- reviewed
- observable
The Collector is not only moving data.
It is enforcing part of the telemetry boundary.
Enrichment adds platform context
Applications may know their service name and version.
The platform may know additional context:
- cluster name
- Namespace
- node
- cloud account
- region
- environment
- workload identity
The Collector can enrich telemetry with this information:
APPLICATION SPAN
service.name:
catalog-service
COLLECTOR ENRICHMENT
environment:
prod
cluster:
bfstore-prod
region:
eu-west-2
The exported span becomes easier to investigate.
An operator can ask:
Are errors limited to one cluster?
Did latency increase in one region?
Is one environment missing telemetry?
Which account produced this signal?
Central enrichment improves consistency.
It also means the Collector configuration becomes part of the telemetry schema.
Changing or removing attributes can break:
- dashboards
- alerts
- queries
- routing rules
- retention policies
Platform-added metadata should be treated as a maintained contract.
Filtering can reduce noise and risk
Not every emitted signal needs long-term storage.
A Collector can filter:
- low-value health-check spans
- noisy development logs
- known duplicate events
- records lacking required attributes
- telemetry from unsupported sources
INCOMING TELEMETRY
│
▼
FILTER POLICY
│
├── retain
└── discard
Filtering can reduce cost and noise.
It can also remove evidence required during an incident.
A rule that looks harmless:
discard successful health checks
may become important when investigating intermittent routing failures.
Filtering policy should therefore answer:
What value does this signal provide?
Why is it safe to discard?
Which environments does the rule affect?
Can the effect be measured?
How can the rule be reversed?
The Collector makes removal easy.
That is precisely why removal needs care.
Transformation can standardise telemetry
Several services may use inconsistent attribute names:
environment
env
deployment_env
A transformation processor can convert them into a shared form:
deployment.environment
This can help during migration or standardisation.
INCONSISTENT INPUTS
│
▼
TRANSFORMATION
│
▼
COMMON SCHEMA
Transformation is useful when the Collector is bridging old and new instrumentation conventions.
It should not become a permanent excuse for careless application telemetry.
The closer an attribute is to its source, the more accurately the application can assign its meaning.
The Collector can repair structure.
It may not understand the domain well enough to repair intent.
Sensitive data can be removed before export
Telemetry can accidentally contain:
- credentials
- authentication headers
- customer data
- payment details
- request payloads
- internal identifiers
The safest approach is not to emit sensitive information.
But the Collector can provide another defensive layer.
APPLICATION TELEMETRY
│
▼
REDACTION OR ATTRIBUTE REMOVAL
│
▼
BACKEND
For example, it may remove:
http.request.header.authorization
before export.
This reduces the number of systems through which the sensitive value travels.
It does not erase every earlier exposure.
The value may already have existed in:
- application memory
- SDK buffers
- network traffic to the Collector
- local Collector memory
Collector-side redaction is valuable defence in depth.
Source-side prevention remains stronger.
Sampling can happen centrally
A platform may generate too many traces to retain them all.
A Collector gateway can apply tail sampling.
It waits until enough spans from a trace have arrived, then makes a decision using the trace’s outcome.
COMPLETE OR PARTIAL TRACE
│
▼
SAMPLING POLICY
│
├── keep errors
├── keep slow traces
├── keep selected services
└── sample ordinary success
This can preserve traces that are operationally valuable.
For example:
keep all traces where:
status = error
or
duration > 2 seconds
Central sampling prevents every application from implementing a different retention decision.
It introduces state and capacity requirements into the Collector layer.
The Collector must receive enough spans from each trace to decide correctly.
If related spans are split unpredictably across gateways, the sampling decision may see only fragments.
Telemetry routing and load balancing therefore affect the correctness of tail sampling.
The Collector can be deployed as an agent
An agent Collector runs close to workloads.
In Kubernetes, this may mean one Collector per node.
NODE
│
├── APPLICATION A
├── APPLICATION B
└── COLLECTOR AGENT
Applications send telemetry to the nearby agent.
The agent can:
- receive local OTLP traffic
- collect node or container context
- batch records
- forward them centrally
APPLICATIONS
│
▼
LOCAL AGENT
│
▼
CENTRAL GATEWAY
Advantages can include:
- a local export endpoint
- reduced repeated workload configuration
- node-level enrichment
- isolation from brief central network interruption
The agent still consumes resources on each node.
Its failure may affect telemetry from every workload on that node.
The Collector can be deployed as a gateway
A gateway Collector runs as a shared service.
APPLICATION A ──┐
APPLICATION B ──┼──► COLLECTOR GATEWAY
APPLICATION C ──┘
The gateway can provide central:
- authentication
- routing
- filtering
- tail sampling
- export policy
- backend credentials
This concentrates policy.
It also concentrates traffic and failure risk.
The gateway needs:
- sufficient capacity
- horizontal scaling
- load balancing
- availability across failure domains
- bounded queues
- clear ownership
A single undersized gateway can turn observability into a very organised traffic jam.
Agent and gateway layers can work together
A larger platform may use both:
APPLICATIONS
│
▼
NODE AGENTS
│
▼
COLLECTOR GATEWAYS
│
▼
OBSERVABILITY BACKENDS
The agent handles local collection and enrichment.
The gateway handles organisation-wide policy and routing.
This separates responsibilities:
AGENT
local reception
node context
short-distance buffering
GATEWAY
central policy
sampling
routing
backend export
The design is more capable.
It is also more complex.
Every layer adds:
- configuration
- capacity planning
- deployment work
- failure modes
- upgrade requirements
- observability needs
The topology should follow a real platform requirement rather than a desire to collect every Collector deployment pattern like ceremonial teaspoons.
The Collector should not become a hidden business service
The Collector can transform and route data.
That does not mean it should contain application-domain logic.
For example, it may be reasonable to route payment-service telemetry to a restricted backend.
It is less attractive to make the Collector decide:
Was this payment business outcome acceptable?
That meaning belongs closer to the payment domain.
APPLICATION
understands business outcome
COLLECTOR
understands telemetry handling policy
A clear boundary keeps Collector configuration manageable.
It should operate on telemetry attributes and platform policy, not become an alternative application runtime written entirely in processor configuration.
The router needs security controls
Telemetry can reveal valuable information about the platform:
- service names
- architecture
- internal endpoints
- failure patterns
- deployment versions
- customer activity
- business identifiers
The Collector should not be treated as an unauthenticated public bin.
Security questions include:
Which workloads may send telemetry?
How are Collector endpoints authenticated?
Is telemetry encrypted in transit?
Which backends may the Collector access?
Where are exporter credentials stored?
Can one tenant influence another tenant’s pipeline?
The Collector identity should have only the authority required to write to its approved destinations.
Applications should not receive backend credentials merely because they produce telemetry.
APPLICATION IDENTITY
send telemetry to Collector
COLLECTOR IDENTITY
write telemetry to backend
The Collector becomes a trust boundary as well as a routing boundary.
Backend migration becomes a routing change
Suppose bfstore currently sends traces to backend A.
APPLICATIONS
│
▼
COLLECTOR
│
▼
BACKEND A
The platform wants to evaluate backend B.
It can temporarily dual-export:
APPLICATIONS
│
▼
COLLECTOR
│
├──► BACKEND A
└──► BACKEND B
The platform can compare:
- ingestion success
- query experience
- retention behaviour
- storage cost
- trace completeness
- operational complexity
Once backend B is accepted, the old exporter can be removed.
APPLICATIONS
│
▼
COLLECTOR
│
▼
BACKEND B
The application instrumentation and OTLP destination remain stable.
Dashboards, alerts and queries may still require migration.
The Collector reduces one important category of coupling.
It does not make the rest of the observability product magically portable.
The Collector must observe itself
The Collector can lose telemetry because:
- a receiver rejects records
- a processor drops them
- a queue fills
- an exporter fails
- a backend throttles requests
- memory limits are reached
- configuration is invalid
The platform should monitor:
records received
records accepted
records refused
records dropped
records exported
export failures
queue utilisation
memory use
processor latency
A useful conservation question is:
What entered the pipeline?
What left the pipeline?
Where did the difference go?
Not every count will align perfectly because of batching, sampling and asynchronous processing.
Large unexplained differences still deserve investigation.
Missing telemetry should not be interpreted automatically as healthy systems.
Sometimes the messenger has fallen into the ravine.
A possible bfstore telemetry route
Suppose bfstore services export OTLP to a local Collector agent.
CATALOG SERVICE ─────┐
BASKET SERVICE ──────┼──► NODE COLLECTOR
ORDER SERVICE ───────┘
The agent:
- receives OTLP
- adds node and cluster metadata
- batches telemetry
- forwards it to a gateway
NODE COLLECTORS
│
▼
OBSERVABILITY GATEWAY
The gateway:
- removes prohibited attributes
- applies environment routing
- performs tail sampling
- exports each signal
OBSERVABILITY GATEWAY
│
├── logs ───► log storage
├── metrics ─► metrics storage
└── traces ──► trace storage
Production traces may follow a stricter sampling and retention path than development traces.
Security telemetry may go to a restricted destination.
During a backend migration, traces may be copied to both old and new storage systems.
The services continue exporting OTLP to the same platform endpoint.
The Collector layer absorbs the routing change.
Questions I now ask about a Collector pipeline
Entry
Which receivers accept telemetry,
and from which trusted producers?
Processing
Which records are changed,
filtered, sampled or enriched?
Routing
Which attributes determine
the destination?
Export
Which backends receive each signal?
Capacity
What happens when incoming volume
exceeds processing or export capacity?
Failure
How long will the Collector retry,
and when will telemetry be dropped?
Security
How are producers, Collector endpoints
and backend credentials protected?
Privacy
Which sensitive attributes
must be removed before export?
Availability
Which workloads lose telemetry
when one Collector instance fails?
Visibility
Can operators detect rejection,
delay, sampling and loss?
Evolution
Can processing and routing policy change
without rewriting applications?
These questions test whether the Collector is an intentional platform boundary or simply another container occupying a respectable-looking box in the architecture diagram.
The mental model I am keeping
My earlier model was:
OPENTELEMETRY COLLECTOR
a process that forwards telemetry
to the backend
The stronger model is:
TELEMETRY PRODUCERS
│
▼
RECEIVERS
│
▼
PROCESSORS
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
ENRICH FILTER SAMPLE
│ │ │
└───────────────────┼───────────────────┘
▼
ROUTE
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
LOG EXPORTER METRIC EXPORTER TRACE EXPORTER
│ │ │
▼ ▼ ▼
BACKEND BACKEND BACKEND
Receivers answer:
How does telemetry enter
the platform pipeline?
Processors answer:
Which platform policies apply
before storage?
Routing answers:
Which destination should receive
this telemetry?
Exporters answer:
How is the signal delivered
to that destination?
Memory limits answer:
How is the Collector protected
when downstream systems slow down?
Sampling answers:
Which operational stories
will be retained?
Enrichment answers:
Which platform context should
be added consistently?
Filtering answers:
Which telemetry should not
continue through the pipeline?
Security answers:
Who may send, process
and store the data?
Collector telemetry answers:
Is the observability pipeline
itself healthy?
The OpenTelemetry Collector gives the platform a place to make telemetry-handling decisions without embedding them into every service.
Applications describe what happened.
The Collector decides how that account should travel.
It can add routing labels, remove dangerous fields, group records, control memory, select traces and deliver signals to several destinations.
That makes it more than a pipe.
It is a programmable checkpoint between telemetry production and telemetry storage.
The Collector becomes valuable not because telemetry passes through it, but because the platform gains one observable, governable place to decide where that telemetry goes and what must happen before it arrives.