Getting AWS telemetry into a self-hosted observability platform

bfstore’s application services run in Amazon EKS.

The platform also uses AWS-managed services and control planes:

• Application Load Balancers
• Network Load Balancers
• Amazon RDS for MySQL
• Amazon VPC
• AWS Transit Gateway
• AWS CloudTrail
• AWS Config
• AWS IAM
• Amazon EKS
• Route 53 Resolver

The platform is self-hosted.

Its backends may include:

• a Prometheus-compatible metrics system
• Thanos for durable metrics
• ClickHouse or OpenSearch for logs and events
• an OpenTelemetry-compatible trace store
• Grafana for visualisation

The first architecture sketch is tempting:

AWS
 │
 ▼
OpenTelemetry Collector
 │
 ▼
self-hosted observability

It is also too vague to build.

AWS telemetry does not originate through one universal interface.

A bfstore Go service can emit OTLP directly.

An EKS control plane cannot.

An RDS instance exposes metrics through CloudWatch and selected engine logs through CloudWatch Logs.

VPC and Transit Gateway Flow Logs can be delivered to CloudWatch Logs, Amazon S3 or Amazon Data Firehose.

CloudTrail trails primarily deliver durable event files to S3, with optional CloudWatch Logs and EventBridge integrations.

The phrase:

export AWS telemetry

therefore hides several different jobs.

A stronger architecture begins by separating three source classes.

WORKLOAD-NATIVE TELEMETRY

bfstore application logs
application metrics
distributed traces
runtime metrics


PLATFORM TELEMETRY

Kubernetes node metrics
Pod and container metrics
cluster object state
container logs


AWS-MANAGED TELEMETRY

CloudWatch metrics
managed-service logs
flow logs
CloudTrail
AWS Config
AWS service events

Self-hosted observability does not mean pretending CloudWatch does not exist. It means using the most appropriate AWS collection path while retaining control of the platform where telemetry is stored, correlated and analysed.

The source determines the collection path.

The platform determines the final destination.

Collection, transport and storage are different decisions

A telemetry pipeline has several distinct stages.

SOURCE
   │
   ▼
COLLECTION
   │
   ▼
TRANSPORT
   │
   ▼
BUFFER
   │
   ▼
PROCESSING
   │
   ▼
BACKEND

Source

Where did the signal originate?

Examples:

• Go application
• Kubernetes node
• EKS control plane
• RDS MySQL
• VPC network interface
• AWS API call

Collection

Which system can obtain the signal?

Examples:

• OpenTelemetry SDK
• OpenTelemetry Collector
• CloudWatch
• VPC Flow Logs
• CloudTrail
• AWS Config

Transport

How does the signal leave the collection system?

Examples:

• OTLP
• Prometheus remote write
• CloudWatch Logs subscription
• CloudWatch metric stream
• Kinesis Data Streams
• Amazon Data Firehose
• S3 object delivery

Buffer

Where can telemetry wait safely when the destination is unavailable?

Examples:

• Collector sending queue
• Kinesis retention
• Firehose buffering
• S3 landing bucket
• local persistent storage

Processing

What must happen before storage?

Examples:

• decompression
• schema parsing
• field removal
• resource enrichment
• deduplication
• sampling
• format conversion

Backend

Which system answers future questions?

Examples:

• Thanos
• ClickHouse
• OpenSearch
• trace store
• long-term S3 archive

Treating these stages separately prevents one product from owning every concern.

An OpenTelemetry Collector is excellent at receiving, enriching, batching and exporting telemetry, but it is not automatically a durable broker.

S3 provides durable retention, not low-latency metric queries.

CloudWatch is the native source for many AWS-managed signals.

It does not have to be bfstore’s final operational interface.

Application telemetry should take the shortest path

bfstore controls the code of its Go services.

That makes application telemetry the easiest category to design cleanly.

The services can emit:

• traces through OTLP
• metrics through OTLP or Prometheus
• structured logs through stdout or OTLP

The preferred path is:

GO SERVICE
    │
    ▼
LOCAL OR NODE COLLECTOR
    │
    ▼
CLUSTER GATEWAY COLLECTOR
    │
    ▼
PRIVATE OBSERVABILITY INGEST
    │
    ▼
SELF-HOSTED BACKENDS

OpenTelemetry documents an agent-to-gateway pattern in which agent Collectors run close to workloads and gateway Collectors provide centralised processing and export.

For bfstore, the node-level collector can handle:

• container log collection
• node metadata
• kubelet metrics
• local OTLP reception
• initial batching
• node and Pod enrichment

The gateway can handle:

• central filtering
• sensitive-field removal
• tenant or environment attribution
• retry and queue policy
• backend routing
• trace sampling

Tail sampling needs one important qualification.

A tail-sampling processor holds spans while it decides whether to retain the completed trace. When several gateway replicas receive different spans from the same trace, each replica can make an incomplete or conflicting decision.

TRACE SPANS
     │
     ▼
TRACE-ID-AWARE ROUTING
     │
     ▼
ONE TAIL-SAMPLING COLLECTOR
     │
     ▼
SAMPLING DECISION

A horizontally scaled tail-sampling tier should therefore sit behind trace-ID-aware routing, commonly an OpenTelemetry load-balancing exporter. Ordinary round-robin load balancing is not enough.

A simplified gateway configuration might look like:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

  batch:
    timeout: 5s
    send_batch_size: 4096

  resource:
    attributes:
      - key: cloud.provider
        value: aws
        action: upsert

exporters:
  otlp/metrics:
    endpoint: metrics-ingest.observability.internal:4317

  otlp/traces:
    endpoint: traces-ingest.observability.internal:4317

  otlp/logs:
    endpoint: logs-ingest.observability.internal:4317

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/metrics]

    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/traces]

    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/logs]

This example shows pipeline separation only. A production configuration also needs:

• TLS or mTLS
• client authentication
• bounded retry policies
• exporter sending queues
• persistent queue storage where required
• Collector health checks
• queue and rejection monitoring

The Collector’s default in-memory queue protects only short interruptions. Data can still be lost when the queue fills, retries expire or the Collector restarts. Persistent file_storage can preserve queued data across a Collector restart, but it still does not turn the Collector into a dedicated durable broker.

AWS Distro for OpenTelemetry is AWS’s supported distribution of the upstream project and can send data to OTLP-compatible and other observability backends. It does not require CloudWatch to be the destination.

The important principle is:

Do not send bfstore application telemetry through CloudWatch merely to export it again when the application can send OTLP directly to the self-hosted platform.

That extra path would add:

• another ingestion charge
• another schema conversion
• another delivery dependency
• additional delay
• duplicate retention

CloudWatch remains valuable for AWS-managed signals.

Application-native telemetry can take the more direct road.

Kubernetes telemetry belongs close to the node

Application instrumentation does not reveal everything happening around the process.

The platform also needs:

• node CPU and memory
• filesystem capacity
• Pod and container resource use
• container restarts
• kubelet behaviour
• Kubernetes object state
• container stdout and stderr
• scheduling and rollout state

A node-level Collector or equivalent agent can collect much of this.

OpenTelemetry’s Kubernetes guidance identifies the kubeletstats receiver for node, Pod and container metrics, the file-log receiver for container logs, and host-metrics collection for operating-system resources. These node-specific receivers are normally deployed in a DaemonSet so each instance observes its own node.

EKS NODE
│
├── bfstore containers
├── container log files
├── kubelet
├── node filesystem
│
└── telemetry DaemonSet
    ├── file-log receiver
    ├── kubelet metrics
    ├── host metrics
    └── OTLP receiver

Cluster-wide state requires another view.

Metrics such as:

• desired versus available replicas
• failed Jobs
• pending Pods
• node conditions
• resource requests and limits
• deployment generations

are commonly exposed through kube-state-metrics and scraped by a Prometheus-compatible collector.

The useful split is:

NODE-LOCAL OBSERVATION

DaemonSet collector


CLUSTER OBJECT STATE

kube-state-metrics


APPLICATION SEMANTICS

OpenTelemetry SDK

These sources overlap in places.

The platform should choose one authoritative source for each metric family rather than collecting the same series through several agents and later wondering which copy is the original goblin.

AWS-managed signals should follow their native exits

AWS services expose telemetry through different collection and delivery surfaces.

A compact source map makes the architecture clearer than pretending every service begins in CloudWatch Logs.

The reusable patterns are:

AWS SERVICE
    → CLOUDWATCH LOGS
    → SUBSCRIPTION
    → KINESIS OR FIREHOSE


AWS SERVICE
    → FIREHOSE
    → S3 OR HTTPS ADAPTER


AWS SERVICE
    → S3
    → SQS
    → PRIVATE IMPORTER

EKS control-plane logs

bfstore does not operate the EKS control-plane hosts.

The cluster can export five control-plane log types:

• API server
• audit
• authenticator
• controller manager
• scheduler

Amazon EKS sends enabled control-plane logs to CloudWatch Logs. The log types are disabled by default and can be enabled independently.

For bfstore, I would enable at least:

• audit
• authenticator
• api

The controller-manager and scheduler logs should also be enabled where their operational value justifies the volume.

eks_control_plane_logging:
  enabled:
    - api
    - audit
    - authenticator
    - controllerManager
    - scheduler

  cloudwatch_retention_days: 14
  export_to_observability: true
  archive_policy: security-long-term

CloudWatch is the collection surface here.

It does not have to become the permanent analytical home.

RDS logs

RDS can publish selected database logs to CloudWatch Logs.

For MySQL, this can include:

• error log
• general log
• slow-query log
• audit log when the required option is configured

RDS Enhanced Monitoring also writes operating-system metrics into CloudWatch Logs rather than ordinary CloudWatch metric series. Those records need to enter the log pipeline and be parsed into metrics if bfstore wants them in the metrics backend.

This is a useful reminder:

SIGNAL MEANING

does not always equal

AWS TRANSPORT TYPE

An operating-system metric can arrive inside a log stream.

Load-balancer logs

Publication note: When this article was originally published on 18 July 2026, the established ALB access-log path was S3. AWS added vended ALB delivery to CloudWatch Logs, Amazon Data Firehose and S3 on 23 July 2026.

The current ALB integration covers access, connection and health-check logs.

NLB also provides vended delivery to CloudWatch Logs, Firehose and S3, while its legacy S3 path remains available. The choice depends on latency, cost and retention.

LOWER-LATENCY SEARCH

ALB OR NLB
    → CLOUDWATCH LOGS
    → SUBSCRIPTION


MANAGED DELIVERY

ALB OR NLB
    → FIREHOSE
    → S3 OR ADAPTER


DURABLE BATCH

ALB OR NLB
    → S3
    → SQS IMPORTER

Flow logs and Resolver query logs

VPC Flow Logs, Transit Gateway Flow Logs and Route 53 Resolver query logs can all deliver to CloudWatch Logs, Firehose or S3.

For high-volume network records, I would avoid CloudWatch Logs unless CloudWatch querying or subscription processing is specifically useful.

FLOW OR DNS QUERY LOGS
          │
          ▼
      FIREHOSE OR S3
          │
          ▼
    SPECIALISED PARSER
          │
          ▼
CLICKHOUSE / SECURITY ANALYTICS

Flow logs are collected outside the packet path and do not affect traffic throughput or latency.

They are also aggregated and delivered on a best-effort basis.

They are excellent network evidence.

They are not packet capture.

Exporting AWS-managed metrics

Many AWS services publish operational metrics to CloudWatch.

Examples include:

• ALB request and target metrics
• NLB flow metrics
• RDS CPU, storage and connection metrics
• NAT gateway errors and bytes
• Transit Gateway metrics
• EBS volume metrics
• S3 request metrics where enabled
• EKS and EC2 infrastructure metrics

There are two primary export models.

Stream broadly

CloudWatch metric streams continually send selected metrics to Amazon Data Firehose.

Metric streams support JSON and OpenTelemetry formats, including OpenTelemetry 1.0.0. The Firehose delivery stream exists in the same account and Region as the metric stream, although its final destination can be elsewhere.

CLOUDWATCH METRICS
        │
        ▼
METRIC STREAM
        │
        ▼
AMAZON DATA FIREHOSE
        │
        ▼
S3 OR HTTPS ADAPTER
        │
        ▼
NORMALISATION
        │
        ▼
SELF-HOSTED METRICS

Streaming is the stronger choice when bfstore needs:

• broad namespace coverage
• low delivery latency
• automatic inclusion of matching new metrics
• no repeated metric discovery

Metric-stream payloads still need an ingestion component that understands the selected output format and maps AWS dimensions into the target model.

The words OpenTelemetry format do not mean that Firehose sends ordinary OTLP requests to any OTLP endpoint.

The consumer must implement the Firehose delivery contract and the metric-stream record format it actually receives.

Poll selectively

A service can periodically call CloudWatch APIs such as ListMetrics and GetMetricData.

SELF-HOSTED COLLECTOR
        │
        ▼
CLOUDWATCH API
        │
        ▼
GET METRIC DATA
        │
        ▼
PROMETHEUS-COMPATIBLE SERIES

Polling may be suitable when:

• the metric set is small
• a mature CloudWatch exporter already exists
• delivery can be delayed by the scrape interval
• bfstore wants precise metric selection
• a streaming decoder would be premature

Polling becomes harder as the estate grows.

It introduces:

• API request volume
• metric and dimension discovery
• delayed retrieval
• repeated historical windows
• duplicate-point handling
• cross-account credential management
• missing-series detection

The decision is not:

streaming is modern

polling is old

It is:

STREAM

for broad continuous export


POLL

for a curated metric set
or an early implementation

Multi-account metric options

bfstore’s AWS estate spans several accounts.

Three CloudWatch mechanisms should not be treated as interchangeable.

Source-account metric streams

Each account owns its own metric stream and Firehose delivery path.

Advantages:

• explicit source ownership
• independent failure domains
• straightforward account boundaries

Costs:

• repeated configuration
• more delivery streams
• more export monitoring

Cross-account observability with a monitoring-account stream

CloudWatch cross-account observability links source accounts to a monitoring account within a Region.

A metric stream created in the monitoring account can include metrics from linked source accounts. This is the clearest central streaming option when bfstore needs AWS-vended service metrics from several accounts without creating a separate stream in every source account.

SOURCE ACCOUNTS
      │
      ▼
CROSS-ACCOUNT OBSERVABILITY
      │
      ▼
MONITORING ACCOUNT
METRIC STREAM
      │
      ▼
FIREHOSE

This still creates a central policy and failure boundary.

The stream also remains Regional.

Metrics Centralization

CloudWatch Metrics Centralization copies metric data from selected AWS Organization accounts and Regions into a destination account while preserving source account and Region metadata.

The current documentation states that all metrics from the selected source accounts are centralized and explicitly identifies custom metrics, Embedded Metric Format metrics and OpenTelemetry metrics. Selective metric filtering is not currently supported.

Centralized metrics can be used by CloudWatch features including metric streams.

SELECTED ORGANIZATION ACCOUNTS
             │
             ▼
METRICS CENTRALIZATION
             │
             ▼
DESTINATION ACCOUNT COPY
             │
             ▼
OPTIONAL METRIC STREAM

This is a replication model, not merely permission to view source-account metrics. Use it when a CloudWatch-owned central copy is valuable and centralizing every supported metric from the selected accounts is acceptable. It is not a universal replacement for source-account streams or cross-account observability.

The architecture should compare:

• whether metrics are viewed or copied
• whether filtering is required
• whether AWS-vended metrics are in scope
• Regional coverage
• stream ownership
• replication cost
• shared failure boundaries

Firehose is a useful buffer, with one private-network catch

Amazon Data Firehose can buffer, transform and deliver records to several destinations.

For custom HTTP delivery, it can:

• send HTTPS requests
• retry failed delivery for a configured duration
• back up failed or all data to S3
• record delivery errors in CloudWatch Logs
• retrieve destination credentials from Secrets Manager

When retry duration expires for an HTTP endpoint, Firehose writes failed data to the configured S3 backup location.

That makes it attractive as a managed edge buffer.

However, Firehose does not support delivery to custom HTTP endpoints inside a VPC. Custom HTTP delivery uses HTTPS on port 443 and requires a Firehose-compatible request and response contract.

This rules out the deceptively neat design:

FIREHOSE
    │
    ▼
PRIVATE OTLP ENDPOINT
inside observability VPC

Firehose cannot simply call that private endpoint.

bfstore has three practical options.

S3 landing zone

FIREHOSE
    │
    ▼
S3 LANDING BUCKET
    │
    ▼
S3 EVENT
    │
    ▼
SQS
    │
    ▼
INGESTION WORKERS
    │
    ▼
SELF-HOSTED BACKEND

This provides durable replay and a private backend, at the cost of additional latency and importer logic.

Public hardened adapter

FIREHOSE
    │
    ▼
PUBLIC HTTPS ADAPTER
    │
    ▼
PRIVATE OBSERVABILITY PIPELINE

The adapter is not the backend itself.

It implements:

• Firehose request contract
• authentication
• request deduplication
• decompression
• payload validation
• rate limiting
• private forwarding

Avoid Firehose HTTP delivery

For CloudWatch Logs, use Kinesis Data Streams and run private consumers.

For a small metric set, poll CloudWatch instead of creating a metric stream.

Metric streams require Firehose, so a private near-real-time metric destination still needs an S3 landing path, a controlled public adapter or a different export model.

The constraint belongs in the design early.

It should not be discovered after bfstore has built a beautiful internal OTLP endpoint that Firehose can admire only from afar.

Exporting AWS logs, events and durable evidence

CloudWatch Logs subscriptions

CloudWatch Logs subscriptions can forward newly ingested events to:

• Kinesis Data Streams
• Amazon Data Firehose
• Lambda
• Amazon OpenSearch Service

The subscription payload is Base64-encoded and GZIP-compressed.

Delivery is at least once, so duplicate events can occur. Retryable failures are retried for a limited period, after which undelivered events can be dropped.

A private self-hosted path can use Kinesis:

CLOUDWATCH LOG GROUPS
        │
        ▼
ACCOUNT-LEVEL SUBSCRIPTION
        │
        ▼
CROSS-ACCOUNT KINESIS STREAM
        │
        ▼
PRIVATE CONSUMER
        │
        ▼
DECOMPRESS AND NORMALISE
        │
        ▼
CLICKHOUSE OR OPENSEARCH

The consumer must decode and decompress the payload, parse the CloudWatch Logs envelope, preserve source metadata, deduplicate where useful, submit events to the backend and checkpoint only after durable acceptance.

Account-level subscriptions reduce the chance that a newly created log group remains unexported.

They also increase the risk of:

• sending unwanted high-volume logs
• exceeding stream capacity
• creating recursive logging loops

The export workflow’s own log groups must be excluded, otherwise the pipeline can ingest its own operational logs and manufacture a surprisingly energetic invoice.

Central searchable logs and the archive are different systems

bfstore already has separate accounts for:

• log archive
• observability

Those accounts should not collapse into one vague “logging account”.

AWS SOURCE
    │
    ├──────────► LOG ARCHIVE
    │             durable evidence
    │             restricted mutation
    │
    └──────────► OBSERVABILITY
                  indexed and queryable
                  operational retention

bfstore-observability exists for:

• dashboards
• alerts
• operational search
• correlations
• service troubleshooting

bfstore-log-archive exists for:

• durable security evidence
• policy-driven retention
• restricted deletion
• central CloudTrail files
• AWS Config history
• selected network and service records
• forensic recovery

An S3 bucket is not immutable merely because it lives in a log-archive account.

Where immutability is required, the design should explicitly include:

• S3 Versioning
• S3 Object Lock
• compliance-mode retention where policy requires it
• restricted bucket and KMS administration
• lifecycle controls
• CloudTrail log-file integrity validation

Object Lock prevents protected object versions from being overwritten or deleted for the retention period. CloudTrail integrity validation helps detect whether delivered files were changed or deleted. It does not itself prevent modification.

Searchability is not durability.

Durability is not immutability.

Each property needs its own control.

CloudTrail and AWS Config

The primary CloudTrail path should remain:

AWS API ACTIVITY
       │
       ▼
ORGANISATION CLOUDTRAIL
       │
       ▼
bfstore-log-archive S3

Selected operational events can also travel through CloudWatch Logs or EventBridge.

The indexed copy improves search and correlation.

It should not replace the authoritative S3 trail and digest files.

AWS Config records resource configuration state and change history.

Its delivery channel can send:

• configuration histories and snapshots to S3
• change notifications to SNS

Config records answer questions such as:

Which security-group rule changed
before the connection failures began?

When was this bucket policy altered?

Which resource configuration existed
during the incident?

They are structured evidence, not ordinary application logs.

The ingestion platform must provide guarantees

The observability platform will sometimes fail.

Reasons include:

• upgrade
• cluster failure
• network partition
• backend overload
• storage outage
• certificate problem
• schema rejection

The source workload should not fail merely because the telemetry destination is unavailable.

APPLICATION
    │
    ▼
TELEMETRY PIPELINE FAILURE
    │
    ▼
APPLICATION CONTINUES
within defined limits

Bounded buffering and replay

A mature design often uses two levels.

SHORT INTERRUPTION

Collector queue


LONGER INTERRUPTION

Kinesis or S3

Collector queues are useful for brief failures.

Kinesis provides a replayable near-real-time stream with consumer lag and checkpoint responsibilities.

Firehose provides managed buffering, retry and S3 backup.

S3 provides the durable landing zone and the clearest replay source.

No buffer should grow without a limit.

An observability outage should not become a workload outage through an infinitely enthusiastic telemetry queue.

At-least-once delivery and deduplication

CloudWatch Logs subscriptions, Firehose, S3 notifications, Kinesis consumers and OpenTelemetry exporters can all retry.

A record may therefore arrive more than once.

A useful CloudWatch log identity can combine:

source account
source Region
log group
log stream
CloudWatch event ID

A Firehose adapter can retain the stable request ID used across retries.

Trace spans carry trace and span identifiers.

Metric duplicates need series, timestamp and value-aware handling.

The platform should state explicitly:

• where deduplication happens
• which signals are deduplicated
• how long the deduplication window lasts
• what happens when the same timestamp carries conflicting values

At-least-once delivery is a reliability feature.

It moves exactly-once fantasies into the consumer’s data model, where they can be dealt with honestly.

Source identity and schema

Every AWS record should preserve enough context to answer:

Which account?

Which Region?

Which environment?

Which service?

Which cluster?

Which resource?

Which original source?

A common envelope could include:

resource:
  cloud.provider: aws
  cloud.account.id: "123456789012"
  cloud.region: eu-west-2
  deployment.environment.name: production
  service.name: order-service
  service.namespace: bfstore
  k8s.cluster.name: bfstore-prod

source:
  type: aws.eks.audit
  aws.log.group: /aws/eks/bfstore-prod/cluster
  pipeline: cloudwatch-kinesis

The common envelope enables cross-source queries.

The original payload preserves source-specific evidence.

Normalisation should not discard fields merely because the first shared schema does not understand them.

Cardinality controls

Useful AWS and application dimensions include:

• load balancer
• target group
• database instance
• Availability Zone
• service
• route
• status
• cluster
• namespace

Dangerous metric labels include:

• customer ID
• order ID
• trace ID
• request ID
• full URL
• exception message
• unbounded SQL text

METRIC LABEL

bounded grouping field


LOG FIELD

searchable event context


TRACE ATTRIBUTE

request-scoped diagnostic context

An order ID is useful in a trace or log.

It is usually catastrophic as a Prometheus label.

Self-hosting does not make cardinality free.

It changes the person receiving the storage invoice and the 03:00 alert.

Security and cost

Private OTLP paths still need authentication.

A private route proves network reachability.

It does not prove the caller is an authorised producer.

The ingestion boundary should use:

• mTLS or short-lived workload identity
• scoped IAM roles
• narrow S3 prefixes and KMS permissions
• request validation
• sensitive-field redaction
• source-specific retention
• access auditing for security telemetry

The architecture should also count every handoff.

EKS AUDIT EVENT
      │
      ▼
CLOUDWATCH INGESTION
      │
      ▼
CLOUDWATCH RETENTION
      │
      ▼
SUBSCRIPTION DELIVERY
      │
      ▼
KINESIS
      │
      ▼
SELF-HOSTED STORAGE

Each copy should have a purpose.

A duplicate without a purpose is just telemetry practising mitosis on the invoice.

My proposed bfstore architecture

I would use several ingestion lanes rather than one universal pipe.

BFSTORE APPLICATIONS
      │
      ▼
OTEL AGENTS AND GATEWAYS
      │
      ▼
PRIVATE OTLP INGEST
      │
      ▼
SELF-HOSTED BACKENDS


AWS CLOUDWATCH METRICS
      │
      ▼
METRIC STREAM OR POLLER
      │
      ▼
NORMALISATION
      │
      ▼
METRICS BACKEND


AWS CLOUDWATCH LOGS
      │
      ▼
CENTRAL SUBSCRIPTION
      │
      ▼
KINESIS
      │
      ▼
PRIVATE LOG CONSUMERS
      │
      ▼
LOG BACKEND


AWS DURABLE SOURCES
      │
      ├── CLOUDTRAIL
      ├── CONFIG
      ├── LOAD-BALANCER LOGS
      ├── FLOW LOGS
      └── RESOLVER QUERY LOGS
              │
              ▼
          CENTRAL S3
              │
              ▼
         SQS IMPORTERS
              │
              ▼
    SEARCH AND ARCHIVE SYSTEMS

Lane 1: Application and Kubernetes telemetry

• Go services instrumented with OpenTelemetry
• Collector DaemonSet on each EKS node
• gateway Collectors
• trace-ID-aware routing before tail sampling
• private OTLP ingestion
• direct export to self-hosted metrics, log and trace backends
• no CloudWatch round trip

Lane 2: AWS service metrics

Initial phase:

• curated CloudWatch polling
• limited namespaces
• explicit metric inventory

Later phase:

• cross-account observability into bfstore-observability
• central monitoring-account metric stream for broad AWS metrics
• Firehose to an S3 landing zone
• private importer converts records into the self-hosted metric model

Metrics Centralization would be evaluated separately where a CloudWatch-owned copy of all supported metrics from selected accounts and Regions is genuinely required.

Lane 3: Managed-service logs

• EKS control-plane and RDS logs to CloudWatch Logs
• selected service logs to CloudWatch Logs
• organization-aware subscription into central Kinesis
• consumers in the observability cluster
• source metadata retained
• duplicates tolerated and removed where useful

Lane 4: Network and DNS telemetry

• VPC Flow Logs to Firehose or S3
• Transit Gateway Flow Logs to Firehose or S3
• Route 53 Resolver query logs to Firehose or S3
• source-specific schemas and parsers
• indexed operational copy
• durable archive copy

Lane 5: Security and configuration evidence

• organization CloudTrail to bfstore-log-archive
• log-file integrity validation
• S3 Object Lock where immutability is required
• selected CloudTrail events copied into the operational backend
• AWS Config histories and snapshots to archive S3
• Config change notifications ingested as structured events

This design uses AWS services where they are the only or strongest source.

It keeps the analytical destination self-hosted.

The pipeline needs its own SLOs

An observability platform cannot be trusted merely because its Grafana home page loads.

The ingestion system needs service-level indicators.

Application OTLP

• accepted spans per second
• rejected telemetry
• exporter queue length
• export failure rate
• end-to-end delay

CloudWatch Logs export

• subscription delivery throttling
• Kinesis iterator age
• consumer lag
• decode failures
• duplicate rate
• last event received per source

Metric streams

• Firehose delivery freshness
• S3 backup count
• decoder failures
• sample delay
• missing expected namespaces

S3 imports

• oldest unprocessed object
• SQS queue age
• failed object count
• replay backlog
• parser-version mismatch

Source coverage

• expected log groups exporting
• expected EKS clusters reporting
• expected accounts represented
• expected Regions represented
• expected CloudTrail files arriving

A meta-signal is:

time since last telemetry
from every expected source

A green ingestion service means little when one production account quietly stopped sending logs three days ago.

The platform should detect absence.

Silence is sometimes excellent news.

In telemetry pipelines, it is often a severed wire wearing a calm expression.

A staged implementation path

Phase 1: Application and Kubernetes telemetry

• instrument two bfstore services
• deploy Collector DaemonSet and gateway
• configure private, authenticated OTLP
• establish common resource attributes
• validate loss, retry and restart behaviour
• add node, kubelet and cluster-state metrics

Phase 2: EKS and selected managed logs

• enable EKS audit, API and authenticator logs
• publish selected RDS logs
• create CloudWatch Logs subscriptions
• run private Kinesis consumers
• preserve source metadata
• exclude export-pipeline log groups

Phase 3: Selected AWS metrics

Begin with:

• ALB
• NLB
• RDS
• NAT Gateway
• Transit Gateway
• EBS

Use polling initially when it provides the quickest testable route.

Measure API cost, delay and maintenance burden.

Phase 4: Durable AWS sources

• organization CloudTrail
• AWS Config
• VPC Flow Logs
• Transit Gateway Flow Logs
• Route 53 Resolver query logs
• load-balancer logs

Use central S3, SQS and replayable importers where batch ingestion is appropriate.

Phase 5: Broader metric streaming

Introduce cross-account observability and a monitoring-account metric stream when:

• the curated poller becomes difficult to maintain
• broader namespace coverage is required
• delivery latency matters
• the Firehose landing and decoding path has been tested

Evaluate Metrics Centralization only as a separate replication decision.

Phase 6: Test failure

Simulate:

• backend outage
• Collector restart
• tail-sampling routing failure
• Kinesis consumer failure
• Firehose delivery failure
• malformed record
• duplicate event
• S3 replay
• one source account stopping export

The pipeline is ready when telemetry failure is:

• detected
• bounded
• recoverable
• unable to break the applications it observes

The mental model I am keeping

My earlier model was:

AWS TELEMETRY
      │
      ▼
CLOUDWATCH
      │
      ▼
GRAFANA

The stronger model is:

                         TELEMETRY SOURCE
                                │
              ┌─────────────────┼─────────────────┐
              │                 │                 │
              ▼                 ▼                 ▼
         APPLICATION        KUBERNETES       AWS-MANAGED
              │                 │                 │
              ▼                 ▼                 ▼
             OTLP         NODE COLLECTOR      NATIVE AWS
                                                COLLECTION
              │                 │                 │
              └─────────────────┼─────────────────┘
                                ▼
                         TRANSPORT CHOICE
                                │
        ┌───────────────┬───────┼────────┬───────────────┐
        │               │       │        │               │
        ▼               ▼       ▼        ▼               ▼
       OTLP          KINESIS  FIREHOSE    S3         API POLLING
        │               │       │        │               │
        └───────────────┴───────┼────────┴───────────────┘
                                ▼
                         DURABLE BUFFER
                                │
                                ▼
                           PROCESSING
                                │
                 ┌──────────────┼──────────────┐
                 │              │              │
                 ▼              ▼              ▼
             ENRICHMENT     DEDUPLICATION    REDACTION
                 │              │              │
                 └──────────────┼──────────────┘
                                ▼
                       SELF-HOSTED BACKENDS
                                │
                 ┌──────────────┼──────────────┐
                 │              │              │
                 ▼              ▼              ▼
              METRICS          LOGS          TRACES
                                │
                                ▼
                             EVIDENCE

OpenTelemetry answers:

How should bfstore-owned
applications describe telemetry?

The node Collector answers:

How does the platform observe
what surrounds the process?

CloudWatch answers:

Where does AWS expose telemetry
for managed resources?

A metric stream answers:

How can broad CloudWatch metrics
leave continuously?

A subscription answers:

How can newly ingested CloudWatch
log events enter another stream?

Kinesis answers:

Where can near-real-time logs wait
for a private consumer?

Firehose answers:

Where can AWS manage buffering,
batching, retry and delivery?

S3 answers:

Which telemetry can be replayed
after the analytical system fails?

Object Lock answers:

Which protected archive objects
must resist deletion or overwrite?

The self-hosted platform answers:

Where are AWS, Kubernetes and
application signals queried together?

And the pipeline SLO answers:

How do we know the observer
has not quietly gone blind?

Getting AWS telemetry into a self-hosted platform does not require rejecting AWS-native observability mechanisms.

It requires refusing to confuse collection with destination.

CloudWatch can be the source without being the home. OpenTelemetry can be the common language without being the durable queue. S3 can be the archive without being the dashboard.

bfstore application traces can travel directly through OTLP.

Kubernetes telemetry can be collected beside the workloads.

EKS control-plane and RDS logs can leave CloudWatch through subscriptions.

AWS metrics can be streamed or polled.

Flow logs and Resolver query logs can land in Firehose or S3.

CloudTrail and Config can retain authoritative evidence in the log-archive account.

Each path can end in the same self-hosted operational platform without pretending that every signal began in the same place.

The result is not one pipe.

It is a small, deliberate railway system.

Every source has the right carriage.

Every important journey has a buffer.

Every record retains its origin.

And when one line closes, the telemetry waits somewhere safer than application memory and wishful thinking.

References and further reading

• AWS Distro for OpenTelemetry
• OpenTelemetry Collector deployment patterns
• Scaling the OpenTelemetry Collector
• OpenTelemetry Collector resiliency
• OpenTelemetry Collector Kubernetes components
• Amazon EKS control-plane logging
• CloudWatch metric streams
• CloudWatch cross-account observability
• CloudWatch Metrics Centralization
• Amazon Data Firehose HTTP delivery
• CloudWatch Logs subscriptions
• CloudWatch Logs cross-account subscriptions
• CloudWatch Logs centralisation
• Application Load Balancer vended logs announcement, 23 July 2026
• Network Load Balancer access logs
• RDS log publication
• VPC Flow Logs
• Transit Gateway Flow Logs
• Route 53 Resolver query logging
• CloudTrail trails
• CloudTrail log-file integrity validation
• AWS Config delivery channels
• S3 Object Lock