CloudWatch can be the telemetry source without being the observability platform

bfstore uses a self-hosted observability platform.

That decision could be interpreted as:

Do not use CloudWatch.

It should not be.

Amazon EKS sends enabled control-plane logs to CloudWatch Logs.

Amazon RDS can publish database logs there.

AWS services publish operational metrics into CloudWatch.

Other managed signals, including flow logs, Resolver query logs and load-balancer logs, can use CloudWatch as one supported destination among several.

Avoiding CloudWatch would not make bfstore more independent.

It would make several AWS-managed components harder to observe.

The stronger position is:

Use CloudWatch where AWS
provides the original signal.

Export the signal when bfstore
needs it in the shared platform.

CloudWatch can be:

an AWS-native collection surface;
a retained source copy for selected logs;
a bounded export-recovery window;
an alarm source;
an export boundary.

It does not have to be the only store, query interface, dashboard system or final retention layer.

Choosing a self-hosted observability platform is a decision about operational ownership. It is not a vow to reject every managed telemetry source.

The architecture becomes clearer once I separate where the signal originates from where engineers investigate it.

Choose the collection path by signal ownership

A telemetry source answers:

Where can this signal
be obtained?

An observability platform answers:

How will we store, correlate,
query and act on signals
from the whole system?

For an RDS MySQL error log:

RDS
  │
  ▼
CloudWatch Logs

For a bfstore application trace:

Go service
  │
  ▼
OpenTelemetry SDK
  │
  ▼
OpenTelemetry Collector

For a VPC Flow Log, the service can deliver to CloudWatch Logs, Amazon S3 or Amazon Data Firehose.

The platform sits later in each journey.

TELEMETRY SOURCES
        │
        ▼
TRANSPORT AND PROCESSING
        │
        ▼
BFSTORE OBSERVABILITY PLATFORM

The platform should not require every signal to have the same first hop.

It should require every important signal to arrive with:

an understood schema;
useful source identity;
bounded delay;
intentional retention;
a recovery method where loss matters;
evidence that the pipeline is operating.

AWS-managed services often choose the source

bfstore controls its application code.

It does not control the EKS control-plane hosts or the RDS filesystem.

When EKS control-plane logging is enabled, AWS sends selected API, audit, authenticator, controller-manager and scheduler logs to CloudWatch Logs.

RDS can continuously publish selected engine logs there.

In these cases, CloudWatch is not an arbitrary detour.

It is part of the managed service’s supported telemetry interface.

Trying to bypass it may require unsupported access, agents that cannot run on the managed host, incomplete telemetry or fragile polling.

The objective is not to remove AWS from AWS-owned signals.

It is to end the path where bfstore’s operational requirements are best served.

Application telemetry can take the direct path

A bfstore Go service can emit traces, metrics and structured logs through OpenTelemetry.

BFSTORE GO SERVICE
        │
        ▼
OTLP
        │
        ▼
OTEL COLLECTOR
        │
        ▼
SELF-HOSTED BACKEND

Sending the same telemetry through CloudWatch first adds delay, format conversion and duplicate ingestion.

That may be justified where a particular AWS integration requires CloudWatch.

It should not be the default merely because the workload runs on AWS.

The split is based on control, not ideology.

Choose one authoritative path per signal

Signal	Authoritative source	Secondary use
Application request latency	OpenTelemetry	CloudWatch only where specifically required
ALB request count	CloudWatch metric	Exported self-hosted copy
Container CPU	Kubernetes or node collector	Container Insights if deliberately retained
EKS audit events	CloudWatch Logs	Searchable platform copy and archive
RDS free storage	CloudWatch metric	Exported self-hosted copy

A common platform does not require every collector to observe everything.

It requires every important question to have one trusted source of evidence.

Design the exit, retention and recovery together

CloudWatch exits have different delivery, replay and networking properties.

Metric streams

CloudWatch metric streams continuously send selected metrics through Amazon Data Firehose.

Supported output formats include JSON and OpenTelemetry 0.7.0 and 1.0.0.

CLOUDWATCH METRICS
        │
        ▼
METRIC STREAM
        │
        ▼
DATA FIREHOSE
        │
        ▼
BFSTORE INGESTION

The OpenTelemetry format does not make the destination an ordinary OTLP receiver.

The consumer must understand both the metric-stream envelope and the Firehose contract.

A metric stream is also not a raw-sample feed. Every stream includes:

Minimum
Maximum
SampleCount
Sum

Additional statistics can be configured.

The adapter must preserve periods, units, dimensions and statistical meaning. It should not flatten every record into an ordinary gauge.

Firehose cannot call a private custom HTTP endpoint

Firehose custom HTTP delivery cannot send directly to an endpoint inside a VPC.

A custom endpoint must be reachable through HTTPS and implement Firehose’s request-and-response contract.

bfstore therefore has three practical options:

Firehose → S3 → private workers

Firehose → hardened public adapter
         → private platform

poll CloudWatch APIs
for a curated metric set

The public adapter should perform narrow functions such as authentication, validation, decompression, deduplication and private downstream forwarding.

This constraint belongs in the architecture before bfstore builds a private endpoint that Firehose cannot reach.

Log subscriptions

CloudWatch Logs subscriptions forward newly ingested events to Kinesis Data Streams, Data Firehose, Lambda or OpenSearch Service.

Delivery is at least once, so duplicates are normal.

For retryable failures, CloudWatch Logs retries for up to 24 hours. After that bounded window, events can be dropped. Non-retryable configuration failures can interrupt delivery without automatically replaying every retained event later.

CLOUDWATCH LOG GROUP
        │
        ▼
SUBSCRIPTION
        │
        ▼
KINESIS OR FIREHOSE
        │
        ▼
SELF-HOSTED LOG PIPELINE

The consumer needs decompression, enrichment, duplicate handling, backpressure, checkpointing and missing-source detection.

A retained log group can support later backfill.

The subscription itself is a near-real-time delivery mechanism, not a durable replay log.

Historical recovery requires a separate process such as exporting a time range to S3 or reading retained events through an API.

API polling

Polling CloudWatch APIs can be sensible when the metric set is small and carefully selected.

It becomes harder as accounts, Regions, namespaces and dimensions grow.

Polling can also recover a missing metric-stream interval.

The decision is not whether streaming sounds more modern.

It is whether the path meets bfstore’s latency, coverage, cost, recovery and operational-complexity requirements.

Multi-account collection uses different mechanisms

Mechanism	Behaviour	Best fit
Logs centralisation	Copies newly arriving matching logs	Central log retention and export
Metrics Centralization	Copies supported custom, EMF and OpenTelemetry metrics	Organisation-owned application metrics
Cross-account observability	Shares source telemetry without a durable central copy	Central viewing and AWS-vended metrics
Source-account exports	Exports directly from each account	Independent ownership and failure domains

CloudWatch Logs centralisation copies only new matching log data after a rule is created. It does not copy pre-existing history automatically.

CloudWatch Metrics Centralization currently supports custom metrics, Embedded Metric Format metrics and OpenTelemetry metrics. It does not provide selective metric filtering inside a chosen source account and should not be presented as a general copying mechanism for every AWS-vended metric.

Cross-account observability is sharing, not retention.

SHARING

central visibility

CENTRALISATION

central copy

EXPORT

movement into another platform

If a source account stops sharing, the monitoring account loses access to that source telemetry.

A monitoring-account metric stream can include metrics from linked source accounts, including AWS-vended metrics, and export them through Firehose.

A distributed source-account export may still be preferable where failure isolation matters.

Retention and backfill should be named

A signal may have:

SOURCE COPY

CloudWatch Logs

OPERATIONAL COPY

self-hosted backend

EVIDENCE COPY

S3 archive

Each copy should have a purpose.

Keeping three identical copies forever is not resilience.

It is a retention policy that forgot to arrive.

Path	Recovery approach
CloudWatch Logs	Export retained events to S3 or read them through APIs
Kinesis	Replay within the configured retention window
Firehose with S3 backup	Reprocess backup objects
Metric stream	Poll CloudWatch for the missing historical interval
Direct OTLP	Use Collector queues or persistent storage where configured

The recovery path should be recorded before an incident requires it.

The export path must be observable

The platform should watch:

subscription delivery errors and throttling;
centralisation errors;
metric-stream health;
Kinesis consumer lag;
Firehose failures and backup objects;
decoding failures;
missing metric namespaces;
source-to-backend delay;
backfill backlog.

source alive?
export alive?
transport alive?
consumer alive?
backend accepting?

A healthy backend does not prove that new telemetry is arriving.

The most useful signal may be:

Time since last expected event
from each account, Region and source.

Absence is part of observability.

The platform must notice when the telescope still exists but somebody has quietly replaced the lens with a biscuit.

The platform is where correlation happens

CloudWatch can tell me:

RDS CPU increased.

The shared platform should help me ask:

Did RDS CPU increase because
order-service traffic rose,
because a deployment changed
the query pattern, or because
a retry storm multiplied work?

That investigation may require:

CloudWatch RDS metrics;
Go application traces;
Kubernetes resource metrics;
deployment events;
Kafka consumer lag;
MySQL slow-query logs;
VPC Flow Logs;
error-budget state.

CloudWatch exposes AWS-native service data.

bfstore’s platform spans application, Kubernetes, AWS, later Azure and GCP, and external dependencies.

The platform is not defined by where the first metric was born.

It is defined by whether an engineer can reconstruct the behaviour of the complete system.

Define which copy and alert are authoritative

Duplicate collection can create duplicate truths.

OpenTelemetry SDK
    → self-hosted metrics

Prometheus scrape
    → self-hosted metrics

CloudWatch agent
    → CloudWatch
    → metric stream
    → self-hosted metrics

The series may differ in timestamps, labels, units, aggregation and missing-data behaviour.

The resulting dashboard becomes a small committee debating how much CPU one Pod used.

Alternative paths may remain for validation or recovery.

They should not all be ingested permanently without a reason.

The same principle applies to alerts.

CloudWatch alarms can remain useful for:

critical AWS-native safety conditions;
Route 53 health integration;
telemetry-export failures;
minimal fallback alerts.

The self-hosted platform can own application SLOs, cross-signal conditions, Kubernetes alerts, multi-cloud correlation and primary operational paging.

For each condition, bfstore should decide:

Which alert owns the page?

Which alert is fallback evidence?

Who changes its threshold?

Two alerting systems should not shout the same sentence into different telephones.

The bfstore boundary

For bfstore, I would use CloudWatch deliberately rather than universally.

CloudWatch-required or primary sources

Use it for:

EKS control-plane logs;
RDS exported engine logs;
RDS and AWS-vended service metrics;
selected AWS-native alarms.

CloudWatch-optional destinations

Evaluate it alongside S3 and Firehose for:

Route 53 VPC Resolver query logs;
VPC and Transit Gateway Flow Logs;
ALB and NLB logs;
other vended logs with several supported destinations.

Publication chronology: When this article was published on 21 July 2026, the established ALB access-log path was Amazon S3. On 23 July 2026, AWS added vended delivery for ALB access, connection and health-check logs to CloudWatch Logs and Data Firehose while retaining S3 delivery.

Direct OpenTelemetry paths

Use them for:

Go application traces;
application metrics;
structured application logs where supported;
Kubernetes workload telemetry;
runtime and service-level signals.

Durable evidence and platform responsibilities

Use S3-centred designs for organisation CloudTrail, AWS Config history, selected flow and load-balancer logs, and replayable telemetry.

Use the self-hosted platform for unified search, dashboards, traces, SLOs, correlation, operational alerting, controlled retention and multi-cloud investigation.

AWS-MANAGED SERVICE
        │
        ▼
AWS-NATIVE SOURCE
        │
        ▼
SUPPORTED EXPORT
        │
        ▼
BFSTORE OBSERVABILITY


BFSTORE APPLICATION
        │
        ▼
OPENTELEMETRY
        │
        ▼
BFSTORE OBSERVABILITY

Both paths are valid.

They converge after collection rather than pretending collection is identical.

Questions I now ask

Ownership

Does bfstore control the system that emits the signal?

Native path

Where does AWS officially expose it?

Export

Which supported mechanism moves it onwards?

Network

Can the destination remain private?

Multi-account model

Is the design copying data, sharing access or exporting from each account?

Authority

Which pipeline produces the trusted version?

Recovery

How will a missing interval be backfilled?

Retention

Why does each copy exist?

Alerting

Which system owns the page?

Failure

How will bfstore detect that export has stopped?

These questions are more useful than:

Are we using CloudWatch
or self-hosting?

The answer can be:

Both, for different jobs.

The mental model I am keeping

My earlier model was:

CLOUDWATCH

AWS observability platform

SELF-HOSTED PLATFORM

alternative to CloudWatch

The stronger model is:

                         BFSTORE SYSTEM
                               │
             ┌─────────────────┴─────────────────┐
             │                                   │
             ▼                                   ▼
       AWS-MANAGED                         BFSTORE-OWNED
          SERVICES                           WORKLOADS
             │                                   │
             ▼                                   ▼
      AWS-NATIVE SOURCE                    OPENTELEMETRY
             │                                   │
             ▼                                   ▼
      SUPPORTED AWS EXIT                  DIRECT OTLP PATH
             │                                   │
             └─────────────────┬─────────────────┘
                               ▼
                     TRANSPORT AND BUFFERING
                               │
                               ▼
                    SELF-HOSTED OBSERVABILITY
                               │
                               ▼
                       SHARED INVESTIGATION

CloudWatch answers:

Where has AWS exposed
this managed-service signal?

OpenTelemetry answers:

How can bfstore-owned workloads
emit portable telemetry?

Metric streams answer:

How can CloudWatch statistics
leave continuously?

Log subscriptions answer:

How can new CloudWatch log events
enter another path?

Cross-account observability answers:

How can one account view
source-account telemetry?

Centralisation answers:

Which supported data should
be copied into one account?

The self-hosted platform answers:

Where will engineers investigate
the complete bfstore system?

The archive answers:

Which copy survives when
the search system does not?

And pipeline monitoring answers:

How do we know the signals
are still arriving?

CloudWatch can be an upstream dependency without becoming the entire observability architecture.

The source of a signal does not own its final meaning.

An RDS metric may begin in CloudWatch.

Its meaning may emerge only when compared with an order-service trace, a Kubernetes rollout and a Kafka backlog.

An EKS audit event may begin in CloudWatch Logs.

Its value may emerge when correlated with a GitHub deployment, an IAM session and a production incident.

CloudWatch supplies the AWS-native piece of the story.

bfstore’s platform assembles the story.

That is not rejecting CloudWatch.

It is giving it a well-defined job and declining to make it wear every hat in the observability cupboard.