A service is unhealthy.
Its Pods are not ready.
The application logs show:
database connection timed out
The first assumption is that the application has a database problem.
The investigation widens.
The database is healthy.
The service identity exists.
The secret reference is correct.
The network policy appears unchanged.
Then the team discovers that the platform controller responsible for publishing connection details has been failing reconciliation for forty minutes.
The workload is broken.
The defect sits in the machinery that prepared the workload’s environment.
APPLICATION FAILURE
│
▼
PLATFORM DEPENDENCY
│
▼
CONTROL-PLANE FAILURE
Application observability shows the symptom.
It may not explain why the platform stopped delivering a required capability.
This distinction becomes more important as the platform takes responsibility for more of the service lifecycle.
A developer may request:
apiVersion: platform.bfstore.dev/v1alpha1
kind: Application
metadata:
name: review-service
spec:
runtime: go
exposure: internal
database:
profile: transactional-mysql
Behind that small declaration, the platform may coordinate:
- repository configuration
- continuous integration
- workload identity
- Kubernetes resources
- database provisioning
- network access
- secret publication
- telemetry
- GitOps deployment
- service registration
Each capability introduces machinery that can fail.
Platform observability is the ability to understand whether the machinery behind the developer platform is delivering its promised outcomes, where a journey is blocked and which operating boundary owns the failure.
The platform is an internal product.
Its observability should begin with the promises that product makes.
Platform observability begins with a product promise
Platform observability does not include every internal system simply because the platform team can see it.
Its boundary is:
The capabilities the platform promises, the machinery that fulfils them and the dependencies whose failure changes those outcomes.
A dependency may belong to another team or provider.
It still belongs in the observability model when its failure affects the platform promise.
A useful contract brings the promise, journey, status and ownership together.
PLATFORM OBSERVABILITY CONTRACT
Capability:
ServiceDatabase
User:
application developer
Requested outcome:
usable development database
Journey:
request accepted
policy evaluated
resource reconciled
connection published
application can connect
Success:
valid request becomes usable
Target:
99% within 15 minutes
Failure states:
rejected
blocked
retrying
failed
Status freshness:
updated within 2 minutes
Dependencies:
platform API
policy engine
Crossplane
cloud identity
provider API
secret publication
network
Owner:
platform-database
Evidence:
request ID
composition version
provider operation
final verification
This creates one reusable model:
PROMISE
│
▼
USER JOURNEY
│
▼
STAGES AND DEPENDENCIES
│
▼
STATUS AND SLIs
│
▼
OWNERSHIP AND ACTION
Observe several layers
A component can be healthy while the service it provides is degraded.
Crossplane Pods:
ready
reconciliation queue:
growing
oldest request:
37 minutes
The process is alive.
The service it provides is degraded.
A useful observation model includes:
RESOURCE HEALTH
processes, Pods, nodes, storage
CONTROL-PLANE HEALTH
queues, reconciliation, API errors
CAPABILITY HEALTH
database creation, deployment,
identity issuance
USER-JOURNEY HEALTH
time from request
to verified outcome
The lower layers help explain the upper ones.
The upper layers reveal whether users are receiving value.
Measure journeys and continuous capabilities differently
Some platform capabilities fulfil a request.
Others are consumed continuously.
Fulfilment journeys
Examples include:
- database provisioning
- application deployment
- certificate issuance
- infrastructure runs
Their model is:
REQUEST
│
▼
PROCESSING
│
▼
READY OUTCOME
Useful indicators include:
- admission-decision latency
- fulfilment duration
- failure rate
- blocked-request age
- actionable-status latency
- cleanup completion
Continuous capabilities
Examples include:
- DNS
- workload identity
- Kafka
- telemetry ingestion
- secret access
- cluster scheduling
Their model is:
AVAILABLE
FRESH
CORRECT
WITHIN CAPACITY
Useful indicators differ.
DNS
resolution success
latency
record freshness
IDENTITY
credential issuance success
issuance latency
revocation propagation
KAFKA
publish success
message age
recoverable backlog
TELEMETRY
accepted records
loss rate
queryability delay
Trying to force every capability into one request-to-ready model hides important behaviour.
Platform SLOs should follow the promise
A platform SLO should protect an outcome users depend on.
For example:
99% of valid development database requests should become usable within fifteen minutes.
Eligibility must be determined at admission using versioned rules.
VALID AT ADMISSION
supported and accepted
UNSUPPORTED AT ADMISSION
rejected with explanation
ACCEPTED BUT LATER FAILED
part of platform reliability
The third category should not be reclassified after the fact to make the SLO look healthier.
Decision latency and fulfilment success should also remain separate.
ADMISSION DECISION
request classified
within 30 seconds
FULFILMENT
accepted request reaches
a verified outcome within target
FAILURE VISIBILITY
blocked or failed request receives
actionable status promptly
A fast failure is operationally better than indefinite waiting.
It is not the same as successful fulfilment.
Follow requests through the control plane
A control plane accepts intent, observes reality and acts to reduce the difference.
Its core behaviour is reconciliation.
Useful signals include:
- reconciliation attempts
- successful and failed reconciliations
- duration
- queue depth
- oldest queued item
- retry count
- rate limiting
- external API errors
- resources stuck in non-ready states
DESIRED STATE
│
▼
RECONCILIATION
│
├── success
├── retry
├── blocked
└── permanent failure
A generic controller-error count is often too vague.
The platform should distinguish:
temporary provider throttling
from:
invalid composition affecting
every new database request
The signals should identify:
- affected capability
- resource
- attempted operation
- failure category
- retry behaviour
- required human action
Status needs meaning and freshness
Status is not merely decoration for kubectl.
It is a user-facing operational interface consumed by portals, CLIs, alerts and support tools.
Good status should explain:
- what is ready
- what is progressing
- what is blocked
- why it is blocked
- whether retry is occurring
- what the user should do next
It should also show whether the status remains current.
status:
observedGeneration: 7
lastReconciledAt: "2025-09-21T10:14:00Z"
conditions:
- type: Ready
status: "False"
reason: ProviderAuthenticationFailed
message: Temporary cloud credentials could not be issued
Useful fields include:
- observed generation
- last transition time
- last successful reconciliation
- next retry
- stable reason code
- correlation identifier
A Ready: true condition written forty minutes ago by a dead controller should not be treated as current truth.
Status needs both meaning and freshness.
Correlation should cross system boundaries
A deployment journey may cross:
SOURCE COMMIT
│
▼
CI BUILD
│
▼
ARTEFACT REGISTRY
│
▼
GITOPS REPOSITORY
│
▼
DEPLOYMENT CONTROLLER
│
▼
KUBERNETES
│
▼
HEALTH VERIFICATION
Shared identifiers help operators follow one change across the machinery:
- service name
- repository
- commit SHA
- artefact digest
- deployment ID
- environment
- running revision
A trace-like delivery model reduces the time spent assembling the story from unrelated dashboards.
Versions and changes need visibility
A platform estate may contain several versions of:
- compositions
- providers
- templates
- controllers
- cluster add-ons
- policies
- telemetry configurations
When a defect affects one version, the platform needs to know which consumers use it.
ServiceDatabase composition v1:
14 resources
ServiceDatabase composition v2:
3 resources
Change events should appear alongside runtime signals.
TIME
│
├── platform release
├── provider errors rise
├── reconciliation queue grows
└── user requests miss SLO
Correlation does not prove causation.
It narrows the investigation.
Observe shared runtime dependencies
Workloads continuously depend on shared platform capabilities.
The observability model should connect their health to affected services and journeys.
Kubernetes and capacity
Useful signals include:
- API errors and latency
- admission latency
- scheduler failures
- unschedulable Pods
- node pressure
- storage provisioning
- DNS
- ingress
- certificate issuance
Capacity views should show:
- headroom
- growth trend
- scaling limits
- blocked expansion
- placement constraints
- largest consumers
- fairness across tenants
Aggregate utilisation can hide one team or environment waiting for hours.
A platform-wide median can look healthy while one tenant is starved.
Network
An intended path may fail at several layers.
REQUEST TIMEOUT
│
├── DNS?
├── route?
├── firewall?
├── endpoint?
├── TLS?
└── application?
Network flow evidence, control-plane configuration and service telemetry should help identify where communication stopped.
The goal is not to retain every packet.
It is to understand whether an intended path works.
Identity and policy
Identity observability should distinguish:
AUTHENTICATION FAILURE
identity could not be established
AUTHORISATION FAILURE
identity established,
but the action was denied
Useful signals include:
- token-issuance failures
- certificate expiry
- role-assumption errors
- group-synchronisation delay
- policy denials
- revocation delay
- emergency-role use
Policy decisions should include:
- policy identifier and version
- affected capability
- reason
- environment
- remediation path
- exception reference
Repeated denials may indicate unsafe requests.
They may also expose poor documentation, a defective template or a policy error.
Kafka and databases
Kafka signals need business context.
lag:
100,000 messages
means little without:
- message age
- expected processing rate
- consumer owner
- affected journey
- recoverability of the backlog
Database capability health should include more than CPU and connection count.
It should also cover:
- provisioning duration
- credential rotation
- connection publication
- backup state
- restore-contract evidence
- supported-version compliance
A healthy database process does not prove the platform database capability is healthy.
Dependency maps need evidence
A dependency map should answer:
What is affected now?
What may be affected next?
Who owns the dependency?
Which fallback exists?
Each relationship should ideally record:
- declared or observed source
- freshness
- confidence
- owner
Dependency:
order-service → Kafka
Source:
service catalog and traces
Observed recently:
yes
Last seen:
4 minutes ago
A decorative graph is worse than no graph when responders assume it is complete.
Treat telemetry as a platform service
The observability pipeline is itself a continuous capability.
APPLICATION
│
▼
COLLECTOR
│
▼
PROCESSOR
│
▼
BACKEND
│
▼
QUERY
Each stage can fail.
Useful indicators include:
- accepted records
- rejected records
- queue depth
- dropped records
- exporter errors
- retries
- processing latency
- backend ingestion
- queryability delay
A Collector reporting healthy while its exporter drops every batch is not delivering a healthy service.
Telemetry freshness changes operational truth
Suppose an incident begins at 10:00.
Metrics appear at 10:02.
Logs appear at 10:07.
Traces appear at 10:15.
The systems provide different views of the same event at different times.
An SLO might state:
99% of accepted platform metrics should be queryable within two minutes.
The platform should distinguish:
signal not emitted
signal rejected
signal stored late
signal inaccessible
query incorrect
Synthetic query tests can verify that known telemetry remains discoverable through the supported query path.
Silence is a state
error count:
0
could mean:
no errors
or:
no data
Useful controls include:
- heartbeat metrics
- last-seen timestamps
- scrape-target health
- exporter success
- synthetic signals
- dead-man switches
The pipeline should also be observed from outside itself.
A Collector cannot be the sole authority on whether its exports are reaching the backend.
Out-of-band probes and independent alert paths reduce that circular dependency.
Sampling, cardinality and retention are operational controls
Sampling determines which trace evidence survives.
Cardinality affects cost and shared backend capacity.
Retention determines which questions can still be answered later.
The platform should observe:
- active series
- label growth
- ingestion rate
- rejected telemetry
- traces received and retained
- sampling rules
- cost by source
- retention by purpose
A missing failed trace may mean no failure occurred.
It may also mean the sampling policy discarded it.
Telemetry needs security and recovery
Platform telemetry may contain identities, deployment metadata, resource names and security decisions.
It needs:
- role-based access
- redaction
- retention controls
- audit evidence
- protection from secret leakage
Version-controlled configuration can recreate dashboards, alerts and pipelines.
Stored telemetry requires its own recovery decisions.
The observability system should not disappear completely during the incident it is meant to explain.
Turn evidence into ownership and action
A weak alert describes a component.
crossplane-provider Pod restarted 3 times
A stronger alert describes the platform impact.
Development database provisioning is degraded.
12 valid requests have waited
longer than 15 minutes.
Suspected dependency:
cloud provider identity
Supporting evidence:
token issuance failures
seven blocked reconciliations
Owner:
platform-control-plane
The wording should separate evidence from inference.
A suspected dependency is not a proven root cause.
Alert grouping should preserve the incident shape without pretending certainty.
IDENTITY SERVICE FAILURE
│
├── deployments affected
├── database provisioning affected
├── secret access affected
└── human access affected
The response system should avoid sending four independent expeditions to the same crater.
Ownership must remain current
Useful ownership fields include:
- service
- capability
- team
- environment
- runbook
- escalation path
A dashboard naming a team that stopped owning the service six months ago is not operational documentation.
It is a forwarding address for ghosts.
Runbooks should begin with the signal
A runbook should explain:
- what the alert means
- which users may be affected
- which dependencies to inspect
- which safe actions exist
- when to escalate
- how to confirm recovery
The alert, dashboard and runbook should use the same terminology.
Partial failure should remain visible
A platform may be degraded without being entirely down.
Existing workloads:
healthy
New deployments:
degraded
New database requests:
unavailable
Telemetry ingestion:
healthy
This helps users decide whether to wait, retry or use an alternative path.
One red platform status hides useful remaining capability.
Synthetic journeys provide product assurance
A scheduled synthetic may:
- create a temporary development resource
- wait for reconciliation
- verify readiness
- retrieve connection details
- perform a safe operation
- delete the resource
- verify cleanup
CREATE
│
▼
READY
│
▼
USE
│
▼
DELETE
│
▼
CLEANUP VERIFIED
This exercises the platform API, policy, controller, provider, identity, network and deletion path.
The test needs:
- dedicated isolation
- bounded profiles
- predictable naming
- automatic expiry
- cost limits
- cleanup alerts
- a named owner
A successful synthetic does not prove every user request works.
It demonstrates that the supported path still functions under the exercised conditions.
A bfstore database-provisioning journey
Suppose a developer requests:
kind: ServiceDatabase
metadata:
name: review
spec:
environment: development
profile: transactional
The expected journey is:
request accepted
│
▼
policy passed
│
▼
Crossplane reconciles
│
▼
cloud database created
│
▼
connection details published
│
▼
review-service connects
The observability contract records:
Capability:
ServiceDatabase
Request:
db-review-dev-042
Service:
review-service
Composition:
service-database-v3
Provider:
aws-development
Submitted:
10:00
Ready:
10:09
Duration:
9 minutes
Verification:
connection succeeded
If the request fails:
State:
blocked
Stage:
provider authentication
Reason:
temporary credential
could not be issued
Last reconciled:
10:07
Next retry:
10:09
Affected requests:
7
Owner:
platform-control-plane
The SLO sees valid requests approaching the fifteen-minute threshold.
Backstage displays the authoritative status.
The operator follows the request across:
- platform API
- policy
- Crossplane
- identity
- provider
- secret publication
- application verification
This is platform observability working as one story rather than several unrelated signal piles.
Questions I now ask
Promise
Which platform capability
and user outcome are protected?
Type
Is this a fulfilment journey
or continuous capability?
Path
Which stages, dependencies
and versions participate?
Signals
Which SLIs distinguish success,
rejection, blockage and stale status?
Correlation
Can one request be followed
across the complete path?
Telemetry
Can missing or delayed data
be distinguished from health?
Ownership
Which users are affected,
and who can act?
Recovery
Can the observability path
survive and be restored?
These questions shift platform observability from component inventory to product understanding.
The mental model I am keeping
My earlier model was:
PLATFORM OBSERVABILITY
monitor the cluster
and platform tools
The stronger model is:
PLATFORM PROMISE
│
▼
USER OR CONSUMER
│
▼
CAPABILITY JOURNEY
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
CONTROL PLANE DEPENDENCIES VERSIONS
│ │ │
└─────────────────┼─────────────────┘
▼
VERIFIED OUTCOME
│
▼
STATUS, SLIs AND EVIDENCE
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
IMPACT OWNER ACTION
│
▼
LEARN AND IMPROVE
A platform exists to remove repeated infrastructure work from application teams.
That does not remove the infrastructure.
It concentrates responsibility for it.
The more the platform automates, the more important it becomes to understand:
- which request entered the system
- which controller acted
- which policy decided
- which dependency failed
- which user journey was affected
- which team owns recovery
A green cluster dashboard cannot answer those questions.
Neither can application telemetry alone.
Platform observability watches the complete path from user intent to verified capability, then makes failure visible at the boundary where somebody can understand and act on it.
Once the organisation builds machinery to create, deploy, connect, secure and observe workloads, that machinery becomes a production system in its own right.
It needs its own promises.
It needs its own evidence.
It needs somebody to notice when the ropes, pulleys and rather opinionated controllers stop carrying the show.