Field notes

Notes

Shorter learning logs, reading notes, and architectural sparks before they become essays.

Browse Notes

55 entries in the Cloud Sandbox.

Year

2026

16 entries

25 Jul 2026

about 18 minutes min read

Simple routing versus failover routing in a private zone

Simple routing publishes the ordinary destination for a private DNS name. Failover routing adds an active-passive decision between prepared primary and secondary destinations. The right choice depends on the failure boundary, trustworthy branch signals, secondary readiness and client caching behaviour.

21 Jul 2026

about 14 minutes min read

CloudWatch can be the telemetry source without being the observability platform

Using a self-hosted observability platform does not require ignoring CloudWatch. For many AWS-managed services, CloudWatch is the native telemetry source. bfstore can export those signals while retaining control over correlation, retention, querying and operational experience.

6 Jul 2026

about 19 minutes min read

Private hosted zones don't require Resolver endpoints

Route 53 private hosted zones already resolve through AmazonProvidedDNS inside VPCs that receive the zone. VPC Resolver endpoints and Global Resolver extend private DNS beyond that native VPC resolver view; neither is required for ordinary resolution inside an associated VPC.

27 Jun 2026

about 14 minutes min read

AmazonProvidedDNS: the resolver already hiding inside the VPC

Every Amazon VPC already contains access to Route 53 VPC Resolver through AmazonProvidedDNS. Understanding its addresses, DHCP configuration, VPC attributes, private hosted zones, forwarding boundaries and query evidence makes AWS DNS architecture much easier to reason about.

23 Jun 2026

about 19 minutes min read

Why CPU limits can make latency weird

Kubernetes CPU limits are enforced as time-based cgroup quotas. A Go service can exhaust its allowance during a short burst and then be throttled, producing tail-latency spikes even when average CPU usage looks reasonable.

22 Jun 2026

about 16 minutes min read

How golden paths become platforms

A golden path begins as a recommended route through software delivery. It becomes a platform when that route is discoverable, self-service, supported, observable and continuously improved through feedback from the teams using it.

8 Jun 2026

about 18 minutes min read

NAT gateways have failure domains and bills

AWS manages the NAT gateway service, but architects still choose its availability mode, routing scope, cost boundary and operational blast radius. Zonal, regional, distributed and centralised egress designs each move failure and expense rather than removing them.

12 May 2026

about 19 minutes min read

IPv6 compression rules I keep forgetting

IPv6 addresses are easier to read once their text-compression rules become mechanical. Remove leading zeros within each group, compress one longest run of zero groups, prefer the leftmost run when tied and expand to exactly eight groups when checking an address.

29 Apr 2026

about 15 minutes min read

Reading a route table from destination outward

An AWS route table is easier to understand when read as a destination lookup rather than a sequence of rows. Start with the packet destination, find the longest matching prefix, follow the selected target and then verify the onward and return paths.

8 Apr 2026

about 16 minutes min read

The CIDR arithmetic I actually use

Most day-to-day IPv4 subnetting does not require writing all 32 bits. A small set of calculations covering host bits, address counts, block increments, alignment and AWS reservations is enough to reason confidently about CIDR ranges.

26 Mar 2026

about 20 minutes min read

Break-glass access should feel deliberately inconvenient

Emergency AWS access should not be the easiest route to privileged authority. A strong break-glass process introduces deliberate activation friction, independent recovery paths, narrow roles, immediate alerts and a tested return to normal operation.

10 Mar 2026

about 16 minutes min read

Human access and workload identity should not look alike

Humans and software may both receive temporary AWS credentials, but their identity journeys should remain visibly different. Workforce federation, interactive sessions and organisational groups solve different problems from workload trust, runtime roles and automatic credential delivery.

24 Feb 2026

about 21 minutes min read

Why a dedicated policy-test account is useful

A dedicated AWS policy-test account provides a controlled place to exercise Service Control Policies with representative identities and workflows before those policies reach development, production or organisation-wide boundaries.

9 Feb 2026

about 16 minutes min read

“Allowed by the SCP” does not mean “allowed”

A service control policy defines the maximum permissions available inside an AWS member account. An action passing the SCP layer still needs a valid grant path and must survive every other applicable permission limit and explicit deny.

27 Jan 2026

about 14 minutes min read

AWS OUs are governance scopes, not folders

AWS organisational units should group accounts that require the same inherited controls and operational treatment. Their architecture lies in the governance contract accounts receive, not in how tidy the hierarchy looks.

11 Jan 2026

about 14 minutes min read

Why the AWS management account should be boring

The AWS Organizations management account owns the organisation and several organisation-level operations. Keeping it quiet, tightly accessed and free of ordinary workloads reduces risk around an account that service control policies cannot constrain.

Year

2025

7 entries

28 Dec 2025

about 16 minutes min read

Change management can be boring and automated

Change management does not need to mean queues, weekly approval meetings and ceremonial forms. Routine changes can follow tested, observable and pre-authorised paths, leaving human judgement for exceptional risk.

23 Nov 2025

about 15 minutes min read

Provider-native doesn't have to mean developer-hostile

Provider-native architecture can preserve provider-specific identity, networking, data and security capabilities without exposing every cloud primitive to application teams. A platform can translate developer intent into precise, versioned and well-operated native implementations.

19 Oct 2025

about 17 minutes min read

High-cardinality telemetry: useful until it becomes expensive

High-cardinality attributes can make telemetry precise and searchable, but uncontrolled dimensions create enormous metric series, storage costs and slow queries. Good observability preserves useful context without turning every unique value into a permanent index.

17 Aug 2025

about 17 minutes min read

Federation is mostly about deciding who gets to assert identity

Identity federation allows one system to rely on identity claims issued by another. The difficult work is deciding which issuer is trusted, which claims are accepted, how they map to local authority and how that trust is revoked.

15 Jun 2025

about 17 minutes min read

RPO and RTO become real during a restore drill

Recovery point and recovery time objectives describe acceptable data loss and service interruption. A restore drill reveals whether the backups, infrastructure, access, procedures and people can actually meet them.

13 Apr 2025

about 12 minutes min read

Guardrails and gates solve different problems

Guardrails preserve approved operating conditions, while gates decide whether work should cross important boundaries. Effective governance uses each where its control model fits rather than turning every safeguard into an approval queue.

9 Feb 2025

about 14 minutes min read

Accounts, clusters and networks are all failure boundaries

Cloud accounts, Kubernetes clusters and network segments contain different kinds of failure, compromise and operational mistakes. Resilient architecture depends on understanding what each boundary can stop, how it can be crossed and where shared dependencies reconnect otherwise isolated systems.

Year

2024

6 entries

29 Dec 2024

about 12 minutes min read

The platform should remove toil, not understanding

A developer platform should automate repetitive work and contain unnecessary infrastructure complexity without hiding the behaviour, risks and ownership engineers need to operate their services safely.

20 Oct 2024

about 17 minutes min read

Error budgets make reliability negotiable

An error budget translates a service-level objective into an explicit allowance for failure. It gives product, platform and service teams a shared basis for deciding when to continue delivery, reduce risk or prioritise reliability.

18 Aug 2024

about 18 minutes min read

The OpenTelemetry Collector as a telemetry router

The OpenTelemetry Collector creates a platform boundary between telemetry producers and storage systems. It can receive, process, route and export signals without making every application understand every backend.

14 Apr 2024

about 11 minutes min read

A golden path isn't the same thing as a mandatory path

A golden path makes the common, safe route through an internal platform easy to discover and use. It guides teams towards supported outcomes without pretending that every workload has identical needs.

11 Feb 2024

about 13 minutes min read

Reconciliation loops are everywhere once you notice them

A reconciliation loop repeatedly observes a system, compares reality with an expectation and acts on the difference. The same pattern appears in Kubernetes, GitOps, autoscaling, health management and operational recovery.

Year

2023

7 entries

29 Dec 2023

about 14 minutes min read

What I now expect from a reproducible cloud environment

A reproducible cloud environment needs more than infrastructure code. Its inputs, state, identities, artefacts, policies, verification and recovery procedures must all be controlled well enough for the environment to be rebuilt and trusted.

26 Nov 2023

about 17 minutes min read

A trace is a story told across processes

A distributed trace reconstructs one operation from spans recorded by several processes. The story survives service boundaries only when trace context travels with the work.

22 Oct 2023

about 11 minutes min read

Build once, promote many: why immutable artefacts matter

Building an artefact once and promoting that exact object through each environment preserves the connection between source code, test evidence and the software that reaches production.

18 Jun 2023

about 15 minutes min read

Terraform state is really an infrastructure database

Terraform state records which real infrastructure objects correspond to configuration addresses. Losing that state does not destroy the infrastructure, but it can leave Terraform unable to manage it safely.

12 Feb 2023

about 25 minutes min read

Namespaces and cgroups: the machinery beneath containers

Linux namespaces give container processes isolated views of the system. Control groups group and account for tasks, allowing configured controllers to prioritise, protect or limit their use of shared resources.

Year

2022

6 entries

27 Nov 2022

about 27 minutes min read

The transactional outbox solves a very particular problem

The transactional outbox closes the failure gap between committing local business state and recording the message that must be published because of it. Delivery, deduplication and workflow completion remain separate responsibilities.

30 Oct 2022

about 22 minutes min read

Dead-letter queues aren't where failed messages go to die

A dead-letter queue protects the healthy processing path by isolating messages that cannot currently be handled. It only becomes a recovery mechanism when failures have owners, evidence and a safe route back into processing.

9 Oct 2022

about 30 minutes min read

Consumer groups finally clicked

A traditional Kafka consumer group is one logical subscriber made from cooperating members. Kafka assigns each subscribed partition to one member at a time, while separate groups consume independently.

14 Aug 2022

about 38 minutes min read

Kafka partitions: ordering has a boundary

Kafka preserves record order within a partition, not across an entire topic. Choosing a partition key therefore defines which events share an ordered history and which may be processed concurrently.

19 Jun 2022

about 33 minutes min read

Commands and events describe different truths

Commands ask an owner to attempt an action. Events report facts that an owner says have already happened. They may travel through the same broker, but they create different contracts.

Year

2021

7 entries

19 Dec 2021

about 28 minutes min read

A retry is not the same thing as recovery

Retrying repeats an operation after failure. Recovery is the larger job of restoring useful service, resolving uncertain state and repairing any consequences left behind.

23 May 2021

about 20 minutes min read

Cron taught me to care about idempotency

Scheduling a Bash script was easy. Making repeated, overlapping and partially failed runs safe was the real automation problem.

21 Mar 2021

about 21 minutes min read

Pipes connect processes, not strings

Looking beneath Bash's vertical bar to see the file descriptors, processes, buffering and exit statuses that make a pipeline work.

Year

2020

6 entries

12 Apr 2020

about 10 minutes min read

A first wander through /proc

Poking around Linux's /proc filesystem to see what a running process looks like from the outside.