A Docker command can start a container on one machine:
docker run \
--name catalog-service \
--publish 50051:50051 \
bfstore/catalog-service:dev
The container runtime prepares a filesystem, creates a process, configures namespaces and cgroups, and connects that process to a network.
That is already a considerable amount of machinery behind one command.
A production application soon asks questions that docker run does not answer by itself:
- Which machine should run this workload?
- How many copies should exist?
- What happens when a machine disappears?
- How do callers find whichever copy is currently ready?
- How is a new version introduced without replacing every process at once?
- Who notices that reality no longer matches the intended system?
- How are configuration, credentials, storage and access policies applied?
- Who keeps checking after the deployment command ends?
I could answer those questions with scripts.
One script might choose a host. Another might restart a failed process. A load balancer configuration might list current instances. A release script might replace them in sequence.
Eventually those scripts would form a system of their own. They would need shared state, coordination, permissions, failure handling and a way to agree about what should be running.
That system is the beginning of an orchestrator.
Kubernetes provides such an orchestration system. Its important contribution is not merely that it starts containers. Docker, containerd, CRI-O and other runtimes already know how to create container processes.
Kubernetes adds a control plane that accepts desired state, records it and continually coordinates machines to move actual state towards it.
Containers give processes portable runtime boundaries. Kubernetes adds a control plane that manages those boundaries across a fleet of machines.
Kubernetes is not a larger docker run.
It is a system for making and continually revisiting decisions about workloads.
1. From a local container to durable cluster intent
When I run a container directly, I issue a local instruction:
Run this image
on this machine
with these ports and mounts
until the process exits.
The intention may exist only in my shell history or deployment script.
HOST
│
▼
CONTAINER RUNTIME
│
▼
CATALOG CONTAINER
If that host disappears, the process disappears with it. Another host does not automatically know that a replacement should exist.
Kubernetes changes the instruction.
Instead of saying:
Start one container now.
I declare something closer to:
The cluster should maintain three catalog replicas
using this image, configuration and policy.
That declaration becomes durable cluster state.
DESIRED STATE
three catalog replicas
ACTUAL STATE
two available catalog replicas
CONTROL PLANE RESPONSE
create another replica
A direct command says:
Do this once.
A desired-state declaration says:
Keep this condition true.
That repeated comparison between declaration and reality is the centre of Kubernetes.
A cluster is several machines, not one large computer
A Kubernetes cluster contains machines with different responsibilities.
KUBERNETES CLUSTER
│
┌─────────────────┴─────────────────┐
│ │
▼ ▼
CONTROL PLANE WORKER NODES
│ │
┌───────┼────────┐ ┌─────────┼─────────┐
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
API server etcd controllers node A node B node C
│ │ │ │
│ Pods Pods Pods
│
└── continually reconciles state
Each worker still has its own:
- Linux kernel;
- CPU and memory;
- filesystems and storage attachments;
- network interfaces;
- container runtime;
- process failures.
Kubernetes treats those machines as a managed pool of capacity. It does not erase their separate failure domains.
One small map of the control model
| Layer | Main question |
|---|---|
| API server | Where is cluster intent accepted and exposed? |
etcd |
Where is authoritative cluster state stored? |
| Controllers | Which drift between desired and observed state needs correction? |
| Scheduler | Which node should receive an unscheduled Pod? |
| Kubelet | How does one node realise its assigned Pods? |
| Container runtime | How are container processes created and managed? |
| Linux kernel | How are namespaces, cgroups, networking and process execution enforced? |
The components cooperate through API state. They do not form one enormous synchronous function call.
2. The API and control loops hold the system together
The Kubernetes API server is the front door to the control plane.
kubectl
deployment pipeline
controller
scheduler
kubelet
│
▼
KUBERNETES API SERVER
When I run:
kubectl apply -f catalog-deployment.yaml
kubectl does not normally log in to every node and start containers directly.
It submits a resource definition to the API.
The API server can:
- authenticate the caller;
- authorise the requested operation;
- validate the resource;
- run admission policy;
- store the accepted state;
- expose that state to controllers, schedulers, nodes and clients.
The API is therefore more than a remote command endpoint.
USER INTENTION
│
▼
KUBERNETES API
│
▼
DURABLE DESIRED STATE
etcd stores authoritative cluster state
The control plane uses etcd as its backing store for information such as:
- declared workloads;
- resource metadata;
- node registrations;
- configuration and Secrets;
- leases;
- controller progress.
Users and workloads should normally interact through the Kubernetes API rather than reading or writing etcd directly.
The API server applies Kubernetes validation, access control and resource semantics.
Losing etcd state can mean losing the cluster’s authoritative record of what should exist. That makes it control-plane data requiring:
- restricted access;
- encryption-at-rest decisions;
- backups and restore tests;
- capacity monitoring;
- quorum health;
- disaster-recovery procedures.
Kubernetes may recreate a failed application process. It cannot reconstruct every desired resource from wishful thinking after its authoritative state is lost.
Resources describe desired conditions
Kubernetes represents intent through API resources such as:
- Pods;
- Deployments;
- StatefulSets;
- DaemonSets;
- Jobs and CronJobs;
- Services;
- ConfigMaps and Secrets;
- Namespaces;
- custom resources.
A typical resource contains:
apiVersion
kind
metadata
spec
For example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog
spec:
replicas: 3
selector:
matchLabels:
app: catalog
template:
metadata:
labels:
app: catalog
spec:
containers:
- name: catalog
image: bfstore/catalog-service:1.0.0
spec.replicas: 3 means that the Deployment should maintain three replicas.
It does not mean that three healthy Pods already exist when the API accepts the object.
spec and status tell different truths
Users usually declare spec.
Controllers and kubelets report observed state through status.
| Field | Meaning |
|---|---|
spec |
What should be true |
status |
What the system currently knows |
metadata.generation |
Which desired-state revision is current |
status.observedGeneration |
Which revision a controller has processed |
A Deployment can exist with:
spec.replicas:
3
status.availableReplicas:
2
That gap is operationally meaningful.
API acceptance means:
The cluster accepted this desired state.
It does not necessarily mean:
The workload is already running successfully.
Deployment systems should observe status and conditions rather than treating a successful API write as completed rollout.
Controllers continually reconcile
A controller repeatedly:
observe current state
compare it with desired state
take a small corrective action
observe again
Suppose a Deployment wants three replicas and only two matching Pods exist.
desired:
3
observed:
2
action:
create one missing Pod
If four exist, the controller may remove one.
The controller does not perform one comparison and retire to a velvet chair. It continues watching because nodes, processes, policies and declarations continue changing.
This repeated reconciliation gives Kubernetes much of its resilience.
Declarative does not mean passive
A declaration such as:
spec:
replicas: 3
activates control loops.
DECLARATION
│
▼
CONTROLLERS
│
▼
CREATE, REPLACE OR REMOVE RESOURCES
│
▼
OBSERVE AGAIN
Kubernetes is declarative because users state the condition they want. It is active because controllers continually work to maintain that condition.
Access and admission protect the control plane
The Kubernetes API can create workloads, mount storage, expose network services and distribute credentials. Access to it is powerful.
The cluster needs to answer:
- Who is the caller?
- Which verb are they requesting?
- On which resource?
- In which Namespace?
- Under which constraints?
Authentication establishes identity. Authorisation, commonly through RBAC, decides what that identity may do.
Admission then validates or modifies accepted requests.
API REQUEST
│
▼
AUTHENTICATION
│
▼
AUTHORISATION
│
▼
ADMISSION
│
├── accepted
├── modified
└── rejected
Admission policy can require:
- approved registries;
- non-root execution;
- resource requests;
- restricted capabilities;
- required labels;
- prohibited host mounts;
- approved storage types.
Policy cannot make application logic correct. It can prevent selected dangerous configurations from becoming cluster state.
The control plane is itself distributed
The API server, scheduler, controllers and etcd can fail independently.
A scheduler outage may prevent new Pods from receiving nodes while existing Pods continue running.
A controller outage may prevent missing replicas from being replaced.
An API outage may prevent new declarations and status reporting.
An etcd problem can threaten authoritative state.
ONE CONTROL-PLANE COMPONENT FAILS
does not necessarily mean
EVERY RUNNING APPLICATION STOPS IMMEDIATELY
The impact depends on which decisions and data paths require the failed component.
The control plane is not one magical Kubernetes daemon. It is a set of cooperating distributed components with its own availability and recovery model.
3. A Pod becomes a process on one selected node
Kubernetes does not schedule individual containers as its smallest workload abstraction.
It schedules Pods.
A Pod contains one or more containers that belong together operationally.
Containers in one Pod usually share:
- one network namespace;
- one IP address;
- one port space;
localhost;- attached volumes;
- placement and lifecycle.
POD
│
├── application container
└── helper container
For a simple service, a Pod may contain one application container. The Pod still matters because Kubernetes schedules, addresses and manages that unit.
The container runtime creates containers.
Kubernetes reasons about the Pod.
Containers in a Pod share one network identity
Suppose a Pod contains a catalog process and a helper process.
POD NETWORK NAMESPACE
│
├── catalog-service
│ listens on 127.0.0.1:50051
│
└── helper
connects to 127.0.0.1:50051
They share the Pod IP and port space.
That is a tighter relationship than two ordinary Docker containers using separate network namespaces.
Containers belong in one Pod when they form one operational unit, not merely because one large diagram box looks tidy.
Pods are disposable identities
A Pod can disappear because:
- its container repeatedly fails;
- its node becomes unavailable;
- a rollout replaces it;
- an operator deletes it;
- a controller creates a different revision.
A replacement is a new Pod.
OLD POD
catalog-7d84f7c9d5-a1b2c
10.20.1.17
NEW POD
catalog-7d84f7c9d5-k8m4p
10.20.3.42
Applications should not depend on one Pod’s permanent name, IP address or writable filesystem.
Higher-level resources preserve intent and discovery over disposable Pods.
Deployments manage interchangeable replicas
A Deployment commonly manages stateless application replicas.
DEPLOYMENT
│
▼
REPLICASET
│
├── POD 1
├── POD 2
└── POD 3
If one Pod disappears, the controller creates a replacement from the Pod template.
This is replacement, not resurrection.
The new Pod does not inherit the old process memory or its ephemeral filesystem.
Any important state stored only inside the failed Pod may be gone.
The scheduler chooses a node
A new Pod initially has no assigned machine.
The scheduler watches for unscheduled Pods, filters unsuitable nodes, ranks the remaining candidates and records one node assignment.
It may consider:
- CPU and memory requests;
- available node capacity;
- taints and tolerations;
- affinity and anti-affinity;
- topology-spread constraints;
- storage requirements;
- policy extensions.
The scheduler chooses placement.
It does not start the containers itself.
Requests guide placement, limits govern runtime use
A container can declare:
resources:
requests:
cpu: "250m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
The main effects differ:
| Setting | Main effect |
|---|---|
| CPU request | Scheduler capacity accounting and placement |
| Memory request | Scheduler capacity accounting and placement |
| CPU limit | CPU-time throttling by the node kernel |
| Memory limit | Reactive enforcement that can end in an OOM kill |
| Ephemeral-storage request or limit | Node-local storage scheduling and governance |
A request is not a permanent physical reservation in every operating sense, but it is the scheduler’s planning unit.
A workload can often consume more than its request when spare capacity exists.
CPU limits normally throttle the process when it reaches its quota.
Memory enforcement is reactive. A container may be killed when it exceeds available memory under its cgroup conditions.
Kubernetes decides where the Pod should run.
The Linux kernel governs consumption on the chosen node.
The kubelet realises assigned Pods
Each worker node runs a kubelet.
The kubelet watches the API for Pods assigned to its node and works to make them real.
Its responsibilities include:
- obtaining Pod specifications;
- preparing volumes;
- requesting container creation;
- running probes;
- reporting Pod and node status;
- stopping containers when Pods are removed;
- restarting containers according to policy.
CONTROL PLANE INTENT
│
▼
KUBELET
│
▼
NODE RUNTIME ACTION
The kubelet is the control plane’s representative on the node.
CRI separates Kubernetes from one runtime product
The kubelet communicates with a container runtime through the Container Runtime Interface.
KUBELET
│
│ CRI
▼
CONTAINERD, CRI-O OR ANOTHER CRI RUNTIME
│
▼
OCI RUNTIME
│
▼
LINUX PROCESS
Kubernetes removed its built-in dockershim integration in version 1.24. Modern clusters expect a CRI-compatible runtime.
Docker may build the image, but Docker Engine does not need to be the node runtime. Containerd, CRI-O or another CRI implementation can run a compatible OCI image.
“Docker image” should not be confused with “Docker Engine is the Kubernetes runtime.”
The control plane is not in every customer request
Once a Pod is running, ordinary application traffic does not normally pass through the API server.
CUSTOMER REQUEST
│
▼
APPLICATION SERVICE
│
▼
POD
The control plane manages resources.
The application data plane carries customer traffic.
A temporary control-plane outage can block deployment, scaling or rescheduling while existing Pods continue serving traffic on their nodes.
4. Services connect callers to changing Pods
Pods are replaceable and their IP addresses change.
A Kubernetes Service provides a stable network identity for a logical group of Pods.
apiVersion: v1
kind: Service
metadata:
name: catalog
spec:
selector:
app: catalog
ports:
- name: grpc
port: 50051
targetPort: grpc
The selector finds Pods with matching labels.
SERVICE: catalog
│
├── catalog Pod A
├── catalog Pod B
└── catalog Pod C
Callers can use a stable DNS name such as:
catalog:50051
rather than tracking individual Pod addresses.
The Service does not create application replicas.
The Deployment does that.
The Service provides stable discovery over whichever ready replicas currently match.
Labels create relationships
Labels are key-value metadata used by selectors.
metadata:
labels:
app: catalog
environment: dev
component: api
A Deployment selector identifies the Pods it manages.
A Service selector identifies the Pods that should receive traffic.
Network policy, observability and operations can use the same metadata.
Labels are not decorative stickers. Poorly chosen labels can connect resources that were never meant to meet, a small administrative haiku with surprisingly loud consequences.
EndpointSlices represent current backends
For selector-backed Services, controllers maintain EndpointSlice resources that represent current backend addresses and readiness information.
Conceptually:
SERVICE SELECTOR
│
▼
ENDPOINTSLICES
│
├── ready Pod A
├── ready Pod B
└── not-ready Pod C
The Service API declares the stable logical connection.
EndpointSlices describe the current set of backends.
A Service data-plane implementation then forwards traffic.
Services need a data plane
A Service object cannot transport one byte by the force of its YAML charisma.
Something must implement forwarding from the Service identity to suitable Pod endpoints.
Historically this has often been kube-proxy, using kernel networking mechanisms. Some network implementations provide a different data plane, including eBPF-based systems.
SERVICE ADDRESS
│
▼
SERVICE DATA PLANE
│
├──► ready Pod A
├──► ready Pod B
└──► ready Pod C
The API describes intent.
Node and network components implement it.
A Service balances connections, not every application operation
For ordinary short-lived connections, Service forwarding may spread connections across several backends.
Protocols with long-lived connections need more care.
gRPC commonly multiplexes many RPCs over one HTTP/2 connection. A client channel can establish one connection through the Service and then send many RPCs to the same Pod.
ONE gRPC CHANNEL
│
▼
ONE SELECTED POD
│
├── RPC 1
├── RPC 2
└── RPC 3
Three ready Pods therefore do not automatically imply evenly distributed RPCs.
A gRPC system may need:
- several client channels;
- client-side service discovery and balancing;
- an L7 proxy;
- another gRPC-aware balancing strategy.
The Service gives stable discovery and connection forwarding. It does not understand every application protocol’s request-level semantics.
Readiness controls traffic eligibility
A process can be running without being ready to serve.
It may still be:
- warming caches;
- loading configuration;
- establishing dependencies;
- draining;
- waiting for a required resource.
A readiness probe reports whether the Pod should currently receive Service traffic.
PROCESS RUNNING
│
▼
READINESS PROBE
│
├── succeeds ──► eligible backend
└── fails ─────► removed from ready endpoints
A readiness failure normally does not restart the container.
It changes traffic eligibility.
CNI implementations build Pod connectivity
Kubernetes defines a Pod networking model. A Container Network Interface implementation realises that model on nodes.
The implementation may use:
- routes;
- bridges;
- encapsulation;
- cloud network interfaces;
- eBPF;
- network-policy enforcement.
KUBERNETES API
declares networking intent
NETWORK IMPLEMENTATION
builds the data path
The exact behaviour depends on the selected networking system.
NetworkPolicy requires enforcement
A NetworkPolicy declares allowed traffic at the API level.
It has an effect only when the cluster’s network implementation supports and enforces the NetworkPolicy API.
A useful starting point is often:
- default-deny ingress;
- default-deny egress where practical;
- explicit allowances for required peers, DNS and external dependencies.
Kubernetes Namespaces do not automatically isolate network traffic.
The policy object and the enforcing implementation are both required.
Gateway and Ingress expose applications
External traffic often needs another layer:
EXTERNAL CLIENT
│
▼
LOAD BALANCER OR GATEWAY
│
▼
SERVICE
│
▼
PODS
An API resource describes routing intent.
A controller and data-plane implementation make it real.
Ingress remains widely deployed and supported. Its API is stable but frozen.
For new platform designs, Kubernetes recommends the Gateway API, which provides a newer model for infrastructure and application routing responsibilities.
Creating an Ingress or Gateway resource without a matching implementation may produce an elegant declaration and no usable traffic path.
5. Kubernetes replaces processes but does not know business truth
Kubernetes can observe process exits, probe results, Pod absence and node status.
It cannot automatically understand every business failure.
That boundary appears at several recovery levels.
Liveness asks whether restart is useful
A liveness probe asks whether the container appears unable to recover without restart.
LIVENESS FAILURE
│
▼
KUBELET RESTARTS CONTAINER
A poor liveness probe can create an outage by repeatedly killing a slow but recoverable process.
For example, tying liveness directly to a temporary database outage can cause every application replica to restart while the database is already under pressure.
The probe needs a precise meaning:
| Probe | Question |
|---|---|
| Startup | Has initialisation completed? |
| Readiness | Should this Pod receive traffic now? |
| Liveness | Is restarting this container an appropriate recovery action? |
Startup probes protect slow initialisation
A startup probe delays liveness and readiness processing until startup succeeds.
CONTAINER STARTS
│
▼
STARTUP PROBE
│
├── still starting
│
└── succeeds
│
▼
LIVENESS AND READINESS
Without it, a liveness probe may kill an application before normal startup completes.
Built-in gRPC probes have a specific contract
A built-in gRPC probe is not a general application RPC.
The application must implement the standard gRPC Health Checking Protocol on the configured numeric port.
Built-in gRPC probes do not use named ports and do not provide arbitrary authentication or TLS settings.
A manifest using:
readinessProbe:
grpc:
port: 50051
therefore assumes that the process exposes that standard health service on port 50051.
Restarting a container is not replacing a Pod
If one container exits, the kubelet can restart it inside the same Pod.
POD
│
├── container attempt 1
├── container attempt 2
└── container attempt 3
If the Pod is deleted or its node is considered lost, a controller creates a replacement Pod.
CONTAINER RESTART
same Pod identity,
new process
POD REPLACEMENT
new Pod identity,
possibly new node and IP
Neither action repairs corrupt business data automatically.
They restore workload processes.
An unreachable node creates uncertainty
A node can become unreachable because it is dead or because the network between it and the control plane has failed.
The control plane cannot always prove which is true.
Pods normally tolerate NotReady or Unreachable conditions for a period before eviction and replacement.
During a partition, the old workload may still be running on the isolated node while the control plane eventually creates a replacement elsewhere.
CONTROL PLANE
old instance is unreachable
REALITY
old instance may still be running
For a stateless request-serving replica, duplicate instances may be acceptable.
For singleton workers, schedulers, payment processors, stateful leaders or other one-owner effects, the design may require:
- a lease;
- a fencing token;
- external leader election;
- database uniqueness;
- workflow-generation checks;
- storage attachment fencing.
Rescheduling proves that the control plane wants another instance.
It does not prove that the old instance has stopped producing effects.
Kubernetes Events are recent observations
Kubernetes can record Event resources for occurrences such as:
- failed scheduling;
- image pull errors;
- probe failures;
- container restarts;
- volume attachment failures.
These are operational observations, not bfstore domain events such as OrderConfirmed.
They are also short-lived, best-effort diagnostics rather than a durable audit trail.
Durable operational history belongs in:
- logs;
- metrics;
- traces;
- audit records;
- external monitoring systems.
Self-healing has a named boundary
Kubernetes can often respond to:
- a process exiting;
- a liveness probe failing;
- a Pod disappearing;
- a node becoming unavailable;
- a replica count falling below the declaration.
It cannot automatically determine that:
- a payment was captured twice;
- an event was semantically wrong;
- a database row is corrupt;
- a migration damaged data;
- an order workflow is stuck;
- a customer received duplicate notifications.
Kubernetes heals infrastructure state that its controllers understand.
Application correctness still belongs to the application.
It may now behave badly on several nodes with excellent automation.
6. Deployments coordinate change
A Deployment manages a Pod template and rollout history for interchangeable replicas.
Changing the image from:
bfstore/catalog-service:1.0.0
to:
bfstore/catalog-service:1.1.0
causes the Deployment controller to create a new ReplicaSet and gradually replace old Pods.
OLD REPLICASET
version 1.0.0
│
▼
scaled down
NEW REPLICASET
version 1.1.0
│
▼
scaled up
During the rollout, both versions may exist.
That means adjacent versions may need to coexist with:
- the same Service;
- the same database;
- the same event streams;
- the same Protobuf contracts;
- the same clients;
- the same configuration.
A Kubernetes rollout does not make incompatible versions compatible.
It coordinates their replacement.
Readiness determines availability
A new Pod should not count as available merely because its process exists.
NEW POD
│
▼
NOT READY
│
▼
INITIALISATION COMPLETES
│
▼
READY
│
▼
RECEIVES TRAFFIC
Deployment strategy controls values such as:
maxSurge;maxUnavailable;- progress deadlines.
If new Pods never become ready, a rollout can stall rather than replacing every working old Pod.
The application still needs truthful health signals.
Graceful shutdown is part of the rollout
When a Pod terminates, the application should:
- stop accepting new work;
- drain eligible in-flight requests;
- leave consumer groups cleanly;
- flush essential telemetry;
- exit before the termination grace period ends.
terminationGracePeriodSeconds gives the process time to respond before forced termination.
The Service endpoint update and process termination are distributed actions. Applications should tolerate a small period in which traffic and shutdown transitions overlap.
Configuration changes need an update model
ConfigMaps separate ordinary configuration from the image.
Secrets distribute sensitive configuration.
Changing a ConfigMap or Secret does not guarantee that every existing process reloads it immediately.
The update model may be:
- application watches mounted files;
- application reloads on signal;
- a rollout restarts Pods;
- a controller updates a workload revision.
The design should define which behaviour is expected.
Rollback restores workload configuration
A Deployment rollback can restore an earlier Pod template.
It cannot automatically undo:
- database migrations;
- messages already published;
- payments already captured;
- files already transformed;
- external API calls;
- customer-visible actions.
KUBERNETES ROLLBACK
restore earlier workload configuration
BUSINESS ROLLBACK
not automatically provided
Kubernetes replaces processes.
The application owns the consequences produced by those processes.
7. Availability requires placement, security and storage design
Three replicas reduce dependence on one process.
They do not by themselves guarantee availability.
Placement must cross failure domains
If all three Pods run on one node, one node failure removes every replica.
NODE A
catalog A
catalog B
catalog C
Affinity, anti-affinity and topology-spread constraints can distribute replicas across nodes or zones.
ZONE A ZONE B ZONE C
catalog A catalog B catalog C
Strict rules can also leave Pods unscheduled when suitable capacity does not exist.
Policy does not manufacture machines.
PodDisruptionBudgets protect only some disruptions
A PodDisruptionBudget can limit how many selected Pods are voluntarily disrupted at once during operations such as node draining.
It does not:
- prevent an involuntary node failure;
- create missing capacity;
- guarantee application availability;
- constrain every possible deletion path.
A PDB is useful planned-disruption policy, not a general uptime guarantee.
Shared dependencies still dominate availability
Three healthy application Pods can all depend on one unavailable database.
THREE CATALOG PODS
│
▼
ONE FAILED DATABASE
Replica count is one part of availability.
Topology, dependency design, capacity and storage recovery still matter.
Kubernetes Namespaces group API resources
A Kubernetes Namespace is an API-level scope.
It is not a Linux kernel namespace.
| Linux namespace | Kubernetes Namespace |
|---|---|
| Isolates a process view on one node | Groups and scopes API resources |
| Enforced by the kernel | Enforced through Kubernetes API semantics and policy |
| Covers resources such as PIDs or networking | Covers names, RBAC, quotas and policy organisation |
Different Kubernetes Namespaces may still share nodes, kernels, control-plane components, networks and storage systems.
Namespaces support organisation and policy. They do not automatically provide complete tenant isolation.
Secrets are distribution objects, not magical vaults
Kubernetes Secrets provide a standard API object for distributing sensitive values.
By default, Secret data is stored unencrypted in etcd unless the cluster is configured for encryption at rest.
Base64 is encoding, not encryption.
The design should include:
- encryption at rest;
- least-privilege RBAC;
- rotation;
- audit;
- restricted backup access;
- external secret systems where appropriate;
- avoiding accidental exposure through environment variables and logs.
Permission to create Pods in a Namespace can also provide indirect access to Secrets that those Pods are allowed to mount.
Secret access must be considered alongside workload-creation permissions.
Workloads should receive minimal API credentials
Pods normally use a ServiceAccount identity when calling the Kubernetes API.
A workload that does not need API access should disable automatic credential mounting:
automountServiceAccountToken: false
A workload that does need API access should receive a dedicated ServiceAccount with the smallest required RBAC permissions.
Running inside the cluster is not a reason to receive broad control-plane authority.
Security context expresses process constraints
A workload can request controls such as:
- non-root execution;
- no privilege escalation;
- dropped capabilities;
- a read-only root filesystem;
- a runtime-default seccomp profile.
These settings strengthen the container boundary. They do not replace image security, NetworkPolicy, secrets management or application security.
Persistent volumes separate data from Pod lifecycle
Pod filesystems are generally ephemeral.
Persistent data can be attached through storage resources.
POD
│
▼
PERSISTENT VOLUME CLAIM
│
▼
PERSISTENT VOLUME
│
▼
STORAGE SYSTEM
A replacement Pod may attach the same persistent storage, subject to topology and access constraints.
A PersistentVolume is persistence plumbing.
It is not a completed data-protection strategy.
The storage system still needs:
- replication;
- backups and restore tests;
- encryption;
- capacity monitoring;
- corruption handling;
- failure-domain analysis.
8. Different workloads need different controllers
Kubernetes resources become useful because controllers understand their desired conditions.
Common workload controllers
| Resource | Desired condition |
|---|---|
| Deployment | Maintain interchangeable replicas and coordinate updates |
| StatefulSet | Maintain stable ordinal identities and storage associations |
| DaemonSet | Maintain a Pod on each eligible node |
| Job | Reach successful completion |
| CronJob | Create Jobs according to a schedule |
A StatefulSet can preserve names such as:
mysql-0
mysql-1
mysql-2
and associate them with persistent volumes.
It does not make a database correct or highly available automatically. The database still needs replication semantics, leader election, quorum design, backups, recovery and upgrade procedures.
A Job represents work intended to finish, such as:
- a migration;
- a report;
- a batch import;
- a repair;
- reconciliation.
A DaemonSet expresses desired state relative to nodes rather than one fixed replica count.
Custom resources extend the control model
A custom resource might describe:
DatabaseBackup
KafkaTopic
ApplicationEnvironment
A custom controller observes that resource and performs reconciliation.
CUSTOM RESOURCE
│
▼
CUSTOM CONTROLLER
│
▼
CLUSTER OR EXTERNAL ACTIONS
This is the operator pattern at its simplest.
The YAML does not contain the behaviour.
The controller does.
The lessons from distributed idempotency return here wearing a Kubernetes badge. Reconciliation may run many times, so each pass should be safe and should expose progress through status.
Kubernetes is a substrate, not a complete platform
A usable production platform still needs decisions and implementations for:
- cluster provisioning;
- identity and access;
- networking;
- gateways and certificates;
- storage;
- secrets;
- image registries;
- logging, metrics and traces;
- policy;
- backups;
- upgrades;
- cost controls;
- developer workflows.
KUBERNETES
workload and control-plane primitives
PLATFORM
Kubernetes
+
approved patterns
+
automation
+
security
+
observability
+
developer experience
+
operations
Installing a cluster creates a powerful machine room.
It does not label every switch or train every operator.
Kubernetes does not build images
A typical delivery path is:
SOURCE CODE
│
▼
CI BUILD
│
▼
CONTAINER IMAGE
│
▼
IMAGE REGISTRY
│
▼
KUBERNETES DEPLOYMENT
│
▼
PODS
The build system produces the artifact.
The registry stores and distributes it.
Kubernetes declares and operates the runtime workload.
A release should use an immutable image identity. A readable version tag is useful, but production deployment records should also retain or pin the resolved digest.
9. A bfstore declaration, review checklist and mental model
A more complete bfstore catalog Deployment might look like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog
labels:
app: catalog
spec:
replicas: 3
selector:
matchLabels:
app: catalog
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: catalog
spec:
automountServiceAccountToken: false
terminationGracePeriodSeconds: 30
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: catalog
image: bfstore/catalog-service:1.0.0
ports:
- name: grpc
containerPort: 50051
envFrom:
- configMapRef:
name: catalog-config
- secretRef:
name: catalog-database
resources:
requests:
cpu: "250m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
startupProbe:
grpc:
port: 50051
periodSeconds: 5
failureThreshold: 30
readinessProbe:
grpc:
port: 50051
periodSeconds: 5
livenessProbe:
grpc:
port: 50051
periodSeconds: 10
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop:
- ALL
And its Service:
apiVersion: v1
kind: Service
metadata:
name: catalog
spec:
selector:
app: catalog
ports:
- name: grpc
port: 50051
targetPort: grpc
The gRPC probes assume that the service implements the standard gRPC Health Checking Protocol on numeric port 50051.
For a reproducible production release, the deployment process should record or pin the image digest rather than relying only on a mutable tag.
Surrounding resources may include:
- topology-spread constraints;
- a PodDisruptionBudget;
- a default-deny NetworkPolicy with explicit allowances;
- a dedicated ServiceAccount when API access is required;
- a Gateway or selected Ingress implementation;
- explicit Secret access and rotation policy.
Following the declaration through the system
| Stage | Main owner |
|---|---|
| Accept and validate resource | API server |
| Persist desired state | API server and etcd |
| Create or update ReplicaSet | Deployment controller |
| Create Pod resources | ReplicaSet controller |
| Choose nodes | Scheduler |
| Realise assigned Pods | Kubelet and container runtime |
| Publish current backends | EndpointSlice controller and Service data plane |
| Report progress | Controllers and kubelets |
| Correct later drift | The same control loops, repeatedly |
This is not one monolithic operation.
DECLARATION
│
▼
API STATE
│
▼
CONTROLLER ACTIONS
│
▼
SCHEDULER DECISION
│
▼
NODE ACTION
│
▼
RUNNING PROCESS
The control plane is itself event-driven and reconciliatory.
The resource can exist while the workload does not
A declared Deployment may remain unrealised because of:
| Condition | Example |
|---|---|
| Unschedulable | No suitable node has enough requested capacity |
| Image pull failure | Missing image, registry outage or invalid credentials |
| Container crash | Process exits during startup |
| Readiness failure | Process runs but cannot serve traffic |
| Missing configuration | Required ConfigMap or Secret is absent |
| Storage failure | Required volume cannot be attached |
| Policy rejection | Admission or security rules reject the workload |
The desired state remains recorded.
Status, conditions, Events and external telemetry explain why reality has not caught up.
Kubernetes review checklist
| Concern | Question |
|---|---|
| Workload controller | Which controller matches this workload’s lifecycle? |
| Pod boundary | Which containers genuinely need shared networking, storage and placement? |
| Image identity | Which immutable image digest should run? |
| Replicas | How many instances are needed, and across which failure domains? |
| Requests | How much capacity should the scheduler plan for? |
| Limits | Which runtime CPU, memory and storage boundaries apply? |
| Placement | Which nodes and zones are eligible, preferred or prohibited? |
| Disruption | Which voluntary disruptions should a PDB limit? |
| Readiness | When should this Pod receive traffic? |
| Liveness | Which condition truly requires process restart? |
| Startup | How long may normal initialisation take? |
| Shutdown | Can the process drain before the grace period expires? |
| Service discovery | Which stable name should callers use? |
| Protocol balancing | Do long-lived connections require application-aware balancing? |
| Network policy | Does the selected CNI enforce the declared rules? |
| External routing | Which Gateway or Ingress implementation makes the route real? |
| Configuration | Which values live outside the image, and how are changes applied? |
| Secrets | Are Secrets encrypted at rest, tightly authorised and rotated? |
| API credentials | Does the Pod need a ServiceAccount token at all? |
| Security context | Can it run non-root, without added capabilities and with seccomp? |
| Storage | Which data must outlive Pod replacement, and how is it recovered? |
| Rollout | Can old and new versions coexist safely? |
| Node uncertainty | Can a replaced singleton still be running on an isolated node? |
| Observability | Can operators distinguish desired state, observed state and business health? |
| Recovery | What remains broken after Kubernetes successfully restarts the process? |
The last question prevents infrastructure automation from being mistaken for application recovery.
The mental model I am keeping
My original model was:
KUBERNETES
software that runs Docker containers
on several machines
The stronger model is:
KUBERNETES API
│
▼
DESIRED STATE
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
CONTROLLERS SCHEDULER KUBELETS
│ │ │
▼ ▼ ▼
create and adjust choose nodes manage assigned Pods
│ │ │
└────────────────┼────────────────┘
▼
CONTAINER RUNTIMES
│
▼
LINUX PROCESSES
The API server answers:
Where is cluster intent accepted and exposed?
etcd answers:
Where is authoritative cluster state stored?
Controllers answer:
How is drift between desire and reality corrected?
The scheduler answers:
Which node should receive this Pod?
The kubelet answers:
How does one node realise its assigned workload?
The container runtime answers:
How are the containers actually created?
A Deployment answers:
How many interchangeable replicas should exist,
and how should their configuration change?
A Service answers:
How do callers find changing replicas
through one stable identity?
Probes answer:
Has this process started,
should it receive traffic,
and is restart a useful recovery action?
Kubernetes Namespaces, RBAC, admission and NetworkPolicy answer parts of:
How are resources grouped,
who may change them,
and which declared connections are allowed?
And the complete control plane answers the most important question:
Who keeps checking after the deployment command ends?
That is the movement from containers to Kubernetes.
A container gives one process a portable, governed runtime environment.
Kubernetes places that environment inside a continuously managed system of intent, observation and correction.
It does not make failures disappear.
It gives many infrastructure failures a controller, a recorded desired state and another attempt.
It still cannot always prove what happened on an unreachable machine or inside the business process.
That is where stable operation identities, fencing, durable data and application recovery continue beyond the control plane.
References and further reading
Kubernetes concepts
Introduces Kubernetes architecture, workloads, Services, storage, configuration and cluster behaviour.
Kubernetes components
Describes control-plane and node components, including the API server, etcd, scheduler, controller manager and kubelet.
Kubernetes API overview
Explains interaction with Kubernetes resources through the API.
Pods
Documents Pods as the smallest deployable Kubernetes workload unit and explains their shared networking and storage model.
Deployments
Documents replica management, rolling updates and rollout status.
Services
Explains stable network access to changing groups of Pods.
EndpointSlices
Documents the scalable API used to represent Service backends.
Container Runtime Interface
Describes the interface between the kubelet and container runtimes.
Container runtimes
Documents CRI-compatible node runtime configuration and the modern runtime model.
Resource management for Pods and containers
Documents CPU, memory and storage requests and limits.
Liveness, readiness and startup probes
Explains probe semantics, including built-in gRPC probe requirements.
Secrets
Documents Secret storage, access and recommended protection.
Service accounts
Explains workload API identities and token mounting.
NetworkPolicy
Documents L3 and L4 policy intent and the need for an enforcing network implementation.
Gateway API
Introduces the current Kubernetes model for advanced traffic routing.
Ingress
Documents the supported but frozen Ingress API.
Disruptions and PodDisruptionBudgets
Explains voluntary and involuntary disruption and the scope of PDB protection.
Taints and tolerations
Documents node conditions, tolerations and eviction behaviour.
Seccomp for Kubernetes
Documents seccomp profile configuration for Pods and containers.
gRPC load balancing
Explains how long-lived gRPC channels affect request distribution and balancing choices.
Open Container Initiative runtime specification
Defines the lower-level runtime configuration used to create container processes.