My first mental model of continuous integration and continuous delivery was a script with better branding.
build application
run tests
copy files to server
restart process
The script might be executed by Jenkins, GitHub Actions or another automation platform, but the underlying picture remained:
SOURCE CODE
│
▼
DEPLOYMENT SCRIPT
│
▼
SERVER
That model explains part of a delivery pipeline.
It does not explain the wider delivery system.
A production delivery system must answer questions such as:
- Which source revision and artefact are being released?
- Which tests, scans and policy checks passed?
- Who proposed, approved and deployed the change?
- Can the same artefact move through staging and production?
- What happens when health deteriorates?
- Is rollback safe after data or external systems have changed?
- Can the organisation reconstruct what happened later?
Those questions turn CI/CD from a sequence of commands into a control system.
CI/CD governs the movement of a change from source code to running software using evidence, policy, controlled authority and feedback.
The pipeline is the visible workflow.
The wider system also includes source control, builders, artefact stores, policy, deployment identities, runtime controllers, telemetry and release records.
A useful control-system model is:
TARGET STATE
approved version should run
│
▼
DELIVERY POLICY AND WORKFLOW
│
▼
DEPLOYMENT API OR RECONCILER
│
▼
RUNNING SYSTEM
│
▼
ROLLOUT, SERVICE AND BUSINESS SIGNALS
│
└──────── feedback ────────┘
The deployment command is one control signal inside that system.
It is not the whole system.
Continuous integration produces evidence
Continuous integration begins when developers combine changes frequently and automation evaluates the result.
A stronger model than “build, test, deploy” is:
SOURCE REVISION
│
▼
BUILD CANDIDATE ARTEFACT
│
├──► UNIT AND INTEGRATION TESTS
├──► STATIC AND DEPENDENCY ANALYSIS
└──► ARTEFACT SCAN
│
▼
ELIGIBLE ARTEFACT
Some checks run against source or intermediate outputs, but the evidence must remain connected to the exact artefact proposed for release.
A CI run might record:
source revision:
8f31c2a
unit tests:
passed
integration tests:
passed
dependency policy:
passed
container scan:
passed
artefact digest:
sha256:7b91...
This evidence supports a decision:
Is this exact output eligible to become a release candidate?
CI reduces uncertainty by checking properties that can be evaluated before deployment. It does not prove perfect production behaviour.
A healthy CI system should make failed evidence visible rather than converting every warning into decorative pipeline confetti.
Build once, then promote through evidence and policy
A weak delivery process rebuilds the application separately in every environment.
DEV
build revision 8f31c2a
STAGING
build revision 8f31c2a again
PRODUCTION
build revision 8f31c2a again
Those builds may use different dependencies, builders, repositories, compiler versions or generated files.
The source revision may be the same while the artefact differs.
A stronger process builds one immutable artefact and promotes that exact object.
SOURCE REVISION
│
▼
BUILD ONCE
│
▼
IMAGE DIGEST
sha256:7b91...
│
├──► development
├──► staging
└──► production
A mutable tag such as:
catalog:latest
can point to different content over time.
A digest identifies the content itself:
bfstore/catalog-service@sha256:7b91...
This lets a production deployment claim:
The artefact being deployed is the same artefact that passed the required checks.
Promotion should move trusted identity, not rebuild hope.
The artefact is only one part of a release, however.
Environment-specific configuration may remain outside the image:
application artefact:
sha256:7b91...
deployment declaration:
commit a42d6e1
environment configuration:
version prod-184
The executable artefact stays constant while the declaration and configuration remain controlled, versioned and traceable.
Promotion then becomes a policy decision.
Development may prove that the container starts, integration may verify dependencies, and staging may exercise production-shaped configuration. Production may require accumulated evidence or human approval.
ARTEFACT
│
▼
DEV EVIDENCE
│
▼
STAGING EVIDENCE
│
▼
PRODUCTION DECISION
Continuous delivery means qualifying changes remain releasable through this process.
Continuous deployment goes further by allowing qualifying changes to reach production automatically.
CONTINUOUS DELIVERY
every qualifying change can be released
CONTINUOUS DEPLOYMENT
every qualifying change is released automatically
Neither is inherently more mature. The right level of automation depends on risk, test confidence, reversibility, regulation and operational readiness.
Delivery authority should be narrow
A CI/CD platform can execute commands.
It should not automatically receive unrestricted authority over every environment.
A deployment identity might need permission to:
update one application workload
read one release artefact
observe one rollout
It should not need permission to modify IAM, delete the cluster, read unrelated secrets or administer production databases.
The access path should follow a deliberate IAM model:
PIPELINE IDENTITY
│
▼
TRUST POLICY
│
▼
TEMPORARY DEPLOYMENT ROLE
│
▼
NARROW ENVIRONMENT PERMISSIONS
Development, staging and production should use separate authority boundaries.
The combined controls of the CI platform and cloud trust policy might require:
- an approved repository;
- a protected branch or environment;
- a named deployment workflow;
- reviewed workflow changes;
- an authorised production approval.
The CI platform is not trusted merely because it is called CI.
A particular workflow receives a particular authority because its identity and context satisfy deliberate controls.
Prefer declared state where the platform supports it
In a Kubernetes environment, a delivery workflow does not need to log into worker nodes and restart containers manually.
It can submit a desired-state change:
spec:
template:
spec:
containers:
- name: catalog
image: bfstore/catalog-service@sha256:7b91...
Kubernetes controllers then create replacement Pods, schedule them, run probes, remove old replicas and report rollout status.
DELIVERY WORKFLOW
│
│ approved desired-state change
▼
KUBERNETES API
│
▼
DEPLOYMENT CONTROLLER
│
▼
ROLLING REPLACEMENT
This creates a clean division of responsibility.
The delivery system selects the approved artefact. The runtime platform reconciles workloads towards that declaration.
There are two common delivery models:
PUSH-BASED DELIVERY
workflow submits the approved declaration
PULL-BASED DELIVERY
workflow updates a versioned declaration
a separate reconciler observes and applies it
An imperative deployment API can also be safe when identity is controlled, intent is repeatable, outcomes are observed and the operation is recorded.
Submitting a declaration or receiving an API acknowledgement is not proof that deployment succeeded.
Feedback makes delivery a control system
Without feedback, a workflow sends instructions and assumes success.
DEPLOY
│
▼
COMMAND ACCEPTED
│
▼
PIPELINE GREEN
But the new version may fail to start, remain unready, increase errors, exhaust memory, time out against dependencies or break a business flow.
A stronger loop is:
DESIRED CHANGE
│
▼
DEPLOY
│
▼
OBSERVE
│
▼
COMPARE WITH ACCEPTANCE CRITERIA
│
├── healthy ──► continue
└── unhealthy
│
▼
stop, reverse or escalate
Feedback can come from several layers:
| Layer | Question |
|---|---|
| Deployment status | Did the required replicas become ready? |
| Application health | Are requests succeeding? |
| Performance | Did latency or resource consumption deteriorate? |
| Business signals | Are checkouts, payments or orders completing? |
| Safety signals | Did error budgets, security rules or data checks fail? |
A green Pod is not sufficient evidence that the customer-facing capability is healthy.
Acceptance criteria also need time and context.
For example:
error rate:
no more than 0.5 percentage points
above the stable version
for ten minutes
latency:
p95 remains within the approved threshold
for the full canary window
A useful promotion rule defines:
- the comparison baseline;
- the threshold;
- the observation window;
- the response to missing or inconclusive telemetry.
Missing evidence should not silently count as success.
Progressive delivery limits exposure
A deployment does not need to replace every instance at once.
Progressive delivery exposes a new version gradually:
NEW VERSION
5% of traffic
│
▼
observe
│
▼
25% of traffic
│
▼
observe
│
▼
100% of traffic
A canary may begin with a small instance group, blue-green delivery may prepare a parallel environment, and feature flags can separate deployment from activation.
These approaches do not remove risk.
They limit how much of the system experiences it before more evidence is available.
Promotion should use predefined criteria, not several engineers staring at graphs until somebody feels spiritually reassured.
Rollback is not always reversal
A delivery system can deploy the previous image version:
catalog 1.8.0
│
▼
catalog 1.7.3
That restores earlier application code.
It may not restore the earlier system.
The newer version may already have:
- changed a database schema;
- written new data;
- published events;
- called external providers;
- invalidated caches;
- changed stored configuration.
APPLICATION ROLLBACK
restore previous executable
SYSTEM REVERSAL
may require data repair,
schema compatibility or compensation
Safe delivery therefore depends on backwards and forwards compatibility.
A database change might follow an expand-and-contract path:
1. add new schema without removing old fields
2. deploy code that can use both forms
3. migrate data
4. move all readers and writers
5. remove old schema later
The delivery system can automate executable replacement.
Application and data design determine whether that replacement is safe.
Provenance and release evidence preserve the chain
A production release should leave a durable chain of evidence:
SOURCE
│
▼
BUILD
│
▼
ARTEFACT
│
▼
POLICY AND APPROVAL
│
▼
DEPLOYMENT
│
▼
RUNNING VERSION
Useful records include source and review details, builder identity, test results, artefact digest, deployment declaration, configuration version, deployment identity, target environment and runtime result.
Provenance connects an artefact to its source, builder, workflow, dependencies and output digest.
A deployment policy might require that an image:
- comes from an approved registry;
- was built by an approved workflow;
- corresponds to reviewed source;
- has a valid signature or attestation;
- passed required checks.
Scanning and provenance answer different questions:
VULNERABILITY SCAN
What known problems were detected?
PROVENANCE
Where did this artefact come from?
A clean malicious image is still malicious.
A well-provenanced image can still contain a vulnerable dependency.
Both forms of evidence matter.
CI/CD must defend its own inputs
Delivery systems process untrusted or semi-trusted input.
A pull request may modify application code, build scripts, workflow definitions, dependencies, infrastructure configuration or deployment manifests.
If an untrusted pull request can access production credentials, it may alter the workflow to exfiltrate them.
A secure system separates contexts:
PULL REQUEST VALIDATION
untrusted code
limited credentials
no production deployment
PROTECTED DEPLOYMENT
trusted revision
approved workflow
temporary production role
Sensitive authority should not be exposed merely because a job runs inside an approved repository.
Trust should consider the event type, source branch, fork status, protected environment, reviewed workflow and immutable revision.
The build system is part of the software supply chain.
Compromising it can produce trusted-looking malicious artefacts.
Delivery has partial failures and several kinds of success
A delivery workflow calls many independent systems. A remote operation may succeed while its response is lost, and a retry may repeat an upload or deployment request.
The delivery system therefore inherits distributed-systems problems:
timeouts
partial failure
duplicate attempts
stale observations
unknown outcomes
eventual consistency
Stable build, artefact, release and deployment identities help retries resume the same logical operation instead of creating another one.
CI success and delivery success must also remain distinct:
build result
artefact eligibility
deployment submission
rollout convergence
runtime health
business outcome
A workflow can pass while production is unhealthy, or fail after an external deployment action succeeded. One green or red badge cannot represent every stage.
A pipeline run is an operational record, not the complete truth about the service.
A possible bfstore delivery flow
Suppose a change is made to the bfstore catalog service.
Continuous integration
1. Check out the exact commit.
2. Verify formatting and generated Protobuf code.
3. Run unit and integration tests.
4. Run static and dependency analysis.
5. Build the Go binary and container image.
6. Scan the final image.
7. Publish it by immutable digest.
8. Record provenance and test evidence.
The result is:
image:
bfstore/catalog-service@sha256:7b91...
source:
commit 8f31c2a
status:
eligible for development
Development and staging
The development declaration is updated to the digest. The workflow waits for rollout readiness and a catalog smoke test.
The same digest then moves to staging, where tests verify gRPC and database compatibility, Kafka publication, consumer integration and retry behaviour.
Production
A protected environment requires approval.
The workflow obtains a temporary production role through OIDC federation and updates only the catalog workload. A canary receives limited traffic, and promotion continues only while the defined technical and business criteria remain healthy for the observation window.
Completion
The release record stores:
source commit
image digest
deployment declaration
configuration version
approver
deployment session
rollout result
production verification
The workflow has not merely copied a binary.
The delivery system has controlled the change’s movement through evidence, policy, trust boundaries and runtime feedback.
A CI/CD control-system review
| Area | Review question |
|---|---|
| Identity | Which source revision, artefact, configuration and workflow are involved? |
| Evidence | Which tests, scans and provenance claims make the release eligible? |
| Authority | Which identity may change the environment, and what may it change? |
| Promotion | Does the same artefact move between environments under explicit policy? |
| Feedback | Which rollout, service and business signals control progression? |
| Exposure | Can delivery stop before every user receives the change? |
| Compatibility | Can old and new versions coexist safely? |
| Recovery | How are retries, partial failure, rollback and compensation handled? |
| Audit | Can the running release be traced back to source, evidence and deployment identity? |
A delivery system becomes trustworthy when it can explain why a change was allowed to progress and what evidence made it stop.
The mental model I am keeping
My earlier model was:
CI/CD
build
test
deploy
The stronger model is:
SOURCE CHANGE
│
▼
EVIDENCE
│
▼
IMMUTABLE ARTEFACT
│
▼
PROMOTION POLICY
│
▼
TEMPORARY AUTHORITY
│
▼
DESIRED OR CONTROLLED CHANGE
│
▼
RUNTIME FEEDBACK
│
├──► continue
└──► stop, reverse or repair
A deployment script sends instructions to a target.
A CI/CD control system asks why this version is eligible, which identity may deploy it, which environment may receive it, whether the system became healthy and whether exposure should continue.
That is the difference between automating a command and controlling software delivery.
References and further reading
Continuous Delivery, Jez Humble and David Farley Explores deployment pipelines, build artefacts, automated testing, environment promotion and reliable software releases.
Accelerate, Nicole Forsgren, Jez Humble and Gene Kim Presents research into software-delivery performance, deployment frequency, lead time, recovery and organisational capability.
GitHub Actions security hardening Documents workflow permissions, untrusted input, third-party actions, OIDC and secure use of CI credentials.
GitHub Actions OIDC federation Explains obtaining short-lived cloud credentials without storing permanent access keys.
Kubernetes Deployments Documents declarative rollouts, rolling updates, status and revision management.
OpenGitOps principles Describes declarative desired state, versioning, automated reconciliation and continuous observation.
SLSA framework Defines software supply-chain levels and provenance practices for trustworthy artefact production.
Argo Rollouts Documents progressive-delivery strategies including canary and blue-green rollouts with metric-driven analysis.