A repository contains the desired version of the bfstore catalog service:
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog
spec:
replicas: 3
template:
spec:
containers:
- name: catalog
image: bfstore/catalog-service@sha256:7b91...
A pull request changes the image digest:
- image: bfstore/catalog-service@sha256:24ac...
+ image: bfstore/catalog-service@sha256:7b91...
The pull request is reviewed and merged.
A controller notices the change and updates the cluster.
CHANGE GIT
│
▼
RECONCILER OBSERVES
│
▼
CLUSTER CHANGES
That sequence looks simple, but the delivery model has changed.
The deployment was not initiated by an engineer running a command from a laptop.
The CI system did not need permanent credentials capable of changing the cluster.
The approved production version became visible in a reviewed, versioned record.
The cluster was changed by a controller operating with authority inside the delivery boundary.
If someone later edits the Deployment manually, the controller can notice that the running state no longer matches the approved declaration.
Git has become more than a place where deployment files are stored.
GitOps treats version-controlled declarations as the approved authority for the resources and fields assigned to a reconciliation system. Automated controllers continually work to bring those parts of the running environment towards the declared state.
Git is not the source of every fact about the environment.
It is authoritative for the durable intent placed within its ownership boundary.
The merge does not directly perform every infrastructure action.
It changes the versioned instruction that the control system is responsible for enforcing.
Git needs an active reconciliation loop
A repository containing Kubernetes YAML is not automatically a GitOps system.
Without a reconciler, the repository is documentation and potential input.
GIT REPOSITORY
desired Deployment exists
│
✕
│ no active reconciler
▼
CLUSTER
remains unchanged
An engineer might still apply the file manually:
kubectl apply -f deployment.yaml
That uses Git as a storage location.
A GitOps system adds an active control loop:
VERSIONED DECLARATIONS
│
▼
RECONCILER
│
│ authorised resources and fields
▼
TARGET ENVIRONMENT
The controller repeatedly asks:
- Which approved revision should I observe?
- Which resources and fields do I own?
- What state currently exists in the target?
- Which differences fall inside my reconciliation boundary?
- Which permitted actions can reduce those differences?
- Did the environment converge successfully?
Git supplies the declaration.
The reconciler supplies the behaviour.
The environment reports whether the result became healthy reality.
The repository becomes an operational control surface
In a conventional deployment workflow, a pipeline might authenticate to the cluster and execute:
deploy image 1.8.0
With GitOps, the control input becomes a versioned change to desired state:
production catalog image should be:
sha256:7b91...
Instead of saying:
Perform these deployment steps now.
the operator says:
This is the state production should maintain.
The reconciler still makes imperative API calls underneath.
Pods are created.
Deployments are updated.
Resources are deleted.
Declarative intent does not remove operational consequences.
It changes where those consequences are initiated and governed.
A pull request changing production configuration may cause:
- workloads to restart
- images to change
- replicas to scale
- network policy to tighten
- credentials to be referenced differently
- resources to be deleted
- traffic to move
A tiny diff can represent a large runtime event:
- replicas: 3
+ replicas: 6
The repository therefore needs controls appropriate to a production interface:
- protected branches
- required reviews
- policy checks
- schema validation
- clear ownership
- environment-specific approval
- auditable merge history
A production change should answer:
Who proposed it?
Who reviewed it?
Which checks passed?
Which environment will reconcile it?
Which controller is authorised to apply it?
What material runtime effect should we expect?
Compromising the repository, workflow or review process can become a route to changing the environment.
The Git platform is part of the production control plane.
Merge, observation, convergence and health are different states
A merged pull request proves that Git accepted a new desired state.
It does not prove that the target environment reached it.
A useful status model distinguishes:
MERGED
approved intent changed
OBSERVED
controller fetched the revision
APPLIED
platform accepted the submitted objects
CONVERGED
resources reached the requested state
VERIFIED
service remained healthy and useful
Each transition can take time or fail.
Convergence may be blocked because:
- the image does not exist
- admission policy rejects the manifest
- the controller lacks permission
- the cluster API is unavailable
- a resource definition is invalid
- a dependency is missing
- the workload never becomes ready
GitOps is eventually consistent.
During reconciliation:
GIT
new desired state
CLUSTER
partly old,
partly new,
still converging
The controller should report an immutable observed revision, not only a branch name:
repository:
bfstore-environments
reference:
main
observed commit:
a42d6e1
Operators also need:
- reconciliation result
- resource readiness
- controller errors
- drift
- suspended resources
- last successful revision
A green merge button is not a production health signal.
Git records intent.
Runtime systems report whether that intent became healthy reality.
Reconcilers need bounded authority
In the usual GitOps model, an agent associated with the target environment observes approved declarations and reconciles them using its own constrained identity.
The agent may run inside the cluster, beside it or in a managed control plane.
The important distinction is authority, not merely network direction.
CI IDENTITY
build artefact
publish artefact
propose Git change
RECONCILER IDENTITY
read approved declarations
manage assigned resources
inside one target boundary
This can reduce the need to distribute powerful environment credentials to external CI workers.
It also makes the reconciler a privileged component that must be protected, monitored and updated carefully.
An application reconciler might manage:
- Deployments
- Services
- ConfigMaps
- approved custom resources
within selected Namespaces.
It may not need authority to:
- modify cluster-wide identity
- alter admission policy
- delete the cluster
- read every Secret
- manage unrelated environments
Separate controllers or identities may own separate boundaries:
PLATFORM RECONCILER
cluster add-ons
shared controllers
platform configuration
APPLICATION RECONCILER
bfstore workloads
service configuration
SECURITY RECONCILER
approved policy resources
GitOps does not remove IAM.
It depends on carefully designed trust and permissions.
Controllers need explicit resource and field ownership
A cluster already contains several control loops:
- Deployment controllers
- StatefulSet controllers
- schedulers
- autoscalers
- operators
- policy controllers
- GitOps reconcilers
They can legitimately own different parts of the same system.
Suppose Git declares:
minimum replicas:
3
maximum replicas:
12
target utilisation:
70%
The autoscaler controls:
current replicas
Git declares the autoscaling policy.
The autoscaler makes the live decision.
If Git and the autoscaler both attempt to own the same replica field, they can fight.
The same issue appears with:
- mutating admission
- controllers that add defaults
- operators that manage generated child resources
- fields intentionally ignored during comparison
Drift exists only relative to the resources and fields the reconciler is configured to own.
A difference is not automatically unauthorised merely because Git does not contain the final runtime value.
Desired state is not one universal number.
It is a set of ownership claims shared between cooperating control loops.
Drift and emergency changes need an operating model
Suppose Git declares:
spec:
replicas: 3
An operator manually changes the Deployment to eight replicas.
The reconciler may restore three because Git remains authoritative for that field.
The problem is not automation failure.
It is disagreement over authority.
A useful operating model distinguishes:
UNAUTHORISED DRIFT
reconcile automatically
LEGITIMATE EXTERNAL CONTROLLER
respect assigned ownership
EMERGENCY HUMAN CHANGE
pause, record, codify and resume
APPROVED EXCEPTION
document and configure explicitly
During an incident, editing Git first may be too slow or impossible.
An operator may need to:
- scale a service immediately
- disable a failing component
- block dangerous traffic
- suspend reconciliation
- apply a temporary policy
One possible procedure is:
- pause reconciliation for the affected resource
- make the emergency change
- record the reason, owner and expiry
- update Git to reflect the intended durable state
- review the resulting difference
- resume reconciliation
- verify convergence
GitOps should reduce undocumented drift, not obstruct recovery for the sake of ritual purity.
Temporary authority should not become invisible permanent state.
Promotion and rollback change environment intent
A build pipeline may produce:
bfstore/catalog-service@sha256:7b91...
Development can declare that digest first.
After validation, staging declares the same digest.
Production later declares it through another reviewed change.
ONE ARTEFACT
sha256:7b91...
│
├── development declaration
├── staging declaration
└── production declaration
Promotion is represented as a change in environment intent.
The application is not rebuilt for each environment.
The artefact remains stable.
What changes is the set of environments that approve it.
Weak declaration:
image: bfstore/catalog-service:latest
The tag may move while the Git revision remains unchanged.
Stronger declaration:
image: bfstore/catalog-service@sha256:7b91...
Now the Git diff identifies the exact application content being promoted.
Rollback follows the same model.
A revert can restore the previous digest:
- image: bfstore/catalog-service@sha256:7b91...
+ image: bfstore/catalog-service@sha256:24ac...
The controller moves the environment towards the earlier declaration.
But Git cannot reverse every system effect.
The newer version may already have:
- changed a database schema
- written data
- published events
- called external systems
- altered durable workflow state
Git can restore an earlier declaration.
System recovery may still require migration, repair or compensation.
Git history is a powerful operational record.
It is not a transaction log for the entire business.
The repository model defines ownership and review
Possible repository designs include:
APPLICATION REPOSITORY
source code
tests
build definition
ENVIRONMENT REPOSITORY
deployment declarations
environment configuration
release digests
Another organisation may keep source and deployment definitions together.
Neither arrangement is universally correct.
The useful questions are:
- who owns application code?
- who owns production intent?
- who may approve promotion?
- which changes should share a pull request?
- which repository can the reconciler read?
- where are environment boundaries enforced?
- how are cross-repository changes coordinated?
Branches are not automatically good environment boundaries.
A directory-based model can keep environments visible in one history:
main
└── environments/
├── dev/
├── staging/
└── prod/
Promotion becomes a pull request changing one environment directory.
The important decision is not directory aesthetics.
It is which representation is authoritative and who may approve changes to it.
Repositories may contain:
- raw YAML
- Helm values
- Kustomize overlays
- generated manifests
- custom templates
Review should show both the human-authored input and the material rendered difference the reconciler is expected to apply.
Useful evidence includes:
input revision
renderer and version
rendered output or digest
created and deleted resources
image changes
permission changes
policy results
target environment
GitOps should make change more inspectable, not bury the real effect beneath templating fog.
Deletion belongs in this review model too.
Removing one declaration may cause a reconciler to prune a valuable resource.
Declarative systems make deletion look calm.
The runtime consequences remain very real.
Secrets and sensitive configuration need separate authority
Plaintext secrets should not be committed to Git.
Git retains history, so removing the current file does not reliably remove earlier values.
Possible patterns include:
- encrypted secret documents
- references to an external secret manager
- sealed-secret mechanisms
- controller-generated credentials
- short-lived workload identity
The repository might declare:
databaseCredential:
secretReference: bfstore/prod/catalog/database
The secret value remains in a controlled system.
GIT
declares which secret is required
SECRET MANAGER
stores sensitive value
WORKLOAD IDENTITY
controls who may retrieve it
Encrypted secret files remain sensitive operational material.
Their security depends on governance and separation of the decryption keys.
A reference alone is also not enough for recovery.
The operating model must explain:
- who controls decryption
- how the value is restored or regenerated
- how rotation is performed
- which workloads may retrieve it
- what happens when the referenced secret is missing
GitOps does not make secret management disappear.
It makes the relationship between declared workloads and secret authority explicit.
Git records durable intent, not runtime status
Git should not contain every changing runtime value.
Values that usually belong outside Git include:
- current Pod names
- allocated IP addresses
- active replica identities
- health status
- temporary credentials
- queue depth
- recent error rate
- current autoscaler decisions
Git records durable intent:
desired image
resource constraints
service configuration
policy
minimum availability
routing relationships
The runtime reports status:
available replicas
current conditions
allocated identifiers
recent failures
observed generation
This mirrors the distinction between specification and status.
Copying volatile status back into Git would create constant noise and unclear authority.
Observability verifies the result
A reconciler may report that every resource is ready while the service still fails customers.
GitOps status should connect to broader observability:
GIT CHANGE
│
▼
RECONCILIATION
│
▼
DEPLOYMENT STATUS
│
▼
METRICS, LOGS AND TRACES
│
▼
BUSINESS VERIFICATION
A production promotion should expose:
- Git commit
- application digest
- reconciliation time
- rollout status
- service version
- error rate
- latency
- critical workflow results
This lets an operator ask:
Did behaviour change after
this desired-state revision?
Git provides the approved change record.
Telemetry provides the runtime consequence.
A delivery control system needs both.
Bootstrap defines the outer ownership boundary
A GitOps controller manages resources from Git.
Something must initially create:
- the cluster
- the controller
- its identity
- its repository access
- its first configuration
BOOTSTRAP AUTOMATION
│
▼
CLUSTER
│
▼
GITOPS CONTROLLER
│
▼
RECONCILED RESOURCES
This creates a boundary between bootstrap infrastructure and reconciled infrastructure.
For bfstore, Terraform might create:
- the network
- the Kubernetes cluster
- the controller identity
- the initial GitOps installation
The GitOps controller might then manage:
- platform add-ons
- application Namespaces
- bfstore workloads
- environment configuration
That boundary should be documented and tested.
Otherwise Terraform and GitOps may both attempt to manage the same resources.
Bootstrap should be reproducible too.
A controller cannot recover the environment when nobody remembers how the controller itself entered the world.
A compact bfstore flow
Suppose a catalog change is approved.
CI
tests source
builds one immutable image
records evidence
PROMOTION CHANGE
updates the development declaration
to the new digest
RECONCILER
observes immutable Git commit
and applies assigned resources
RUNTIME
reports convergence and service health
LATER PROMOTION
moves the same digest
to staging and production
The release remains traceable through:
source commit
image digest
Git promotion commit
reconciler revision
Deployment revision
running Pods
runtime telemetry
GitOps has not replaced CI, Kubernetes, IAM or observability.
It connects them through a versioned desired-state boundary.
A GitOps review
| Area | Review question |
|---|---|
| Authority | Which declarations, resources and fields are authoritative in Git? |
| Reconciliation | Which agent observes and corrects each environment? |
| Identity | What may each reconciler manage? |
| Governance | Who proposes, reviews and merges production intent? |
| Rendering | Can reviewers see the actual material change? |
| Artefacts | Are releases identified immutably? |
| Drift | How are manual changes, other controllers and exceptions handled? |
| Destruction | Which changes can prune valuable resources? |
| Secrets | Where do values live, and how are they recovered or rotated? |
| Status | Can operators identify the observed commit and convergence state? |
| Verification | How is healthy service behaviour proven? |
| Bootstrap | How are the controller and its access recreated? |
GitOps is useful when the path from proposal to running state remains understandable.
The mental model I am keeping
My earlier model was:
GITOPS
store Kubernetes YAML in Git
deploy it automatically
The stronger model is:
PROPOSED CHANGE
│
▼
REVIEW AND POLICY
│
▼
VERSIONED INTENT
│
▼
RECONCILER
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
OBSERVE COMPARE CORRECT
│ │ │
└──────────────┼──────────────┘
▼
RUNNING ENVIRONMENT
│
▼
STATUS AND TELEMETRY
GitOps changes Git from a historical record into an operational interface.
A commit can become a declaration of production intent.
A merge can authorise a controller to alter the environment.
A revert can ask the platform to restore an earlier configuration.
That power comes with responsibility.
The repository must be protected.
The controller must be constrained.
The desired-state boundary must be clear.
Runtime health must remain visible outside Git.
GitOps does not mean the entire system lives inside a repository.
Git cannot store the current health of every Pod, the outcome of every payment or the complete truth of the cloud platform.
It stores the durable claim the environment is expected to defend.
The reconciler reads that claim.
The platform attempts to realise it.
Telemetry tells us whether the result works.
That is what changes when Git represents desired state:
A merge no longer merely describes what engineers intend to deploy. It changes the versioned instruction that an automated control system is responsible for keeping true.
References and further reading
OpenGitOps principles Defines the core principles of declarative, versioned, automatically pulled and continuously reconciled desired state.
Kubernetes declarative management Introduces declarative object management and the distinction between configuration intent and running state.
Kubernetes controllers Explains the control loops that move Kubernetes resources towards desired state.
Flux documentation Documents a GitOps toolkit for reconciling Kubernetes clusters from version-controlled sources.
Argo CD documentation Documents declarative continuous delivery, application reconciliation, health and synchronisation status.