Testing AWS SCPs instead of hoping they work

A new Service Control Policy is written for the bfstore AWS organisation.

Its purpose is straightforward:

Workload-account administrators
must not disable protected audit logging.

The JSON parses successfully.

The policy editor shows no syntax error.

A reviewer reads the document and agrees with its intention.

The policy is attached to the Production OU.

The intended operation fails:

cloudtrail:StopLogging

Access denied

The policy appears to work.

Several hours later, a security workflow attempts to update the organisation trail.

It also fails.

The backup service cannot complete one of its cross-account operations.

A deployment pipeline receives an access-denied error from an API call nobody realised belonged to its normal release path.

The SCP successfully denied something.

It did not necessarily enforce the intended boundary.

POLICY ACCEPTED BY AWS
          │
          ▼
EXPECTED ACTION DENIED
          │
          ▼
POLICY DECLARED SUCCESSFUL
          │
          ▼
UNRELATED WORKFLOW FAILS LATER

The missing step was behavioural testing.

An SCP is correct only when it denies the behaviour the organisation intends to prohibit while preserving every required path through the accounts it governs.

A passing SCP test is therefore not one command result.

It is a reproducible relationship between:

BEHAVIOURAL CONTRACT

REPRESENTATIVE FIXTURE

DEPLOYED CONTROL STATE

EXECUTION

EVIDENCE

That is the testing model I want to keep.

Define the behavioural contract first

A strong test begins with intended behaviour, not the JSON statement.

For example:

control:
  name: protect-cloudtrail
  owner: bfstore-security
  scope: workload-accounts

intent:
  protected_behaviour:
    - organisation audit logging remains enabled
    - workload administrators cannot alter the protected trail

permitted_behaviour:
  - security automation can inspect trail status
  - approved security administrators can maintain the trail
  - member accounts continue sending audit events

prohibited_behaviour:
  - workload administrator stops logging
  - workload administrator deletes the trail
  - workload administrator redirects log delivery

recovery:
  rollback_owner: bfstore-organisation-administrator
  rollback_action: detach candidate SCP from policy-test target

The SCP implementation is one attempt to satisfy this contract.

That distinction matters.

Weak test design:

The policy denies cloudtrail:StopLogging.

Test cloudtrail:StopLogging.

Stronger test design:

The organisation requires durable audit logging.

Test the ways relevant principals
could weaken, stop or redirect it.

Also test the workflows required
to operate and receive it.

The first proves one policy statement behaves as written.

The second tests the security outcome.

Represent each test as an evidence bundle

A useful test case records all five parts explicitly.

test:
  id: workload-admin-cannot-stop-trail
  control: protect-cloudtrail
  type: negative

fixture:
  account: bfstore-policy-test
  principal: PolicyTestAdministrator
  region: eu-west-2
  resource: organisation-trail

control_state:
  candidate_policy: protect-cloudtrail-v3
  target: bfstore-policy-test
  hierarchy_snapshot: scp-path-20260217-001

expectation:
  result: denied
  source: service-control-policy

evidence:
  preserve_error: true
  preserve_identity: true
  preserve_attachment_snapshot: true
  preserve_request_id: true

cleanup:
  required: false

The bundle answers:

What behaviour was expected?

Which principal and resource exercised it?

Which policy and attachment state applied?

What operation ran?

Which evidence proves the result?

A test report missing any of those answers is difficult to reproduce and easy to misinterpret.

Validate the policy document

Static validation remains the first gate.

An SCP must be valid JSON and use syntax supported by AWS Organizations.

IAM Access Analyzer can validate a policy using the SERVICE_CONTROL_POLICY policy type.

aws accessanalyzer validate-policy \
  --policy-type SERVICE_CONTROL_POLICY \
  --policy-document file://policies/protect-cloudtrail.json

CI should fail on validation errors.

Warnings should require review.

Validation can detect:

malformed JSON
unsupported policy elements
invalid actions
invalid condition keys
malformed ARNs
policy grammar problems

It proves:

Can AWS interpret this document?

It does not prove:

Does this policy protect
the intended behaviour?

Does it affect the intended principals?

Do required workflows still function?

Can the organisation recover?

Valid JSON proves very little about platform behaviour.

Build representative fixtures

A policy-test account should be safe enough to absorb failure and realistic enough to exercise actual control paths.

Experimentation
└── PolicyStaging
    └── bfstore-policy-test

The account should not contain production data or critical services.

It should contain representative:

roles
trust relationships
resources
service integrations
automation paths
logging
backup
emergency access

Representative identities

Testing from one administrator role is not enough.

Useful fixtures include:

PolicyTestAdministrator

PolicyTestDeveloper

PolicyTestReadOnlyInvestigator

PolicyTestDeploymentRole

PolicyTestSecurityRole

PolicyTestBackupRole

PolicyTestBreakGlassRole

A break-glass role is an emergency access path.

It is not automatically a path around SCPs.

Unless a policy deliberately exempts it, a break-glass role in a member account remains inside the applicable organisational ceiling.

Test separately:

SESSION ACQUISITION

Can the authorised incident process
obtain the role?


EMERGENCY ACTION

Which operations remain possible
after the role is assumed?

Where a policy exempts a role, test:

exact approved ARN
a near-matching role name
the same role name in another account
a changed role path
an unapproved session context
monitoring and expiry

An exception is part of the control.

It deserves its own tests.

Representative resources and contexts

Fixtures should cover:

exact protected resources
disposable resources
approved and unapproved Regions
required and absent tags
approved and unapproved network paths
same-account and cross-account calls
service-linked-role participation

The dangerous failure is often not the obvious opposite value.

It is the request context the policy author did not realise could exist.

Verify identity and consequence before every test

A surprising number of tests run against the wrong profile, role, account or Region.

Every run should begin with:

aws sts get-caller-identity

The harness should verify:

account ID
principal ARN
expected role
Region
approved test target

It should also classify the test action.

NON-MUTATING

describe
list
read


REVERSIBLE

create disposable fixture
change temporary configuration
clean up afterwards


DESTRUCTIVE OR IRREVERSIBLE

close account
leave organisation
delete evidence
destroy keys

Destructive tests should require:

approved disposable account

approved resource prefix

explicit destructive-test flag

documented disposal or cleanup plan

A lifecycle control protecting both:

organizations:LeaveOrganization

account:CloseAccount

may need a disposable organisation or account fixture rather than a routine policy-test account.

Running a destructive test in the wrong account can turn a policy exercise into the incident the guardrail was intended to prevent.

Capture the deployed control state

A source-controlled JSON file does not prove which policy AWS enforced.

The test must capture the deployed state.

That includes:

policy ID
policy document digest
direct attachment target
complete hierarchy path
ancestor attachments
account-level attachments
capture time
test run ID

ListPoliciesForTarget returns only policies directly attached to the selected target.

ListParents can be used to walk from an account through parent OUs to the organisation root.

DescribeEffectivePolicy does not return one aggregated SCP document because SCPs are authorization policies rather than supported management-policy types.

The harness therefore needs to reconstruct the path deliberately.

snapshot:
  id: scp-path-20260217-001
  account: bfstore-policy-test
  captured_at: 2026-02-17T10:30:00Z

  path:
    - root: r-abcd
    - ou: ou-experimentation
    - ou: ou-policy-staging
    - account: "123456789012"

  policies:
    - id: p-root-baseline
      attached_to: r-abcd
      digest: sha256:...
    - id: p-experimentation
      attached_to: ou-experimentation
      digest: sha256:...
    - id: p-protect-cloudtrail-v3
      attached_to: "123456789012"
      digest: sha256:...

Every result should reference one snapshot ID.

That answers:

Which exact organisation state
produced this result?

Wait for a stable deployed state

AWS Organizations is distributed.

A policy change may take effect while related API observations still take time to converge across endpoints.

The test flow should therefore be:

ATTACH OR UPDATE POLICY
          │
          ▼
POLL DEPLOYED STATE
          │
          ▼
CONFIRM EXPECTED TARGET
          │
          ▼
WAIT FOR STABLE OBSERVATION
          │
          ▼
BEGIN BEHAVIOURAL TEST

The harness should:

poll rather than use a blind fixed sleep
use a bounded timeout
confirm the expected policy digest
confirm the expected attachment
record the first stable observation
treat timeout as an infrastructure failure

A rapid baseline, attach, test and rollback sequence can otherwise measure propagation timing rather than policy behaviour.

Isolate the candidate SCP

A denied request does not prove the candidate SCP caused it.

Possible denial sources include:

missing IAM permission
permissions boundary
session policy
resource policy
RCP
VPC endpoint policy
another SCP
service-specific control

The cleanest pattern is:

BASELINE

action succeeds without candidate SCP

ENFORCEMENT

action fails with candidate SCP

ROLLBACK

action succeeds after controlled removal

This may use:

the same account before attachment
an equivalent account outside the staging OU
a disposable fixture with controlled attachment changes

The test identity should have only the permission required to exercise the candidate control.

Do not create one broadly privileged test role for every policy.

Use three denial outcomes

A production harness should distinguish:

DENIED_BY_CANDIDATE_SCP

expected control observed

DENIED_BY_OTHER_CONTROL

authorization failed,
candidate not proven

FAILED_FOR_NON_AUTH_REASON

fixture or operation invalid

A generic AccessDenied is not enough.

Where available, capture:

AWS error code
full message
request ID
policy type
policy ARN
caller identity
CloudTrail event
baseline result
candidate result

Enhanced denial messages can sometimes identify an SCP explicitly.

They are useful evidence, not universal coverage.

When the message is inconclusive, the controlled baseline and rollback sequence becomes essential.

Test at three levels

Atomic controls

Atomic tests verify one intended rule.

workload administrator cannot
stop protected logging

member-account administrator cannot
leave the organisation

unapproved Region request
is denied

Atomic tests make failures easier to attribute.

Complete journeys

A platform workflow usually depends on several APIs.

An EKS deployment may require:

role assumption
state access
artefact retrieval
Kubernetes API access
load-balancer integration
KMS operations
log delivery
telemetry publication

The visible deployment action may remain allowed while a supporting API is blocked.

Journey tests should cover:

application deployment completes

backup and restore complete

security findings reach
the delegated administrator

audit logs arrive
in the log archive

approved platform automation
provisions a service

The atomic test tells us which rule failed.

The journey test tells us whether the platform still works.

Recovery paths

Test recovery before broad rollout.

Required paths may include:

policy detachment
account movement
organisation administration
emergency role assumption
log access
backup execution
restore
incident investigation

Rollback must not depend on a principal inside the affected member account performing an action the candidate SCP now denies.

BAD

use a workload administrator
to remove the organisation guardrail


BETTER

use protected organisation administration
to detach the policy from the staging target

Guardrails are less reassuring when the only key to the gate has been locked on the other side.

Test conditions at their boundaries

Conditions need edge-case testing.

A Region policy permitting eu-west-2 should test:

Case	Expected
Approved regional action in `eu-west-2`	Allow
Same action in an unapproved Region	Deny
Required global service action	Allow
Exempted security workflow	Allow
Non-exempted workload request outside Region	Deny
Request Region differs from resource Region	Verify explicitly

A principal exception should test:

exact approved role
near-matching role
same role name elsewhere
changed role path
missing principal context
unapproved session context

A tag condition should test:

expected tag present
tag absent
wrong value
caller attempting to set the exempting tag
unsupported resource type
missing context key

MATCHING CASE

NON-MATCHING CASE

MISSING-KEY CASE

NEAR-MATCH CASE

ABUSE CASE

Boundary testing should explore the context space, not only the expected happy path and its obvious opposite.

Promote through evidence

A rollout should begin in the smallest useful governance scope.

POLICY DOCUMENT
      │
      ▼
STATIC VALIDATION
      │
      ▼
POLICY-STAGING ACCOUNT
      │
      ▼
REPRESENTATIVE NON-PRODUCTION
      │
      ▼
PRODUCTION CANARY
      │
      ▼
PRODUCTION OU
      │
      ▼
BROADER ORGANISATION SCOPE

Change one meaningful variable at a time.

Do not introduce a Region restriction, CloudTrail deny, role exception and service allowlist in one release unless the evidence can isolate each behaviour.

Promotion should not depend on:

Nobody complained.

Silence may mean the workflow never ran.

Use measurable observation.

observation:
  duration: 72h
  required_workflow_runs:
    deployment: 3
    backup: 1
    security_aggregation: 1
    identity_access: 3
  unexplained_denials_allowed: 0
  unresolved_incidents_allowed: 0

Promotion evidence should record:

policy version
attachment snapshot
principals tested
denied operations
successful workflows
recovery result
observation activity
remaining untested paths
accepted residual risk
approvers

The evidence should be useful to an engineer, not ceremonial paperwork.

Make the harness trustworthy

The harness becomes part of the control evidence.

It needs engineering discipline.

Structured outcomes

Return:

PASS

DENIED_BY_CANDIDATE

DENIED_BY_OTHER_CONTROL

FAILED_FIXTURE

FAILED_CLEANUP

SKIPPED

Unique run identity

Every run should have an ID included in:

fixture names
resource tags
logs
evidence files
cleanup records

Timeouts

Every AWS command and polling loop needs a bounded timeout.

A hung CLI process should not block rollback.

Cleanup quarantine

When cleanup fails:

mark the run failed
preserve evidence
tag the remaining resource
alert the owner
stop later tests that require a clean fixture

Concurrency protection

Only one pipeline should mutate the same policy, attachment or test account at a time.

Use a lock around:

policy updates
attachment changes
account movement
rollback

The organisational control plane should not be tested by two pipelines racing to define reality.

Safe shell behaviour

A shell implementation should:

use set -Eeuo pipefail
preserve stdout and stderr
verify caller identity
separate personas
refuse destructive tests outside approved fixtures
clean up idempotently
produce machine-readable reports

A green tick produced by an unreliable harness is merely a cheerful rumour.

The bfstore test repository

aws/
└── organizations/
    └── scp/
        ├── policies/
        │   ├── protect-account-lifecycle.json
        │   ├── protect-cloudtrail.json
        │   └── restrict-regions.json
        │
        ├── contracts/
        │   ├── protect-account-lifecycle.yaml
        │   ├── protect-cloudtrail.yaml
        │   └── restrict-regions.yaml
        │
        ├── tests/
        │   ├── static/
        │   ├── atomic/
        │   ├── journeys/
        │   └── recovery/
        │
        ├── fixtures/
        │   ├── iam/
        │   ├── s3/
        │   └── cloudtrail/
        │
        ├── attachments/
        │   ├── policy-test.yaml
        │   ├── nonproduction.yaml
        │   └── production.yaml
        │
        ├── snapshots/
        ├── reports/
        └── README.md

Each control has:

a behavioural contract
a policy implementation
representative fixtures
attachment definitions
atomic tests
journey tests
recovery tests
state snapshots
promotion evidence

The JSON is only one part of the product.

What bfstore should test first

Protect account lifecycle

Expected deny:

member-account administrator cannot call organizations:LeaveOrganization
member-account root cannot leave directly
member-account administrator cannot call account:CloseAccount
member-account root cannot close the account directly

Expected success:

approved central lifecycle workflow can remove an account
approved central lifecycle workflow can close a disposable account
ordinary account operations continue

Protect audit logging

Expected deny:

workload administrator cannot stop protected logging
workload administrator cannot delete or redirect the protected trail

Expected success:

log delivery continues
approved security administration works
investigation access remains available

Restrict Regions

Expected deny:

representative regional resource creation outside the approved set

Expected success:

approved Region operations
required global service operations
identity, DNS, security and recovery workflows

Protect security integrations

Expected deny:

workload administrator cannot disable organisation-managed security configuration

Expected success:

delegated administration works
finding aggregation continues
member accounts can use the protected service normally

Preserve recovery

Expected success:

backup creation
cross-account copy
restore
emergency investigation
policy detachment through protected organisation administration

These tests prove the platform can still be deployed, observed, secured and recovered after the guardrail is introduced.

Questions I now ask

Contract

Which required and prohibited
behaviours define the control?

Fixtures

Which principals, resources
and contexts exercise it?

State

Which policy content and hierarchy
snapshot are under test?

Baseline

Would the action succeed
without the candidate SCP?

Isolation

Did the candidate cause the denial,
or did another control?

Journeys

Which delivery, security
and recovery paths must survive?

Conditions

Which boundary, missing-key
and abuse cases exist?

Recovery

Can rollback occur through
an independent tested path?

Evidence

Can another engineer reproduce
the result from the report?

These questions turn a policy attachment into a controlled engineering release.

The mental model I am keeping

My earlier model was:

WRITE SCP
    │
    ▼
CHECK JSON
    │
    ▼
ATTACH POLICY
    │
    ▼
TRY ONE DENIED ACTION
    │
    ▼
DONE

The stronger model is:

                    ORGANISATIONAL REQUIREMENT
                                │
                                ▼
                       BEHAVIOURAL CONTRACT
                                │
                                ▼
                    REPRESENTATIVE FIXTURES
                                │
                                ▼
                    DEPLOYED STATE SNAPSHOT
                                │
              ┌─────────────────┼─────────────────┐
              │                 │                 │
              ▼                 ▼                 ▼
        ATOMIC TESTS       JOURNEY TESTS      RECOVERY TESTS
              │                 │                 │
              └─────────────────┼─────────────────┘
                                ▼
                         PRESERVED EVIDENCE
                                │
                                ▼
                       PROGRESSIVE PROMOTION
                                │
                                ▼
                    OBSERVATION AND REGRESSION

The behavioural contract answers:

Which outcomes must succeed
and which must fail?

The fixtures answer:

Which real principals, resources
and contexts exercise the control?

The state snapshot answers:

Which exact policy and attachment
environment produced the result?

Atomic tests answer:

Does the intended rule behave correctly?

Journey tests answer:

Can the platform still deploy,
observe and secure services?

Recovery tests answer:

Can the organisation repair
a harmful policy safely?

Evidence answers:

Can another engineer verify
and reproduce the result?

Promotion answers:

How much of the organisation
should experience the next consequence?

A Service Control Policy may contain only a few lines.

Those lines can change the maximum authority available to every governed principal beneath an OU.

The smaller the policy document looks, the easier it is to underestimate the size of the system it can affect.

Testing an SCP means proving more than denial.

It means proving that the intended boundary exists, that required workflows remain intact and that the organisation can recover when the policy itself is wrong.

The alternative is attaching the JSON, trying one command and waiting for production to complete the test suite.

Production is remarkably thorough.

Its reporting style is less considerate.

Define the behavioural contract first

Represent each test as an evidence bundle

Validate the policy document

Build representative fixtures

Representative identities

Representative resources and contexts

Verify identity and consequence before every test

Capture the deployed control state

Wait for a stable deployed state

Isolate the candidate SCP

Use three denial outcomes

Test at three levels

Atomic controls

Complete journeys

Recovery paths

Test conditions at their boundaries

Promote through evidence

Make the harness trustworthy

Structured outcomes

Unique run identity

Timeouts

Cleanup quarantine

Concurrency protection

Safe shell behaviour

The bfstore test repository

What bfstore should test first

Protect account lifecycle

Protect audit logging

Restrict Regions

Protect security integrations

Preserve recovery

Questions I now ask

Contract

Fixtures

State

Baseline

Isolation

Journeys

Conditions

Recovery

Evidence

The mental model I am keeping

References and further reading