What failure means once a network sits between two processes

When one process calls a function inside itself, failure feels comparatively contained.

caller
  │
  ▼
function
  │
  ├── returns value
  └── returns error

Local software is hardly a meadow of perfect certainty. A function can start asynchronous work, write to a database, publish a message, mutate shared state or crash before returning.

The advantage of the local call is not perfect knowledge. It is that ordinary return and error propagation share one immediate control flow.

Once a network sits between the caller and the work, the picture changes.

CLIENT PROCESS
      │
      ▼
client library
      │
      ▼
network
      │
      ▼
server library
      │
      ▼
SERVER PROCESS
      │
      ▼
business operation

Now the caller can observe an error without knowing exactly where the interaction stopped.

A timeout might mean that:

• the request never left the client;
• it left but never reached the server;
• the server received it but did not begin processing;
• the server began processing but did not finish;
• the server finished successfully;
• the response was lost;
• or the response arrived after the caller stopped waiting.

The caller sees one outcome:

No successful response arrived.

The remote system may be in several different states.

That seems to be the central change introduced by a network:

Failure becomes an observation made from one participant’s position, not necessarily a complete description of the whole interaction.

1. A network separates observation from outcome

Suppose a local function returns:

order, err := createOrder(request)

If err is nil, the caller receives the order. If it is not, the function can often describe what happened within the same execution context: validation failed, a database constraint rejected the change, a transaction rolled back or a required record was not found.

Across a network, the client does not call the server function directly. It sends bytes describing a request. The server later sends bytes describing a response.

CLIENT
  │
  │ request bytes
  ▼
NETWORK
  │
  ▼
SERVER
  │
  │ response bytes
  ▼
NETWORK
  │
  ▼
CLIENT

Anything that interrupts either journey can separate what happened remotely from what the caller learned.

“The request failed” hides useful information

A remote interaction can stop at several layers:

1. name resolution;
2. connection establishment;
3. secure-channel negotiation;
4. protocol parsing;
5. application acceptance;
6. business processing;
7. persistence;
8. response transmission;
9. client receipt.

Each layer can succeed while a later one fails.

For example, all of these may be true:

• DNS succeeded;
• the TCP connection succeeded;
• TLS succeeded;
• the HTTP request was valid;
• the application accepted the request;
• the database commit succeeded;
• the response connection then failed.

The client may still report:

connection reset

So I need to distinguish at least three broad outcomes:

These outcomes are related. They are not equivalent.

A successful connection proves very little

A TCP connection tells the client that it reached a listening endpoint. It does not prove that:

• the correct application is healthy;
• the application can reach its database;
• the request is valid;
• the server has enough capacity;
• or the business action will succeed.

Likewise, successfully writing request bytes to a local socket does not prove that the remote application read, interpreted, accepted or durably acted on them.

TCP provides a reliable, ordered byte stream between endpoints while the connection remains viable under its protocol rules. It understands sequence numbers, acknowledgements, retransmission, flow control and connection state.

It does not understand:

• CreateOrder;
• ReserveStock;
• ChargePayment;
• SendEmail.

TCP gives the receiving application an ordered byte stream or a connection failure. It does not give the sender proof that the peer application interpreted, accepted or committed the requested business effect.

BUSINESS MEANING
    CreateOrder
         │
         ▼
APPLICATION PROTOCOL
    HTTP or gRPC
         │
         ▼
TRANSPORT
    TCP byte stream
         │
         ▼
NETWORK
    packets and routes

Each layer offers its own guarantees. Higher layers still need their own acknowledgement semantics.

The classic ambiguous outcome

Suppose a client creates an order.

CLIENT
  │
  │ CreateOrder
  ▼
SERVER
  │
  ├── validate request
  ├── begin transaction
  ├── insert order
  ├── commit transaction
  └── send response

Now imagine the connection disappears after the database commit but before the response reaches the client.

CLIENT                         SERVER

send CreateOrder
       ───────────────────────►

                               create order 5001
                               commit order 5001

       ◄──── response begins

connection fails

client observes:
    no response

server state:
    order 5001 exists

The client knows:

I did not receive confirmation.

It does not know:

The order was not created.

Those statements are different.

A useful outcome vocabulary is:

That final state is uncomfortable, but it is real.

2. What the client can and cannot know

No response is not a negative response

Compare two stock-reservation interactions.

In the first, inventory explicitly rejects the request:

CLIENT
  │
  │ ReserveStock
  ▼
INVENTORY
  │
  │ insufficient stock
  ▼
CLIENT

The client knows that inventory evaluated the request and returned a business rejection.

In the second, no response arrives:

CLIENT
  │
  │ ReserveStock
  ▼
?
  │
  │ no response before deadline
  ▼
CLIENT

The client does not know whether inventory:

• never saw the request;
• rejected it but lost the reply;
• reserved the stock;
• or is still processing it.

Treating the timeout as reservation rejected can cause the caller to make the wrong business decision.

Under a well-designed contract, an explicit validation or business rejection should mean that the requested change was not applied. Silence does not provide that guarantee.

Incomplete responses are failures too

A response can begin without completing.

Suppose an HTTP response declares:

HTTP/1.1 200 OK
Content-Length: 12800

The client receives only 4,000 bytes before the connection closes. The response is incomplete.

The initial 200 OK does not make the truncated body a successful representation.

A streamed RPC may likewise deliver several messages and then fail:

message 1 received
message 2 received
message 3 received
stream fails

The application must decide whether partial results are discarded, usable, checkpointed, retried from the beginning or resumed from a known position.

“Some data arrived” and “the operation completed” are separate statements.

The point of failure changes what is knowable

Even a complete response must be interpreted according to its application contract.

202 Accepted means the request was accepted for processing. It does not mean that later processing succeeded.

200 OK does not guarantee that every downstream side effect completed unless the API explicitly defines success that way.

3. Deadlines, cancellation and failure detection

A timeout does not mean rollback

Suppose a caller gives an RPC a two-second deadline.

request begins
      │
      ▼
two-second deadline
      │
      ▼
client stops waiting

The client receives deadline exceeded.

That is a local conclusion:

The call did not complete from the client’s perspective within the permitted time.

It does not mean that:

• the server never received the request;
• the server stopped all work;
• the database transaction rolled back;
• or no business effect occurred.

The server may observe cancellation and stop. It may already have completed an irreversible step. Cancellation may not reach it promptly.

A deadline bounds waiting. It does not provide distributed rollback.

The client and server can both be correct

Suppose the server completes an RPC just after the client’s deadline.

Those conclusions appear contradictory, but they describe different local events.

Distributed systems contain several observers. A status reported by one observer does not automatically summarise every participant’s state.

Cancellation requires cooperation

When the client stops waiting, it may send a cancellation signal or close the connection.

CLIENT DEADLINE
      │
      ▼
cancellation propagated
      │
      ▼
SERVER
  │
  ├── stop unnecessary work
  ├── release resources
  └── avoid starting further downstream work

But cancellation does not reverse effects already committed, published or sent.

It means:

The caller no longer requires this interaction to continue.

It does not mean:

Pretend the interaction never began.

Where reversal is required, the workflow needs explicit compensation.

Failure detection takes time

Some failures are immediate, such as connection refusal. Others are silent.

If packets disappear without a response, the client cannot immediately distinguish between:

• a slow network;
• a slow server;
• a lost request;
• a lost response;
• a failed destination.

It waits until a deadline or protocol mechanism concludes that progress has not occurred.

Failure detection is therefore often suspected after a time bound, not observed at the exact moment the remote process failed.

Short and long timeouts fail differently

A timeout that is too short can create false failures.

server begins work
      │
      ▼
500 ms passes
      │
      ▼
client reports timeout
      │
      ▼
server completes at 700 ms

That can trigger unnecessary retries and duplicate work.

A timeout that is too long delays recovery and consumes connections, threads or goroutines, memory and user patience.

The value should reflect:

• the normal latency distribution;
• the user or caller deadline;
• downstream work;
• the retry budget;
• the resource cost;
• the value of a late result.

A timeout is part of the system design, not seasoning sprinkled over the client constructor.

Downstream deadlines need a margin

If service A has 500 ms remaining and gives service B the full 500 ms, A may have no time left to process B’s response and answer its own caller.

A service should normally propagate a smaller remaining deadline downstream, preserving time for local processing and response transmission.

UPSTREAM DEADLINE
│
├── local processing
├── downstream attempt
└── response margin

Every layer should consume part of one finite budget, not restart the clock.

4. Retries depend on operation semantics

After a timeout, a client may retry:

attempt 1
    │
    └── no response

attempt 2
    │
    └── response received

Attempt two is a new request. It does not continue attempt one from the uncertain point.

The server may observe both attempts. If the first already created an order, the second may create another.

The retry converted uncertainty into duplicate work.

Retryability is a property of the operation, not merely the error.

Safe and idempotent are different

In HTTP terminology:

• a safe method is intended for retrieval rather than a requested state change;
• an idempotent method has the same intended effect whether an identical request is applied once or several times.

Appropriately designed GET and HEAD requests are safe. Appropriately designed PUT and DELETE requests are idempotent.

For example:

PUT /customers/72/preferences HTTP/1.1
Content-Type: application/json

{
  "email_updates": false
}

Applying that request twice should still produce:

email_updates = false

By contrast:

POST /orders HTTP/1.1

may create two different orders when repeated.

An application can make a particular POST operation safe to retry by assigning it a stable logical identity. That does not make POST generically idempotent to arbitrary intermediaries. HTTP method semantics tell generic infrastructure what it may assume; the application contract tells a particular client how to retry a particular operation.

Reads are easier to retry, but not free

A query such as GetProduct is often safe to repeat, but repeated reads may still affect:

• audit records;
• metrics;
• billing;
• rate-limit counters;
• cache state;
• access timestamps.

A read retry may also observe newer data.

attempt 1 would have returned:
    price £229

attempt 2 returns:
    price £249

For ordinary current-state queries, that may be acceptable. Snapshot-sensitive work may need a version, revision, timestamp or snapshot identifier.

Automatic retries need boundaries

A retry policy should decide:

• which errors are transient;
• which operations are safe to repeat;
• how many attempts are allowed;
• how long the complete operation may take;
• how long to wait between attempts;
• how overload will be avoided.

A useful policy is bounded, selective, delayed and observable.

A dangerous policy is:

on any error:
    retry immediately forever

At scale, retries can prevent the recovering service from recovering.

Backoff and jitter control load

A client can increase the delay between attempts:

attempt 1
   │
   └── wait 100 ms

attempt 2
   │
   └── wait 200 ms

attempt 3
   │
   └── wait 400 ms

Jitter spreads clients across a wider interval instead of producing rhythmic retry avalanches.

Backoff controls load. It does not make a non-idempotent operation safe.

The retry budget belongs inside the deadline

Suppose the caller’s complete deadline is two seconds. The retry policy should not accidentally perform three two-second attempts.

TOTAL OPERATION DEADLINE
│
├── attempt 1
├── backoff
├── attempt 2
└── remaining time

Connection establishment, request transmission, server processing, response reception, backoff and retries all consume one finite budget.

Not retrying trades duplicate risk for unresolved work

A client may choose to send once and never retry. That limits duplicate attempts from that client, but it does not guarantee at-most-once behaviour across gateways, libraries or other infrastructure.

It also allows uncertain work to remain unresolved.

request sent
      │
      ▼
response lost
      │
      ▼
no retry

Perhaps the operation succeeded. Perhaps it did not.

Not retrying trades duplicate risk for omission or unresolved outcome risk. It can be appropriate when duplication is more harmful and another reconciliation mechanism exists.

At-least-once attempts make the opposite trade. They increase the chance that work reaches a receiver, but require duplicate-safe processing through idempotency keys, unique constraints, state-machine checks or conditional writes.

5. Idempotency controls duplicate effects

The business usually cares about:

• exactly one order;
• exactly one payment capture;
• exactly one stock reservation.

Those are application effects, not transport properties.

A request may be transmitted, received or processed more than once while the business effect occurs once if the receiver recognises duplicates.

DELIVERY COUNT

How many times did the request arrive?


PROCESSING COUNT

How many attempts did the receiver make?


EFFECT COUNT

How many business changes became authoritative?

Reliable systems focus on controlling the effect count.

Idempotency belongs to business identity

Suppose a client assigns one stable operation identifier:

checkout-7f82

It sends:

POST /orders HTTP/1.1
Idempotency-Key: checkout-7f82
Content-Type: application/json

{
  "customer_id": "72",
  "basket_id": "basket-913"
}

The server records:

idempotency key:
    checkout-7f82

request fingerprint:
    customer 72
    basket basket-913

result:
    order 5001

A retry with the same key and equivalent request can return the original result.

attempt 1
    checkout-7f82
        │
        └── creates order 5001


attempt 2
    checkout-7f82
        │
        └── returns order 5001

The retry is still another network request. The server recognises that it represents the same logical operation.

The idempotency record needs an atomic relationship

The server must avoid:

create order 5001
      │
      ▼
crash
      │
      ▼
idempotency key not recorded

Where possible, the business result and idempotency record should commit in the same local transaction.

BEGIN
  │
  ├── claim idempotency key
  ├── validate request fingerprint
  ├── create order
  ├── store result for key
  │
COMMIT

The wider response can still be lost, but a retry can recover the existing result.

Concurrent attempts need one winner

A transaction boundary alone is not enough if two requests with the same key arrive simultaneously.

REQUEST A                     REQUEST B

check key: absent             check key: absent
create order                  create order
store key                     store key

Both may attempt the business effect unless the database enforces one winner.

A robust design may use:

• a unique constraint scoped by client or tenant and idempotency key;
• an in_progress claim inserted before doing the work;
• row locking;
• serializable execution;
• or another atomic compare-and-set mechanism.

A duplicate arriving while the first request is still running also needs a policy. It may wait, receive in progress, or return a retryable response.

Reusing a key for different work should be rejected

A useful policy is:

An idempotency key identifies one logical operation. It should not become a reusable coupon for whichever request arrives next.

6. Unknown outcomes need somewhere to live

Blind retry is not the only response to uncertainty.

Operation resources make progress queryable

After an uncertain CreateOrder, the client can query by logical operation identifier:

GET /order-operations/checkout-7f82 HTTP/1.1

Possible responses include:

• not found;
• pending;
• completed with order 5001;
• failed conclusively;
• outcome still unknown.

CreateOrder
    │
    └── response lost

GetOperation(checkout-7f82)
    │
    └── completed: order 5001

An operation resource separates starting the work from observing its state. It is especially useful for long-running or asynchronous operations.

Reconciliation resolves uncertainty later

Suppose the application calls a payment provider and loses the response.

It may reconcile later using:

• a merchant reference;
• a provider transaction ID;
• an idempotency key;
• a settlement report;
• a webhook;
• a status query.

payment attempt
      │
      ▼
outcome uncertain
      │
      ▼
record pending state
      │
      ▼
query or receive provider result
      │
      ▼
resolve to authorised, declined or failed

The application should not collapse unknown into declined merely because the request timed out.

State machines make uncertainty explicit

A payment workflow may need more than one Boolean.

Possible states include:

• not_started;
• pending;
• authorised;
• declined;
• capture_pending;
• captured;
• failed;
• outcome_unknown.

PAYMENT ATTEMPT
      │
      ▼
pending
  │       │
  │       ├── explicit decline ──► declined
  │       │
  │       ├── success ───────────► authorised
  │       │
  │       └── connection lost ───► outcome_unknown
  │
  └── reconciliation
          │
          ├── found success ─────► authorised
          └── found failure ─────► failed

This is less tidy than payment_successful = true|false. It is more honest.

7. Precise errors and observability

Error categories should preserve useful knowledge

A client library might expose:

• invalid request;
• unauthenticated;
• permission denied;
• not found;
• conflict;
• resource exhausted;
• unavailable;
• deadline exceeded;
• cancelled;
• internal error;
• unknown.

These categories support different decisions.

Errors should describe what was observed and help the caller decide what remains safe.

Error messages should not overclaim

Compare:

Order creation failed.

with:

We could not confirm whether the order was created. Check the order status before trying again.

The second is less satisfyingly final. It is more accurate when the outcome is unknown.

Likewise, Payment declined should be reserved for an explicit decline. A provider timeout should not be translated into a business rejection.

Error language is part of the distributed contract.

Request IDs and idempotency keys have different jobs

A request ID identifies one transport or processing attempt.

attempt 1:
    request ID req-91a3

attempt 2:
    request ID req-b102

An idempotency key identifies the logical operation shared by those attempts.

checkout-7f82
│
├── req-91a3
└── req-b102

The client can log:

request req-91a3 timed out

while the server logs:

request req-91a3 created order 5001

and the database records:

idempotency key checkout-7f82
order 5001

Now an operator can connect the client observation, server processing and business result.

Without correlation, the investigation becomes a timestamp-themed guessing competition.

Distributed tracing shows a path, not omniscience

A trace may connect:

CreateOrder
│
├── gateway request
├── order validation
├── ReserveStock RPC
├── database transaction
└── response write

That helps locate latency and errors, but tracing still depends on telemetry being generated, propagated, exported and retained.

A crashed process may lose buffered telemetry. A partition may interrupt export. An external provider may not participate.

Observability improves knowledge. It does not remove uncertainty.

Health checks answer a different question

A health endpoint can return healthy one millisecond before the process crashes.

A process may also be alive while its database is unavailable, its worker pool is exhausted or one tenant is misconfigured.

A health result is evidence about one test at one moment. It does not prove that the next business operation will succeed, and it does not reveal whether an earlier request committed.

8. Failure policy belongs to the operation

There is no universal network-error handler that can infer every business consequence.

A practical outcome table can help preserve what the client actually knows.

The table does not eliminate uncertainty. It stops one observation from claiming more than it proves.

9. The mental model I am keeping

My old model was:

remote call
    │
    ├── success
    └── failure

The new model is:

                         REMOTE INTERACTION
                                  │
              ┌───────────────────┼───────────────────┐
              │                   │                   │
              ▼                   ▼                   ▼
       CLIENT OBSERVATION   SERVER OBSERVATION   BUSINESS STATE
              │                   │                   │
              ├── response        ├── received        ├── unchanged
              ├── timeout         ├── processing      ├── pending
              ├── reset           ├── completed       ├── committed
              └── error           └── failed          └── compensated

Those views can differ.

A client timeout can coexist with a committed server transaction.

A server success can coexist with a client-side failure.

A successful transport exchange can carry an application rejection.

An accepted asynchronous request can later produce a failed business outcome.

So a network error should make me ask:

• At which layer did the failure appear?
• What does this participant know?
• What remains uncertain?
• Could the operation already have changed state?
• Is repeating it safe?
• How will duplicates be recognised?
• Can the result be queried or reconciled?
• What should the user be told?

The useful distinction is no longer simply worked or did not work.

It is:

• known success;
• known rejection;
• known non-execution;
• still in progress;
• outcome unknown.

Once a network sits between two processes, failure is not always a clean ending.

Sometimes it is a gap between what one process did and what the other process managed to learn.

Reliable distributed systems do not fill that gap with optimism. They give it:

• deadlines;
• precise error semantics;
• stable operation identities;
• idempotent handling;
• bounded retries;
• observable state;
• reconciliation.

The network does not merely add another place where an operation can fail.

It adds another place where certainty can be lost.

References and further reading

The original article cited the standards current when it was written in November 2021. The current specifications below supersede the earlier HTTP and TCP RFCs while preserving the same core concepts discussed here.

HTTP semantics, messaging and retries

RFC 9110: HTTP Semantics
Defines HTTP method semantics, including safe and idempotent methods, response meanings and the distinction between successful handling, rejection and acceptance for later processing. It supersedes RFC 7231.

RFC 9112: HTTP/1.1
Defines HTTP/1.1 message framing, incomplete messages, connection management and behaviour around premature connection closure. It supersedes the relevant parts of RFC 7230.

TCP connection behaviour

RFC 9293: Transmission Control Protocol
The current TCP specification, describing reliable ordered byte-stream delivery, connection state, acknowledgements, retransmission, closure and reset behaviour. It obsoletes RFC 793.

POSIX.1-2017: connect()
Documents establishing a socket connection and errors including refusal and timeout.

POSIX.1-2017: send()
Documents sending data through a socket and the transport-level errors visible to an application.

POSIX.1-2017: recv()
Documents receiving socket data, short reads, connection termination, reset and timeout behaviour.

RPC deadlines and local outcomes

gRPC: Core concepts, architecture and lifecycle
Explains RPC lifecycles, cancellation, deadlines and how clients and servers can reach different local conclusions about the same RPC.

gRPC and Deadlines
Describes why callers should define deadlines and illustrates how a server can complete successfully after the client has already concluded that its deadline was exceeded.

Retry behaviour

gRPC: Retry Design
Defines gRPC’s client-retry design, including retryable status codes, attempt limits, backoff, throttling and the distinction between transparent and configured retries.

Error vocabulary

gRPC Status Codes
Defines gRPC’s standard status codes, including INVALID_ARGUMENT, DEADLINE_EXCEEDED, ALREADY_EXISTS, RESOURCE_EXHAUSTED, UNAVAILABLE and UNKNOWN.