Cloud Sandbox
All writing

HTTP without the hand-waving

Following an HTTP request from a URL to a response to understand what clients and servers actually exchange beneath the browser.

about 22 minutes min read

For a long time my mental model of the web was something like:

type URL
   │
   ▼
browser
   │
   ▼
internet
   │
   ▼
website appears

This model has the advantage of being short.

Unfortunately, almost everything interesting is hiding inside the word internet.

After looking at how data moves through a network, and then at what TCP and UDP actually provide, HTTP seems like the next layer worth pulling apart.

Because HTTP is not the network.

It is not TCP.

It is not HTML.

And it is definitely not the browser.

HTTP is an application-level protocol that gives clients and servers a common way to talk about resources using requests and responses.

So what actually happens when I type:

http://example.com/notes

into a browser?

HTTP starts above the transport layer

The network stack I have been building so far looks roughly like:

application
     │
     ▼
transport
     │
     ▼
IP
     │
     ▼
link

HTTP belongs at the top:

HTTP
 │
 ▼
TCP
 │
 ▼
IP
 │
 ▼
Ethernet / other link

For ordinary HTTP/1.1 traffic, TCP commonly provides the transport underneath.

That means HTTP does not itself need to solve problems such as:

packet loss
reordering
retransmission
flow control

TCP already provides the ordered byte stream in which the HTTP messages can travel.

HTTP can concentrate on a different problem:

What does the client want, and how should the server describe the result?

That separation is starting to become a recurring theme in networking.

Each protocol layer solves a narrower problem and relies on another layer below it.

First, the URL is not the request

Suppose I enter:

http://example.com/products?id=42

That string contains several different pieces of information.

Very roughly:

http://example.com/products?id=42
│      │           │         │
│      │           │         └── query
│      │           │
│      │           └──────────── path
│      │
│      └──────────────────────── host
│
└─────────────────────────────── scheme

The URL helps the client work out:

  • which protocol scheme is involved;
  • which host it needs to contact;
  • which resource it wants;
  • additional information associated with that request.

But the browser does not simply transmit this entire string across the network and hope somebody understands it.

It has work to do first.

The host name needs an address

The client needs to communicate with:

example.com

Networks ultimately route using addresses rather than the friendly domain name I typed.

So the system may need DNS to resolve the hostname to an IP address.

Conceptually:

example.com
     │
     │ DNS
     ▼
IP address

DNS and HTTP are therefore different protocols solving different problems.

DNS helps answer:

Where is example.com?

HTTP will later help answer:

What do I want from the HTTP service there?

This distinction seems obvious after writing it down.

It was considerably fuzzier in my head before.

Then the client needs a connection

Once the destination address is known, the client can establish transport communication with the server.

For HTTP over TCP, the broad sequence is:

client
  │
  │ resolve hostname
  ▼
server IP
  │
  │ establish TCP connection
  ▼
TCP byte stream
  │
  │ send HTTP request
  ▼
server

If I am using plain HTTP, the HTTP messages can travel over that connection without encryption.

HTTPS adds another layer, which I will come back to later.

For now:

HTTP
  │
  ▼
TCP

is enough.

An HTTP request is structured text

One of the nicest things about HTTP/1.1 is that its wire format is readable enough to inspect.

A very small request might resemble:

GET /products?id=42 HTTP/1.1
Host: example.com
Accept: text/html

That blank line at the bottom matters.

An HTTP/1.1 message consists of:

start line
headers
blank line
optional message body

For a request:

┌────────────────────────────────────┐
│ GET /products?id=42 HTTP/1.1       │  request line
├────────────────────────────────────┤
│ Host: example.com                  │
│ Accept: text/html                  │  headers
├────────────────────────────────────┤
│                                    │  blank line
├────────────────────────────────────┤
│ optional body                      │
└────────────────────────────────────┘

So beneath all the browser buttons, address bars and progress indicators is a protocol with a surprisingly understandable structure.

The request line says what I want to do

The first line:

GET /products?id=42 HTTP/1.1

contains three important things:

GET
│
└── method

/products?id=42
│
└── request target

HTTP/1.1
│
└── protocol version

The method describes the intention of the request.

GET means something different from POST.

POST means something different from DELETE.

HTTP therefore separates:

which resource?

from:

what do you want to do with it?

Conceptually:

METHOD + RESOURCE

GET     /products/42
POST    /orders
DELETE  /basket/items/7

The method is not merely decorative text.

It has defined semantics.

Methods have meaning

Some familiar HTTP methods include:

GET

Request a representation of a resource.

GET /products/42 HTTP/1.1

Similar to GET, but the server does not send the response body that a corresponding GET would have returned.

That can be useful when a client wants response metadata without transferring the representation itself.

POST

Submit data to a resource for processing according to that resource’s semantics.

POST /orders HTTP/1.1

PUT

Request that the supplied representation create or replace the state of the target resource.

DELETE

Request removal of the association between the target resource and its current functionality.

There are others, but even these few expose something important.

HTTP is not:

send URL
receive webpage

It is a resource interaction protocol.

Web pages happen to be one very visible use of it.

Safe and idempotent are different ideas

The HTTP specification also gives methods properties.

One distinction I want to remember is between:

safe

and:

idempotent

A safe method is defined as essentially read-only from the client’s requested semantics.

GET and HEAD are examples.

An idempotent method is one where making the same intended request multiple times should have the same intended effect as making it once.

For example, PUT and DELETE are defined as idempotent even though they are not safe.

So:

safe       → am I asking to change server state?
idempotent → does repeating the same request change the intended effect?

This feels like a subtle distinction now.

I suspect retries will make it much more important later.

Headers carry metadata and control information

After the request line come header fields:

Host: example.com
Accept: text/html
User-Agent: SomeClient/1.0

Headers allow additional information to accompany the request.

For example:

Host
   → which host is being addressed?

Accept
   → which response media types can I handle?

User-Agent
   → information about the client

Other headers can participate in:

  • authentication;
  • caching;
  • conditional requests;
  • content negotiation;
  • connection handling;
  • cookies;
  • representation metadata.

This is another useful separation:

request method
      │
      └── intention

request target
      │
      └── resource

headers
      │
      └── metadata / controls

body
      │
      └── optional content

Why does HTTP/1.1 need Host?

This header looked redundant to me at first:

Host: example.com

Surely I already connected to the server?

But one IP address can serve HTTP requests for more than one hostname.

For example, the same server infrastructure might handle:

shop.example.com

blog.example.com

docs.example.com

The destination IP alone is therefore not necessarily enough for the HTTP server to determine which logical host the request targets.

The HTTP/1.1 Host header carries that information.

This is one reason multiple websites can share infrastructure without each requiring a unique IP address.

Another piece of the network puzzle clicks into place.

Requests can have bodies

Not every HTTP request has a message body.

A GET often does not.

But something like:

POST /orders HTTP/1.1
Host: example.com
Content-Type: application/json
Content-Length: 35

{"product_id":42,"quantity":1}

contains one.

The body is application data.

The headers describe things about it.

For example:

Content-Type: application/json

tells the recipient how the representation should be interpreted.

So HTTP gives applications a standard envelope around arbitrary content.

That content might be:

HTML
JSON
an image
plain text
binary data
something else entirely

HTTP does not require everything to be HTML.

Despite the name Hypertext Transfer Protocol, its use has become much broader than transferring hypertext documents.

Then the server responds

Once the server has interpreted the request and performed whatever processing is appropriate, it sends an HTTP response.

A small HTTP/1.1 response might look like:

HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 5

Hello

Again, the structure is recognisable:

status line
headers
blank line
optional body

Or visually:

┌────────────────────────────────┐
│ HTTP/1.1 200 OK                │
├────────────────────────────────┤
│ Content-Type: text/plain       │
│ Content-Length: 5              │
├────────────────────────────────┤
│                                │
├────────────────────────────────┤
│ Hello                          │
└────────────────────────────────┘

So the conversation is fundamentally:

CLIENT

request
   │
   ▼
SERVER

response
   │
   ▼
CLIENT

That request/response pattern is HTTP’s central rhythm.

The status code is machine-readable meaning

The response begins:

HTTP/1.1 200 OK

The important machine-readable part is:

200

HTTP status codes are grouped into classes:

1xx  informational

2xx  successful

3xx  redirection

4xx  client error

5xx  server error

Some familiar examples:

200 OK
201 Created
204 No Content

301 Moved Permanently
302 Found

400 Bad Request
401 Unauthorized
403 Forbidden
404 Not Found

500 Internal Server Error
502 Bad Gateway
503 Service Unavailable
504 Gateway Timeout

That gives HTTP clients a protocol-level way of interpreting the broad outcome of a request.

This feels related to the exit-status idea I explored in Bash.

A human may read:

Not Found

but software primarily needs the defined status:

404

Another machine-readable contract.

200 does not mean everything everywhere succeeded

It is tempting to treat:

200

as HTTP’s version of:

everything is fine

But the status describes the result according to the semantics of this HTTP request.

An application could still return a technically successful HTTP response containing an application-level failure:

HTTP/1.1 200 OK
Content-Type: application/json

{"success":false}

Whether that is good API design is another conversation.

The important distinction is:

HTTP status
       │
       └── HTTP-level outcome

application content
       │
       └── application-specific meaning

Layers again.

Apparently they are everywhere.

A response body is a representation

When I request:

GET /products/42 HTTP/1.1

the server does not necessarily hand me some literal internal object called:

product 42

Instead, HTTP works with representations of resources.

The same resource could potentially be represented as:

HTML
JSON
XML
plain text
an image

depending on the application and negotiation involved.

So:

resource
    │
    ├── HTML representation
    ├── JSON representation
    └── something else

This separation between a resource and a representation of that resource seems small, but it makes the HTTP model much more general.

Content negotiation

Suppose a client sends:

Accept: application/json

It is expressing a preference about the representation it can accept.

Another client might ask for:

Accept: text/html

A server can use request information such as Accept when selecting an appropriate representation.

That means a resource is not necessarily tied permanently to one wire format.

The request and response can negotiate characteristics of the representation.

HTTP has quietly become much more than:

send me that file.

HTTP is described as stateless

This sentence appears constantly:

HTTP is stateless.

I used to interpret that as:

Websites cannot remember anything.

Clearly that cannot be right.

I can log into a site, add something to a basket and return to another page without introducing myself again every time.

The more precise idea is that HTTP itself is designed so that requests can be understood according to the information associated with those requests rather than depending on some implicit conversation state built into the basic protocol.

Applications can layer state management on top.

Cookies are one important mechanism for doing that.

A server might respond with:

Set-Cookie: session=abc123

The client can later send:

Cookie: session=abc123

Now the application can associate separate HTTP requests with some ongoing state.

Conceptually:

HTTP
    mostly stateless request/response protocol

         +

cookies / tokens / application storage

         ↓

stateful application experience

So:

HTTP is stateless

does not mean:

web applications cannot maintain state

It means the state is not magically inherent in HTTP’s basic request/response interaction.

One TCP connection can carry more than one request

Another old mental model of mine was:

HTTP request
   │
   ├── create TCP connection
   ├── send request
   ├── receive response
   └── destroy TCP connection

every single time.

HTTP/1.1 supports persistent connections.

A connection can remain open and carry multiple requests and responses rather than paying the cost of establishing a fresh TCP connection for every object.

Conceptually:

TCP connection
│
├── request 1
│   response 1
│
├── request 2
│   response 2
│
├── request 3
│   response 3
│
└── ...

This matters because loading even a seemingly simple web page may involve fetching:

HTML
CSS
JavaScript
images
fonts
API data

A browser may therefore make many HTTP requests while rendering what I casually think of as:

one webpage.

A webpage is usually many HTTP conversations

Suppose I visit:

https://example.com/

The initial response might contain HTML:

<link rel="stylesheet" href="/styles.css">
<script src="/app.js"></script>
<img src="/logo.png">

The browser then discovers that it needs more resources.

Conceptually:

GET /
   │
   ▼
HTML
   │
   ├── GET /styles.css
   ├── GET /app.js
   └── GET /logo.png

So the browser loading:

one page

may generate:

many HTTP requests

The webpage is an application-level composition of those resources.

HTTP simply carries the individual interactions.

This makes browser developer tools suddenly seem much more interesting.

curl removes some of the browser magic

A browser does so much work that it can obscure the protocol underneath.

curl gives me a simpler way to look at HTTP.

For example:

curl -i http://example.com/

The -i option includes response headers in the output.

I can see something conceptually like:

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: ...

<!doctype html>
...

For more detail:

curl -v http://example.com/

shows information about the connection and request/response exchange.

That makes HTTP feel much less like browser magic and much more like what it actually is:

a protocol spoken by software

The browser is simply a very sophisticated HTTP client, among many other things.

HTTP can have intermediaries

There is another wrinkle.

The path is not always:

client ─────────────► origin server

HTTP explicitly allows intermediaries.

The actual path might look more like:

client
   │
   ▼
proxy
   │
   ▼
gateway
   │
   ▼
origin server

Or perhaps:

client
   │
   ▼
cache
   │
   ├── already has usable response
   │        │
   │        └── return it
   │
   └── otherwise contact server

This means the server directly answering my request may not always be the origin system that ultimately owns the resource.

Proxies, gateways and caches can participate in HTTP communication.

That becomes particularly interesting when I start thinking about:

reverse proxies
load balancers
CDNs
API gateways

All apparently waiting further down this road.

Caching is part of HTTP

Caching is not merely a browser trick bolted onto the side.

HTTP defines caching behaviour and headers that let clients, servers and intermediaries reason about whether stored responses may be reused.

Headers such as:

Cache-Control: max-age=3600

can describe caching policy.

Validators such as:

ETag: "abc123"

can help a client ask whether its stored representation is still valid.

That can turn:

send me the entire resource again

into something closer to:

has this changed?

Caching therefore affects:

latency
bandwidth
server load
freshness

which feels like a surprisingly large amount of behaviour hiding inside HTTP metadata.

Another future rabbit hole successfully identified.

HTTPS is HTTP plus transport security

Then there is:

https://

It is tempting to think HTTPS is a completely different application protocol.

Conceptually, though, HTTPS is HTTP communicated through TLS.

The stack becomes:

HTTP
 │
 ▼
TLS
 │
 ▼
TCP
 │
 ▼
IP

TLS provides security properties for the connection, including confidentiality and integrity, and allows the client to authenticate the server through the certificate-based mechanisms used on the web.

The HTTP semantics above it remain recognisably HTTP:

GET
POST
headers
status codes
representations

The important distinction is:

HTTP
    → application semantics

TLS
    → secure communication channel

Again: different layers, different jobs.

I am sensing a pattern.

HTTP/2 changes the wire format, not the meaning of GET

There is one more modern complication worth noting.

HTTP/2 already exists.

Unlike HTTP/1.1’s human-readable textual message framing, HTTP/2 uses a binary framing layer and can multiplex multiple streams over one connection.

So this:

GET / HTTP/1.1
Host: example.com

is useful for understanding HTTP/1.1, but it should not make me assume every version of HTTP looks like human-readable text on the wire.

HTTP/2 changes how messages are framed and transported.

But important HTTP concepts remain:

requests
responses
methods
status codes
headers
resources
representations

This gives me another useful distinction:

HTTP semantics
       │
       └── what the request means

HTTP wire format
       │
       └── how that meaning is encoded

Those aren’t necessarily the same thing.

Following one request end to end

I can now build a much more useful picture of:

http://example.com/products/42

than I had before.

USER
 │
 │ enters URL
 ▼
CLIENT / BROWSER
 │
 ├── interpret URL
 │
 ├── identify scheme: http
 │
 ├── identify host: example.com
 │
 └── identify target: /products/42
 │
 ▼
DNS
 │
 └── example.com → IP address
 │
 ▼
TCP
 │
 └── establish transport connection
 │
 ▼
HTTP REQUEST

GET /products/42 HTTP/1.1
Host: example.com
Accept: text/html

 │
 ▼
HTTP SERVER
 │
 ├── parse request
 ├── interpret GET
 ├── identify resource
 ├── perform application work
 └── choose representation
 │
 ▼
HTTP RESPONSE

HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: ...

<html>
...
</html>

 │
 ▼
TCP
 │
 ▼
CLIENT / BROWSER
 │
 ├── interpret status
 ├── inspect headers
 ├── process body
 ├── parse HTML
 └── discover more resources
       │
       ├── GET /styles.css
       ├── GET /app.js
       └── GET /logo.png

That is considerably more machinery than:

browser gets website

but still manageable.

The mental model I’m keeping

HTTP sits above the network plumbing I have been learning.

It does not decide how Ethernet delivers a frame.

It does not choose the route an IP packet follows.

It does not provide TCP’s retransmission machinery.

Instead, HTTP establishes a shared application-level vocabulary:

resource
method
request
headers
body
status
response
representation

The basic conversation is:

CLIENT

"What do I want?"

      │
      │ HTTP request
      ▼

SERVER

"What happened,
and what am I returning?"

      │
      │ HTTP response
      ▼

CLIENT

And beneath that conversation:

HTTP
 │
 ▼
TCP
 │
 ▼
IP
 │
 ▼
link

or, when security is added:

HTTP
 │
 ▼
TLS
 │
 ▼
TCP
 │
 ▼
IP
 │
 ▼
link

So HTTP is neither mysterious nor particularly magical.

It is a carefully defined agreement between software components about how to express requests and responses.

The browser adds a huge amount of machinery around that agreement.

But underneath the buttons, tabs, JavaScript and rendered pages, two pieces of software are still exchanging protocol messages.

And now that I can see those messages, the web feels considerably less cloudy.

References and further reading

The following specifications were the primary technical references for this article and reflect the HTTP standards in use when this note was written.

HTTP/1.1 message format and connections

RFC 7230: Hypertext Transfer Protocol (HTTP/1.1) — Message Syntax and Routing Defines HTTP/1.1 message syntax, request and response structure, connection management, message framing, the Host header and HTTP intermediaries.

HTTP/1.1 semantics, methods and status codes

RFC 7231: Hypertext Transfer Protocol (HTTP/1.1) — Semantics and Content Defines HTTP resources and representations, request methods including GET, POST, PUT and DELETE, safe and idempotent method properties, content negotiation and response status codes.

HTTP caching

RFC 7234: Hypertext Transfer Protocol (HTTP/1.1) — Caching Defines HTTP caching behaviour, freshness, validation and cache-related controls.

URIs

RFC 3986: Uniform Resource Identifier (URI) — Generic Syntax Defines the generic URI syntax behind components such as schemes, authorities, paths, queries and fragments.

Cookies and HTTP state

RFC 6265: HTTP State Management Mechanism Defines the Cookie and Set-Cookie header fields used by HTTP applications to maintain state across otherwise largely stateless HTTP interactions.

HTTP over TLS

RFC 2818: HTTP Over TLS Describes the use of HTTP over TLS, the basis of HTTPS as used when this article was written.

TLS 1.3

RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3 Defines TLS 1.3 and the secure channel properties used by modern HTTPS connections.

HTTP/2

RFC 7540: Hypertext Transfer Protocol Version 2 (HTTP/2) Defines HTTP/2’s binary framing layer, streams and multiplexing while preserving the broader HTTP request/response semantics.

Linux command-line exploration

curl documentation Official documentation for curl, the command-line client used here to inspect HTTP exchanges without a browser hiding most of the protocol.