Idempotency Keys

The third lesson measured that POST is not idempotent: sending the same request twice produced two loan records. Back then, that was noted as POST’s nature and left alone. Once the eighth lesson closed the read side, this gap remained.

The gap comes from a real situation. The client sends a request to create a loan record, and the network drops. The network-error class of the error contract from the Application Architecture course described exactly this: the request may have reached the server, or it may not have. If the response was lost on the way back, the record was created but the client does not know it. If the client retries, there is a risk of a second record; if it does not, the member may end up without the book. Neither option is acceptable.

This lesson builds a third way: attaching a client-generated identity to the request.

What the Key Does

An idempotency key is the unique name a client gives to an operation’s intent. It travels in the Idempotency-Key header and has three rules.

The key is generated per intent, not per request. A key is generated when the user presses the “issue loan” button; three attempts made because the network dropped all carry that same key. If the user wants to check out a second book, a new key is generated.

The key is not enough on its own. The server must also store which operation the key is tied to: which address, with which body. Otherwise, if the client mistakenly reuses the same key on a different request, that request gets silently swallowed.

The key is stored together with the response. When the second request arrives, the operation is not redone; the response generated the first time is returned as is. The client learns the result even if it never saw the first response.

// key-server.mjs — loan creation guarded by an idempotency key
import { createServer } from "node:http";
import { DatabaseSync } from "node:sqlite";
import { createHash } from "node:crypto";

const db = new DatabaseSync("library.db");
db.exec(`CREATE TABLE IF NOT EXISTS idempotency (
  key       TEXT PRIMARY KEY,
  path      TEXT NOT NULL,
  digest    TEXT NOT NULL,
  status    INTEGER,
  response  TEXT,
  createdAt TEXT NOT NULL
)`);

const readBody = (request) => new Promise((resolve) => {
  let data = ""; request.on("data", (p) => (data += p));
  request.on("end", () => resolve(data));
});
const send = (response, status, data, extra = {}) => {
  response.writeHead(status, { "content-type": "application/json; charset=utf-8", ...extra });
  response.end(JSON.stringify(data));
};

const server = createServer(async (request, response) => {
  const path = new URL(request.url, "http://127.0.0.1").pathname;
  if (request.method !== "POST" || path !== "/loans") return send(response, 404, { error: "path_not_found" });

  const raw = await readBody(request);
  const key = request.headers["idempotency-key"];
  const digest = createHash("sha256").update(raw).digest("hex").slice(0, 16);
  const body = JSON.parse(raw);

  // No key: every call produces a new record.
  if (!key) return send(response, ...create(body));

  const record = db.prepare("SELECT * FROM idempotency WHERE key = ?").get(key);
  if (record) {
    // Same key, different body: the client reused the key.
    if (record.path !== path || record.digest !== digest)
      return send(response, 409, { error: "key_conflict", key });
    return send(response, record.status, JSON.parse(record.response), { "idempotency-replayed": "true" });
  }

  // Reserving the key and doing the work happen in one block, with no wait in between.
  db.prepare("INSERT INTO idempotency (key, path, digest, createdAt) VALUES (?,?,?,datetime('now'))")
    .run(key, path, digest);
  const [status, data] = create(body);
  db.prepare("UPDATE idempotency SET status = ?, response = ? WHERE key = ?")
    .run(status, JSON.stringify(data), key);
  send(response, status, data);
});

function create(body) {
  if (!db.prepare("SELECT 1 FROM book WHERE isbn = ?").get(body.isbn))
    return [422, { error: "validation", field: "isbn" }];
  const s = db.prepare("INSERT INTO loan (member, isbn, issuedAt, return) VALUES (?,?,?,NULL)")
              .run(body.member, body.isbn, body.issuedAt);
  return [201, { id: Number(s.lastInsertRowid), member: body.member, isbn: body.isbn, issuedAt: body.issuedAt }];
}

server.listen(8483, "127.0.0.1", () => console.log("key server 127.0.0.1:8483"));

Measurement

The same body is sent twice: first without a key, then with one. The catalog is set up with the catalog.sql file from the seventh lesson.

# The same request is sent twice: first without a key, then with an idempotency key.
rm -f library.db && sqlite3 library.db < schema.sql && sqlite3 library.db < catalog.sql
sqlite3 library.db "DELETE FROM loan;"
node key-server.mjs & server=$!
sleep 0.4

BODY='{"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"}'
say() { printf '%-30s loan rows: %s\n' "$1" "$(sqlite3 library.db 'SELECT COUNT(*) FROM loan;')"; }
send() { curl -sS -D - -o /tmp/body.$$ -X POST -H 'content-type: application/json' "$@" \
            -d "$BODY" http://127.0.0.1:8483/loans \
          | grep -iE '^(HTTP/|idempotency-replayed)' | tr -d '\r'; cat /tmp/body.$$; echo; rm -f /tmp/body.$$; }

echo "--- no key, twice ---"
send; send; say "no key x2"

sqlite3 library.db "DELETE FROM loan; DELETE FROM idempotency;"
echo "--- same key, twice ---"
send -H 'Idempotency-Key: OD-7f21'
send -H 'Idempotency-Key: OD-7f21'
say "with key x2"

echo "--- same key, different body ---"
curl -sS -o /dev/null -w 'status: %{http_code}\n' -X POST -H 'content-type: application/json' \
  -H 'Idempotency-Key: OD-7f21' -d '{"member":"U-1002","isbn":"K-03","issuedAt":"2026-03-11"}' \
  http://127.0.0.1:8483/loans
say "after conflict"

kill $server

key server 127.0.0.1:8483
--- no key, twice ---
HTTP/1.1 201 Created
{"id":1,"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"}
HTTP/1.1 201 Created
{"id":2,"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"}
no key x2                      loan rows: 2
--- same key, twice ---
HTTP/1.1 201 Created
{"id":1,"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"}
HTTP/1.1 201 Created
idempotency-replayed: true
{"id":1,"member":"U-1001","isbn":"K-02","issuedAt":"2026-03-10"}
with key x2                    loan rows: 1
--- same key, different body ---
status: 409
after conflict                 loan rows: 1

The two requests without a key produced two records and returned two different identities. The two requests with a key produced a single record and returned the same body: the second response’s id field is also 1. The second response also carries the Idempotency-Replayed header; this header is not mandatory, but it lets the client and the logs distinguish the situation.

The last section tests reusing the key. The same OD-7f21 key arrived with a different body, and the request was rejected with 409; the loan record count stayed at one. Had the body digest not been stored, this request would have been silently swallowed and the client would have assumed a second loan was created. Rejecting it is the right behavior: only the client can fix a key used incorrectly.

Concurrent Requests

Two requests arriving one after another is the easy case. The hard case is both arriving at the same time: while the client times out and retries, the first request may still be being processed.

# Five requests with the same key are sent concurrently; the resulting records are counted.
rm -f library.db response-*.json && sqlite3 library.db < schema.sql && sqlite3 library.db < catalog.sql
sqlite3 library.db "DELETE FROM loan;"
node key-server.mjs & server=$!
sleep 0.4

pids=""
for i in 1 2 3 4 5; do
  curl -sS -o "response-$i.json" -X POST -H 'content-type: application/json' \
    -H 'Idempotency-Key: OD-9c04' \
    -d '{"member":"U-1002","isbn":"K-05","issuedAt":"2026-03-12"}' \
    http://127.0.0.1:8483/loans &
  pids="$pids $!"
done
wait $pids

for f in response-*.json; do cat "$f"; echo; done | sort -u > variants.txt
echo "distinct response bodies : $(wc -l < variants.txt | tr -d ' ')"
echo "body                      : $(cat variants.txt)"
echo "loan rows                 : $(sqlite3 library.db 'SELECT COUNT(*) FROM loan;')"

echo "--- idempotency store age ---"
sqlite3 library.db "
  INSERT INTO idempotency (key, path, digest, status, response, createdAt)
  VALUES ('OD-old','/loans','0000',201,'{}',datetime('now','-30 day'));
  SELECT key || '  age: ' || CAST(julianday('now') - julianday(createdAt) AS INT) || ' days'
    FROM idempotency ORDER BY createdAt;
  DELETE FROM idempotency WHERE createdAt < datetime('now','-7 day');
  SELECT 'remaining after cleanup: ' || COUNT(*) FROM idempotency;"

kill $server

key server 127.0.0.1:8483
distinct response bodies : 1
body                      : {"id":1,"member":"U-1002","isbn":"K-05","issuedAt":"2026-03-12"}
loan rows                 : 1
--- idempotency store age ---
OD-old  age: 30 days
OD-9c04  age: 0 days
remaining after cleanup: 1

Five concurrent requests produced a single record, and all five got the same body. What makes this work is the code’s structure: there is no wait point between recording the key and doing the work. If a wait were introduced in between, both requests could see “no key” and both could create a record.

This arrangement being sufficient on a single-process server does not mean it stays sufficient in a multi-process deployment. The general fix is writing the key as a primary key: the second write attempt is rejected by the database, and that request is routed to wait and reread, or to an “operation in progress” response. How races of this kind are isolated at the transaction level is the subject of the Transactions topic in the Advanced SQL course.

The Key Store’s Lifetime

The output’s last section shows the key store aging. A key written thirty days ago was cleaned up under a seven-day retention window.

The retention decision has two bounds. The lower bound is set by the client’s longest retry window: if a client is still retrying twenty-four hours later, the key must be kept that long. The retry budget defined in the Retry and Backoff lesson of the Application Architecture course gives this window. The upper bound is set by storage cost and the size of the response bodies written into the table.

Once the window closes, a request arriving with the same key counts as a new operation and produces a new record. This is the design’s accepted limit; the duration is chosen with this in mind.

One last boundary is scope. A key must be unique together with the authenticated client, not on its own: two different clients producing the same key string must not lead one’s request to receive the other’s response. In the table above, the key alone is the primary key; in a real deployment, the client’s identity also becomes part of the key.

Summary

• Under a network error, whether the request reached the server is unknown; retrying risks a repeated effect, not retrying risks a missing one.
• An idempotency key is generated per intent, not per request; every attempt of the same operation carries the same key.
• The server stores the key together with the path and a body digest; in the measurement, two requests without a key produced two records, two requests with one produced one record, and the second response matched the first exactly.
• The same key arriving with a different body is rejected with 409; silently swallowing it would show the client an operation that never happened as if it had.
• No wait must fall between recording the key and doing the work; five concurrent requests produced a single record under this structure.
• The key store’s retention period cannot be shorter than the client’s longest retry window; once the window closes, the same key counts as a new operation.

Next Step

Across nine lessons, the client was assumed to build every address itself: the collection address, the page cursor, the field list, even the address of a newly created record. To do this, the client embeds the contract’s shape into itself — and when the contract changes, every client gets updated separately. The next lesson loosens this dependency: the response carries within itself the addresses the client will use in its next request. Which operations a loan record is open to is also read from the same place; a returned record no longer offers a return link.