How we bill each event exactly once

Every guide to usage-based billing ends the same way: use idempotency keys. The Stripe-style write-ups cover the API request. Airbnb's Orpheus framework dedupes payment requests against a sharded idempotency database. Metering vendors like m3ter and Lago promise they dedupe events for you.

The advice is right, and it stops one step short of where the money goes wrong. When Gunnar Morling's post on idempotency keys hit Hacker News last winter, the most-upvoted objection was precisely that step:

The article glosses over the hardest bit. How do you do that when the processing isn't persisted to the same database — what if the side effect is outside the transaction?

For a product that bills per event, the meter is that side effect. Carbon charges by the event, our analytics rows live in ClickHouse, the payloads live in S3, and the usage meter lives in DynamoDB — three stores, none of which can join the others in a transaction.

This is the story of how we made that safe anyway: one DynamoDB transaction that commits the meter and the idempotency checkpoint together, and a worker where every crash-recovery path falls out of that single field.

A per-event bill is two records, and a crash can split them

Billing an event means writing two facts: the month's counter moved, and this specific event is what moved it. The first is the meter; the second is the idempotency record that makes a retry harmless.

Write them separately and the crash between them writes your incident report for you. Meter first, and a worker death before the record means the retry charges again. Record first, and a death before the meter means the event ships free — and the record claims otherwise, forever.

No ordering fixes this. The only fix is making the two records move together — which is where most write-ups wave at "wrap it in a transaction" and move on. The rest of this piece is what that actually takes when the transaction can only cover one of your three stores.

Where the authoritative decision lives

Carbon's write path has two halves with opposite contracts. The front door authenticates, gives the SDK a fast answer, and enqueues — returning 202 the moment the event is durably queued. Its billing gate is advisory: it exists to return a quick 402 instead of a silent drop, and it fails open on any dependency error, because a courtesy check should never lose a customer's event.

The queue worker is the other half: slow, idempotent, and authoritative. It is the only place free-tier quota is actually consumed. The queue between them is at-least-once with bounded retry, so the worker's whole design assumption is that any step can run twice.

The asymmetry is deliberate. Of the three stores, only DynamoDB gives us a conditional-write primitive we control, so the ledger there is the one authority on "have I seen this event?" — ClickHouse's ReplacingMergeTree dedup is an eventual, merge-time backstop, never the guarantee. Everything downstream of the ledger is reconstructible; the ledger is not allowed to be wrong.

Step zero: claim the event

Before billing anything, the worker claims the event — a conditional put on the ledger, keyed by space and event id:

// analytics/packages/event-ledger/src/modules/claimEventForInsert.ts
await args.client.send(
  new PutItemCommand({
    ConditionExpression: "attribute_not_exists(pk)",
    Item: {
      createdAtMs: { N: String(now) },
      pk: { S: key.pk },
      rowInsertedAtMs: { N: String(args.rowInsertedAtMs) },
      sk: { S: key.sk },
      status: { S: "INSERTING" },
      updatedAtMs: { N: String(now) },
    },
    TableName: args.tableName,
  }),
);

attribute_not_exists(pk) makes this a first-writer-wins gate for all time: exactly one delivery of an event ever wins the claim and runs the pipeline from the top. Every other delivery loses, reads the ledger, and recovers instead — more on that below.

One detail here quietly carries the whole dedup story: the claim persists rowInsertedAtMs, the timestamp the warehouse row will be stamped with. Every retry reuses it, so a replayed insert produces a byte-identical row — which is the only reason ClickHouse's merge-time dedup can actually collapse duplicates instead of keeping two near-identical ones.

The transaction: meter and checkpoint, both or neither

For a free-tier space, billing the event is one TransactWriteItems carrying two conditional updates — the month's counter in the free-usage table, and a checkpoint field on the event's own ledger item:

// core/packages/dynamodb/src/modules/billingFreeUsage/consumeMonthlyUsageForEvent.ts
// (abridged: item keys and bound values elided)
new TransactWriteItemsCommand({
  TransactItems: [
    {
      Update: {
        ConditionExpression: "attribute_not_exists(#eventCount) OR #eventCount < :limit",
        UpdateExpression:
          "SET #eventCount = if_not_exists(#eventCount, :zero) + :increment, periodMonth = :periodMonth, spaceId = :spaceId, updatedAtMs = :now",
      },
    },
    {
      Update: {
        ConditionExpression: "#status = :inserting AND attribute_not_exists(billingAcceptedAtMs)",
        UpdateExpression: "SET billingAcceptedAtMs = :now, updatedAtMs = :now",
      },
    },
  ],
})

DynamoDB transactions are all-or-nothing across tables in a region, so there is no state where the counter moved and the checkpoint didn't, or the reverse. The meter and the checkpoint cannot diverge because they were never two writes.

Read the two conditions again, because each one protects the other update's invariant. The counter's guard (eventCount < :limit) means the checkpoint can never be stamped past the 1,000-event monthly cap. The checkpoint's guard (attribute_not_exists(billingAcceptedAtMs)) means a redelivered, already-billed event can never move the counter again. That second condition is the entire double-charge defence, expressed as one line.

One exception, two meanings

There's a subtlety hiding in the failure path. When either condition fails, DynamoDB cancels the whole transaction with the same TransactionCanceledException — and the two worlds that produce it demand opposite responses. If the month is at its limit, the event must be rejected. If a previous attempt of this same event already committed, the event is already paid for and must be accepted without charging again.

The worker resolves the ambiguity with one read: get the event's ledger item, projecting a single attribute. Checkpoint present → return accepted; a prior attempt did the work. Checkpoint absent → nothing ever billed this event, so the cancellation can only mean the limit is genuinely exhausted → reject.

The exception does carry a CancellationReasons list naming which item's condition failed, and we deliberately don't branch on it. The reasons describe this attempt; the checkpoint's presence states the durable truth regardless of which attempt wrote it. One authoritative re-read beats parsing a per-attempt error shape — and it costs a single projected GetItem on a path that is already the slow, rare one.

The checkpoint is also the resume pointer

Here is where the design pays for itself twice. The worker runs three ordered steps — acceptBilling, archiveAndInsert, markInserted — and billingAcceptedAtMs is not just the double-charge guard. It is the marker that tells a retry where to re-enter.

A redelivered event whose ledger still reads INSERTING means some earlier worker died mid-flight. The recovering worker reads the ledger and asks ClickHouse one question — does this row exist? — and those two facts pick the exact re-entry point. Checkpoint absent, no row: nothing durable happened, start at billing. Checkpoint present, no row: billing is done, skip it and finish storage. Row present: everything but the final mark happened; just close out.

Nothing is inferred from how far the previous attempt "probably" got. Every re-entry point derives from durable state — the ledger's status, one field's presence, one existence probe.

The whole decision table

The full dispatch lives in one ordered list of eight scenarios in event-scenarios.ts, matched top to bottom; the first match wins. Our publish pipeline doesn't render tables, so here it is as the diagram it deserves to be anyway:

Rows six and seven are the article's thesis in table form: two states identical except for one field's presence, dispatching to "bill now" versus "never bill again."

The three ack-only rows matter just as much. Rejection is written to the ledger terminally — a space at its limit produces REJECTED events, and every redelivery of one acks instantly instead of re-asking the billing question forever.

Order is correctness: archive before insert

Step two has an internal ordering that never varies: the full payload goes to the S3 archive before the lean row goes to ClickHouse. A crash between them leaves an archived payload with no analytics row — recoverable, invisible, fine. The reverse would leave an analytics row whose payload exists nowhere, which is a permanent lie in the product.

The same step owns the poison-row path. ClickHouse can quarantine a row whose shape has drifted from the schema; retrying that forever would just expire the message silently, and our queue has no dead-letter facility. So repeated quarantines move the event to a terminal UNINSERTABLE status and fire an alert — a designed terminal, not an accident. Nothing is lost, because the archive step already ran: the runbook is fix the drift, re-drive the row from blob.

What each tier pays at the gate

The transaction is the expensive instrument, so only the tier that needs it runs it. Internal spaces — the synthetic canary that end-to-end-tests the live pipeline — stamp the checkpoint and consume nothing. Active Plus spaces do the same, because Plus usage is billed, not capped: it's counted later, off a narrow events_billing table, by a reporting cron that never touches this hot path. An inactive Plus space rejects terminally. Only free spaces — the default for any unknown space — run the meter transaction at all.

The numbers the gate enforces are the ones on the pricing page: Free is 1,000 events a month at $0 with no overage; Plus is $28 a month with 100,000 events included and $0.00028 per additional event.

The tests are the spec

None of the above survives refactoring on good intentions. It survives because every crash window has a test that kills the worker there and asserts what the retry does — 23 tests on the worker in event-queue-worker.test.ts, 3 on the transaction in billing-free-usage.test.ts.

The one to steal is the test named "recovers inserting events with accepted billing without consuming usage again." It kills the worker in the exact window between the transaction and storage, redelivers, and asserts the meter moved once. If you adopt any single assertion from this article, adopt that one — it is the difference between believing your billing is exactly-once and knowing where it isn't.

The transaction tests are equally literal: one asserts the two ConditionExpressions verbatim, one races an already-billed event through the cancellation path and demands accepted, one exhausts the limit and demands rejected.

What this costs, and what we don't claim

The design has real costs and its guarantee has real edges. In order:

• Transactions cost double. DynamoDB performs two underlying writes per transactional item — prepare and commit — so the billing write is 2× the capacity of a plain update, and a cancelled transaction still consumes it. We pay that only on the free tier, where the hard cap lives.
• DynamoDB's own idempotency wouldn't do this job. ClientRequestToken dedupes API retries for 10 minutes; a worker that died and got redelivered an hour later is a new caller with a new token. The checkpoint is idempotency with no expiry, owned by our data model instead of the SDK.
• We did not solve exactly-once delivery. Nobody has; the queue is at-least-once and always will be. The guarantee is narrower and sufficient: first-writer-wins processing, and billing that commits atomically with its own receipt.
• ClickHouse dedup stays a backstop. ReplacingMergeTree collapses duplicates eventually, at merge time, only for identical rows — useful insurance, and never the mechanism we reason from.
• The disambiguating re-read is eventually consistent by default. In the window that matters it is answering a question about the event's own prior attempt, whose transaction committed before the queue could possibly redeliver — and the failure mode is a thrown error and another redelivery, never a second charge, because the transaction's own condition still guards the meter.

The pattern, portable

If you're metering anything — events, tokens, seats — the shape transfers whole:

1. Pick the one store you own that has conditional writes, and make it the only authority on "have I processed this?"
2. Claim first: a conditional insert that exactly one delivery can win, carrying any timestamps replays will need to be deterministic.
3. Commit the meter and the checkpoint in one transaction, each update's condition guarding the other's invariant.
4. On a cancelled transaction, resolve the ambiguity from durable state — the checkpoint's presence — not from the error's shape.
5. Make the same checkpoint your resume pointer, and enumerate the crash windows as an explicit, ordered scenario table.
6. Write one test per window that kills the worker there and counts the money afterward.

Everything downstream of that checkpoint — the archive, the analytics row, the dashboard a customer actually sees — can then afford to be merely idempotent, because the one thing that moves money moves exactly once.

That dashboard is the public demo — synthetic data, real pipeline. The meter behind it moved once per event, and we can point to the field that proves it.