Payments Engine

Idempotency

A withdrawal request that times out has an unknown outcome — the payout may have been created, or not. Idempotency is what turns that unknown into a question you can simply ask again, and it is the single most important thing to get right in an integration.

The model

You supply an Idempotency-Key per logical withdrawal. The engine records the key together with a fingerprint of the instruction — operator, payout ID, asset, amount, destination — inside the same transaction that creates the payout. A later request with the same key is compared against that fingerprint.

The engine seesIt answersMeaning
A key it has never seen201 · outcome: "ACCEPTED"A new payout was created.
A known key, identical instruction200 · outcome: "REPLAYED"The original payout is returned. Nothing new happened.
A known key, different instruction409Refused. Your bookkeeping produced two meanings for one key.

Choosing a key

  • One key per withdrawal, not per attempt. A new key on retry is exactly how a double payment happens — the engine has no way to know the two requests meant the same thing.
  • Derive it from something you already store. Your own withdrawal row's primary key is ideal: it is stable across retries, across restarts and across a redeploy mid-request.
  • Never a timestamp, a UUID generated at call time, or a hash of the payload. The first two change on every attempt; the third makes a legitimate correction indistinguishable from a new payment.
node
// Wrong — a new key every attempt. Three timeouts, three payouts.
await payments.payouts.submit(request, randomUUID());

// Wrong — the key changes if the customer edits the destination.
await payments.payouts.submit(request, sha256(JSON.stringify(request)));

// Right — stable, and it already exists in your database.
await payments.payouts.submit(request, `withdrawal-${withdrawal.id}`);

Handling all three outcomes

node
import { PaymentsApiError } from '@payments/sdk';

try {
  const result = await payments.payouts.submit({
    payoutId: `pyo_wd${withdrawal.id}`,
    asset: withdrawal.asset,
    amountMinor: withdrawal.amountMinor,   // a string from your DB
    destination: withdrawal.destination
  }, `withdrawal-${withdrawal.id}`);

  // Both outcomes are success. REPLAYED means an earlier attempt got through
  // even though you never saw the response — which is the normal case after
  // a timeout, and requires no special handling.
  await markSubmitted(withdrawal.id, result.payoutId);

} catch (error) {
  if (error instanceof PaymentsApiError && error.isIdempotencyConflict) {
    // Do NOT retry with a fresh key. Two different instructions have been
    // sent under one key: something upstream is wrong. Halt and alert.
    await flagForReview(withdrawal.id, error.problem.requestId);
    return;
  }
  throw error;  // transport and 5xx are already retried by the SDK
}

Why a 409 is never something to route around

A conflict means one key now stands for two different payment instructions. The engine cannot know which one you meant, and neither can your retry logic. Retrying with a fresh key resolves the error and pays the customer twice — which is precisely the failure the key existed to prevent. Stop, and find out which instruction was correct.

In practice a 409 almost always means one of: your withdrawal row was mutated after the first submission; two workers picked up the same withdrawal and computed different amounts; or a currency conversion ran again at a different rate. All three are worth finding.

What makes the guarantee hold

The key and its fingerprint are written in the same transaction as the payout, under UNIQUE (operator_id, idempotency_key). Three properties follow, and none of them depends on the application behaving:

  • It survives a restart. The record is in Postgres, not in a cache — a process that dies between writing the payout and answering you still refuses the duplicate.
  • It is one guarantee across instances. Ten API processes share one unique index, not ten in-memory maps.
  • It survives a deploy. A request that started against the old version and retried against the new one resolves to the same payout.

The adversarial test for this fires a hundred concurrent submissions of the same key and asserts that exactly one payout exists afterwards. It is also the test that caught a real bug: an ON CONFLICT (payout_id) clause that worked serially and failed under concurrency, because the table carries two unique constraints and the conflict arrived on the other one.