The idempotency key is created at the boundary where the caller first commits to an intent, typically the client or the public API edge, and is carried unchanged through every downstream hop. Generating a fresh key inside each service defeats the purpose entirely: a retry then looks like a new intent to everything past the first hop, and the payment is made twice while every individual service believes it behaved correctly.
Idempotency exists because networks fail after the work is done, not only before. A timeout tells the caller nothing about whether the payment executed, so the only safe behaviour is to retry. And the only thing that makes retrying safe is the receiver's ability to recognise that it has seen this exact intent before and return the original result instead of doing it again.
That recognition has to be durable and it has to happen before the side effect. A key checked against an in-memory cache, or checked after the transfer is initiated, is decoration. In practice this means the key is written to the same transactional store as the operation itself, in the same transaction, so there is no window in which the work is done but the key is not recorded.
The key's lifetime should exceed the longest plausible retry window of every caller, including a human clicking a button again after lunch and a batch job resuming the next morning. Expiring keys after a few minutes turns a correctness mechanism into a probabilistic one.
Where the key comes from matters as much as where it is stored. A key derived from the payment's own contents, such as a hash of amount, beneficiary and reference, collapses two genuinely separate payments of the same amount to the same person into one, which is a real scenario in payroll and refunds. The key must identify the intent, not the values.