The same payment, twice

The tidy instinct is to deduplicate: if two rows agree on date, amount and description, keep one. It produces a cleaner table and it is wrong most of the time. A merchant who pays the same supplier twice on a Tuesday, a card used twice at the same shop, two identical monthly fees debited together — all of these are ordinary, and all of them look exactly like a duplicate.

Deleting one of those rows removes real money from the file. If it was a debit, financing burden reads lighter than it is and a deal gets funded that should not have been. If it was a credit, revenue reads lower and a decent merchant gets declined. So identical rows stay, and a row is only removed when something other than its resemblance says it is a reprint.

There is one duplicate that banks create deliberately. The transaction ledger lists a cheque as it cleared, and a later section of the same statement reproduces the paid cheques as images, each with its own caption. One payment, two lines, in two different parts of the document.

What makes them catchable is that they do not look alike. The ledger row says something like “Check 683”; the image caption says “#683”. A text comparison sees two unrelated rows — which is exactly why a text comparison is not the test. The pairing is made on the cheque number and the amount: the same serial, the same money, in one statement. The register entry is the payment; the caption is a picture of it.

The rule around that is narrow on purpose. A line has to actually look like a cheque reference rather than merely contain a number, or ordinary electronic descriptors carrying a long reference would start reading as cheques and real payments would vanish. The cost of over-matching here is the same cost as over-deduplicating anywhere else, and it lands on the merchant.

Files repeat too. A broker sends the pack twice, or a merchant re-downloads a month they already provided. The obvious defence is to hash the file, and it barely works: banking portals regenerate a PDF on every download, so two copies of one statement are byte-different documents with identical contents.

So the fingerprint is taken over what the statement says — its rows, their dates, their amounts — rather than over the bytes it happens to be packaged in. Re-uploading the same month is recognised as the same month, and the analysis you already have comes back instead of being computed again.

The useful part of that design is its converse. A corrected statement covering the same period has different rows, so it fingerprints differently and is treated as what it is: a new document that deserves a fresh read. Sameness is defined by content, which means a real change is never mistaken for a repeat.

Duplicates have one more property worth relying on: they break the balance. A row counted twice puts the extracted transactions out of step with the statement’s own printed totals, and where a running balance is printed on each line, it stops following from the row above.

That is why the arithmetic layer matters here and not only for altered documents. A duplicate nobody noticed and an edit somebody made produce the same symptom, and the same check finds both. When the totals do not reconcile, the ratios are withheld rather than published over a transaction set known to be wrong.

If a burden figure looks heavier than the deal you know, the first thing worth scanning is the debit side for repeated amounts on the same day from the same payee. Real double-pulls happen — some funders debit twice a day, deliberately — and they are supposed to be counted. A cheque duplicated by the statement’s own layout is not.

And if the totals do not reconcile at all, that is the more important finding. It usually means a page of the statement is missing from the upload rather than that anything was double-counted, and the fix is a complete document rather than a deduplication rule.