Thirty correct findings, and only one looked like a mistake

A small tool checks the arithmetic in price lists. Real published files found what its tests never did.

There is one question everybody wants to ask of a price list. Is this pricing right? Is the discount too generous, is tier three too cheap, is anyone making money on it?

A price list cannot answer any of that, because it does not state a cost. Every one of those questions needs a number the document does not contain. A tool that answers them anyway is guessing in a confident voice.

So when I had a tool built for price lists, that one fact became its whole scope. It is called Adds Up. It never says a price is right, wrong, high, low, fair or profitable. It looks only for statements inside a single document that cannot all be true at once. A larger quantity priced higher than a smaller one. The same item priced twice, differently. A printed discount that does not turn the list price into the net price printed beside it. A minimum above its own maximum.

Here is what happened when it met real files.

29,521 tariff price lists

The first test was the OpenEI U.S. Utility Rate Database, a public domain export of electricity tariffs. It took every rate structure with two or more tiers: 29,521 price lists, from 18,049 distinct tariffs published by 2,062 utilities. Nothing crashed.

The volume check reported 15,487 findings on 12,050 of those documents. Thirty of them were drawn with a fixed random seed, one per tariff, and each was read by hand against that tariff’s own printed tier table. All thirty quoted the right two rates in the right order.

Thirty is a small sample, so there is a second test that does not depend on it. A separate script sorted every tier ladder in the export by shape, straight from the raw data, without asking the tool. Of 16,954 falling ladders, which is an ordinary volume discount, the tool reported on none. Of 515 flat ones, none. Of 10,711 rising ladders it reported on 10,709. The two it missed are among 316 documents whose rate column it refused to read, because the numbers in it could be written two ways and nothing in the file settled which.

The arithmetic held at scale. The next finding was less comfortable.

Thirty right, one mistake

Of those thirty correct findings, twenty nine were not mistakes.

Eleven had a first band priced at zero, which is an allowance already covered by a fixed charge, or the point above which a demand charge starts. Eighteen were inclining block tariffs, where a utility charges more per unit as you use more, on purpose, to encourage conservation. One looked like a genuine defect: an industrial ladder whose thresholds ran 3,000, 10,000 and 200,000, then dropped to 200, 400 and 600. Two rate blocks joined into one.

The README puts it plainly. On that sample of thirty, the check was 100% arithmetically right and about 3% useful. That sentence is published because both halves of it are true.

It also means this corpus says nothing about how often an inverted tier is a mistake. A utility tariff is the wrong document to ask.

200 hospital files

The second corpus exists because of a federal rule: every US hospital must publish a file of its standard charges. The tool was run over 200 of them. What mattered was the column headings: thousands of them, written by people who had never heard of this tool. The 173 files that really were CSV used 8,519 distinct column names between them. The other 27 turned out to be JSON served from an address ending in .csv, and the tool now refuses those by name.

The findings from the first run were read back by hand, one per document for each check and up to thirty per check, against the exact row and column each one named. That reading found three defects in the tool. The test suite had been green the whole time.

The largest was about commas. One hospital’s file leaves commas unquoted inside free text, so 180,084 of its 1,342,453 rows split into more cells than the header has, and every value after the break slides under a heading that is not its own. The tool read those rows anyway and said nothing. Every one of that file’s 1,517 minimum above maximum findings came from a shifted row. On that first run they were 36.7% of the check’s findings across the whole corpus.

Now a row that does not fit the header line is not read, and the report says how many were set aside and in which file. On the second run that was 190,549 rows in 19 files, against 20,513,338 rows read. For the two checks that problem hit, the corpus wide counts fell by exactly the numbers the hand reading had proved false, and by nothing else.

Refusing to guess has a price

One file writes every value in its percentage columns as a fraction such as 0.8. That is 80% in one convention and 0.8% in the other, and nothing in the file says which. The tool now refuses a column like that. In that one file it lost 91,188 findings. The hand audit had judged that file’s sampled finding correct, and the file’s own dollar column shows the rows do contradict themselves.

The README counts that loss instead of hiding it. The design choice stands anyway: a finding that rests on a guess about what a number means is worse than no finding.

The same logic sits behind the exit code. A review commissioned to break the tool’s promises found a price list with a real inverted tier that still came back clean, because its price column had a heading the tool did not know, and a check that compares no numbers had run and passed. Now a clean result requires that at least two numbers were actually compared.

What it still cannot tell you

Two of the seven checks, and half of a third, have never been measured against a real document. The discount tier and bundle checks found nothing to run on in either corpus, and rest on their unit tests alone. No currency column appeared in either corpus either, so the currency half of the unit check is in the same position. They may be wrong in ways nobody has seen.

Real price lists are hard to find in public. A keyword search of six open data portals returned 3,802 spreadsheet download links. Of 180 picked at random, 54 could be read as tables, and three of those were genuinely price lists.

The hand audit of the second hospital run found no wrong finding, and that is a statement about a sample: for three of the checks the sample is every document that produced a finding at all, which sounds complete and is 4, 12 and 16 documents.

The tool reads CSV, TSV and .xlsx files, not PDF. It reads one file at a time, so it cannot compare this quarter’s list with last quarter’s, or with the invoice. It holds the whole table in memory. And nothing in it says anything about margin, ever.

A note on how it was built, since somebody always asks. I am not an engineer. AI agents wrote the code and the documentation. My part was deciding what the tool had to prove before it went out, which real files it had to meet, and that the numbers which do not flatter it get published beside the ones that do.

If you work with price lists, run it on one. It installs with pip install adds-up, runs on your own machine, needs no account and makes no network call. A column heading it did not recognise stops the check that needed it, and the report names the word it was looking for. That word, and where you saw it, is the most useful thing you could send back as an issue on the repository. Send the heading, not the file. The vocabulary only grows from headings that really exist.

The repository, with every figure above and the scripts that produced them: https://github.com/Waiga/adds-up