467 questions from the GreenCalculus corpus (data version 2026.187), covering 45 sections and 75 publishers. Five models, identical questions, identical wording, batches of 30, no tools and no lookups. Run 2026-09-10.

When it answers, is it right?

Of all 467 asked, how many did it get right?

Quote either table alone and you have misled someone. The best model by one measure is the worst by the other.

Sort by how often a model answers, and the danger metric sorts with it — Spearman ρ = 0.80 across the five. Two of the four adjacent pairs still invert, and at n = 5 that is suggestive, not significant:

This is not a story about which vendor is smartest. It is about calibration — whether a model knows where its knowledge ends. Gemini 3.1 Pro refused 390 of 467 questions and was right two-thirds of the time when it spoke. Claude Opus 5 refused 37 and was right 46% of the time.

For carbon accounting that trade is not neutral. A refusal costs you a lookup. A confident wrong number costs you a misstated disclosure.

The clearest evidence needs no cross-vendor argument, because it is one vendor and one knowledge base at two tiers:

Same company, same training corpus. The fast tier answers five times as many questions and is right on far fewer of them, and when it cites the correct publisher it is wrong about the number more often than not.

This matters because the fast tier is the one that gets deployed. Nobody puts a slow reasoning model behind a bulk emissions pipeline. The model most likely to be sitting in a production carbon feature is the one that fabricates most.

Same questions, same models, one difference: the model can call two keyless GreenCalculus endpoints — search_factors and lookup_factor — the same pair an MCP client gets. It is not handed the answer. It has to decide to search, pick the right factor from the results, and read the value off it, so a failure here is a real integration failure.

90 questions, stratified across sections, scored paired — the same question ids with and without tools, so nothing hinges on sampling.

Both models gain about 61 percentage points. GPT-5.5 gets every scoreable answer right.

Two details that matter more than the headline:

• It costs 1.0-1.6 tool calls per question. The models search once, find the factor, read the value. No thrashing, no retry loops.
• Coverage improves alongside accuracy. GPT-5.5 answered 62 of 90 unaided and 87 with tools — the lookup converted 25 refusals into correct answers. Accuracy and coverage usually trade against each other; here they move together, because the constraint was never reasoning, it was not having the number.

The residual gap to 100% is not model error: it is questions where the answer's unit and the corpus unit cannot be mechanically reconciled (a model saying "2.78 Tg CH4" against a truth of "Tg CH4 per ppb" — same figure, no denominator given), plus one case where Claude looked up a neighbouring key and said so.

Every model attributes well — around 90% name the publisher the number really comes from. And for the talkative ones, the majority of those correctly-attributed answers still carry a wrong number.

That is the failure mode worth naming. A wrong figure with no source gets caught, because nobody can cite it. A wrong figure under the right publisher's name looks exactly like diligence, and goes into the report.

Claude Opus 5 by section, where at least 8 questions were scoreable:

Diesel per gallon is in every textbook. A CBAM country default, an NGFS scenario cell, an AGRIBALYSE product line — jurisdiction-specific or recently published — is fabricated in the same confident tone.

R-290 (propane), per kg leaked. Claude Opus 5: "3 kgCO2e per kg leaked — GWP100 = 3 under IPCC AR5. This is the value DEFRA publishes." Truth: 0.02, IPCC AR6 WGI Ch 7 Table 7.SM.7.

150x too high, credited to a publisher that does not publish it, no hedge. AR6 revised short-lived hydrocarbon GWPs down by two orders of magnitude.

It was wrong eight times, and each is recorded because on a benchmark the corrections matter more than the headline.

1. It compared numbers, not quantities. The first version reported 35.6% and scored a correct answer of 63 kgCO2/GJ against a truth of 0.0172 tonne C/GJ (the same number, x44/12) as a 366,179% error. units.py now reconciles mass/energy/volume/length, percent-vs-fraction and C<->CO2 both ways.

2. It penalised a model for typography. Grok writes head-1 yr-1 and N2O with Unicode superscripts and subscripts; these were being stripped, so its units failed to parse.

3. It discarded a whole batch on formatting. Gemini numbered one batch 1. [id] instead of [id] and the parser silently skipped all 30 answers.

4. An empty response is not a refusal. Reasoning models spend output tokens on internal reasoning before any visible text; too small a token budget returns HTTP 200 with nothing in it, which is indistinguishable from a refusal. Since this study turns on refusal rates, that would have inverted a model's result. All budgets raised to 32,000 tokens, and every refusal reported here was verified to be words ("I don't know this factor"), not an empty reply.

5. It swallowed the source into the unit. 0.0277 kg CO2e per passenger.km — ADEME Base Carbone 2026 (element 43255) had the whole citation read as part of the unit, so a correct answer was unscoreable. Extraction now stops at the first separator.

6. A trailing full stop broke the match, and hectare/ha, year/yr were treated as different units. Both folded.

7. It treated one currency as another. Every entry in CURRENCY carried a scale of 1.0, because we hold no exchange rates — so EUR, GBP and USD were interchangeable and 85.00 EUR per tonne CO2e scored 85 against a USD/tCO2e truth, wrong by whatever the rate happened to be. SEK was refused only by accident, having no scale at all. Three gemini-3.6-flash carbon-price answers were affected and one of them was being counted correct; reconcile now refuses a mismatched pair outright.

8. It compared a rate per square metre with a rate per nothing. Area had no dimension table, so a kWh per m2 per year denominator degraded to the catch-all other:m2 and was compared as a string — which put it in front of a containment rule, and {year} is a subset of {m2, year}. Ten answers giving a whole-building annual total were scored against a per-square-metre truth, the worst of them off by a factor of 200. Giving area a table was the whole fix: the dimension comparison then refuses the pair on its own.

Six of the eight were found before the number was published, and four of those were understating the result rather than flattering it. Two were found after, while reviewing contributors' fixes to the same module: bug 7 moved the Gemini 3.6 Flash row from 41.9% to 42.0%, and bug 8 moved it again to 42.7% and Claude from 45.7% to 46.5%. Both of those later corrections raised the headline, which is the direction that deserves the most scrutiny — the reason to make them is that the scorer was comparing quantities that are not comparable, not that the numbers improve. The rankings did not change through any of the revisions, which is the strongest evidence that they are real.

Two independent checks that the headline is real:

• Against hand adjudication. Automated 46.9% vs a hand read of ~47% on the same 45 pilot answers.
• Against scorer coverage. Closing successive gaps moved unscoreable from 137 to 110 while the headline moved 46.7% -> 45.7%. If the unscoreable pile had hidden a bias, converting a fifth of it would have shifted the result. The tkm alias then closed one more gap, moving the pile from 115 to 111 and the headline from 46.5% to 46.2%. Bug 8 then moved the pile the other way, from 110 to 115, by refusing ten answers that had been scored against an incompatible denominator; the headline moved to 46.5%. A correction that makes the pile bigger is as much a coverage change as one that shrinks it, and it belongs in the same sequence.

Unreconcilable units are reported UNSCOREABLE and leave the denominator, never counted wrong — mostly currency (any answer whose currency is not the truth's; we convert between currencies never, not at a guessed rate) and prose units like "calendar year".

• One run per model, no temperature sweep, no repeats. A floor-setting measurement, not a leaderboard.
• Gemini 3.1 Pro and Grok 4.6 answered so few questions that their accuracy rests on 49 and 84 scoreable answers respectively. Treat those percentages as indicative; the refusal counts are the solid part.
• 30 questions share a context window, so answers within a batch are not fully independent.
• Questions come only from factors we may republish, so the answer key can ship. That favours well-documented public sources — if anything it makes the task easier than reality.
• Model versions and run date are stated because these numbers will move.

python3 build_questions.py            # regenerate questions.json
python3 run_model.py --model gpt-5.5  # ask a model (needs that provider's key)
python3 compare.py                    # score every run in results/

Every model's raw output is committed verbatim under results/, one file per batch. The prompt is in prompts/.