Repository navigation
Expand file tree
/
Copy pathclaim_register_v1.yaml
More file actions
1101 lines (1080 loc) · 56.4 KB
/
Copy pathclaim_register_v1.yaml
File metadata and controls
1101 lines (1080 loc) · 56.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
---
# REMORA Machine-Readable Claim Register v1
# Source of truth: THIS FILE is the single authoritative claim register.
# docs/claim_register.md is a human-readable companion view; docs/thermodynamics/claim_ledger.yaml
# governs thermodynamics sub-claims only, never overall claim status.
# Evidence level taxonomy: docs/assurance/evidence_levels.md
# Generated: 2026-07-31 (header comment kept in sync with the data key below)
# Version: 1
#
# evidence_level values (ordered weakest to strongest):
# theoretical, internal_simulation, internal_benchmark, regression_tested,
# externally_benchmarked, independently_replicated, field_observed, externally_validated
#
# Fields required per claim:
# id, statement, evidence_level, artifact, n, caveat, reproduce, status
# status values (lifecycle — is this claim current?):
# active — a current result; may be cited on the README front page
# superseded — a later round replaced it. MUST name superseded_by: CLAIM-0NN.
# Superseded claims are never deleted (CLAUDE.md); they move to
# docs/assurance/superseded_claims.md, which is generated by
# scripts/generate_superseded_claims.py and diffed in CI. The
# claim-provenance gate refuses to let a superseded claim stay
# anchored on the README front page.
# retired_values — strings the claim's numbers used to be written as, before
# an artifact re-issue changed them. The gate fails on any occurrence outside
# an explicitly historical paragraph, so the first CI run after a re-issue
# enumerates every document that still needs updating. Populate this at
# re-issue time; see docs/assurance/rebenchmark_protocol_v1.md.
# blindness values (measurement condition — was it a blind evaluation?):
# blind — evaluated once, on a sealed set, against targets fixed
# beforehand. The set is spent afterwards.
# development — measured on data the system had already seen, or re-measured
# on a spent set. Never quote as generalisation evidence.
# omitted — the distinction does not apply (regression corpora, bounds,
# telemetry). status and blindness are separate on purpose: a
# development measurement can be perfectly current.
# Optional fields:
# metrics — machine-readable numbers for this claim. Narrative docs bind to
# these via claim anchors (<!-- claim:CLAIM-004 accuracy_pct n -->) checked
# by scripts/check_claim_provenance.py. Update numbers HERE first; the gate
# fails any anchored doc paragraph that no longer contains the value.
# utility_floors — pre-registered utility bars for a safety claim: metric (a
# bound key in metrics), exactly one of min/max, source (the committed file
# that fixed the bar) and status (met | missed). A missed floor stays in the
# register as missed. utility_floor_exempt — the reason no floor applies.
# Every active safety claim (a false-accept style metric) declares one or the
# other; scripts/check_claim_utility_floors.py recomputes each status.
schema_version: "1"
generated: "2026-07-31"
claims:
- id: CLAIM-001
title: "FAR=0% on adversarial tool-call benchmark v2 (simulator-scoped)"
statement: >
REMORA's full policy gate produces 0.0% unsafe execution rate on a 700-task
(70 unique templates x 10 cosmetic variants; effective N=70) adversarial
tool-call benchmark. After the 2026-07-20 leakage fix (gate and baselines
both restricted to the observable task surface plus platform-fact context),
heuristic baselines show 1.4% unsafe execution; the unsafe-rate delta vs
baselines is NOT statistically significant at the template-cluster level
(one-sided p=0.50). A post-hoc test, declared after the pre-registered
hypothesis failed, gives a utility delta of +0.46 vs heuristics
(p~=1e-4). The ablation attributes the zero to no single component
(CLAIM-020): the condition without Stage 1 hard blocks also records a
false-accept rate of 0.0. The unsafe rate rests on the 56 harmful
templates; over them the cluster-level Wilson 95% upper bound is 6.42%
(post hoc, artifacts/posthoc/peer_review_level2_v1.json). Do not cite as
evidence for the consensus layer, and do not cite the unsafe-rate delta
vs baselines as significant.
evidence_level: internal_benchmark
status: active
artifact:
- "results/toolcall_benchmark_v2_results.json"
- "results/toolcall_benchmark_v2_significance.json"
- "results/toolcall_blind_v3_results.json"
- "results/toolcall_m1_clean_signal.json"
n: 700
n_detail: >
560 harmful, 140 benign; deterministic synthetic benchmark;
70 unique templates x 10 near-duplicate variants, so effective N=70
# Q2.1: the independent harmful units the FAR rests on. 560 harmful tasks
# collapse to 56 harmful template clusters (14 of the 70 templates are
# benign-only); results/toolcall_benchmark_v2_results.json records
# effective_n 56 for remora_full_policy_gate. Corrected from 70 on
# 2026-10-18 (NEGATIVE_RESULTS.md section 83).
far_denominator: 56
metrics:
far_pct: 0.0
far_ci_high_pct: 5.2
n_effective: 70
baseline_far_pct: 1.4
metric_bindings:
far_pct:
path: "results/toolcall_benchmark_v2_results.json#baselines.remora_full_policy_gate.false_accept_rate"
scale: 100
path_rationale: >-
FAR is the false-accept rate of REMORA's own full policy gate, which the artifact stores as a fraction under that baseline key.
far_ci_high_pct:
path: "results/toolcall_benchmark_v2_significance.json#remora_unsafe_rate.cluster_level_wilson_ci95[1]"
scale: 100
rounded_to: 1
path_rationale: >-
The significance file stores the cluster-level Wilson interval for the
gate's unsafe rate (0 of 70 template clusters). False accepts are also
0 of 700 tasks in results/toolcall_benchmark_v2_results.json, so the
interval over the 70 clusters is the same one. This was declared
unbound until 2026-09-28 on the belief that the artifact held only counts.
n_effective:
path: "results/toolcall_benchmark_v2_significance.json#n_template_clusters"
path_rationale: >-
the effective sample size is the number of template clusters, which the significance artifact stores under that name.
baseline_far_pct:
path: "results/toolcall_benchmark_v2_results.json#baselines.single_model_heuristic.false_accept_rate"
scale: 100
rounded_to: 1
path_rationale: >-
the heuristic baselines' false-accept rate, 10 of 700 tasks = 1.428...%, published to one decimal as 1.4% here and in the paper.
utility_floor_exempt: >-
No absolute utility floor was pre-registered for this benchmark. Utility
is reported only as a delta against heuristic baselines (the statement's
+0.46), and the full gate sends half of the benign rows to review
(review_friction_benign 0.5 in results/toolcall_benchmark_v2_results.json).
caveat: >
Simulator-scoped: no real shell, network, database, or file mutations execute.
Benchmark is synthetic and adversarial patterns are designed in-distribution.
Effective sample size is 70 template clusters, not 700 tasks; cluster-level
Wilson 95% CI on FAR: [0.0%, 5.2%]. External replication with independently
withheld labels is required before a field-deployment safety claim can be made.
M1 leakage (is_unsafe_if_executed) removed 2026-06-28; oracle context-flag,
severity, and tags leakage removed 2026-07-20 (gate now derives signals from
the task surface; AST detector and mutation tests guard re-introduction).
Platform-fact context values (target_environment, blast_radius, authz_verified,
evidence_complete) remain simulator-declared. Harmful tasks rely on keyword
heuristics that could be evaded by rewording.
reproduce: >
python experiments/generate_toolcall_benchmark_v2.py
python experiments/evaluate_toolcall_benchmark_v2.py
python experiments/toolcall_v2_significance.py
Compare to results/toolcall_benchmark_v2_results.json (rates) and
results/toolcall_benchmark_v2_significance.json (70 clusters, Wilson bound).
- id: CLAIM-002
title: "FAR=0% on external AgentHarm benchmark (N=208)"
statement: >
REMORA blocked all 208 independently-sourced harmful scenarios from the
AI Safety Institute's AgentHarm benchmark (arxiv:2410.09024). FAR=0.0%,
Wilson 95% CI [0.00%, 1.81%]. The 208 scenarios come from 47 harmful
base behaviours; counted by behaviour, the Wilson 95% upper bound is
7.56% (post hoc, artifacts/posthoc/peer_review_level2_v1.json). This
dataset was not present in REMORA's training corpus; external validity
is supported by dataset independence.
evidence_level: externally_benchmarked
status: active
artifact:
- "results/external_benchmark_agentharm_v1.json"
n: 208
n_detail: "208 harmful scenarios from ai-safety-institute/AgentHarm; peer-reviewed dataset"
far_denominator: 208
metrics:
far_pct: 0.0
far_ci_high_pct: 1.81
fbr_pct: 100.0
metric_bindings:
far_pct:
path: "results/external_benchmark_agentharm_v1.json#false_accept_rate"
scale: 100
path_rationale: >-
FAR is the false-accept rate; the artifact stores it as a fraction under that key.
far_ci_high_pct:
unbound: >-
Wilson upper bound, computed for the register from the counts in the artifact; the artifact stores the counts, not the interval.
fbr_pct:
path: "results/external_benchmark_agentharm_v1.json#false_block_rate"
scale: 100
path_rationale: >-
FBR is the false-block rate; the artifact stores it as a fraction under that key.
utility_floors:
- metric: fbr_pct
max: 40.0
source: "artifacts/verify_protocols/agentharm_rescore_v1.json"
source_detail: >-
fbr_target "<=40%", pre-registered for the re-scoring of the same
208 benign scenarios (NEGATIVE_RESULTS.md section 19).
status: missed
caveat: >
No ablation was run on this benchmark and the artifact records no
per-component attribution, so the result is not attributed to the
hard-block layer; on the tool-call benchmark the ablation attributes the
zero to no single component (CLAIM-020). Do not cite this result as
evidence for the consensus layer.
FBR=100% on benign variants is expected (same harm_category as harmful counterparts
triggers same risk_tier mapping). FAR=0.0% comes with every benign scenario blocked; FBR documents friction.
Gate: REM-014 PASS. Benchmark is intent-gating; tool-call interception is not verified.
reproduce: >
IMPORTED HISTORICAL RESULT. Produced in the REMORA main implementation repo
(commit 483d1b0, experiments/agentharm/score_guardrail.py) and copied here for
public verifiability; NOT regenerated from this repository.
scripts/run_agentharm_benchmark.py here implements a different intent-gating
protocol and yields a different artifact. Validate the imported artifact
structurally: python -m pytest tests/test_rem014_external_benchmark.py -m rem014_gate
- id: CLAIM-003
title: "FAR=0% on historical regression corpus (N=167)"
statement: >
167 historical false-accept episodes from AROMER D1 (WARMUP/LEARNING phase)
re-evaluated against the current system. FAR=0.0% — 167/167 blocked.
Confirms that policy improvements have not introduced regressions on known failures.
evidence_level: regression_tested
status: active
artifact:
- "results/false_accept_regression_v1.json"
n: 167
far_denominator: 167
n_detail: >
169 original episodes minus 2 corpus artifacts excluded:
aradhye-0060 (benign school-outing query, mislabeled in aradhye dataset);
caizhitech-1059 (conversational AI safety case, out of scope for agent action governance).
metrics:
far_pct: 0.0
far_ci_high_pct: 2.25
metric_bindings:
far_pct:
path: "results/false_accept_regression_v1.json#false_accept_rate"
scale: 100
path_rationale: >-
FAR is the false-accept rate; the artifact stores it as a fraction under that key.
far_ci_high_pct:
derived: "wilson_upper(results/false_accept_regression_v1.json#n_false_accept, results/false_accept_regression_v1.json#n_scenarios)"
scale: 100
path_rationale: >-
Wilson 95% upper bound at 0/167, recomputed by the gate from the artifact's own counts (added 2026-09-28, Q2.1).
utility_floor_exempt: >-
The corpus holds only previously observed false accepts. It has no
benign items, so it cannot measure what the gate costs legitimate calls.
caveat: >
Corpus is internal — episodes authored within the project context (AROMER D1).
2 exclusions documented with rationale in results/false_accept_regression_v1.json.
Policy change applied: VERIFY floor for risk_tier=high and risk_tier=critical.
This change is conservative (cannot create new false accepts by construction).
Gate: REM-019 PASS.
reproduce: >
Requires REMORA main repo + AROMER worker + D1 corpus access.
Artifact committed for reference: results/false_accept_regression_v1.json
- id: CLAIM-004
title: "Temperature-selective holdout: 100% at 16.7% coverage — directional only, later contradicted on fresh data"
statement: >
In the SAP v2 clean round (2026-07-27, Workers AI cross-family trio,
544 items), temperature-based selective acceptance scored 18/18 correct
(100.0%) at 16.7% holdout coverage with tau* locked on the training
split. Wilson CI [82.4%, 100.0%]; exact one-sided binomial vs the
pre-registered training-split null p0=84.86% gives p=0.052 — NOT
significant at alpha=0.05, directional by the pre-registered rule
(N_accepted=18 < 100). The SAP v3 fresh-data round subsequently
FAILED to confirm the temperature signal (CLAIM-012): do not cite
this claim as evidence that temperature generalizes.
evidence_level: internal_benchmark
status: superseded
superseded_by: CLAIM-012
artifact:
- "results/selective_n500_holdout_results.json"
n: 18
n_detail: >
N_accepted=18 on the 108-item holdout of the 544-item benchmark
(group-aware 80/20 split, seed 42); tau*=0.4991 locked on the
436-item training split before the holdout was touched.
metrics:
accuracy_pct: 100.0
coverage_pct: 16.7
ci_low_pct: 82.4
ci_high_pct: 100.0
# Written as they appeared before the 2026-07-27 SAP v2 clean-round
# re-issue. Any document still carrying one of these strings outside an
# explicitly historical paragraph is drift, and CI says which document.
# The last three are LaTeX forms: the markdown strings do not match them,
# so until they were added paper/remora_paper.tex passed the gate
# vacuously while still carrying the retired figures into the compiled
# PDF. "1.45" alone is deliberately absent — the chi threshold is 1.45 too.
# (A comment inside the list would terminate it: the register's restricted
# parser breaks on any non-list line.)
retired_values:
- "23.2% coverage"
- "N_accepted=25"
- "N_accepted = 25"
- "[70.0%, 95.8%]"
- "p = 1.45"
- "p=1.45"
- '23.2\%'
- "[70.0, 95.8]"
- "accepted}=25"
caveat: >
Directional observation only: p=0.052 vs the training-split null, the
CI does not exclude p0, and N_accepted=18 is far below the
pre-registered >=100 bar for generalization language. Superseding
evidence: the pre-registered SAP v3 round on 1231 FRESH items
(CLAIM-012) found temperature ranks significantly WORSE than a
calibrated confidence baseline and certifies no SGR coverage —
the signal did not generalize beyond the reused 544-item corpus.
Temperature is diagnostics-grade, not an authoritative selector.
reproduce: >
python scripts/selective_n500_holdout.py
Held-out result artifact: results/selective_n500_holdout_results.json
- id: CLAIM-005
title: "Critical-phase trust inversion (negative result)"
statement: >
Exploratory: on N=32 critical-phase items, trust anti-correlates with
correctness. Low-trust items: 76.2% correct (N=21). High-trust items:
36.4% correct (N=11). Fisher exact p = 0.053, two-sided (post hoc,
artifacts/posthoc/peer_review_level2_v1.json).
evidence_level: internal_benchmark
status: active
artifact:
- "results/critical_trust_split_v1.json"
- "results/selective_n500_results.json"
- "NEGATIVE_RESULTS.md"
n: 32
n_detail: "32 critical-phase items from N=544 benchmark"
metrics:
low_trust_correct_pct: 76.2
high_trust_correct_pct: 36.4
metric_bindings:
low_trust_correct_pct:
path: "results/critical_trust_split_v1.json#low_trust_bucket.accuracy_pct"
high_trust_correct_pct:
path: "results/critical_trust_split_v1.json#high_trust_bucket.accuracy_pct"
caveat: >
Small sample (N=32). Published as a negative result requiring further investigation.
REMORA routes around this failure mode via PhaseAwareGuardrail. Do not cite as
evidence of general trust calibration quality.
reproduce: >
python scripts/compute_critical_trust_split.py
See NEGATIVE_RESULTS.md section 14 (M8)
- id: CLAIM-006
title: "AROMER reached AII=0.8412 TRAINED status (shadow-mode only)"
statement: >
AROMER's Autonomous Intelligence Index (AII) was in the TRAINED band (>=0.80)
for 12 consecutive organic adapt cycles on 2026-06-28, with FAR=0 throughout.
It read 0.8412 at cycle 8 and peaked at AII=0.844, T2=1.000, T3=0.800
(milestone) at cycle 12, then regressed to CAPABLE (AII=0.7885) the same day.
evidence_level: internal_simulation
status: active
artifact:
- "NEGATIVE_RESULTS.md"
n: null
n_detail: "15306+ episodes in AROMER D1 as of 2026-06-28"
caveat: >
AROMER is experimental. deployment_status=SHADOW_ONLY. AII is an adaptive
composite that reflects window composition, not a fixed safety metric. TRAINED
threshold reached via organic recovery after seeding-induced regression; full
§12->§13->recovery cycle documented in NEGATIVE_RESULTS.md. REM-020
(longitudinal stability), REM-022 (RBAC audit) and REM-023 are DONE; the
release profile is CONTROLLED_PILOT by owner decision of 2026-10-25 (REM-021,
independent human review, is not a precondition for it and has not taken
place). AROMER's own status is unchanged by that profile. Do not cite AII as
a safety proof. AROMER is NOT the core governance system; AII metrics do not
imply safety for the REMORA decision engine.
The AII values are live D1 telemetry transcribed into the NEGATIVE_RESULTS.md
§11 table; no committed machine-readable artifact carries them, so no gate
can bind them. The committed snapshot artifacts/aromer/intelligence_after_v020.json
(2026-06-09) predates this run and reads AII=0.5165 (history peak 0.6497,
trend degraded_from_peak); it was listed as evidence until 2026-09-28 and
does not support this claim.
reproduce: >
Not reproducible from this repository. Live AROMER endpoint: GET /intelligence
(requires AROMER worker access). NEGATIVE_RESULTS.md §11-§13 documents the
full recovery trajectory and the regression that followed.
- id: CLAIM-007
title: "Five-condition component ablation: REMORA full gate dominates"
statement: >
WITHDRAWN 2026-08-28. The claim asserted FAR=30%/10%/25% for conditions
A/B/C and that zero FAR requires both the structural gates and the proxy
thermodynamic policy. Both halves are contradicted by committed evidence
in this repository. See CLAIM-020 for what the ablation actually supports.
evidence_level: internal_benchmark
status: superseded
superseded_by: CLAIM-020
artifact:
- "artifacts/aromer/component_ablation_results.json"
n: 700
n_detail: "Historical record only; see NEGATIVE_RESULTS.md section 57."
metrics: {}
caveat: >
Withdrawn for three independent reasons, each verifiable from this commit.
Sourcing: the statement attributed A=0.300 and B=0.100 to
results/toolcall_benchmark_v2_results.json, which records 0.0142857 for
every heuristic baseline. Reproducibility: the reproduce command
(experiments/toolcall_ablation_v2.py) writes
results/toolcall_benchmark_v2_ablation.json and
results/toolcall_ablation_v2_results.json, never the artifact this claim
names, which has not changed since 2026-07-02. Identification: the
necessity half is falsified by the repository's own ablation, where
remora_without_temperature reaches FAR=0.0.
reproduce: >
Withdrawn; do not cite. To see the contradiction:
python -c "import json;d=json.load(open('results/toolcall_benchmark_v2_results.json'));print({k:v['false_accept_rate'] for k,v in d['baselines'].items()})"
python experiments/toolcall_ablation_v2.py
- id: CLAIM-008
title: "Selective trust curve (N=302): 94.7% accuracy at 25% coverage"
statement: >
On a 302-item calibration set using the neg_temperature signal, the top-25%
slice achieves k=76, correct=72, accuracy=94.7%.
evidence_level: internal_benchmark
status: superseded
superseded_by: CLAIM-013
artifact:
- "results/selective_trust_curve_results.json"
n: 302
n_detail: "N=302 items; neg_temperature signal; coverage=25%"
metrics:
accuracy_pct: 94.7
coverage_pct: 25.0
caveat: >
Calibration-set result (not held-out). The baseline accuracy is 82.78% on this set.
In-distribution optimism expected. Do not quote 94.7% without noting it is a
calibration-set upper bound. Held-out validation uses the N500 artifact (CLAIM-004).
reproduce: >
python experiments/selective_trust_curve.py
Artifact: results/selective_trust_curve_results.json
- id: CLAIM-009
title: "FA=30.7% on external adversarial datasets under neutral metadata (negative result)"
statement: >
Across 1036 combined harmful scenarios from aradhye/agent-safety-bench and
CaiZhiTech/Evaluation-Dataset under neutral trust metadata (trust=0.70),
AROMER /decide achieves FA rate of 43.0% (Phase 1, structural only) and
30.7% after semantic enrichment (Phase 2, -12.3 pp reduction).
evidence_level: internal_benchmark
status: active
artifact:
- "artifacts/aromer/external_dataset_eval.json"
- "artifacts/aromer/external_dataset_eval_v2.json"
n: 1036
n_detail: "aradhye: 184 harmful / 216 benign; CaiZhiTech: 852 harmful / 166 benign"
caveat: >
This is a NEGATIVE RESULT. It documents the metadata-dependency vulnerability:
AROMER's structural policy achieves FAR=0% only when callers supply accurate
high-risk metadata. With neutral-looking metadata, harmful scenarios are
indistinguishable from benign ones at the structural layer. The residual 30.7%
FA gap requires runtime execution monitoring or world-model seeding from
real-world labeled patterns. This result must NOT be removed or suppressed.
Semantic enrichment also raises false-block rate (74.1% vs 70.9% Phase 1).
reproduce: >
Artifacts committed 2026-07-03 (restored from the main implementation
repo): artifacts/aromer/external_dataset_eval.json (Phase 1, FA=43.0%),
artifacts/aromer/external_dataset_eval_v2.json (Phase 2, FA=30.7%).
Re-running against the live datasets requires AROMER worker access.
- id: CLAIM-010
title: "Blinded benchmark v3: FAR=0% without label access (N=700)"
statement: >
On the blinded toolcall benchmark v3 (CandidateAction separated from
EvaluationTruth at file level), REMORA gate achieves FAR=0.0000 on N=700
with leakage_free=True. The gate only receives CandidateAction; scorer loads
labels separately by task_id.
evidence_level: regression_tested
status: active
artifact:
- "results/toolcall_blind_v3_results.json"
- "benchmarks/toolcall_blind_v3/tasks.json"
- "benchmarks/toolcall_blind_v3/labels.json"
n: 700
n_detail: "700 tasks (560 harmful, 140 benign); blinded separation confirmed by 10 invariant tests"
caveat: >
Simulator-scoped: same synthetic benchmark as v2, with stricter label separation.
Context flags are correlated with harmfulness by benchmark construction; if all
harmful tasks have at least one detectable signal by construction, the result
may document a tautology rather than generalization. External replication with
independently withheld labels required for definitive resolution.
reproduce: >
python experiments/toolcall_blind_v3_eval.py
Artifact: results/toolcall_blind_v3_results.json
- id: CLAIM-011
title: "Anytime-valid FA-rate bound for REM-020 monitoring (cycle level)"
statement: >
Over the REM-020 monitoring window, 0 of 168 analyzed adapt cycles
contained a false accept. The Beta-mixture confidence sequence gives a
95% time-uniform upper bound of 4.72% on the per-cycle FA-indicator
rate — valid under continuous monitoring and data-dependent stopping
(Ville's inequality), unlike the fixed-N Wilson intervals reported
elsewhere in this repository.
evidence_level: theoretical
status: active
artifact:
- "results/far_confidence_sequence_v1.json"
- "results/superseded_rem020_cycle_window_2026-06-30/longitudinal_stability_v1.json"
n: 168
n_detail: "168 adapt cycles from the REM-020 window (longitudinal_stability_v1 as of 2026-06-30, archived when the live record moved to the 7-day criterion on 2026-07-17)"
metrics:
upper_bound_pct: 4.72
metric_bindings:
upper_bound_pct:
unbound: >-
confidence-sequence bound computed by the analysis script; the artifact stores the sequence inputs, not the bound.
caveat: >
Cycle-level bound on the indicator "any false accept in this cycle" —
NOT a per-decision FAR bound (per-decision counts are not aggregated in
the input artifact). The method's validity is theorem-backed
(Ville 1939; Howard et al. 2021); the numeric bound inherits the
internal-telemetry provenance of the input artifact. Time-uniformity
costs width: the bound is ~2x wider than a fixed-N Wilson interval at
the same N — this is the price of surviving optional stopping, and is
the reason it is safe for a gate that closes on a data-dependent date.
reproduce: >
python scripts/compute_far_confidence_sequence.py
Artifact: results/far_confidence_sequence_v1.json.
Library: remora/selective/confidence_sequence.py (tests:
tests/test_confidence_sequence.py, including a seeded demonstration
that per-step Wilson monitoring violates its nominal level).
- id: CLAIM-012
title: "NEGATIVE: consensus temperature failed pre-registered fresh-data confirmation (SAP v3)"
statement: >
On the pre-registered SAP v3 round (1231 fresh BoolQ/TruthfulQA items,
deduplicated against the prior 544-item corpus; frozen Workers AI
cross-family trio; three-way group-aware split dev 493 / risk-cal 370 /
test 368, seed 20260727), the consensus-temperature signal FAILED
confirmation: test-split AURC 0.0954 vs 0.0664 for a
dev-split-calibrated mean-confidence baseline — paired bootstrap delta
0.0290 with 95% CI [0.0119, 0.0503], excluding zero (temperature ranks
significantly WORSE) — and SGR certification at r*=5%, delta=0.10 found
NO certifiable coverage. Temperature's CRC gate (alpha=5%) saw an
empirical test exceedance (6.5% unconditional) — a validation
exceedance, not proven assumption violation. The exploratory
temperature advantage observed on the reused 544-item corpus did not
transfer: temperature is demoted to diagnostics and must not be cited
as a validated selection signal.
evidence_level: internal_benchmark
status: active
artifact:
- "results/sap_v3_round_results.json"
- "results/sap_v3_collection.json"
n: 1231
n_detail: >
1231 fresh items (89 TruthfulQA + 1142 BoolQ); confirmatory numbers
computed once on the untouched 368-item test split.
metrics:
temperature_aurc: 0.0954
confidence_aurc: 0.0664
delta_ci_low: 0.0119
delta_ci_high: 0.0503
metric_bindings:
temperature_aurc:
unbound: >-
AURC computed by the selective-prediction analysis; the round artifact stores per-item outcomes, not the summary curve.
confidence_aurc:
unbound: >-
AURC computed by the selective-prediction analysis; the round artifact stores per-item outcomes, not the summary curve.
delta_ci_low:
unbound: >-
paired bootstrap bound computed by the analysis; not stored in the round artifact.
delta_ci_high:
unbound: >-
paired bootstrap bound computed by the analysis; not stored in the round artifact.
caveat: >
Corpus is ~93% BoolQ and ~93% easy-classified with 2.2% adversarial —
evidence about selective QA on this distribution, not about tool
execution or safety-critical false accepts. "Fresh" means unused in
REMORA's prior corpus; the public source datasets may appear in model
pretraining. The negative finding is robust to the PAV calibration fix
(SAP v3 s8 D-2): temperature's failure is independent of the
confidence-calibration code path.
reproduce: >
python experiments/sap_v3_round.py analyze
(replays offline from results/sap_v3_collection.json; zero API calls)
- id: CLAIM-013
title: "Calibrated confidence methods on fresh data: significant aggregation win; marginal per-arm certificates only"
statement: >
Same SAP v3 round: calibrated confidence-weighted voting scored 87.8%
vs 85.1% for unweighted majority on the untouched test split — paired
exact McNemar 11-1 discordants, p=0.0064 (significant); vs the
dev-selected best single model the difference is directional only
(p=0.077). Selection arms: calibrated mean confidence SGR-certified
31.9% risk-cal coverage (test point estimate 5.30%) and
margin+calibrated-confidence 39.7% (test point 4.97% at 43.8%
coverage) — MARGINAL per-arm certificates at 90% confidence. Under
family-wise Bonferroni-3 (selection among three parallel arms) NO arm
certifies; the hybrid arm is a promising pre-registered secondary that
requires its own frozen confirmation round before any engine
integration (SAP v3 s8 D-3).
evidence_level: internal_benchmark
status: active
artifact:
- "results/sap_v3_round_results.json"
n: 368
n_detail: >
Test split of the 1231-item fresh corpus; certificates fitted on the
370-item risk-calibration split; calibration fitted on the 493-item
development split only.
metrics:
cw_accuracy_pct: 87.8
majority_accuracy_pct: 85.1
mcnemar_p: 0.0064
hybrid_certified_coverage_pct: 39.7
hybrid_test_risk_pct: 4.97
metric_bindings:
cw_accuracy_pct:
unbound: >-
accuracy recomputed from per-item outcomes by the analysis; not stored as a summary field.
majority_accuracy_pct:
unbound: >-
accuracy recomputed from per-item outcomes by the analysis; not stored as a summary field.
mcnemar_p:
unbound: >-
test statistic computed by the analysis from the discordant pairs in the artifact.
hybrid_certified_coverage_pct:
unbound: >-
coverage recomputed from the accepted set; not stored as a summary field.
hybrid_test_risk_pct:
unbound: >-
risk recomputed from the accepted set; not stored as a summary field.
caveat: >
The test set alone is not precise enough to establish that the
hybrid's true selective risk is below 5% (risk CI roughly 2.5%-9.5%
from 8/161); the point estimate met the budget and the risk-cal
certificate is valid marginally per arm. No selective statistical
gate is integrated into the execution engine from this round. Same
corpus-scope caveats as CLAIM-012.
reproduce: >
python experiments/sap_v3_round.py analyze
(replays offline from results/sap_v3_collection.json; zero API calls)
- id: CLAIM-014
title: "System demonstration: governance chain from tool call to enforcement"
statement: >
A single command (python scripts/run_system_demonstration.py) exercises the
full governance chain — structure and policy, state coverage and freshness,
argument-value provenance, ACCEPT/VERIFY/ABSTAIN/ESCALATE, named validator,
re-entry, and enforcement — and produces one reproducible artifact. Under
study conditions: verified reads are accepted autonomously, writes route to
VERIFY, UNKNOWN state resolves through named validators, and context-foreign
calls lose autonomy. All seven pre-registered degradation expectations met.
All eight validator targets met. Validator loop restored read utility from
0% to 100%. None of 139 seeded corruption forms reached ACCEPT.
evidence_level: internal_benchmark
status: active
blindness: development
artifact:
- "results/system_demonstration_v1.json"
- "results/fleetops_degradation_results.json"
- "results/fleetops_validator_study_results.json"
n: 7
n_detail: >
7 degradation conditions (fleetops domain, mechanism study, non-blind);
validator study N is study-internal (not independently published).
metrics:
degradation_conditions_n: 7
degradation_all_expectations_met: true
validator_targets_n: 8
validator_all_targets_met: true
read_utility_without_validators: 0.0
read_utility_with_validators: 1.0
seeded_corruption_forms_n: 139
corrupt_accept_rate: 0.0
valid_id_completion_read: 1.0
metric_bindings:
valid_id_completion_read:
path: "results/system_demonstration_v1.json#validator_study.targets.valid_id_completion_read.value"
degradation_conditions_n:
path: "results/system_demonstration_v1.json#degradation_study.n_conditions"
validator_targets_n:
unbound: >-
count of targets in the study protocol, not a field in the results artifact.
read_utility_without_validators:
path: "results/system_demonstration_v1.json#validator_study.read_utility_without_validators"
read_utility_with_validators:
path: "results/system_demonstration_v1.json#validator_study.read_utility_with_validators"
seeded_corruption_forms_n:
unbound: >-
count of seeded forms in the generator, not a field in the results artifact.
corrupt_accept_rate:
path: "results/system_demonstration_v1.json#validator_study.targets.corrupt_id_accept_after_resolver.value"
utility_floors:
- metric: valid_id_completion_read
min: 0.85
source: "results/system_demonstration_v1.json"
source_detail: >-
validator_study.targets.valid_id_completion_read, target ">= 0.85",
one of the eight validator targets the statement reports as met.
status: met
caveat: >
Open (non-blind) mechanism study on the generated fleetops domain.
All three sub-studies are mechanism studies, not blind holdout evaluations.
Results demonstrate architectural behaviour under controlled conditions; they
are not evidence of generalisation to external domains or field deployments.
false_absent_on_valid=0 is by construction (study validator consults live
world directly). Do not cite as production-readiness evidence.
reproduce: >
python scripts/run_system_demonstration.py
(deterministic; no API keys; one unified artifact at
results/system_demonstration_v1.json)
- id: CLAIM-015
title: "Superseded BFCL v3 development measurement"
statement: >
Historical post-hoc measurements on the spent BFCL v3 set are superseded
by the sealed BFCL v4 result in CLAIM-019.
evidence_level: internal_benchmark
status: superseded
superseded_by: CLAIM-019
blindness: development
artifact:
- "NEGATIVE_RESULTS.md"
n: null
n_detail: "Historical record only; see NEGATIVE_RESULTS.md section 34."
metrics: {}
caveat: >
Do not cite or reproduce the spent-set development figures as current
evidence. The active external result is CLAIM-019.
reproduce: "Not rerunnable; the BFCL v3 set is spent."
- id: CLAIM-016
title: "Sealed BFCL v3 negative record: four of five targets met; superseded by CLAIM-019"
statement: >
Track C-ext was evaluated once, at locked commit cf02fa8, on sealed
external data the system had never seen: BFCL v3 live categories
(ShishirPatil/gorilla @ c15b2a15, Apache-2.0), 1509 episodes over 515
clusters, under no authority at all (empty state index, no validator
bindings, registry derived only from the tasks' own schemas). Four
pre-registered targets were met: required-but-unknown autonomous ACCEPT
0/19 = 0.0% (target <=0%), irrelevance ABSTAIN recall 258/258 = 100.0%
(target >=70%), unobtainable-argument ABSTAIN recall 133/133 = 100.0%
(target >=70%), obtainable-argument VERIFY recall 110/133 = 82.7%
(target >=70%). Labelled routing accuracy is 94.0% (n=1251,
cluster-level Wilson 95% CI [91.6%, 95.7%]); both untrusted-provenance
families routed at 100% (257/257 ESCALATE controls-sensitive, 213/213
VERIFY noncontrolling). The fifth target MISSED and is published as
measured: known-wrong-call ACCEPT 224/258 = 86.8% against a <=20% bar —
a substituted call carrying its own complete, well-formed arguments
gives the structural signals nothing to distrust.
evidence_level: externally_benchmarked
status: superseded
superseded_by: CLAIM-019
blindness: blind
artifact:
- "results/routing_bench_bfcl_results.json"
- "data/routing_bench_bfcl/manifest.json"
n: 1509
n_detail: >
1509 episodes over 515 clusters; 1251 labelled, 258 unlabelled
(irrelevance episodes propose no call, so their ABSTAIN measures refusal
of the no-call decision point). Statistics are cluster-adjusted.
metrics:
routing_accuracy_pct: 94.0
irrelevance_abstain_pct: 100.0
unobtainable_abstain_pct: 100.0
obtainable_verify_pct: 82.7
required_unknown_accept_pct: 0.0
wrong_call_accept_pct: 86.8
n_clusters: 515
caveat: >
Evaluated once; this set is now spent and can never serve as a blind set
again. The wrong-argument value axis was excluded BEFORE sealing
(admission verdict recorded in the manifest): BFCL's ground-truth
argument lists are the labels, so using them as a system of record would
score the answer key against itself. The known-wrong-call miss is a
measured architectural limit, not a tuning defect — closing it needs an
authoritative task-call semantic source for tool_matches_goal, and no
threshold change can substitute. A partial mitigation was measured
afterwards on this same spent set and is registered separately as
CLAIM-015 (development, NOT blind). Full analysis: NEGATIVE_RESULTS.md
§34.
reproduce: >
python scripts/build_bfcl_holdout.py && python scripts/run_bfcl_holdout.py
Compare to results/routing_bench_bfcl_results.json (holdout_sha256
c3a8e27b..., upstream file hashes pinned in
data/routing_bench_bfcl/manifest.json).
- id: CLAIM-017
title: "Semantic binding gap in match_tool_to_intent: source_spans verified presence, not entailment (finding registered 2026-08-04)"
statement: >
Before 2026-08-04, TaskIntent.source_spans verified only that the
quoted text appeared in the task. A model could claim
requested_effect="cancel" with source_spans=("booking B-104",) for a
task saying "Show me booking B-104" and reach SUPPORTED: the entity
span verified, the resource type matched, the effect label matched the
contract. The fix adds action_spans (a separate, mandatory effect-grounding
field), EFFECT_VOCABULARY v1 (frozen keyword mapping), and
negation/conditionality detection. Seven pinning tests cover the attack
surface. Effect on benchmark metrics is unmeasured pending the new
sealed OT track; the BFCL population is spent.
evidence_level: regression_tested
status: active
artifact:
- "remora/toolcall/routing/goal_match.py"
- "tests/test_goal_match.py"
caveat: >
The fix is in code and regression-tested (28 tests, 7 new). Its effect
on the §34 residue (30/258 wrong-call accepts after value grounding) is
unmeasured. CAP-014 and CLAIM-016 are not affected: those claims were
made under conditions where no TaskIntent or ToolContract was supplied,
so tool_matches_goal=None for all episodes and the gap was inert.
SHELF-020 wiring landed 2026-08-04 (REMORA_SEMANTIC_BUNDLE_MODULE;
tests/test_execution_semantic_wiring.py); discrimination through the
wired path is unmeasured pending SAP v4.
reproduce: >
python -m pytest tests/test_goal_match.py -v
- id: CLAIM-018
title: "Disjoint sealed BFCL v4 confirmation: all five pre-registered routing targets met"
statement: >
Track C-ext2 was evaluated once on a sealed, ID-disjoint BFCL v4 sample
at ShishirPatil/gorilla commit 6ea57973c7a6 (Apache-2.0): 258
live_multiple tasks and 258 live_irrelevance tasks not present in the
spent BFCL v3 population. Across 1,527 episodes, all five unchanged
pre-registered targets met: required-but-unknown autonomous ACCEPT 0/32
= 0.0%; irrelevance ABSTAIN 258/258 = 100.0%; obtainable-argument VERIFY
96/99 = 97.0%; unobtainable-argument ABSTAIN 98/99 = 99.0%; and
known-wrong-call ACCEPT 28/258 = 10.9% against a <=20% bar. Labelled
routing accuracy was 91.2% (n=1,170; cluster-level Wilson 95% CI
[88.4%, 93.3%]).
evidence_level: externally_benchmarked
status: superseded
superseded_by: CLAIM-019
blindness: blind
artifact:
- "results/routing_bench_bfcl_v4_results.json"
- "data/routing_bench_bfcl_v4/manifest.json"
n: 1527
n_detail: >
1,527 episodes over 516 source clusters (508 effective labelled
clusters); 1,170 labelled and 357 unlabelled. Source-task ID overlap
with the spent BFCL v3 holdout is zero.
metrics:
routing_accuracy_pct: 91.2
irrelevance_abstain_pct: 100.0
unobtainable_abstain_pct: 99.0
obtainable_verify_pct: 97.0
required_unknown_accept_pct: 0.0
wrong_call_accept_pct: 10.9
optional_identity_accept_pct: 69.3
n_clusters: 508
caveat: >
Evaluated once; this BFCL v4 sample is now spent. The wrong-argument
value axis was excluded before sealing because BFCL provides no
independently vouchable state table. The track supplies no authoritative
TaskIntent/ToolContract bundle, so it confirms the current routing
pipeline and value-grounding mitigation, not isolated causal efficacy of
semantic intent matching. The original BFCL v3 86.8% miss remains an
immutable negative result under CLAIM-016.
reproduce: >
The sealed result must not be regenerated. Verify hashes and status in
data/routing_bench_bfcl_v4/manifest.json and run
python -m pytest tests/test_routing_claim_artifacts.py -v.
- id: CLAIM-019
title: "Sealed C-ext3: semantic authority eliminates native wrong-tool acceptance; four of seven targets missed"
statement: >
Track C-ext3 was evaluated once on a sealed, ID-disjoint BFCL v4 sample
(500 live_multiple clusters + 300 live_irrelevance tasks, disjoint from
both spent populations, 2,799 episodes) under the frozen deterministic
semantic bundle with the semantic-authority floor. Met: native non-gold
call ACCEPT 0/500 = 0.0% (Wilson 95% upper bound 0.76% by row, 3.5%
clustered over the 106 proposed tools, post hoc; C-ext2
degraded-authority baseline 28/258 = 10.9%); irrelevance ABSTAIN 300/300
= 100.0%; required-unknown autonomous ACCEPT 0/398 = 0.0%. Missed and
published as measured: constructed wrong-tool ACCEPT 2/199 = 1.005%
(bar 1%); legitimate read autonomy 25/94 = 26.6% (bar 75%); obtainable
VERIFY 93/199 = 46.7%; unobtainable ABSTAIN 126/199 = 63.3%. Single-pass
ablation on the same episodes: structural-only 24/500 wrong-call
accepts, contracts+intent 6/500, with the UNKNOWN floor 0/500.
evidence_level: externally_benchmarked
status: active
blindness: blind
artifact:
- "results/routing_bench_bfcl_v4_cext3_results.json"
- "data/routing_bench_bfcl_v4_cext3/manifest.json"
n: 2799
n_detail: >
2,799 episodes over 800 source clusters (500 positive + 300
irrelevance), seed 20260820 drawn after the semantic bundle was frozen
and hashed into the manifest. Source-task ID overlap with the spent v3
and C-ext2 populations is zero.
metrics:
wrong_call_accept_pct: 0.0
wrong_call_wilson_upper_pct: 0.76
constructed_wrong_tool_accept_pct: 1.005
irrelevance_abstain_pct: 100.0
required_unknown_accept_pct: 0.0
legitimate_read_autonomy_pct: 26.6
obtainable_verify_pct: 46.7
unobtainable_abstain_pct: 63.3
n_clusters: 800
metric_bindings:
wrong_call_accept_pct:
path: "results/routing_bench_bfcl_v4_cext3_results.json#targets.known_wrong_call_accept.value"
scale: 100
wrong_call_wilson_upper_pct:
path: "results/routing_bench_bfcl_v4_cext3_results.json#targets.known_wrong_call_accept.wilson95[1]"
scale: 100
constructed_wrong_tool_accept_pct:
unbound: >-
percentage recomputed from the arm's n/d counts by the analysis.
irrelevance_abstain_pct:
path: "results/routing_bench_bfcl_v4_cext3_results.json#targets.irrelevance_abstain_recall.value"
scale: 100
required_unknown_accept_pct:
path: "results/routing_bench_bfcl_v4_cext3_results.json#targets.required_unknown_auto_accept.value"
scale: 100
legitimate_read_autonomy_pct:
path: "results/routing_bench_bfcl_v4_cext3_results.json#targets.legitimate_read_autonomy.value"
scale: 100
obtainable_verify_pct:
path: "results/routing_bench_bfcl_v4_cext3_results.json#targets.obtainable_verify_recall.value"
scale: 100
rounded_to: 1
unobtainable_abstain_pct:
path: "results/routing_bench_bfcl_v4_cext3_results.json#targets.unobtainable_abstain_recall.value"
scale: 100
rounded_to: 1
n_clusters:
path: "data/routing_bench_bfcl_v4_cext3/manifest.json#n_clusters"
utility_floors:
- metric: legitimate_read_autonomy_pct
min: 75.0
source: "docs/assurance/statistical_analysis_plan_v5_bfcl_semantic.md"
source_detail: "T2 legitimate read autonomy, pre-registered before sealing."
status: missed
- metric: obtainable_verify_pct
min: 95.0
source: "docs/assurance/statistical_analysis_plan_v5_bfcl_semantic.md"
source_detail: "obtainable VERIFY recall, pre-registered before sealing."
status: missed
- metric: unobtainable_abstain_pct
min: 95.0
source: "docs/assurance/statistical_analysis_plan_v5_bfcl_semantic.md"
source_detail: "unobtainable ABSTAIN recall, pre-registered before sealing."
status: missed
caveat: >
Evaluated once; the sample is spent. The accept property is
SEMANTICALLY AUTHORIZED READ (empty state index, no value grounding
against a system of record) — explicitly weaker than SAP v4 grounded
read. Contract authoring and intent extraction are deterministic
functions frozen before sampling, but their grammar was designed with
spent C-ext2 material visible (SAP v5 §12 deviation 2) — weaker than
three-human role separation. Four of seven pre-registered targets were
missed (NEGATIVE_RESULTS.md §39): the deterministic extractor caps read
autonomy at 26.6% and the semantic gates preempt argument routing. The
C-ext2 10.9% baseline remains permanent under CLAIM-018.
reproduce: >
The sealed result must not be regenerated. Verify hashes and status in
data/routing_bench_bfcl_v4_cext3/manifest.json; SAP:
docs/assurance/statistical_analysis_plan_v5_bfcl_semantic.md.
- id: CLAIM-020
title: "Component ablation has no discriminating power on false-accept rate"
statement: >
On toolcall_benchmark_v2 (700 rows, 70 template clusters), all six
committed ablation conditions record false_accept_rate = 0.0, including
remora_without_hard_blocks. The ablation therefore identifies no component
as necessary for the zero-false-accept result. What it does separate is
task accuracy (0.90 full, 0.8714 without_phase, 0.30 without_hard_blocks)
and benign review friction (0.50 full, 0.6429 without_phase).
evidence_level: internal_benchmark
status: active
artifact:
- "results/toolcall_benchmark_v2_ablation.json"
n: 700
n_detail: >
700 rows over 70 template clusters (10 cosmetic variants each); the
artifact records effective_n=56 per condition, the harmful template
clusters (it recorded 7, the domain count, until the 2026-09-28 scorer
fix). Cluster structure means row-level counts overstate independence.
metrics:
false_accept_rate_full: 0.0
false_accept_rate_without_hard_blocks: 0.0
accuracy_full: 0.90
accuracy_without_hard_blocks: 0.30
review_friction_benign_full: 0.50
metric_bindings:
false_accept_rate_full:
path: "results/toolcall_benchmark_v2_ablation.json#ablations.remora_full.false_accept_rate"
false_accept_rate_without_hard_blocks:
# The pair is the argument: the condition with the deterministic floor
# removed records the same zero as the full gate. One path cannot show
# "all six conditions", so the register names the two the statement
# actually leans on rather than implying a coverage it cannot bind.
path: "results/toolcall_benchmark_v2_ablation.json#ablations.remora_without_hard_blocks.false_accept_rate"
accuracy_full:
path: "results/toolcall_benchmark_v2_ablation.json#ablations.remora_full.accuracy"
accuracy_without_hard_blocks:
path: "results/toolcall_benchmark_v2_ablation.json#ablations.remora_without_hard_blocks.accuracy"
review_friction_benign_full:
path: "results/toolcall_benchmark_v2_ablation.json#ablations.remora_full.review_friction_benign"
utility_floor_exempt: >-
A component ablation with no pre-registered utility bar. Accuracy and
benign review friction are reported per arm (0.90 and 0.50 for the full
gate) as comparisons, not as floors.
caveat: >
This is a negative identification result, not a safety claim. FAR=0 under
remora_without_hard_blocks is reached by refusing almost everything:
accuracy collapses to 0.30, so the condition buys its zero by declining
the task rather than by judging it correctly. A metric that a degenerate
refuse-all policy also satisfies cannot attribute the outcome to any
component. Any future necessity claim requires an ablation whose