Skip to content

Commit 544a96b

Browse files
abrignoniclaude
abrignoni
and
claude
committed
Check artifact notes for unsupported claim language
Brings this core in line with ALEAPP abrignoni#1242. Extends check_claim_language.py to the notes field, which reaches the report and the LAVA manifest and was not covered, and closes the habit stem in the claim vocabulary: the open spelling also matched "habitat". Notes get their own vocabulary. The completeness words come out, because a note legitimately states what it was tested against, and "read by" comes out because in a note it means read by the code rather than by a person. A match preceded by a negation inside the same clause is not reported; suppressed matches are counted and printed under --verbose rather than dropped silently. The new test pins the vocabulary to a literal and is byte-identical in all five cores, so a change made in one repo alone fails that repo's own CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 4125908 commit 544a96b

5 files changed

Lines changed: 282 additions & 14 deletions

File tree

‎admin/scripts/check_claim_language.py‎

Lines changed: 97 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -107,11 +107,70 @@
107107
r'|\balways\b'
108108
r'|\breliable'
109109
r'|\bvisited\b'
110-
r'|\bhabit',
110+
r'|\bhabits?\b',
111111
re.IGNORECASE)
112112

113+
# `notes` reaches the examiner too, in the report and in the artifact info modal, so
114+
# the same standard applies to it. It cannot use the same vocabulary, because notes do
115+
# a job name and description do not: they state what was tested. "empty on all 18
116+
# copies tested" and "NULL for every account tested" are the coverage discipline this
117+
# project asks for, not claims about a person, and the completeness words fire on every
118+
# one of them. Measured 2026-08-29: the full vocabulary flags 368 of the 1,477
119+
# artifacts carrying notes across the five cores, dominated by `every` and `all`,
120+
# almost all describing a test set.
121+
#
122+
# `read by` comes out for the same reason. In a note it means read by the code, not by
123+
# a person: "columns are read by position", "not read by this artifact".
124+
#
125+
# What is left is attribution and certainty, which mean the same thing in a note as in
126+
# a description.
127+
NOTES_PATTERN = re.compile(
128+
r'\bthe user (?:searched|typed|viewed|visited|opened|selected|deleted|read|sent'
129+
r'|created|hid|chose)\b'
130+
r'|\buser[- ](?:created|entered|typed|searched|selected|initiated)\b'
131+
r'|\bsearched by\b'
132+
r'|\btyped by\b'
133+
r'|\bviewed by\b'
134+
r'|\bmanually\b'
135+
r'|\bproves?\b'
136+
r'|\bdefinitively\b'
137+
r'|\balways\b'
138+
r'|\breliable'
139+
r'|\bvisited\b'
140+
r'|\bhabits?\b',
141+
re.IGNORECASE)
142+
143+
# A note that *denies* a claim uses the same words as one that makes it: "not terms the
144+
# user searched for", "does not establish that the user viewed them". That denial is the
145+
# wording this project asks for, so matching it and demanding an allowlist entry would
146+
# tax the correct behaviour and grow the allowlist without bound. A match in `notes`
147+
# preceded by a negation inside the same clause is therefore not reported.
148+
#
149+
# The window is deliberately short. A negation two sentences back says nothing about
150+
# this clause, and a long window would swallow real claims. Suppressed matches are
151+
# counted and printed under --verbose, because a check that narrows its own scope
152+
# silently is worse than no check.
153+
NEGATION_WINDOW = 60
154+
NEGATION_PATTERN = re.compile(
155+
r"\b(not|no|never|nor|neither|without|cannot|rather than|instead of"
156+
r"|isn't|doesn't|don't|does not|do not)\b",
157+
re.IGNORECASE)
158+
159+
160+
def negated(text, start):
161+
"""True when a negation appears close enough before `start` to govern it."""
162+
window = text[max(0, start - NEGATION_WINDOW):start]
163+
# A sentence boundary ends the clause, so a negation before it does not govern.
164+
window = window.rsplit('. ', 1)[-1]
165+
return bool(NEGATION_PATTERN.search(window))
166+
167+
113168
# Fields that reach the examiner through the report and the LAVA manifest.
114-
CHECKED_FIELDS = ('name', 'description')
169+
CHECKED_FIELDS = {
170+
'name': CLAIM_PATTERN,
171+
'description': CLAIM_PATTERN,
172+
'notes': NOTES_PATTERN,
173+
}
115174

116175
# Reviewed exceptions, keyed by (filename, artifact_key, field). Every entry
117176
# needs a comment justifying it. See the module docstring before adding one.
@@ -154,6 +213,26 @@
154213
# A Foursquare list is user-created by the definition of the feature, and
155214
# the artifact reads the account holder's own list table.
156215
('foursquareSwarm.py', 'foursquare_swarm_saved_lists', 'description'),
216+
217+
# "one row per visited page" names the record ZBROWSINGHISTORYENTRYMANAGEDOBJECT
218+
# holds, alongside the aggregate visit count the app's own column carries. It
219+
# describes the table, not what the account holder did.
220+
('duckduckgo.py', 'duckduckgo_history', 'notes'),
221+
222+
# "enumerated beside an account reads as something the user chose" is the reason
223+
# the downloaded catalogue is summarised rather than listed. The denial follows
224+
# the phrase instead of preceding it, so the negation lookback cannot see it.
225+
('googleSheets.py', 'google_sheets_templates', 'notes'),
226+
227+
# "originally typed by the user" is quoted verbatim from the CREATE TABLE comment
228+
# stored in Apple's own database, which the note says it is quoting.
229+
('keyboard.py', 'keyboardAutocorrectionRejections', 'notes'),
230+
('keyboard.py', 'keyboardInlineCompletionRejections', 'notes'),
231+
232+
# "It does not establish that the profile was displayed to the user, that the user
233+
# viewed it, or that the user acted on it" is a denial governing a list. The
234+
# negation sits before the first item and the lookback does not span to the third.
235+
('tinder.py', 'tinderRecommendations', 'notes'),
157236
}
158237

159238
STANDARD_NOTE = (
@@ -207,30 +286,36 @@ def load_artifacts(path):
207286

208287

209288
def scan_file(path):
210-
"""Return (matches, skip_reason) for one artifact module.
289+
"""Return (matches, skip_reason, negated_count) for one artifact module.
211290
212291
Each match is a (rel_path, artifact_key, field, text, matched_terms,
213292
allowlisted) tuple.
214293
"""
215294
artifacts, skip_reason = load_artifacts(path)
216295
if artifacts is None:
217-
return [], skip_reason
296+
return [], skip_reason, 0
218297

219298
filename = os.path.basename(path)
220299
matches = []
300+
negated_count = 0
221301
for artifact_key, entry in artifacts.items():
222302
if not isinstance(entry, dict):
223303
continue
224-
for field in CHECKED_FIELDS:
304+
for field, pattern in CHECKED_FIELDS.items():
225305
text = entry.get(field)
226306
if not isinstance(text, str):
227307
continue
228-
terms = CLAIM_PATTERN.findall(text)
308+
hits = list(pattern.finditer(text))
309+
if field == 'notes':
310+
kept = [hit for hit in hits if not negated(text, hit.start())]
311+
negated_count += len(hits) - len(kept)
312+
hits = kept
313+
terms = [hit.group(0) for hit in hits]
229314
if not terms:
230315
continue
231316
allowlisted = (filename, str(artifact_key), field) in ALLOWLIST
232317
matches.append((path, str(artifact_key), field, text, terms, allowlisted))
233-
return matches, None
318+
return matches, None, negated_count
234319

235320

236321
def repo_root():
@@ -268,9 +353,11 @@ def main():
268353
allowlisted = []
269354
skipped = []
270355
fired = set()
356+
negated_total = 0
271357
for path in paths:
272358
rel_path = os.path.relpath(path, root)
273-
matches, skip_reason = scan_file(path)
359+
matches, skip_reason, negated_here = scan_file(path)
360+
negated_total += negated_here
274361
if skip_reason:
275362
skipped.append((rel_path, skip_reason))
276363
continue
@@ -301,6 +388,8 @@ def main():
301388
f'{len(skipped)} skipped.')
302389
print(f'Allowlist holds {len(ALLOWLIST)} entr(ies); {len(allowlisted)} fired '
303390
f'this run.')
391+
print(f'{negated_total} match(es) in notes were preceded by a negation and '
392+
f'not reported.')
304393
print()
305394

306395
if stale:

‎admin/test/scripts/test_check_claim_language.py‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
"""Prove the claim checker reads notes, reads them with the right vocabulary, and
2+
that the vocabulary is the same one every core enforces.
3+
4+
Extending the check to `notes` is only worth having if three things hold at once, and
5+
each fails silently on its own:
6+
7+
* notes are actually matched, or the field is nominally covered and never read;
8+
* the completeness words do not apply to notes, or the check fires on every artifact
9+
that states what it was tested against, which is the wording this project asks for;
10+
* a denial is not treated as a claim, or the same wording is taxed and the allowlist
11+
grows every time somebody writes a limitation down correctly.
12+
13+
The vocabulary is pinned to a literal here. The five cores each carry their own copy of
14+
the checker, and they had already drifted: four spelled the habit stem open, so
15+
"habitat" matched, and only one had it closed. A checker that quietly enforces a
16+
different standard per repo is worse than a strict one, and nothing else compares them.
17+
This file is identical in all five, so a change made in one repo alone fails that
18+
repo's own CI.
19+
20+
The expected values are spelled out rather than derived from the patterns under test.
21+
A fixture built from the constant it verifies moves with the bug.
22+
"""
23+
import importlib.util
24+
import pathlib
25+
import sys
26+
import unittest
27+
28+
REPO_ROOT = pathlib.Path(__file__).resolve().parents[3]
29+
_MODULE_PATH = REPO_ROOT / 'admin' / 'scripts' / 'check_claim_language.py'
30+
31+
# admin/scripts is not a package, so load the module from its path.
32+
_spec = importlib.util.spec_from_file_location('check_claim_language', _MODULE_PATH)
33+
ccl = importlib.util.module_from_spec(_spec)
34+
sys.modules['check_claim_language'] = ccl
35+
_spec.loader.exec_module(ccl)
36+
37+
# The vocabulary every core is expected to enforce, written out rather than imported.
38+
EXPECTED_CLAIM = (
39+
r'\ball\b|\bevery\b|\bcomplete|\bfull list\b|\bentire\b|'
40+
r'\bthe user (?:searched|typed|viewed|visited|opened|selected|deleted|'
41+
r'read|sent|created|hid|chose)\b|\buser[- ](?:created|entered|typed|'
42+
r'searched|selected|initiated)\b|\bsearched by\b|\btyped by\b|'
43+
r'\bviewed by\b|\bread by\b|\bmanually\b|\bproves?\b|\bdefinitively\b|'
44+
r'\balways\b|\breliable|\bvisited\b|\bhabits?\b'
45+
)
46+
EXPECTED_NOTES = (
47+
r'\bthe user (?:searched|typed|viewed|visited|opened|selected|deleted|'
48+
r'read|sent|created|hid|chose)\b|\buser[- ](?:created|entered|typed|'
49+
r'searched|selected|initiated)\b|\bsearched by\b|\btyped by\b|'
50+
r'\bviewed by\b|\bmanually\b|\bproves?\b|\bdefinitively\b|\balways\b|'
51+
r'\breliable|\bvisited\b|\bhabits?\b'
52+
)
53+
EXPECTED_NEGATION = (
54+
r'\b(not|no|never|nor|neither|without|cannot|rather than|instead of|'
55+
r"isn't|doesn't|don't|does not|do not)\b"
56+
)
57+
EXPECTED_FIELDS = ('description', 'name', 'notes')
58+
59+
60+
def fires(field, text):
61+
"""True when the vocabulary for `field` reports a match in `text`, negation applied."""
62+
pattern = ccl.CHECKED_FIELDS[field]
63+
hits = list(pattern.finditer(text))
64+
if field == 'notes':
65+
hits = [hit for hit in hits if not ccl.negated(text, hit.start())]
66+
return bool(hits)
67+
68+
69+
class VocabularyParity(unittest.TestCase):
70+
"""The five cores must enforce one standard. Drift in any of them fails here."""
71+
72+
def test_claim_vocabulary_is_the_shared_one(self):
73+
self.assertEqual(ccl.CLAIM_PATTERN.pattern, EXPECTED_CLAIM)
74+
75+
def test_notes_vocabulary_is_the_shared_one(self):
76+
self.assertEqual(ccl.NOTES_PATTERN.pattern, EXPECTED_NOTES)
77+
78+
def test_negation_vocabulary_is_the_shared_one(self):
79+
self.assertEqual(ccl.NEGATION_PATTERN.pattern, EXPECTED_NEGATION)
80+
self.assertEqual(ccl.NEGATION_WINDOW, 60)
81+
82+
def test_the_same_three_fields_are_checked(self):
83+
self.assertEqual(tuple(sorted(ccl.CHECKED_FIELDS)), EXPECTED_FIELDS)
84+
85+
def test_the_habit_stem_is_closed(self):
86+
# The open stem also matches "habitat". Four cores carried that until 2026-08-29.
87+
self.assertFalse(ccl.CLAIM_PATTERN.search('habitat'))
88+
self.assertFalse(ccl.NOTES_PATTERN.search('habitat'))
89+
self.assertTrue(ccl.CLAIM_PATTERN.search('habits'))
90+
91+
92+
class FieldCoverage(unittest.TestCase):
93+
def test_notes_is_checked(self):
94+
self.assertIn('notes', ccl.CHECKED_FIELDS)
95+
96+
def test_name_and_description_keep_the_full_vocabulary(self):
97+
self.assertIs(ccl.CHECKED_FIELDS['name'], ccl.CLAIM_PATTERN)
98+
self.assertIs(ccl.CHECKED_FIELDS['description'], ccl.CLAIM_PATTERN)
99+
100+
def test_notes_uses_its_own_vocabulary(self):
101+
self.assertIs(ccl.CHECKED_FIELDS['notes'], ccl.NOTES_PATTERN)
102+
103+
104+
class NotesVocabulary(unittest.TestCase):
105+
# Attribution and certainty mean the same thing in a note as in a description.
106+
CLAIMS = (
107+
'the term the user typed into the search box',
108+
'a page the user visited',
109+
'a term the user entered',
110+
'this proves the account holder sent it',
111+
'the column is always populated',
112+
'a reliable record of the conversation',
113+
)
114+
# Notes state what was tested. These must stay silent or the check punishes the
115+
# coverage discipline it exists alongside.
116+
COVERAGE = (
117+
'empty on all 18 copies tested',
118+
'NULL for every account tested',
119+
'the complete table list is given above',
120+
'present on the entire corpus',
121+
'columns are read by position over a select star',
122+
'the sidecar is not read by this artifact',
123+
)
124+
125+
def test_claims_fire(self):
126+
for text in self.CLAIMS:
127+
with self.subTest(text=text):
128+
self.assertTrue(fires('notes', text))
129+
130+
def test_coverage_wording_stays_silent(self):
131+
for text in self.COVERAGE:
132+
with self.subTest(text=text):
133+
self.assertFalse(fires('notes', text))
134+
135+
def test_notes_drops_exactly_the_two_documented_classes(self):
136+
"""The two vocabularies differ only where the module says they do."""
137+
removed = ('all', 'every', 'complete', 'entire', 'full list', 'read by')
138+
for word in removed:
139+
with self.subTest(word=word, vocabulary='description'):
140+
self.assertTrue(ccl.CLAIM_PATTERN.search(word))
141+
with self.subTest(word=word, vocabulary='notes'):
142+
self.assertFalse(ccl.NOTES_PATTERN.search(word))
143+
kept = ('the user typed', 'typed by', 'manually', 'proves', 'always',
144+
'reliable', 'visited', 'habits', 'user-created')
145+
for word in kept:
146+
with self.subTest(word=word):
147+
self.assertTrue(ccl.CLAIM_PATTERN.search(word))
148+
self.assertTrue(ccl.NOTES_PATTERN.search(word))
149+
150+
def test_description_still_flags_completeness(self):
151+
# The narrowing applies to notes only. Regression guard on the original behaviour.
152+
self.assertTrue(fires('description', 'every message in the conversation'))
153+
self.assertTrue(fires('description', 'all sites the user visited'))
154+
155+
156+
class Negation(unittest.TestCase):
157+
def test_denial_in_the_same_clause_is_not_a_claim(self):
158+
for text in (
159+
'this store holds suggestions the app downloaded, not terms the user searched for',
160+
'their presence does not establish that the user viewed them',
161+
'they evidence the app running rather than anything the user chose',
162+
'the app stores no user created images',
163+
):
164+
with self.subTest(text=text):
165+
self.assertFalse(fires('notes', text))
166+
167+
def test_a_negation_in_the_previous_sentence_does_not_carry(self):
168+
# "not" governs its own clause. A full stop ends it, so the claim after it stands.
169+
text = 'The table is not a cache. It records the term the user typed.'
170+
self.assertTrue(fires('notes', text))
171+
172+
def test_a_distant_negation_does_not_carry(self):
173+
# Sixty characters is the window; padding past it must leave the claim visible.
174+
text = 'not' + ' x' * 45 + ' the term the user typed'
175+
self.assertTrue(fires('notes', text))
176+
177+
178+
if __name__ == '__main__':
179+
unittest.main()

‎scripts/artifacts/biomeLocationVisit.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,15 +14,15 @@
1414
"is relied on.",
1515
"author": "@abrignoni, @mattiaepi (Mattia Epifani)",
1616
"creation_date": "2026-07-25",
17-
"last_update_date": "2026-08-20",
17+
"last_update_date": "2026-08-29",
1818
"requirements": "none",
1919
"category": "Biome",
2020
"notes": "Timestamp reliability, observed in the sample data: every record in a stream "
2121
"file shared one identical SEGB write time while the visits themselves spanned "
2222
"weeks, so the whole batch was flushed at once. The detection timestamp "
2323
"(field 1) ran from hours to more than a day after the recorded departure and "
2424
"up to 28 days before the SEGB write. Arrival and departure are internally "
25-
"consistent (arrival always precedes departure, spans from about half an hour "
25+
"consistent (arrival precedes departure, spans from about half an hour "
2626
"to just over a day) and are the most usable pair, but they are still not "
2727
"independently verified. In tested samples the coordinates, accuracy and "
2828
"point of interest details decoded consistently. Latitude, longitude, "

‎scripts/artifacts/logarchive.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -392,7 +392,7 @@
392392
"VBUS power and CON_DET physical-connection states",
393393
"author": "@AlexisBrignoni",
394394
"creation_date": "2026-08-01",
395-
"last_update_date": "2026-08-02",
395+
"last_update_date": "2026-08-29",
396396
"requirements": "logarchive module must be executed first",
397397
"category": "Unified Logs",
398398
"notes": "Documented at "
@@ -402,7 +402,7 @@
402402
"form. The 'USB Power (VBUS) Present' pattern was added 2026-08-02 as "
403403
"version insurance, not as a fix: the cited CarPlay research, revised for "
404404
"iOS 26.6, quotes that line without the shim prefix and treats it as the "
405-
"most consistent connection marker, with 'Present: 0' the reliable detach "
405+
"most consistent connection marker, with 'Present: 0' treated as the detach "
406406
"signal while CON_DET can remain 1. On our images every VBUS line did "
407407
"carry the shim prefix and was therefore already collected (30 records on "
408408
"iOS 18.7, 40 on iOS 17.1, none without the prefix), so the pattern is "

‎scripts/artifacts/ornetBrowser.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,12 @@
2727
"Tor connection mode and any custom bridge the user configured",
2828
"author": "@AlexisBrignoni, Claude",
2929
"creation_date": "2026-08-07",
30-
"last_update_date": "2026-08-15",
30+
"last_update_date": "2026-08-29",
3131
"requirements": "none",
3232
"category": "OrNET Browser",
3333
"notes": "Values are read directly from the app preferences plist. Each configured custom "
3434
"bridge is reported on its own row; a bridge line contains the transport, the "
35-
"relay address and its fingerprint as the user entered them. The app with "
35+
"relay address and its fingerprint, reported as stored. The app with "
3636
"bundle id ch.b-eng.tor names itself OrNET Browser in its bundle's Info.plist, "
3737
"read from the tested image; iLEAPP releases up to v2026.3.0 labelled these "
3838
"artifacts Onion Browser, which is a different app.",

0 commit comments

Comments
 (0)