Three gates, scored separately and in this order of importance:
- Fidelity. No literal changed (
check_fidelity.pyreports no errors) and the content the case requires is still there. - Restraint. Text that was already good comes back unchanged; edits stay within the case's change or length limit.
- Target. The specific problem in the case is gone, and the wording the case requires is present.
All three are deterministic. grade.py uses substring and ratio checks plus the fidelity checker; no model judges another model's output.
Naturalness is not scored automatically, on purpose. Language-model judges favour longer and more polished text, and one Turkish humanizer project found that blind "which reads more human" judging rewarded fabricated sentences. See the protocol at the end of this file.
| File | Purpose |
|---|---|
cases.json |
67 cases: Turkish IT (17), developer English (10), already-good input (16), adversarial (18), translation (4), drafting (2). Each lists the rules it exercises. |
references.json |
One acceptable output for every case that is not "unchanged". |
grade.py |
Grades an outputs file; --self-test grades the references. |
run.py |
Runs each case through an agent command, one fresh call per case. |
triggers.json |
20 prompts for testing whether the skill description triggers when it should and stays quiet when it should not. |
Check that the cases and references agree:
python evals/grade.py --self-testRun the cases with the skill, then without it, and compare:
python evals/run.py --cmd "claude -p" --point-to-skill --out evals/runs/with_skill.jsonpython evals/run.py --cmd "claude -p" --out evals/runs/without_skill.jsonpython evals/grade.py evals/runs/with_skill.json --baseline evals/runs/without_skill.json--cmd can be any command that reads a prompt on standard input and prints the reply. The baseline matters: a check that passes without the skill tells you nothing about the skill.
Model output varies between runs. Run each arm at least three times before drawing a conclusion from a difference of a few cases.
| Field | Meaning |
|---|---|
prompt, input |
What the user types, and the text they want handled. |
context |
The channel, as the prompt states or implies it. |
kind |
revise, unchanged, translate or draft. |
unchanged |
The output must equal the input. |
must_keep |
Exact strings that must appear (fidelity). |
must_not |
Strings that must not appear, compared case-insensitively (target). |
must_have_any |
Groups of alternatives; at least one from each group must appear (target). |
max_change |
Upper limit on the share of words changed (restraint). |
max_len_ratio |
Upper limit on output length relative to input (restraint). |
max_first_line |
Upper limit on the first line's length, for commit subjects (target). |
rules |
Rule IDs from the skill that the case exercises. |
The checks are deliberately loose about wording. A case fails only on things that are wrong whatever the phrasing; it does not require the reference output.
Cases passing, out of 67, graded with the cases as tightened after the first human review (below). "Held-out" excludes the 21 cases that also appear as worked examples in the skill files, since a model that has just read the skill has seen their answers. Outputs are in evals/runs/.
| Model | Arm | All gates | Held-out (46) | Fidelity | Restraint | Target |
|---|---|---|---|---|---|---|
| Claude Sonnet | without skill | 49 | 33 | 66 | 55 | 61 |
| Claude Sonnet | with skill, run 1 | 65 | 44 | 67 | 67 | 65 |
| Claude Sonnet | with skill, run 2 | 67 | 46 | 67 | 67 | 67 |
| Claude Haiku | without skill | 36 | 26 | 61 | 47 | 57 |
| Claude Haiku | with skill, run 1 | 53 | 33 | 64 | 57 | 64 |
| Claude Haiku | with skill, run 2 | 47 | 27 | 63 | 53 | 61 |
| Claude Haiku | with skill, run 3 | 52 | 33 | 67 | 62 | 57 |
What the runs showed:
- Without the skill, the most common failure on both models was rewriting text that was already fine: 12 of Sonnet's 17 failures and 20 of Haiku's 30 were restraint failures. Haiku without the skill also dropped both Turkish evidentials in translation and edited a code block.
- With the skill, Sonnet passed every case in both runs.
- With the skill, Haiku improved but was not reliable. Across its runs it changed a URL fragment and a code comment (run 1), mixed English verb forms into Turkish sentences (run 2), and left some translated terms unfixed and dropped the evidentials in translation (run 3).
Read these numbers with their limits:
- The skill was revised between runs, using failures from these same cases. Run 2 followed a change that restricts term rules to prose and limits "default" rules to drafted text; run 3 followed rule T0 (Turkish stays the grammatical frame). The with-skill numbers are therefore not from an untouched test set.
- Each arm handled all 67 requests in one agent context, with an instruction to treat them independently. That is weaker than
run.py, which makes one fresh call per case, and it probably helps restraint, because the agent sees many cases side by side.run.pyitself was tested only with a stub command. - Baselines were run once. Haiku's three with-skill runs span 47 to 54, which shows how large run-to-run variation is.
- The cases and the skill have the same author. The checks are deterministic, but they test what the author thought to test.
- Passing does not mean natural. One Turkish developer has reviewed a sample so far (below).
2026-10-05, one reviewer (Turkish backend developer, the project author). Six Sonnet outputs where the author of the rules was unsure:
| Case | Verdict | What changed |
|---|---|---|
tr-frontend-01 "state yönetimi" |
natural | nothing |
tr-infra-01 "log rotation yapılandırılmamış" |
unnatural; would write "ayarlı değil" | R1 now covers bookish verbs such as "yapılandırmak"; case tightened |
draft-tr-01 "prod'a deploy edildi" |
would write "prod'a deploy"; teams have their own environment tags (preprod, DRC) | T1 note on environment names |
x-en-tr-02 "Katılabileceğini onaylar mısın?" |
natural | nothing |
tr-jira-01 "Bu görevin amacı, ..." |
too generic; would start with the problem | R2 entry for "Bu görevin amacı"; case tightened |
x-tr-en-02 "-mış" as "Apparently" |
correct reading | nothing |
2026-10-05, same reviewer, questions on unverified rules.
- Used by the team: build almak, release çıkmak, log basmak. Not used: deploy almak ("deploy yapacağız", "prod'a deploy ettik/edildi", "prod'a çıktık" instead).
- Seen from Turkish colleagues in English: "I controlled the logs", "Have a nice work", "according to me". Not seen: "open/close the flag".
- Commits: English and Turkish both occur in the team. Commits and PRs should give the subject and the reason fast; the skill now has explicit commit and PR rules drawn from STE (specific first line, why in the body, short one-idea sentences).
One reviewer is one team's usage. Findings from a single reviewer are recorded as [P] and change a rule only where they agree with the counts or with S/R rules already in the skill.
For the question "would a Turkish developer write this?", use people:
- Take outputs from two arms (with and without the skill, or two versions of the skill).
- Show each reviewer the original text, the channel, and the two outputs in random order, unlabelled.
- Ask three questions: Which would you send? Does either change what the original says? Is either unnatural for this channel, and where?
- Reviewers should be Turkish developers who write these messages at work. Three reviewers per pair is a reasonable minimum.
Showing the original is essential: a reviewer who cannot see the source cannot notice that the more "human" version invented something.