Skip to content

Latest commit

 

History

5 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Flaky Test Quarantine Register

Offline, read-only validation of a locally maintained quarantine register and complete test-outcome export. It produces an exact-match CI policy preview; it does not modify CI, fetch failures, or write files.

Run

Node.js 22+; no dependencies or network calls.

node bin/flaky-test-quarantine-register.mjs --root examples/pass --input register.json
node bin/flaky-test-quarantine-register.mjs --root examples/fail --input register.json
npm run check

--help prints usage. Normal runs emit one JSON report on stdout: exit 0=pass, 1=fail, 2=incomplete or invalid configuration. Invalid options or root produce empty stdout; an unreadable, undecodable, malformed, or out-of-root named input produces an incomplete report. The named input is realpath-confined under the root. UTF-8 decoding is strict and duplicate decoded JSON keys are rejected. No files are written.

Register format and decisions

The operator supplies one complete JSON snapshot:

{
  "schemaVersion": "1",
  "asOf": "2026-09-26",
  "complete": {"registry": true, "outcomes": true},
  "quarantines": [{"suite": "unit", "testId": "flaky-case", "owner": "team-a", "evidenceId": "failure-1", "expiresOn": "2026-09-26", "recovery": {"requiredPasses": 2}}],
  "outcomes": [{"suite": "unit", "testId": "flaky-case", "status": "fail", "consecutivePasses": 0}]
}

The producer must assert both completeness flags from actual exports; the tool cannot prove completeness itself. evidenceId is a required opaque reference to local failure evidence, not the evidence contents. Each quarantine needs a current outcome. A quarantine is valid through expiresOn, inclusive. On later asOf dates it fails policy. Matching uses the exact pair (suite, testId)—not a prefix, suffix, or test name alone. A new failing test without its own active quarantine fails, even if its name or suite resembles another. An active quarantine remains until a passing outcome reaches requiredPasses consecutive passes; then the preview marks it recovered and no longer suppresses it. Failed outcomes require a zero pass streak; passing outcomes require at least one. Duplicate identities, malformed records, incomplete exports, or missing outcomes are incomplete, never pass. Optional top-level metadata is non-semantic.

ciPolicyInput contains logical pointers into the exact register file named at invocation, not raw test identities. It is populated only for a pass report; failed or incomplete evaluations produce no actionable quarantine pointers. A CI adapter must resolve those pointers against that same validated snapshot and use the declared exact-suite-and-test-id rule; the report by itself is not a standalone CI allowlist. This avoids putting private test names or evidence IDs in diagnostics. A recovered pointer means the exception can be removed from CI; it does not rewrite the register. Findings use fixed messages, @register as logical source provenance, and JSON pointers. Findings sort by (pointer, ruleId) in UTF-16 code-unit order.

Limits and non-goals

Input ≤65,536 bytes, ≤100 quarantines, ≤200 outcomes, opaque IDs ≤128 UTF-16 units, pass streak and recovery requirement 1–100 (failed streak exactly 0), JSON depth ≤16, and evaluation time ≤5,000 ms through an injected monotonic clock. Each inclusive maximum accepts N and rejects N+1. Dates are strict YYYY-MM-DD calendar dates. No automatic quarantine creation, CI writes, live test runs, legal retention opinion, or inference of flaky behavior is performed.

About

Register quarantined tests with owner, reason, expiry and recovery evidence.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages