Tags: sqlc-dev/darkwing
Tags
Milestone 6: hardening — error fidelity, difftest, benchmarks, fuzzing ( #5) * Match the pinned build's parser error messages over the negative corpus Milestone 6, error fidelity: every must-reject corpus case now renders the oracle's recorded first line verbatim, gated by the new TestCorpusErrorMessages in errors_test.go alongside hand-written position (Offset/Line/Column) pins. Message fixes to reach zero divergence: - Whole-query UTF-8 validation before parsing ("Invalid UTF-8 in query (byte sequence mismatch)"), via a new port of Utf8Proc::Analyze that classifies byte mismatches vs invalid codepoints and, like upstream, accepts overlong encodings; escape-string literals reuse it for the "byte mismatch"/"invalid unicode" suffixes. - Column-without-type errors quote the name per KeywordHelper::WriteOptionallyQuoted (keywords and non-plain identifiers quoted, plain ones bare). - Chained-comparison ban transforms its operands first so deeper errors (e.g. Empty subscript '[]') win, and uses upstream's wording. - Window frames validate the start bound before the end bound. - Interval range specifiers name the range (YEAR TO MONTH is not supported) instead of a generic message. - RESET options name the last offending option; PIVOT-on-subquery renders the expression's source text; plus upstream wording for CREATE TABLE AS, OR REPLACE/IF NOT EXISTS, OR REPLACE|IGNORE with ON CONFLICT, COMMENT ON qualification depth, and NEAREST counts (which now also reject non-positive literals, retiring the last nearest_errors todo). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QQHouvcnCnGT1MuHvwPJrj * Add milestone 6 benchmarks, fuzz target and parallel-parse race gate - bench_test.go: tokenize + parse throughput on simple/complex/mixed statement workloads, plus upstream's pathological unmatched-parens packrat case as both a benchmark and a fast-failure regression test (19 and 40 parens complete in milliseconds thanks to memoization). - fuzz_test.go: FuzzParse holds Parse to its public invariants on any input - errors are *parser.Error with in-range offset and 1-based line/column, successful parses return non-nil statements whose spans tile the input. Seeds cover the dialect's corners (FROM-first, PIVOT, list comprehensions, dollar quotes, invalid UTF-8, chained comparisons); a plain test keeps the seeds honest in ordinary runs (3-minute local fuzz run: 2.4M execs, no failures). - TestParallelParse hammers the shared compiled engine from concurrent goroutines so the -race dev loop and CI gate the engine's immutability and per-call packrat state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QQHouvcnCnGT1MuHvwPJrj * Add cmd/difftest, the mutation differential fuzzer, and fix what it found Milestone 6, differential loop: cmd/difftest mutates corpus seed statements at token-span boundaries (delete, duplicate, swap adjacent, cross-seed replace, truncate, splice) and compares darkwing's verdict - and, for syntax errors, the message first line - against the pinned DuckDB CLI. Deterministic per -seed, parallel oracle workers, bounded by -n or -duration; a nightly GitHub Actions lane runs it (plus a native fuzz pass) while the DuckDB nightly artifact still matches the pin, and skips green once the artifact drifts. Known oracle imprecisions are filtered, not reported: nested-SQL functions whose bind-time inner parse raises Parser Error, bind-time ParserExceptions outside the parse pipeline (the corpus todo category), statements the CLI shell intercepts (column-0 '#' comments, dot commands), and multi-statement mutants where a post-parse error aborts the batch CLI before later statements parse - duckdbsrc.Verdict now records that with PostParseError. Real divergences it caught, all fixed with regression tests: - SyntaxError rendered token spellings through Go %q escaping; upstream puts the raw spelling between plain quotes (embedded quotes and backslashes verbatim). - COLLATE rejected constant collation names; upstream accepts constants via Value::ToString ('x' COLLATE 'nocase', even COLLATE 3) and only fails lookup at bind time, and raises NotImplementedException (post- parse) for other expression shapes, so darkwing now accepts those too. - Number tokens whose text has no value ("2.03.0") raised a parser error; upstream accepts the token and throws InvalidInputException while constructing the value, so darkwing records the raw text as a cast and accepts. - Dollar-quoted string tokens record the rewritten literal's length (upstream behavior pinned by query_location_length in the serialize goldens), which could push a statement span past the end of the input; consumedEnd now clamps so statement spans keep their tiling contract. Validated with ~24k mutants across seven RNG seeds (zero disagreements) and a corpus-wide serialize sweep against the live pinned CLI (25,982 statements, zero mismatches). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QQHouvcnCnGT1MuHvwPJrj --------- Co-authored-by: Claude <noreply@anthropic.com>