fix: preserve consumed tokens in unsupported statement recovery - #2752

AI disclosure: this change was prepared with AI coding agents, reviewed and revised line by line by me.

What

UnsupportedStatement error recovery now retains tokens already consumed by a failed production and avoids dereferencing an unassigned statement. Single-statement IF recovery returns the recovered statement instead of a partially parsed IF node.

Why

CCJSqlParserUtil.parse("INSERT INTO t (a", p -> p.withUnsupportedStatements(true)) throws JSQLParserException with a NullPointerException cause. Statement-list recovery can return an empty UnsupportedStatement for the same input, losing the SQL needed to inspect or handle the failure.

How

1. Record the token preceding each statement before parsing it.
2. Reuse the existing recovery boundary, then collect the linked token images through that boundary, excluding the final separator or EOF.
3. Construct UnsupportedStatement from those images instead of serializing a partial AST. Clear the prior single IF result only when unsupported recovery succeeds.
4. Add token-text, parser-entry, recovery-option and incremental-boundary regressions.

Root cause

A nested production can consume tokens and throw before SingleStatement returns an AST. The single-statement catch calls stm.toString() while stm is null. All three recovery catches combine AST text with only the unconsumed suffix, so consumed tokens are lost or rewritten. The single IF return also prioritizes its prior AST over the recovered result.

Testing

• UnsupportedStatementRecoveryTest adds 49 cases. On upstream master bb55bb9d, 35 fail and 14 normal controls pass; the fix passes all 49. Cases cover single and list parsing, configured Reader/InputStream parsers, EOF/separators, first and later failures, following statements, original token case/punctuation, quoted semicolons, recovery options, typed SELECT/IF nodes and parse/deparse behavior.
• Forcing all three unsupported-recovery guards true is rejected by seven tests for default errors, null/error recovery and single-statement EOF rejection; the two selected normal SELECT/IF guards remain green. Restoring the grammar passes again. Three existing incremental-prefix shapes are explicitly asserted as separate typed-prefix and unsupported-suffix nodes on both baseline and fix.
• JDK 17 targeted Gradle run, including UnsupportedStatementTest, StatementsTest, IfElseStatementTest, CCJSqlParserUtilTest and SqlRoutineBodyBoundaryTest: 160 cases, zero failures/errors and three existing skipped cases. SpotlessCheck passes. The upstream fix: require end of input when parsing a single statement #2733 default EOF and SingleStatement incremental controls pass.
• Full Gradle check passes with 9451 cases, zero failures/errors and 25 existing skipped cases; fresh Maven clean verify and SpotlessCheck pass with 9433 cases, zero failures/errors and 25 existing skipped cases. The strict changed-Java license check scans one header and passes; the unchanged grammar header was checked manually. Maven retains 33 preexisting SQL resource header warnings, all on files byte-identical to bb55.
• Project JMH parseSQLStatements, unchanged 54-statement corpus, version=latest: interleaved states, three forks per state, two one-second warmups and five one-second measurements per fork; 15 samples per state. Baseline 28.608 ms/op, 99.9% CI [21.591,35.624]; fixed 30.525 ms/op, CI [21.976,39.073]. The intervals overlap; no measurable regression in this benchmark. This benchmark uses valid SQL and does not measure invalid-input recovery throughput. No database-server or Windows/macOS local test matrix was run.

Behavior notes

• Recovered text contains the complete lexical token sequence, with the existing space-joined formatting. Source case and punctuation previously lost through partial-AST serialization are retained. Original whitespace and comments are not reconstructed.
• Unsupported recovery retains its existing priority over errorRecovery. Disabling unsupported recovery preserves checked parsing errors, or null statements and parseErrors when errorRecovery is enabled. Default single-statement EOF rejection and SingleStatement incremental parsing are unchanged.
• Later statement-list iterations can still parse a valid prefix and capture its remaining suffix as a separate UnsupportedStatement without raising ParseException. This change preserves that existing boundary behavior and only changes the recovery catches. No new tokens, syntax or public model APIs are added.

Verification of the original issue

No existing issue is linked. The failure was reproduced on upstream master bb55bb9de8377de9880b14e4fe3b3cc94bf63498; the locally verified fixed commit is b7c937df24869b84e8900049211e0f158e56b063. This builds on Andreas Reichel's UnsupportedStatement capture in 063d2442, while preserving the strict single-statement EOF contract restored by #2733. Thanks to Andreas Reichel for establishing that recovery path; this change completes its handling of already-consumed tokens and does not claim to fix #1984 or #2681 again.

CCJSqlParserUtil.parse("INSERT INTO t (a", p -> p.withUnsupportedStatements(true));
CCJSqlParserUtil.parse("UPDATE t SET", p -> p.withUnsupportedStatements(true));
CCJSqlParserUtil.parseStatements("SELECT 0; INSERT INTO t (a; SELECT 2;",
        p -> p.withUnsupportedStatements(true));

The first two return UnsupportedStatement with text INSERT INTO t ( a and UPDATE t SET, respectively. The third retains SELECT 0, the complete unsupported INSERT token sequence and SELECT 2 as three separate statements.