[format] Drop ORC float bounds that hide NaN or -0.0 - #10337
kalayciburak wants to merge 2 commits into
Conversation
ORC records double min/max with primitive comparisons. A NaN never replaces a finite bound, and -0.0 never replaces +0.0, so a predicate can skip a file that still contains a matching row. Omit those bounds. A missing bound is not used for skipping. Fixes apache#10334
|
[P1] The signed-zero fix still skips positive zero when ORC reports a negative-zero upper bound. On head There is also a second pruning layer: in an isolated control that additionally rejects This is a remaining case of the existing bug described by #10334, not a new regression introduced by this PR. The current patch fixes the NaN cases in the same comparison, but leaves this signed-zero case unresolved. Validation: all 30 normal JDK 8 tests passed ( |
ORC's primitive max keeps -0.0 when +0.0 is in the file, so equal(+0.0) and greaterThan(-0.0) skip it. Drop that bound. Row-group SearchArgument still hides the other zero, so do not push a signed-zero literal. Fixes apache#10334
Purpose
ORC records FLOAT and DOUBLE min/max with primitive comparisons. A NaN never replaces a finite bound, and -0.0 never replaces +0.0.
OrcSimpleStatsExtractorcopied those bounds into file stats, so a predicate could skip a file that still contained a matching row.When the ORC sum is NaN, or the recorded minimum is +0.0, the bounds are omitted. A missing bound is not used for skipping. Other finite bounds are unchanged.
Omitting the bound when the minimum is +0.0 also drops a correct bound for a column whose smallest value really is +0.0. That is safer than a wrong bound. Reading the column back would keep that bound, at the cost of a scan.
Fixes #10334
Tests
mvn -pl paimon-format -Pfast-build -DwildcardSuites=none -DfailIfNoTests=false -Dtest=OrcFloatingPointStatsTest,OrcSimpleStatsExtractorTest testOrcFloatingPointStatsTestfailed 4/4: max was 1.0 for a written NaN, min was +0.0 for a written -0.0.OrcFloatingPointStatsTest6/0,OrcSimpleStatsExtractorTest5/0.