Skip to content

[format] Drop ORC float bounds that hide NaN or -0.0 - #10337

Open
kalayciburak wants to merge 2 commits into
apache:masterfrom
kalayciburak:fix-orc-floating-point-stats
Open

kalayciburak wants to merge 2 commits into
apache:masterfrom
kalayciburak:fix-orc-floating-point-stats

Conversation

@kalayciburak

Copy link
Copy Markdown

Purpose

ORC records FLOAT and DOUBLE min/max with primitive comparisons. A NaN never replaces a finite bound, and -0.0 never replaces +0.0. OrcSimpleStatsExtractor copied those bounds into file stats, so a predicate could skip a file that still contained a matching row.

When the ORC sum is NaN, or the recorded minimum is +0.0, the bounds are omitted. A missing bound is not used for skipping. Other finite bounds are unchanged.

Omitting the bound when the minimum is +0.0 also drops a correct bound for a column whose smallest value really is +0.0. That is safer than a wrong bound. Reading the column back would keep that bound, at the cost of a scan.

Fixes #10334

Tests

mvn -pl paimon-format -Pfast-build -DwildcardSuites=none -DfailIfNoTests=false -Dtest=OrcFloatingPointStatsTest,OrcSimpleStatsExtractorTest test

  • Before the change, OrcFloatingPointStatsTest failed 4/4: max was 1.0 for a written NaN, min was +0.0 for a written -0.0.
  • After the change: OrcFloatingPointStatsTest 6/0, OrcSimpleStatsExtractorTest 5/0.

ORC records double min/max with primitive comparisons. A NaN never
replaces a finite bound, and -0.0 never replaces +0.0, so a predicate
can skip a file that still contains a matching row. Omit those bounds.
A missing bound is not used for skipping.

Fixes apache#10334
@JingsongLi

Copy link
Copy Markdown
Contributor

[P1] The signed-zero fix still skips positive zero when ORC reports a negative-zero upper bound.

On head c52330babf7b12668472001388daab973a2f06c0, OrcSimpleStatsExtractor.orcFloatingPointBoundsUnusable (line 261) only rejects a +0.0 minimum. Writing [-1.0, -0.0, +0.0] to a real ORC append table produces footer and manifest bounds min=-1.0, max=-0.0, with a finite sum, so this guard accepts the unsafe maximum. An unfiltered read preserves the positive-zero row, but ReadBuilder.withFilter(equal(v, +0.0)) and greaterThan(v, -0.0) both plan zero splits and return no rows. I reproduced this for FLOAT and DOUBLE, with the default write batch size 1024, before and after an actual COMPACT commit. With two input files, the row-level oracle contains two matching rows in both cases.

There is also a second pruning layer: in an isolated control that additionally rejects max=-0.0, the plan retains the file, but ORC SearchArgument row-group filtering still returns zero rows. Only the combined control—dropping the unsafe upper bound and declining signed-zero predicate pushdown in OrcPredicateFunctionVisitor—returns the expected rows. Please cover both layers with actual table-read regression tests; an extractor-only assertion does not verify the required end-to-end behavior.

This is a remaining case of the existing bug described by #10334, not a new regression introduced by this PR. The current patch fixes the NaN cases in the same comparison, but leaves this signed-zero case unresolved. Validation: all 30 normal JDK 8 tests passed (OrcFloatingPointStatsTest, OrcSimpleStatsExtractorTest, OrcFormatReadWriteTest, AppendOnlyTableFileMetaFilterTest); the additional real-ORC matrix has 240 predicate queries, with 8 failures on this head and zero failures in the combined control. No production storage or credentials were used.

ORC's primitive max keeps -0.0 when +0.0 is in the file, so equal(+0.0)
and greaterThan(-0.0) skip it. Drop that bound. Row-group SearchArgument
still hides the other zero, so do not push a signed-zero literal.

Fixes apache#10334
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] ORC: FLOAT/DOUBLE min/max statistics ignore NaN and -0.0, so files containing matching rows are skipped (Parquet is correct)

2 participants