You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f84b82c
Browse filesBrowse the repository at this point in the historyBrowse files
Feat(parquet) : Introduce support for optionally writing Distinct Values to statistics (#25576)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes#123` indicates that this PR will close issue #123.
-->
- Closes#24114.
## Rationale for this change
DataFusion can use NDV (number of distinct values) from Parquet row
group statistics to power optimizations like COUNT(DISTINCT) aggregation
pushdown and cardinality estimation (#24111) . However, currently the
Parquet writers doesn't emit NDV by default, so these statistics are
absent for roundtrip queries.
Two recent arrow-rs additions make it possible to close this gap
end-to-end:
- [arrow-rs #10652 —
`StatisticsConverter::row_group_distinct_counts()`](apache/arrow-rs#10652)
: a typed API for reading NDV from row group metadata
- [arrow-rs #10654 —
`WriterPropertiesBuilder::set_write_row_group_number_distinct_values()`](apache/arrow-rs#10654):
a writer-side option to track and emit NDV during write
This PR wires both into DataFusion so that users can write parquet files
with NDV and have DataFusion read them back correctly.
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
Please explain the problem you are trying to solve in terms of the
user-visible
behavior, rather than the implementation.
For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of
the
implementation. "COUNT(DISTINCT) returns wrong results when the column
contains
nulls" is the user-visible problem.
-->
## What changes are included in this PR?
- summarize_distinct_counts now calls
`StatisticsConverter::row_group_distinct_counts()` instead of manually
walking column chunk statistics.
- Adds
`datafusion.execution.parquet.write_row_group_number_distinct_values`
(bool, default false) to ParquetOptions. When enabled, the ArrowWriter
hashes every non-null value per column per row group and writes the
distinct count into the Parquet statistics footer.
<!--
There is no need to duplicate the description in the issue here, but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## What is the testing strategy for this PR?
- The NDV read change is covered by the 8 existing unit tests in
metadata.rs (test_distinct_count_*), which all continue to pass.
- The write config is tested by
sqllogictest/test_files/parquet_ndv_write.slt, which writes 4 rows with
3 distinct values and verifies:
- Without the option: Distinct is absent from DataSourceExec statistics
- With the option: Distinct=Exact(3) appears in DataSourceExec
statistics
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
Briefly describe how this PR is tested, and point to the specific tests
you added. For example: 'This new feature is covered by the
`sqllogictest` cases added in `foo.slt`'.
If this PR does not add tests, explain why. For example, if the change
is already covered by existing tests, please mention it.
You should also check the `codecov` bot reply on this PR to confirm the
changed code is exercised.
-->
## Are there any user-facing changes?
yes, new `write_row_group_number_distinct_values` to parquet config
options.
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
0 commit comments