Conversation
The projection visitor rebuilds a list of structs from its offsets alone, which cannot represent a null list, so every null list<struct<...>> was turned into an empty list. On the write path this is persisted into the Parquet file and the null is lost for every reader. Rebuild the list from the original validity and offsets buffers instead, which also keeps the original array offset so sliced batches stay correct. Closes apache#3833
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3833
Rationale for this change
ArrowProjectionVisitor.listrebuilds a list whose element is a struct withpa.LargeListArray.from_arrays(list_array.offsets, value_array). The offsets alone cannot express a null list, so every nulllist<struct<...>>came out as an empty list. The visitor runs on both the read and the write path; on write the empty list is persisted into the Parquet file, so the null is gone for every reader, not only for pyiceberg.Reproduction from the issue (pyarrow 25.0.1):
This PR rebuilds the list with
pa.Array.from_buffers, reusing the validity and offsets buffers (and the array offset) of the original list and swapping in the projected element array. Passingmask=list_array.is_null()tofrom_arrayswould fix the unsliced case, but Arrow rejects a mask together with sliced offsets (Null bitmap with offsets slice not supported) forLargeListArray, so sliced batches would start raising. Reusing the buffers keeps the array offset and avoids that.The integration test
test_null_list_and_mapasserted the corrupted value with the correct assertion commented out; the correct assertion is restored.Are these changes tested?
test__to_requested_schema_null_list_of_structsintests/io/test_pyarrow.py, parametrized overpa.list_andpa.large_list, covering a full batch and a sliced one. It fails onmain(At index 2 diff: [] != None) and passes with this change.tests/integration/test_reads.py::test_null_list_and_map.make testandmake lintpass locally.Are there any user-facing changes?
Null lists of structs are now preserved on read and write instead of being turned into empty lists.