Core: Fix rewrite_position_delete_files failure with array/map columns #15079
+144
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… columns This test demonstrates a regression in Iceberg 1.10.0 where rewrite_position_delete_files fails with ValidationException when the table contains array columns with primitive fields. The error occurs because ExpressionUtil.identitySpec() creates a temporary partition spec from schema field IDs, and the new validation added in commit 9fb80b7 rejects fields whose parent type is a list.
ExpressionUtil.identitySpec() now skips fields that cannot be partition sources. Fields nested inside arrays or maps are filtered out by checking that all ancestor types are structs, matching the validation logic in PartitionSpec.checkCompatibility(). Fixes apache#15080
bk-mz
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes
rewrite_position_delete_filesfailing withValidationException: Invalid partition field parentwhen tables contain array or map columns.Root Cause
ExpressionUtil.identitySpec()attempted to create identity partition fields for all primitive fields in the schema, including those nested inside arrays and maps. Since commit 9fb80b7,PartitionSpec.checkCompatibility()validates that partition field parents must be struct types, causing the failure.Fix
identitySpec()now filters out fields that cannot be partition sources by checking that all ancestor types are structs, usingTypeUtil.indexParents()- the same approach used inPartitionSpec.checkCompatibility().Testing
Added test case that reproduces the issue with a table containing
ARRAY<STRUCT<...>>column.Fixes #15080