[755] Support column stats for paimon #767

mao-liu · 2025-12-10T07:01:02Z

Important Read

Please ensure the GitHub issue is mentioned at the beginning of the PR

Closes #755

What is the purpose of the pull request

Adds support for Paimon metadata stats

Brief change log

Adds support for Paimon file-level stats metadata
Adds unit tests for Paimon file stats

Verify this pull request

This change added tests and can be verified as follows:

Extends TestPaimonDataFileExtractorto verify the change.
Manually verified the change by running a job locally.

Dev Notes

build and test this out against a real table
open draft PR against upstream
complete test cases
confirm assumptions with Paimon user group
remove logging

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonDataFileExtractor.java

mao-liu · 2025-12-10T07:05:34Z

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonDataFileExtractor.java

+      List<String> colNames = file.valueStatsCols();
+      //      log.info("valueStatsCols: {}", colNames);
+      if (colNames == null || colNames.isEmpty()) {
+        // if column names are not present, we assume all columns in the schema are present in the same order as the schema - TODO: validate this assumption


Q to Paimon experts: Is this assumption valid?

@mao-liu are you in contact with anyone on the Paimon side to get these questions answered?

@mikedias do you know the answers to any of these Paimon questions on this PR by any chance?

sorry, I don't have these answers...

Hey @the-other-tim-brown , apologies I haven't been very active on this PR until this week.
I have just emailed the Paimon user group about these questions, and hoping to hear back soon.

We have been busy test-driving this change, and happy to report it's working well thus far!

Validated: https://lists.apache.org/thread/ph4xjpvy5qzdbp8h31d2q63kjybchzov

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonDataFileExtractor.java

the-other-tim-brown · 2025-12-10T15:29:42Z

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonDataFileExtractor.java

-    // TODO: Implement logic to extract column stats from the file meta
-    // https://github.com/apache/incubator-xtable/issues/755
-    return Collections.emptyList();
+  private List<ColumnStat> toColumnStats(DataFileMeta file, InternalSchema internalSchema) {


Can we move the column stats conversion to its own class? In the future if we add Paimon as a target then we will also need to convert to the Paimon representation and it would be nice to have all this stats logic in its own class.

Will do, thanks for the early review @the-other-tim-brown !

I do wonder though, if it is even possible to have Paimon as a target... Paimon has a pretty unique file layout, and might not be as easily "tricked" as other formats.

I don't know enough about Paimon to say. Hudi also has a unique native layout structure to allow for update heavy workloads though and we were able to make this work.

Mainly we do this separation to keep the logic isolated though. As not necessarily relevant to Paimon, but if a table format changes how they represent stats in a new version, we can plug in the appropriate converter based on the version.

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonDataFileExtractor.java

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonStatsExtractor.java

mao-liu · 2026-01-09T02:44:03Z

Hey @the-other-tim-brown , we have validated the assumptions in this PR with responses from Paimon maintainers - no more TODOs from me, ready for your review now :)

the-other-tim-brown · 2026-01-12T16:14:26Z

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonStatsExtractor.java

+      Object min = getValue(minValues, i, type, field.getSchema());
+      Object max = getValue(maxValues, i, type, field.getSchema());


If the getValue returns null, should we set the range value to null as well?

good call, added this

the-other-tim-brown · 2026-01-12T16:15:01Z

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonStatsExtractor.java

+      InternalType type = field.getSchema().getDataType();
+      Object min = getValue(minValues, i, type, field.getSchema());
+      Object max = getValue(maxValues, i, type, field.getSchema());
+      Long nullCount = (nullCounts != null && i < nullCounts.size()) ? nullCounts.getLong(i) : 0L;


nitpick: make this a primitive long

the-other-tim-brown · 2026-01-12T16:24:18Z

xtable-core/src/test/java/org/apache/xtable/paimon/TestPaimonDataFileExtractor.java

            TestPaimonTable.createTable("test_table", "level", tempDir, new Configuration(), false);
    paimonTable = testTable.getPaimonTable();
-
-    // just the partition field matters for this test


Just curious why the test setup is changing as part of this PR

The previous tests did not accurately represent the schema of the test tables - the tables had certain columns, but the "testSchema" objects were empty.

The stats extractor rely on the table schema to parse binary stats data from Paimon metadata, hence the change to extract the schema properly

the-other-tim-brown · 2026-01-12T16:25:46Z

xtable-core/src/test/java/org/apache/xtable/paimon/TestPaimonStatsExtractor.java

+    // compaction create commits that are DELETE and ADD on the same file
+    // with `manifest.delete-file-drop-stats` enabled, this means stats are empty after compaction
+    // this is a smoke test to ensure exceptions aren't raised for this scenario
+    // TODO: Question for Paimon experts - is this the expected behaviour?


has this question been answered?

This hasn't been validated with the Paimon community, though empirically this is the behaviour of the manifest.delete-file-drop-stats configuration today.

I have created a bug report at apache/paimon#7026 , and replaced the TODO with a link to the GH issue

the-other-tim-brown · 2026-01-12T16:27:25Z

xtable-core/src/test/java/org/apache/xtable/paimon/TestPaimonStatsExtractor.java

+import org.apache.xtable.model.storage.InternalDataFile;
+
+@Log4j2
+public class TestPaimonStatsExtractor {


Is it possible to add a test case where the schema has nested fields?

If it will cause a lot of changes, feel free to suggest we handle this in a follow up pull request.

Unfortunately, I don't believe Paimon collects stats on nested fields or complex fields

I have added a test case showing this is the case.

the-other-tim-brown · 2026-01-12T16:28:32Z

xtable-core/src/test/java/org/apache/xtable/paimon/TestPaimonDataFileExtractor.java

    // Insert some data to create files
    testTable.insertRows(5);

    List<InternalDataFile> result =


Can we add some basic sanity checks that the stats are non-null/empty for the result?

good call! This has been added

mao-liu · 2026-01-13T10:00:42Z

@the-other-tim-brown thanks for your review!

I have replied to your comments and made the suggested changes, thanks again!

the-other-tim-brown · 2026-01-13T22:25:34Z

@mao-liu things look good overall but it looks like the tests are not passing. Can you take a look?

mao-liu · 2026-01-14T00:04:50Z

This workflow requires approval from a maintainer. Learn more about approving workflows.

@the-other-tim-brown Sorry fixed a test! Missed it somehow on my local. Shall we try the build again?

the-other-tim-brown · 2026-01-14T02:53:05Z

pom.xml

                <version>${spark.version}</version>
                <scope>provided</scope>
            </dependency>
+            <dependency>


@mao-liu the Paimon dependency brings in transitive dependency on spark-catalyst. I had to pin the version to ensure it was consistent to solve the test failure.

Thanks heaps @the-other-tim-brown

The latest CI run also had some more test failures in xtable-utilities, I've fixed them up now

the-other-tim-brown · 2026-01-15T22:53:28Z

I'm tracking the CI issues in this ticket #787

the-other-tim-brown · 2026-01-16T21:19:02Z

I'm tracking the CI issues in this ticket #787

The CI Issue is now resolved, please rebase your PR when you have a chance.

mao-liu · 2026-01-16T21:36:37Z

I'm tracking the CI issues in this ticket #787

The CI Issue is now resolved, please rebase your PR when you have a chance.

Thanks @the-other-tim-brown , done!

mao-liu added 11 commits December 5, 2025 08:42

paimon: wip column stats

678ceac

paimon column stats: fix tests

7139e6b

paimon: fixing some tests

63adfe0

paimon: fixing more tests

ed8ac27

paimon: fix column stats column checks

b08ce6f

paimon: stats: handling timestamp as long

e82dbb9

paimon: add to xtable-utilities

be48db4

paimon: lint

60df096

paimon: ticked version to 1.3.1

3b884f8

paimon: reducing logging

8400413

paimon: annotating assumptions with comments

203b651

mao-liu changed the title ~~Support column stats for paimon~~ #755 - Support column stats for paimon Dec 10, 2025

mao-liu commented Dec 10, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonDataFileExtractor.java Outdated Show resolved Hide resolved

mao-liu commented Dec 10, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonDataFileExtractor.java Outdated Show resolved Hide resolved

mao-liu commented Dec 10, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonDataFileExtractor.java Outdated Show resolved Hide resolved

mao-liu force-pushed the feat/paimon-column-stats branch 3 times, most recently from c0cabf3 to 2757735 Compare December 10, 2025 12:38

paimon: expanded tests for stats extractor

6e927e2

mao-liu force-pushed the feat/paimon-column-stats branch from 2757735 to 6e927e2 Compare December 10, 2025 13:03

the-other-tim-brown reviewed Dec 10, 2025

View reviewed changes

mao-liu added 3 commits December 22, 2025 11:27

paimon: stats extractor to its own class

d0daa21

paimon: handling deleted stats

74a302a

paimon: spotless

86a10e5

mao-liu changed the title ~~#755 - Support column stats for paimon~~ [755] Support column stats for paimon Jan 6, 2026

mao-liu commented Jan 8, 2026

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonStatsExtractor.java Outdated Show resolved Hide resolved

mao-liu commented Jan 8, 2026

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/paimon/PaimonStatsExtractor.java Outdated Show resolved Hide resolved

mao-liu added 2 commits January 8, 2026 20:23

Merge branch 'main' into feat/paimon-column-stats

b97a0e9

paimon: fix tests

73460f6

paimon: tests remove wildcard imports

850dbd8

mao-liu marked this pull request as ready for review January 8, 2026 09:43

paimon: assumptions validated

53aa0e0

mao-liu requested review from mikedias and the-other-tim-brown January 12, 2026 01:49

mikedias approved these changes Jan 12, 2026

View reviewed changes

the-other-tim-brown reviewed Jan 12, 2026

View reviewed changes

mao-liu added 2 commits January 13, 2026 20:13

paimon: code review changes

33692cf

paimon: added test case for lack of stats for complex fields

7dec291

mao-liu force-pushed the feat/paimon-column-stats branch from 98012d5 to ad5b35a Compare January 13, 2026 09:57

paimon: extended tests and updated test TODO with github issue reference

4867167

mao-liu force-pushed the feat/paimon-column-stats branch from ad5b35a to 4867167 Compare January 13, 2026 09:58

mao-liu requested a review from the-other-tim-brown January 13, 2026 10:00

the-other-tim-brown approved these changes Jan 13, 2026

View reviewed changes

paimon: fix tests

480d02c

pin catalyst version

b7fff7f

the-other-tim-brown reviewed Jan 14, 2026

View reviewed changes

paimon: fix tests in xtable-utilities

1183121

mao-liu force-pushed the feat/paimon-column-stats branch from b10488e to 1183121 Compare January 14, 2026 21:29

Merge remote-tracking branch 'origin' into feat/paimon-column-stats

7f91eef

		Object min = getValue(minValues, i, type, field.getSchema());
		Object max = getValue(maxValues, i, type, field.getSchema());

[755] Support column stats for paimon #767

Are you sure you want to change the base?

[755] Support column stats for paimon #767

Uh oh!

Conversation

mao-liu commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Important Read

What is the purpose of the pull request

Brief change log

Verify this pull request

Dev Notes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

the-other-tim-brown Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mao-liu commented Jan 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mao-liu Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mao-liu commented Jan 13, 2026

Uh oh!

the-other-tim-brown commented Jan 13, 2026

Uh oh!

mao-liu commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

the-other-tim-brown commented Jan 15, 2026

Uh oh!

mao-liu commented Dec 10, 2025 •

edited

Loading

the-other-tim-brown Dec 10, 2025 •

edited

Loading

mao-liu Jan 13, 2026 •

edited

Loading

mao-liu commented Jan 14, 2026 •

edited

Loading