docs: Stop generating dynamic docs content in build

[andygrove]

Closes #2582

Summary

Move documentation generation from build time to publish time to prevent merge conflicts in PRs.

Previously, GenerateDocs ran during mvn package and modified docs/source/user-guide/latest/configs.md and compatibility.md in-place. When PRs added new configs or expressions, the regenerated tables would conflict with main.

Now:

• Source docs contain only template markers (no generated content)
• Content is generated at publish time in docs/build.sh
• Release branches freeze generated content via dev/generate-release-docs.sh

Changes

• spark/pom.xml: Remove exec-maven-plugin that ran GenerateDocs during package phase
• docs/build.sh: Add step to compile and run GenerateDocs against temp copy before Sphinx build
• docs/source/user-guide/latest/configs.md: Replace generated tables with empty template markers
• docs/source/user-guide/latest/compatibility.md: Replace generated tables with empty template markers
• dev/generate-release-docs.sh: New script to generate docs when creating a release branch
• dev/release/README.md: Add "Generate Release Documentation" step to release process

How It Works

Branch	Source Docs State	When Generated
main	Empty template markers	At publish time by CI
branch-0.x	Frozen generated content	Once, when branch is created

Release Process Addition

When cutting a new release:

git checkout -b branch-0.13 main
./dev/generate-release-docs.sh
git add docs/source/user-guide/latest/
git commit -m "Generate docs for 0.13.0 release"
git push apache branch-0.13

Test Plan

• Verify make no longer modifies docs
• Verify docs/build.sh generates content correctly (test locally or check CI)
• Verify dev/generate-release-docs.sh works on a test branch

[andygrove]

@snmvaughan fyi

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.07%. Comparing base (f09f8af) to head (31e28c4).
⚠️ Report is 877 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3212      +/-   ##
============================================
+ Coverage     56.12%   60.07%   +3.94%     
- Complexity      976     1437     +461     
============================================
  Files           119      172      +53     
  Lines         11743    15926    +4183     
  Branches       2251     2631     +380     
============================================
+ Hits           6591     9567    +2976     
- Misses         4012     5031    +1019     
- Partials       1140     1328     +188     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[snmvaughan]

Maybe we should use a profile to decide when to generate docs?

[andygrove]

Maybe we should use a profile to decide when to generate docs?

I went with build.sh because the doc generation is tightly coupled to the Sphinx build step, and keeping it there ensures mvn package stays deterministic (no modified files).

The workflow also has two different targets: during normal publishing, we generate docs into a temp folder (so the repo stays clean), but during the release process, we run dev/generate-release-docs.sh once to freeze the generated content into the actual docs folder in the release branch. A Maven profile would only cover the first case cleanly, unless we added multiple profiles perhaps.

[hsiang-c]

(nit) We can follow what README.md suggests git add docs/source/user-guide/latest/

[andygrove]

Updated

[hsiang-c]

LGTM

[parthchandra]

lgtm (please address the comment from @hsiang-c)