Improve documentation for configs related to explain/fallback

[andygrove]

What is the problem the feature request solves?

We currently have multiple ways of reporting information about which parts of a plan are accelerated with Comet, and it is not well-documented.

spark.comet.explainFallback.enabled

When enabled, Comet checks for fallback reasons. If there are no fallback reasons then nothing gets logged. If there are fallback reasons, then Comet shows the verbose extended info plan (regardless of whether spark.comet.explain.verbose.enabled is enabled).

There is no way to show the verbose plan if there are no fallback reasons (although this changes with Spark 4.0.0, which adds support for ExtendedExplainInfo).

spark.comet.logFallbackReasons.enabled

This logs every fallback reason to a single WARN log entry and does not show the plan that it is related to.

spark.comet.explain.verbose.enabled

This property is referenced from ExtendedExplainInfo. The code can be invoked directly from Spark (starting with version 4.0.0) and later if --conf spark.sql.extendedExplainProviders=org.apache.comet.ExtendedExplainInfo is specified.

spark.comet.explain.native.enabled

Shows an explain plan for native plans. This runs for each task and the output goes to the executor logs.

Describe the potential solution

No response

Additional context

No response

[s6r6k]

Hello! Im new to open source contributions. I would like to start off by contributing to this as I feel like it would help me understand this project better. My understanding is that the main goal is to clearly document the different explain, fallback-related configs, how they interact and how behavior differs across Spark versions.

[coderfender]

Sure checkout this link to get started , setting up the project on local IDE https://datafusion.apache.org/comet/

https://datafusion.apache.org/comet/contributor-guide/contributing.html