Investigate failing tests with Spark 4.0.0

[andygrove]

Describe the bug

#1830 adds initial support for Spark 4.0.0, but some tests were ignored and need to be reviewed.

See all references to this issue in the diff file for more information:

$ grep -B 2 "https://github.com/apache/datafusion-comet/issues/1948" dev/diffs/4.0.0.diff 
 
+-- TODO: Disabled due to one of the test failed for Spark4.0
+-- TODO: https://github.com/apache/datafusion-comet/issues/1948
--
 
+-- TODO: Disabled due to one of the test failed for Spark4.0
+-- TODO: https://github.com/apache/datafusion-comet/issues/1948
--
+  test("SPARK-35332: Make cache plan disable configs configurable - check AQE",
+    IgnoreComet("TODO: ignore for first stage of 4.0 " +
+      "https://github.com/apache/datafusion-comet/issues/1948")) {
--
+  test("SPARK-47430 Support GROUP BY MapType",
+    IgnoreComet("TODO: ignore for first stage of 4.0 " +
+      "https://github.com/apache/datafusion-comet/issues/1948")) {
--
+  test("SPARK-43402: FileSourceScanExec supports push down data filter with scalar subquery",
+    IgnoreComet("TODO: ignore for first stage of 4.0, " +
+    "https://github.com/apache/datafusion-comet/issues/1948")) {
--
+  test("hash join should be used for collated strings if sort merge join is not forced",
+    IgnoreComet("TODO: ignore for first stage of 4.0 " +
+      "https://github.com/apache/datafusion-comet/issues/1948")) {
--
+  test("rewrite with collationkey shouldn't disrupt multiple join conditions",
+    IgnoreComet("TODO: ignore for first stage of 4.0 " +
+      "https://github.com/apache/datafusion-comet/issues/1948")) {
--
+  test("SPARK-48655: order by on partition keys should not introduce additional shuffle",
+    IgnoreComet("TODO: ignore for first stage of 4.0 " +
+      "https://github.com/apache/datafusion-comet/issues/1948")) {
--
+  test("root LIMIT preserves data ordering with CollectLimitExec",
+    IgnoreComet("TODO: ignore for first stage of 4.0 " +
+      "https://github.com/apache/datafusion-comet/issues/1948")) {
--
+  test("middle LIMIT preserves data ordering with the extra sort",
+    IgnoreComet("TODO: ignore for first stage of 4.0 " +
+      "https://github.com/apache/datafusion-comet/issues/1948")) {
--
+  test("middle OFFSET preserves data ordering with the extra sort",
+    IgnoreComet("TODO: ignore for first stage of 4.0 " +
+      "https://github.com/apache/datafusion-comet/issues/1948")) {
--
     test (s"$label - ${conf.desc}") {
+
+      assume(!shouldSkip(conf), s"TODO: https://github.com/apache/datafusion-comet/issues/1948 " +
--
+      s"current thread context classloader (${udfInfo.identifier}",
+      IgnoreComet("TODO: ignore for first stage of 4.0 " +
+        "https://github.com/apache/datafusion-comet/issues/1948")) {

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[andygrove]

Files in Comet that need to be reviewed:

./common/src/test/java/org/apache/comet/parquet/TestFileReader.java:      if (!isSpark40Plus()) { // TODO: https://github.com/apache/datafusion-comet/issues/1948

Files in Spark 4.0.0 diff that need to be reviewed:

./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:    "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala:      "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/core/src/test/scala/org/apache/spark/sql/collation/CollationSuite.scala:      "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:      "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala:      "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/core/src/test/scala/org/apache/spark/sql/execution/InsertSortForLimitAndOffsetSuite.scala:      "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/core/src/test/scala/org/apache/spark/sql/execution/InsertSortForLimitAndOffsetSuite.scala:      "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/core/src/test/scala/org/apache/spark/sql/execution/InsertSortForLimitAndOffsetSuite.scala:      "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowIndexSuite.scala:      assume(!shouldSkip(conf), s"TODO: https://github.com/apache/datafusion-comet/issues/1948 " +
./sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:      "https://github.com/apache/datafusion-comet/issues/1948")) {
./sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveUDFDynamicLoadSuite.scala:        "https://github.com/apache/datafusion-comet/issues/1948")) {