[SPARK-41538][SQL] Metadata column should be appended at the end of project list

[gengliangwang]

What changes were proposed in this pull request?

For the following query:

CREATE TABLE table_1 (
  a ARRAY<STRING>,
 s STRUCT<id: STRING>)
USING parquet;

CREATE VIEW view_1 (id)
AS WITH source AS (
    SELECT * FROM table_1
),
renamed AS (
    SELECT
     s.id
    FROM source
)
SELECT id FROM renamed;

with foo AS (
  SELECT 'a' as id
),
bar AS (
  SELECT 'a' as id
)
SELECT
  1
FROM foo
FULL OUTER JOIN bar USING(id)
FULL OUTER JOIN view_1 USING(id)
WHERE foo.id IS NOT NULL

There will be the following error:

class org.apache.spark.sql.types.ArrayType cannot be cast to class org.apache.spark.sql.types.StructType (org.apache.spark.sql.types.ArrayType and org.apache.spark.sql.types.StructType are in unnamed module of loader 'app')
java.lang.ClassCastException: class org.apache.spark.sql.types.ArrayType cannot be cast to class org.apache.spark.sql.types.StructType (org.apache.spark.sql.types.ArrayType and org.apache.spark.sql.types.StructType are in unnamed module of loader 'app')
    at org.apache.spark.sql.catalyst.expressions.GetStructField.childSchema$lzycompute(complexTypeExtractors.scala:108)
    at org.apache.spark.sql.catalyst.expressions.GetStructField.childSchema(complexTypeExtractors.scala:108)

This is caused by the inconsistent metadata column positions in the following two nodes:

• Table relation: at the ending position
• Project list: at the beginning position

When the InlineCTE rule executes, the metadata column in the project is wrongly combined with the table output.

Thus the column a ARRAY<STRING> is casted as s STRUCT<id: STRING> and cause the error.

This PR is to fix the issue by putting the Metadata column at the end of project list, so that it is consistent with the table relation.

Why are the changes needed?

Bug fix

Does this PR introduce any user-facing change?

Yes, it fixes a bug in the analysis rule AddMetadataColumns

How was this patch tested?

New test case

[gengliangwang]

The full outer join will trigger the rule AddMetadataColumns to add meta columns over the whole plan.
Before rule AddMetadataColumns:

!Filter isnotnull(id#223)
+- Project [coalesce(id#227, id#225) AS id#228]
   +- Join FullOuter, (id#227 = id#225)
      :- Project [coalesce(id#223, id#224) AS id#227]
      :  +- Join FullOuter, (id#223 = id#224)
      :     :- SubqueryAlias foo
      :     :  +- CTERelationRef 2, true, [id#223]
      :     +- SubqueryAlias bar
      :        +- CTERelationRef 3, true, [id#224]
      +- SubqueryAlias spark_catalog.default.view_1
         +- View (`spark_catalog`.`default`.`view_1`, [id#225])
            +- Project [cast(id#226 as string) AS id#225]
               +- WithCTE
                  :- CTERelationDef 4, false
                  :  +- SubqueryAlias source
                  :     +- Project [a#218, s#219]
                  :        +- SubqueryAlias spark_catalog.default.table_1
                  :           +- Relation spark_catalog.default.table_1[a#218,s#219] parquet
                  :- CTERelationDef 5, false
                  :  +- SubqueryAlias renamed
                  :     +- Project [s#219.id AS id#226]
                  :        +- SubqueryAlias source
                  :           +- CTERelationRef 4, true, [a#218, s#219]
                  +- Project [id#226]
                     +- SubqueryAlias renamed
                        +- CTERelationRef 5, true, [id#226]

After the rule:

Filter isnotnull(id#223)
+- Project [id#227, id#225, id#223, id#224, coalesce(id#227, id#225) AS id#228]
   +- Join FullOuter, (id#227 = id#225)
      :- Project [id#223, id#224, coalesce(id#223, id#224) AS id#227]
      :  +- Join FullOuter, (id#223 = id#224)
      :     :- SubqueryAlias foo
      :     :  +- CTERelationRef 2, true, [id#223]
      :     +- SubqueryAlias bar
      :        +- CTERelationRef 3, true, [id#224]
      +- SubqueryAlias spark_catalog.default.view_1
         +- View (`spark_catalog`.`default`.`view_1`, [id#225])
            +- Project [cast(id#226 as string) AS id#225]
               +- WithCTE
                  :- CTERelationDef 4, false
                  :  +- SubqueryAlias source
                  :     +- Project [_metadata#221, a#218, s#219]
                  :        +- SubqueryAlias spark_catalog.default.table_1
                  :           +- Relation spark_catalog.default.table_1[a#218,s#219,_metadata#221] parquet
                  :- CTERelationDef 5, false
                  :  +- SubqueryAlias renamed
                  :     +- Project [s#219.id AS id#226]
                  :        +- SubqueryAlias source
                  :           +- CTERelationRef 4, true, [a#218, s#219]
                  +- Project [id#226]
                     +- SubqueryAlias renamed
                        +- CTERelationRef 5, true, [id#226]

[dongjoon-hyun]

Does this cause any user-facing query result change? I guess it's not. Just want to get your confirmation.

[gengliangwang]

I don't think so. The analyzer adds the metadata columns so the parent operators can access theym. If the metadata columns are not used, they will be pruned.

[dongjoon-hyun]

+1, LGTM. Thank you, @gengliangwang .

[MaxGekk]

+1, LGTM. Merging to 3.3/master.
Thank you, @gengliangwang and @dongjoon-hyun @viirya for review.

[cloud-fan]

late LGTM