Skip to content

[SPARK-41708][SQL][FOLLOWUP] Add a new replaceAll to SQLQueryTestHelper#replaceNotIncludedMsg to remove @hashCode #39598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferYang wants to merge 6 commits into from
+7 −6
Closed

[SPARK-41708][SQL][FOLLOWUP] Add a new replaceAll to SQLQueryTestHelper#replaceNotIncludedMsg to remove @hashCode #39598

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7bd488b
Override equals and hashCode for TableIdentifier
LuciferYang Jan 16, 2023
104cdac
update golden files
LuciferYang Jan 16, 2023
38be9a0
remove hashCode and revert change
LuciferYang Jan 16, 2023
b0f8c6c
revert change
LuciferYang Jan 16, 2023
f30204e
update golden files
LuciferYang Jan 16, 2023
bfe25b2
remove @hashCode
LuciferYang Jan 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/identifiers.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,13 @@ case class TableIdentifier(table: String, database: Option[String], catalog: Opt

def this(table: String) = this(table, None, None)
def this(table: String, database: Option[String]) = this(table, database, None)

override def equals(obj: Any): Boolean = obj match {
Copy link
Contributor Author

@LuciferYang LuciferYang Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @cloud-fan @HyukjinKwon @dongjoon-hyun @ulysses-you

The daily build GA Master Scala 2.13 + Hadoop 3 + JDK 8 failed after #39468 merged.

Are there any suggestions for better solutions? For example, override toString of CatalogFileIndex? What is better to print?

Copy link
Contributor Author

@LuciferYang LuciferYang Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently printing is org.apache.spark.sql.execution.datasources.CatalogFileIndex@hashCode

Copy link
Member

@dongjoon-hyun dongjoon-hyun Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The better solution would be to normalize the output by removing CatalogFileIndex@xxx part.

Copy link
Contributor Author

@LuciferYang LuciferYang Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just print org.apache.spark.sql.execution.datasources.CatalogFileIndex?

Copy link
Contributor

@ulysses-you ulysses-you Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can replace CatalogFileIndex@xxx to CatalogFileIndex in SQLQueryTestHelper#replaceNotIncludedMsg to recover test.

Copy link
Contributor Author

@LuciferYang LuciferYang Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ulysses-you do you mean

spark/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala

Line 173 in 2062c74

output

change to use SQLQueryTestHelper.replaceNotIncludedMsg(output) ?

Copy link
Contributor

@ulysses-you ulysses-you Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it had already used

case t: TableIdentifier => table == t.table && database == t.database && catalog == t.catalog
case _ => false
}

override def hashCode(): Int = unquotedString.hashCode
}

/** A fully qualified identifier for a table (i.e., database.tableName) */
Expand Down
6 changes: 3 additions & 3 deletions sql/core/src/test/resources/sql-tests/results/explain-aqe.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1086,20 +1086,20 @@ struct<plan:string>
+- 'UnresolvedRelation [explain_temp4], [], false

== Analyzed Logical Plan ==
InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@7d811218, [key, val]
InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@5c30f6b5, [key, val]
Copy link
Contributor

@cloud-fan cloud-fan Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's not include the hashCode in the golden files.

+- Project [key#x, val#x]
+- SubqueryAlias spark_catalog.default.explain_temp4
+- Relation spark_catalog.default.explain_temp4[key#x,val#x] parquet

== Optimized Logical Plan ==
InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@7d811218, [key, val]
InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@5c30f6b5, [key, val]
+- WriteFiles
+- Sort [val#x ASC NULLS FIRST], false
+- Project [key#x, empty2null(val#x) AS val#x]
+- Relation spark_catalog.default.explain_temp4[key#x,val#x] parquet

== Physical Plan ==
Execute InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@7d811218, [key, val]
Execute InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@5c30f6b5, [key, val]
+- WriteFiles
+- *Sort [val#x ASC NULLS FIRST], false, 0
+- *Project [key#x, empty2null(val#x) AS val#x]
Expand Down
6 changes: 3 additions & 3 deletions sql/core/src/test/resources/sql-tests/results/explain.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1028,20 +1028,20 @@ struct<plan:string>
+- 'UnresolvedRelation [explain_temp4], [], false

== Analyzed Logical Plan ==
InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@7d811218, [key, val]
InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@5c30f6b5, [key, val]
+- Project [key#x, val#x]
+- SubqueryAlias spark_catalog.default.explain_temp4
+- Relation spark_catalog.default.explain_temp4[key#x,val#x] parquet

== Optimized Logical Plan ==
InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@7d811218, [key, val]
InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@5c30f6b5, [key, val]
+- WriteFiles
+- Sort [val#x ASC NULLS FIRST], false
+- Project [key#x, empty2null(val#x) AS val#x]
+- Relation spark_catalog.default.explain_temp4[key#x,val#x] parquet

== Physical Plan ==
Execute InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@7d811218, [key, val]
Execute InsertIntoHadoopFsRelationCommand Location [not included in comparison]/{warehouse_dir}/explain_temp5], Append, `spark_catalog`.`default`.`explain_temp5`, org.apache.spark.sql.execution.datasources.CatalogFileIndex@5c30f6b5, [key, val]
+- WriteFiles
+- *Sort [val#x ASC NULLS FIRST], false, 0
+- *Project [key#x, empty2null(val#x) AS val#x]
Expand Down