[SPARK-37452][SQL] Char and Varchar break backward compatibility between v3.1 and v2 #34697

yaooqinn · 2021-11-24T10:28:16Z

What changes were proposed in this pull request?

We will store table schema in table properties for the read-side to restore. In Spark 3.1, we add char/varchar support natively. In some commands like create table, alter table with these types, the char(x) or varchar(x) will be stored directly to those properties. If a user uses Spark 2 to read such a table it will fail to parse the schema.

FYI, https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala#L136

A table can be a newly created one by Spark 3.1 and later or an existing one modified by Spark 3.1 and on.

Why are the changes needed?

backward compatibility

Does this PR introduce any user-facing change?

That's not necessarily user-facing as a bugfix and only related to internal table properties.

How was this patch tested?

manully

…ween v3 and v2

SparkQA · 2021-11-24T11:45:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50049/

SparkQA · 2021-11-24T12:22:47Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50049/

SparkQA · 2021-11-24T12:24:38Z

Test build #145576 has finished for PR 34697 at commit 1934285.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2021-11-24T12:24:39Z

cc @cloud-fan thanks

SparkQA · 2021-11-24T13:02:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50052/

cloud-fan · 2021-11-24T13:32:25Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala

    val schemaJson = CatalogTable.readLargeTableProp(table.properties, DATASOURCE_SCHEMA)
    if (schemaJson.isDefined) {
      val schemaFromTableProps = DataType.fromJson(schemaJson.get).asInstanceOf[StructType]
+      val rawSchema = CharVarcharUtils.getRawSchema(schemaFromTableProps)


Maybe we should create a method getSchemaFromTableProperties and call it in these places.

SparkQA · 2021-11-24T14:02:51Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50052/

SparkQA · 2021-11-24T14:14:55Z

Test build #145580 has finished for PR 34697 at commit 799a761.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-25T02:26:26Z

Test build #145603 has finished for PR 34697 at commit 0334309.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-25T02:35:46Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50075/

SparkQA · 2021-11-25T04:23:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50079/

SparkQA · 2021-11-25T05:05:13Z

Test build #145605 has finished for PR 34697 at commit d3a284f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-25T05:13:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50079/

mridulm · 2021-11-25T05:34:48Z

FYI @xkrogen, @shardulm94

yaooqinn · 2021-11-29T06:15:18Z

any more concerns from the CCers？

…een v3.1 and v2 ### What changes were proposed in this pull request? We will store table schema in table properties for the read-side to restore. In Spark 3.1, we add char/varchar support natively. In some commands like `create table`, `alter table` with these types, the `char(x)` or `varchar(x)` will be stored directly to those properties. If a user uses Spark 2 to read such a table it will fail to parse the schema. FYI, https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala#L136 A table can be a newly created one by Spark 3.1 and later or an existing one modified by Spark 3.1 and on. ### Why are the changes needed? backward compatibility ### Does this PR introduce _any_ user-facing change? That's not necessarily user-facing as a bugfix and only related to internal table properties. ### How was this patch tested? manully Closes #34697 from yaooqinn/SPARK-37452. Authored-by: Kent Yao <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit 0c3c4e2) Signed-off-by: Kent Yao <[email protected]>

yaooqinn · 2021-11-29T09:21:25Z

merged to master and branch 3.2, will create a backport PR to 3.1 and link it to this later

… between v3.1 and v2 This backports #34697 to 3.1 ### What changes were proposed in this pull request? We will store table schema in table properties for the read-side to restore. In Spark 3.1, we add char/varchar support natively. In some commands like `create table`, `alter table` with these types, the `char(x)` or `varchar(x)` will be stored directly to those properties. If a user uses Spark 2 to read such a table it will fail to parse the schema. FYI, https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala#L136 A table can be a newly created one by Spark 3.1 and later or an existing one modified by Spark 3.1 and on. ### Why are the changes needed? backward compatibility ### Does this PR introduce _any_ user-facing change? That's not necessarily user-facing as a bugfix and only related to internal table properties. ### How was this patch tested? manully Closes #34736 from yaooqinn/PR_TOOL_PICK_PR_34697_BRANCH-3.1. Authored-by: Kent Yao <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…een v3.1 and v2 ### What changes were proposed in this pull request? We will store table schema in table properties for the read-side to restore. In Spark 3.1, we add char/varchar support natively. In some commands like `create table`, `alter table` with these types, the `char(x)` or `varchar(x)` will be stored directly to those properties. If a user uses Spark 2 to read such a table it will fail to parse the schema. FYI, https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala#L136 A table can be a newly created one by Spark 3.1 and later or an existing one modified by Spark 3.1 and on. ### Why are the changes needed? backward compatibility ### Does this PR introduce _any_ user-facing change? That's not necessarily user-facing as a bugfix and only related to internal table properties. ### How was this patch tested? manully Closes apache#34697 from yaooqinn/SPARK-37452. Authored-by: Kent Yao <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit 0c3c4e2) Signed-off-by: Kent Yao <[email protected]>

… between v3.1 and v2 This backports apache#34697 to 3.1 ### What changes were proposed in this pull request? We will store table schema in table properties for the read-side to restore. In Spark 3.1, we add char/varchar support natively. In some commands like `create table`, `alter table` with these types, the `char(x)` or `varchar(x)` will be stored directly to those properties. If a user uses Spark 2 to read such a table it will fail to parse the schema. FYI, https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala#L136 A table can be a newly created one by Spark 3.1 and later or an existing one modified by Spark 3.1 and on. ### Why are the changes needed? backward compatibility ### Does this PR introduce _any_ user-facing change? That's not necessarily user-facing as a bugfix and only related to internal table properties. ### How was this patch tested? manully Closes apache#34736 from yaooqinn/PR_TOOL_PICK_PR_34697_BRANCH-3.1. Authored-by: Kent Yao <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…een v3.1 and v2 ### What changes were proposed in this pull request? We will store table schema in table properties for the read-side to restore. In Spark 3.1, we add char/varchar support natively. In some commands like `create table`, `alter table` with these types, the `char(x)` or `varchar(x)` will be stored directly to those properties. If a user uses Spark 2 to read such a table it will fail to parse the schema. FYI, https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala#L136 A table can be a newly created one by Spark 3.1 and later or an existing one modified by Spark 3.1 and on. ### Why are the changes needed? backward compatibility ### Does this PR introduce _any_ user-facing change? That's not necessarily user-facing as a bugfix and only related to internal table properties. ### How was this patch tested? manully Closes apache#34697 from yaooqinn/SPARK-37452. Authored-by: Kent Yao <[email protected]> Signed-off-by: Kent Yao <[email protected]> (cherry picked from commit 0c3c4e2) Signed-off-by: Kent Yao <[email protected]>

[SPARK-37452][SQL] Char and Varchar breaks backward compatibility bet…

0a479a3

…ween v3 and v2

github-actions bot added the SQL label Nov 24, 2021

[SPARK-37452][SQL] Char and Varchar breaks backward compatibility bet…

1934285

…ween v3 and v2

yaooqinn added 2 commits November 24, 2021 20:13

read path: restore raw schema from json

6e29c55

nit

799a761

cloud-fan reviewed Nov 24, 2021

View reviewed changes

cloud-fan approved these changes Nov 24, 2021

View reviewed changes

yaooqinn changed the title ~~[SPARK-37452][SQL] Char and Varchar breaks backward compatibility between v3.1 and v2~~ [SPARK-37452][SQL] Char and Varchar break backward compatibility between v3.1 and v2 Nov 25, 2021

nit

0334309

fix build

d3a284f

cloud-fan approved these changes Nov 25, 2021

View reviewed changes

yaooqinn closed this in 0c3c4e2 Nov 29, 2021

yaooqinn deleted the SPARK-37452 branch November 29, 2021 09:21

yaooqinn mentioned this pull request Nov 29, 2021

[SPARK-37452][SQL][3.1] Char and Varchar break backward compatibility between v3.1 and v2 #34736

Closed

[SPARK-37452][SQL] Char and Varchar break backward compatibility between v3.1 and v2 #34697

[SPARK-37452][SQL] Char and Varchar break backward compatibility between v3.1 and v2 #34697

Uh oh!

Conversation

yaooqinn commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

yaooqinn commented Nov 24, 2021

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

cloud-fan Nov 24, 2021

Choose a reason for hiding this comment

Uh oh!

yaooqinn Nov 25, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

SparkQA commented Nov 24, 2021

Uh oh!

SparkQA commented Nov 25, 2021

Uh oh!

SparkQA commented Nov 25, 2021

Uh oh!

SparkQA commented Nov 25, 2021

Uh oh!

SparkQA commented Nov 25, 2021

Uh oh!

SparkQA commented Nov 25, 2021

Uh oh!

mridulm commented Nov 25, 2021

Uh oh!

yaooqinn commented Nov 29, 2021

Uh oh!

yaooqinn commented Nov 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yaooqinn commented Nov 24, 2021 •

edited

Loading

yaooqinn commented Nov 29, 2021 •

edited

Loading