Skip to content

[SPARK-48545][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents #46977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dtenedor wants to merge 13 commits into from
Closed

[SPARK-48545][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents #46977

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a7c08a6
commit
dtenedor Jun 12, 2024
4241540
add descriptions
dtenedor Jun 13, 2024
9a36eac
update expression schema suite
dtenedor Jun 13, 2024
ac69741
add examples
dtenedor Jun 14, 2024
1e09801
respond to code review comments
dtenedor Jun 15, 2024
ddbb57f
fix test
dtenedor Jun 18, 2024
16161e4
fix
dtenedor Jun 20, 2024
3bdb1f1
Merge branch 'master' into from-avro
dtenedor Jun 20, 2024
f5e5f64
fix test
dtenedor Jun 20, 2024
d1f1844
fix test
dtenedor Jun 20, 2024
50f728f
fix test
dtenedor Jun 20, 2024
fa7ca52
fix compile
dtenedor Jun 20, 2024
aa26df3
fix compile
dtenedor Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix
  • Loading branch information
@dtenedor
dtenedor committed Jun 20, 2024
commit 16161e41a0fa67673b2edab49b54001490ef4936
6 changes: 3 additions & 3 deletions connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ class AvroFunctionsSuite extends QueryTest with SharedSparkSession {
|select to_avro(s, 42) as result from t
|""".stripMargin)),
errorClass = "DATATYPE_MISMATCH.TYPE_CHECK_FAILURE_WITH_HINT",
parameters = Map("sqlExpr" -> "\"toavrosqlfunction(s, 42)\"",
parameters = Map("sqlExpr" -> "\"toavro(s, 42)\"",
"msg" -> ("The second argument of the TO_AVRO SQL function must be a constant string " +
"containing the JSON representation of the schema to use for converting the value to " +
"AVRO format"),
Expand All @@ -339,7 +339,7 @@ class AvroFunctionsSuite extends QueryTest with SharedSparkSession {
|select from_avro(s, 42, '') as result from t
|""".stripMargin)),
errorClass = "DATATYPE_MISMATCH.TYPE_CHECK_FAILURE_WITH_HINT",
parameters = Map("sqlExpr" -> "\"fromavrosqlfunction(s, 42, )\"",
parameters = Map("sqlExpr" -> "\"fromavro(s, 42, )\"",
"msg" -> ("The second argument of the FROM_AVRO SQL function must be a constant string " +
"containing the JSON representation of the schema to use for converting the value " +
"from AVRO format"),
Expand All @@ -356,7 +356,7 @@ class AvroFunctionsSuite extends QueryTest with SharedSparkSession {
errorClass = "DATATYPE_MISMATCH.TYPE_CHECK_FAILURE_WITH_HINT",
parameters = Map(
"sqlExpr" ->
s"\"fromavrosqlfunction(s, $jsonFormatSchema, 42)\"".stripMargin,
s"\"fromavro(s, $jsonFormatSchema, 42)\"".stripMargin,
"msg" -> ("The third argument of the FROM_AVRO SQL function must be a constant map of " +
"strings to strings containing the options to use for converting the value " +
"from AVRO format"),
Expand Down