Skip to content

[SPARK-48545][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents #46977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dtenedor wants to merge 13 commits into from
Closed

[SPARK-48545][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents #46977

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a7c08a6
commit
dtenedor Jun 12, 2024
4241540
add descriptions
dtenedor Jun 13, 2024
9a36eac
update expression schema suite
dtenedor Jun 13, 2024
ac69741
add examples
dtenedor Jun 14, 2024
1e09801
respond to code review comments
dtenedor Jun 15, 2024
ddbb57f
fix test
dtenedor Jun 18, 2024
16161e4
fix
dtenedor Jun 20, 2024
3bdb1f1
Merge branch 'master' into from-avro
dtenedor Jun 20, 2024
f5e5f64
fix test
dtenedor Jun 20, 2024
d1f1844
fix test
dtenedor Jun 20, 2024
50f728f
fix test
dtenedor Jun 20, 2024
fa7ca52
fix compile
dtenedor Jun 20, 2024
aa26df3
fix compile
dtenedor Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix test
  • Loading branch information
@dtenedor
dtenedor committed Jun 18, 2024
commit ddbb57fdf42c4fa261eac76e2dabfb07ff9cf01a
4 changes: 2 additions & 2 deletions ...yst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromAvroSqlFunctions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ import org.apache.spark.util.Utils
To deserialize the data with a compatible and evolved schema, the expected Avro schema can be
set via the corresponding option.
""",
group = "avro_funcs",
group = "misc_funcs",
since = "4.0.0"
)
// scalastyle:on line.size.limit
Expand Down Expand Up @@ -134,7 +134,7 @@ case class FromAvro(child: Expression, jsonFormatSchema: Expression, options: Ex
> SELECT IS_NULL(_FUNC_(result, '{"type": "record", "name": "struct", "fields": [{ "name": "u", "type": ["int","string"] }]}', MAP() ).u FROM (SELECT FROM_AVRO(result, '{"type": "record", "name": "struct", "fields": [{ "name": "u", "type": ["int","string"] }]}' ) AS result FROM (SELECT NAMED_STRUCT('u', NAMED_STRUCT('member0', member0, 'member1', member1)) AS s FROM VALUES (1, NULL), (NULL, 'a') tab(member0, member1)));
[false]
""",
group = "avro_funcs",
group = "misc_funcs",
since = "4.0.0"
)
// scalastyle:on line.size.limit
Expand Down