[SPARK-25260][SQL] Fix namespace handling in SchemaConverters.toAvroType #22251
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
Test build #95338 has finished for PR 22251 at commit
|
| } | ||
|
|
||
| val childNameSpace = if (nameSpace != "") s"$nameSpace.$recordName" else recordName | ||
| val fieldsAssembler = builder.record(recordName).namespace(nameSpace).fields() |
Member
There was a problem hiding this comment.
+1, this line is the only difference for the whole code change. The namespace here should not be the one with recordName at the end.
HyukjinKwon
reviewed
Aug 28, 2018
| } | ||
| } | ||
|
|
||
| test("check namespace - toAvroType") { |
Member
There was a problem hiding this comment.
@arunmahadevan, can we add a simple end-to-end test as well?
Contributor
Author
There was a problem hiding this comment.
Its sort of covered in the below existing cases. Do you think we need more?
Validate namespace in avro file that has nested records with the same name
conversion to avro and back with namespace
| nullable = false))) | ||
| val employeeType = SchemaConverters.toAvroType(sparkSchema, | ||
| recordName = "employee", | ||
| nameSpace = "foo.bar") |
Member
There was a problem hiding this comment.
nit: could you also add a case for nameSpace as "" ?
Contributor
Author
There was a problem hiding this comment.
Added a test case for toAvroType with empty namespace
gengliangwang
approved these changes
Aug 28, 2018
Member
gengliangwang
left a comment
gengliangwang
left a comment
There was a problem hiding this comment.
LGTM, thanks for the fix!
arunmahadevan
force-pushed
the
avro-fix
branch
from
5330c45 to
2153428
Compare
|
Test build #95367 has finished for PR 22251 at commit
|
Member
|
Merged to master. |
otterc
pushed a commit
to linkedin/spark
that referenced
this pull request
Mar 22, 2023
`toAvroType` converts spark data type to avro schema. It always appends the record name to namespace so its impossible to have an Avro namespace independent of the record name.
When invoked with a spark data type like,
```java
val sparkSchema = StructType(Seq(
StructField("name", StringType, nullable = false),
StructField("address", StructType(Seq(
StructField("city", StringType, nullable = false),
StructField("state", StringType, nullable = false))),
nullable = false)))
// map it to an avro schema with record name "employee" and top level namespace "foo.bar",
val avroSchema = SchemaConverters.toAvroType(sparkSchema, false, "employee", "foo.bar")
// result is
// avroSchema.getName = employee
// avroSchema.getNamespace = foo.bar.employee
// avroSchema.getFullname = foo.bar.employee.employee
```
The patch proposes to fix this so that the result is
```
avroSchema.getName = employee
avroSchema.getNamespace = foo.bar
avroSchema.getFullname = foo.bar.employee
```
New and existing unit tests.
Please review http://spark.apache.org/contributing.html before opening a pull request.
Closes apache#22251 from arunmahadevan/avro-fix.
Authored-by: Arun Mahadevan <[email protected]>
Signed-off-by: hyukjinkwon <[email protected]>
(cherry picked from commit 68ec207)
RB=2106636
BUG=LIHADOOP-53221
G=spark-reviewers
R=mshen,ekrogen
A=ekrogen
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
toAvroTypeconverts spark data type to avro schema. It always appends the record name to namespace so its impossible to have an Avro namespace independent of the record name.When invoked with a spark data type like,
The patch proposes to fix this so that the result is
How was this patch tested?
New and existing unit tests.
Please review http://spark.apache.org/contributing.html before opening a pull request.