Skip to content

[SPARK-23072][SQL][TEST] Add a Unicode schema test for file-based data sources #20266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 7 commits into from
Closed

[SPARK-23072][SQL][TEST] Add a Unicode schema test for file-based data sources #20266

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f9a35f1
[SPARK-23072][SQL] Add a Unicode schema test for file-based data sources
dongjoon-hyun Jan 14, 2018
60b8e43
Create FileBasedDataSourceSuite.
dongjoon-hyun Jan 15, 2018
144c596
Move to under `sql` package.
dongjoon-hyun Jan 15, 2018
5afaa28
Address comments.
dongjoon-hyun Jan 16, 2018
fb708b7
Move SPARK-22146, too.
dongjoon-hyun Jan 16, 2018
8fec65b
Remove unused imports.
dongjoon-hyun Jan 16, 2018
c67809c
Address comments.
dongjoon-hyun Jan 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Address comments.
  • Loading branch information
@dongjoon-hyun
dongjoon-hyun committed Jan 17, 2018
commit c67809c9dbe0a21011649dceededa84d73377d1c
15 changes: 10 additions & 5 deletions sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,19 @@ import org.apache.spark.sql.test.SharedSQLContext
class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
import testImplicits._

Seq("orc", "parquet", "csv", "json", "text").foreach { format =>
private val allFileBasedDataSources = Seq("orc", "parquet", "csv", "json", "text")

allFileBasedDataSources.foreach { format =>
test(s"Writing empty datasets should not fail - $format") {
withTempPath { dir =>
Seq("str").toDS().limit(0).write.format(format).save(dir.getCanonicalPath)
}
}
}

Seq("orc", "parquet", "csv", "json").foreach { format =>
test(s"SPARK-23072 Write and read back unicode schema - $format") {
// `TEXT` data source always has a single column whose name is `value`.
allFileBasedDataSources.filterNot(_ == "text").foreach { format =>
test(s"SPARK-23072 Write and read back unicode column names - $format") {
withTempPath { path =>
val dir = path.getCanonicalPath

Expand All @@ -42,12 +45,14 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
df.write.format(format).option("header", "true").save(dir)
val answerDf = spark.read.format(format).option("header", "true").load(dir)

assert(df.schema === answerDf.schema)
assert(df.schema.sameType(answerDf.schema))
checkAnswer(df, answerDf)
}
}
}

// Only ORC/Parquet support this. `CSV` and `JSON` returns an empty schema.
// `TEXT` data source always has a single column whose name is `value`.
Seq("orc", "parquet").foreach { format =>
Copy link
Member

@gatorsmile gatorsmile Jan 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only these two formats support it? If so, please add the comments.

This is the same comment to the other test cases. Otherwise, add all of them for each test case.

You can define a global Seq to include all the built-in file formats we support.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Jan 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

  1. Only two support this. I added comments.
  2. For the other test cases, I did.
  3. I added a global Seq, allFileBasedDataSources.

test(s"SPARK-15474 Write and read back non-emtpy schema with empty dataframe - $format") {
withTempPath { file =>
Expand All @@ -62,7 +67,7 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
}
}

Seq("orc", "parquet", "csv", "json", "text").foreach { format =>
allFileBasedDataSources.foreach { format =>
test(s"SPARK-22146 read files containing special characters using $format") {
val nameWithSpecialChars = s"sp&cial%chars"
withTempDir { dir =>
Expand Down