Skip to content

[SPARK-24836][SQL] New option for Avro datasource - ignoreExtension #21798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 8 commits into from
Closed

[SPARK-24836][SQL] New option for Avro datasource - ignoreExtension #21798

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cdb26f2
Adding new option - ignoreExtension
MaxGekk Jul 17, 2018
d07ced3
Revert tests for hadoop's config
MaxGekk Jul 17, 2018
e86d231
Adding the ignoreExtension option
MaxGekk Jul 17, 2018
565e599
Ticket's number to tests titles is added
MaxGekk Jul 17, 2018
d1c511c
Merge remote-tracking branch 'origin/master' into avro-ignore-extension
MaxGekk Jul 19, 2018
3bd3475
Moving the ignoreExtension to AvroOptions
MaxGekk Jul 19, 2018
0657508
Moving ignoreExtension to AvroOptions
MaxGekk Jul 20, 2018
3206a20
Adding option name to comment
MaxGekk Jul 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert tests for hadoop's config
  • Loading branch information
@MaxGekk
MaxGekk committed Jul 17, 2018
commit d07ced332ba04ac18f270962d887f1e19e9155ec
26 changes: 25 additions & 1 deletion external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -620,6 +620,19 @@ class AvroSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
spark.read.avro("*/*/*/*/*/*/*/something.avro")
}

intercept[FileNotFoundException] {
withTempPath { dir =>
FileUtils.touch(new File(dir, "test"))
val hadoopConf = spark.sqlContext.sparkContext.hadoopConfiguration
try {
hadoopConf.set(AvroFileFormat.IgnoreFilesWithoutExtensionProperty, "true")
spark.read.avro(dir.toString)
} finally {
hadoopConf.unset(AvroFileFormat.IgnoreFilesWithoutExtensionProperty)
}
}
}

intercept[FileNotFoundException] {
withTempPath { dir =>
FileUtils.touch(new File(dir, "test"))
Expand All @@ -630,7 +643,6 @@ class AvroSuite extends QueryTest with SharedSQLContext with SQLTestUtils {
.avro(dir.toString)
}
}

}

test("SQL test insert overwrite") {
Expand Down Expand Up @@ -689,6 +701,18 @@ class AvroSuite extends QueryTest with SharedSQLContext with SQLTestUtils {

Files.createFile(new File(tempSaveDir, "non-avro").toPath)

val hadoopConf = spark.sqlContext.sparkContext.hadoopConfiguration
val count = try {
hadoopConf.set(AvroFileFormat.IgnoreFilesWithoutExtensionProperty, "true")
val newDf = spark
.read
.avro(tempSaveDir)
newDf.count()
} finally {
hadoopConf.unset(AvroFileFormat.IgnoreFilesWithoutExtensionProperty)
}
assert(count == 8)

val newDf = spark
.read
.option("ignoreExtension", false)
Expand Down