Skip to content

[SPARK-24836][SQL] New option for Avro datasource - ignoreExtension #21798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 8 commits into from
Closed

[SPARK-24836][SQL] New option for Avro datasource - ignoreExtension #21798

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
cdb26f2
Adding new option - ignoreExtension
MaxGekk Jul 17, 2018
d07ced3
Revert tests for hadoop's config
MaxGekk Jul 17, 2018
e86d231
Adding the ignoreExtension option
MaxGekk Jul 17, 2018
565e599
Ticket's number to tests titles is added
MaxGekk Jul 17, 2018
d1c511c
Merge remote-tracking branch 'origin/master' into avro-ignore-extension
MaxGekk Jul 19, 2018
3bd3475
Moving the ignoreExtension to AvroOptions
MaxGekk Jul 19, 2018
0657508
Moving ignoreExtension to AvroOptions
MaxGekk Jul 20, 2018
3206a20
Adding option name to comment
MaxGekk Jul 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Adding option name to comment
  • Loading branch information
@MaxGekk
MaxGekk committed Jul 20, 2018
commit 3206a20fc9f3036e16eca20118e1559d722ff0b9
2 changes: 1 addition & 1 deletion external/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ class AvroOptions(
val recordNamespace: String = parameters.getOrElse("recordNamespace", "")

/**
* The option controls ignoring of files without `.avro` extensions in read.
* The `ignoreExtension` option controls ignoring of files without `.avro` extensions in read.
* If the option is enabled, all files (with and without `.avro` extension) are loaded.
* If the option is not set, the Hadoop's config `avro.mapred.ignore.inputs.without.extension`
* is taken into account. If the former one is not set too, file extensions are ignored.
Expand Down