Skip to content

[SPARK-30509][SQL] Fix deprecation log warning in Avro schema inferring #27200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 5 commits into from
Closed

[SPARK-30509][SQL] Fix deprecation log warning in Avro schema inferring #27200

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
47879d9
Add a test
MaxGekk Jan 14, 2020
0bb33aa
Bug fix
MaxGekk Jan 14, 2020
11cbf2e
Remove .option("header", true) in the added test
MaxGekk Jan 14, 2020
6c53b50
Remove .option("header", true) in another test
MaxGekk Jan 14, 2020
164cf5b
Revert "Remove .option("header", true) in another test"
MaxGekk Jan 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Bug fix
  • Loading branch information
@MaxGekk
MaxGekk committed Jan 14, 2020
commit 0bb33aa15c813af5b12798b666f782cc2feb7741
9 changes: 5 additions & 4 deletions external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import org.apache.hadoop.mapreduce.Job
import org.apache.spark.SparkException
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.avro.AvroOptions.ignoreExtensionKey
import org.apache.spark.sql.execution.datasources.OutputWriterFactory
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
Expand All @@ -42,12 +43,12 @@ object AvroUtils extends Logging {
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
val conf = spark.sessionState.newHadoopConf()
if (options.contains("ignoreExtension")) {
logWarning(s"Option ${AvroOptions.ignoreExtensionKey} is deprecated. Please use the " +
"general data source option pathGlobFilter for filtering file names.")
}
val parsedOptions = new AvroOptions(options, conf)

if (parsedOptions.parameters.contains(ignoreExtensionKey)) {
logWarning(s"Option $ignoreExtensionKey is deprecated. Please use the " +
"general data source option pathGlobFilter for filtering file names.")
}
// User can specify an optional avro json schema.
val avroSchema = parsedOptions.schema
.map(new Schema.Parser().parse)
Expand Down