Skip to content

[SPARK-30490][SQL] Eliminate compiler warnings in Avro datasource #27174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 3 commits into from
Closed

[SPARK-30490][SQL] Eliminate compiler warnings in Avro datasource #27174

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4e1065a
Output log warning
MaxGekk Jan 11, 2020
78077eb
Remove @deprecated
MaxGekk Jan 11, 2020
e25f195
Remove logWarning from inferSchema()
MaxGekk Jan 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Remove logWarning from inferSchema()
  • Loading branch information
@MaxGekk
MaxGekk committed Jan 11, 2020
commit e25f1954d62972476e3bb4bdb58fa6fd8921d51c
4 changes: 0 additions & 4 deletions external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,6 @@ object AvroUtils extends Logging {
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
val conf = spark.sessionState.newHadoopConf()
if (options.contains("ignoreExtension")) {
Copy link
Member

@HyukjinKwon HyukjinKwon Jan 13, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MaxGekk, let's just remove this option after branch-3.0 is cut out.

Copy link
Member Author

@MaxGekk MaxGekk Jan 13, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be deprecated explicitly for users before removing? It should be mentioned in docs at least if we don't want to output a warning like in the PR.

Copy link
Member

@HyukjinKwon HyukjinKwon Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it still shows the warning properly although it only shows during schema inference. Yeah, can you simply fix the doc and say it's deprecated at docs/sql-data-sources-avro.md?

Copy link
Member

@gengliangwang gengliangwang Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 with @HyukjinKwon
Let's remove the option and document it in the future, instead of creating such changes. If we merge this one, then there might be some other options we have to do the same thing.

Copy link
Member Author

@MaxGekk MaxGekk Jan 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the PR #27194 for docs

logWarning(s"Option ${AvroOptions.ignoreExtensionKey} is deprecated. Please use the " +
"general data source option pathGlobFilter for filtering file names.")
}
val parsedOptions = new AvroOptions(options, conf)

// User can specify an optional avro json schema.
Expand Down