Skip to content

[SPARK-25425][SQL][BACKPORT-2.3] Extra options should override session options in DataSource V2 #22489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 4 commits into from
Closed

[SPARK-25425][SQL][BACKPORT-2.3] Extra options should override session options in DataSource V2 #22489

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0cc1741
[SPARK-25425][SQL] Extra options should override session options in D…
MaxGekk Sep 16, 2018
744c5b8
Adding missing import after merge to 2.4
MaxGekk Sep 19, 2018
c87a6de
Fix merge
MaxGekk Sep 19, 2018
f8b5aa6
Adding options to Reader
MaxGekk Sep 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Adding options to Reader
  • Loading branch information
@MaxGekk
MaxGekk committed Sep 20, 2018
commit f8b5aa6b3d1fc9e953bcfef8f23aeb6f3ab7818d
21 changes: 11 additions & 10 deletions sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -326,9 +326,9 @@ class DataSourceV2Suite extends QueryTest with SharedSQLContext {
.option(optionName, false)
.format(classOf[DataSourceV2WithSessionConfig].getName).load()
val options = df.queryExecution.optimizedPlan.collectFirst {
case d: DataSourceV2Relation => d.options
case DataSourceV2Relation(_, SimpleDataSourceV2Reader(options)) => options
}
assert(options.get.get(optionName) == Some("false"))
assert(options.get.getBoolean(optionName, true) == false)
}
}

Expand All @@ -350,17 +350,18 @@ class DataSourceV2Suite extends QueryTest with SharedSQLContext {
}
}

class SimpleDataSourceV2 extends DataSourceV2 with ReadSupport {

class Reader extends DataSourceReader {
override def readSchema(): StructType = new StructType().add("i", "int").add("j", "int")
case class SimpleDataSourceV2Reader(options: DataSourceOptions) extends DataSourceReader {
Copy link
Member

@dongjoon-hyun dongjoon-hyun Sep 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this non-trivial backporting, @MaxGekk !

override def readSchema(): StructType = new StructType().add("i", "int").add("j", "int")

override def createDataReaderFactories(): JList[DataReaderFactory[Row]] = {
java.util.Arrays.asList(new SimpleDataReaderFactory(0, 5), new SimpleDataReaderFactory(5, 10))
}
override def createDataReaderFactories(): JList[DataReaderFactory[Row]] = {
java.util.Arrays.asList(new SimpleDataReaderFactory(0, 5), new SimpleDataReaderFactory(5, 10))
}
}

override def createReader(options: DataSourceOptions): DataSourceReader = new Reader
class SimpleDataSourceV2 extends DataSourceV2 with ReadSupport {
override def createReader(options: DataSourceOptions): DataSourceReader = {
SimpleDataSourceV2Reader(options)
}
}

class SimpleDataReaderFactory(start: Int, end: Int)
Expand Down