Skip to content

[SPARK-20427][SQL] Read JDBC table use custom schema #18266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 20 commits into from
+222 −10
Closed

[SPARK-20427][SQL] Read JDBC table use custom schema #18266

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
871c303
Read JDBC table use custom schema
wangyum Jun 11, 2017
0444c4d
Fix test error.
wangyum Jun 11, 2017
06881e8
Revert api.
wangyum Jun 15, 2017
ffaee42
Improve custom schema.
wangyum Jun 23, 2017
a984f3b
Improve custom schema.
wangyum Jun 23, 2017
e0fc6b4
Throw exception if custom schema field names does not match table col…
wangyum Jul 4, 2017
9e6f7cf
Merge branch 'master' into SPARK-20427
wangyum Jul 4, 2017
5fdd2bb
Remove SPARK-16848: jdbc API throws an exception for user specified s…
wangyum Jul 4, 2017
87df014
Support DDL-like JDBC custom schema.
wangyum Aug 16, 2017
63d3244
Merge branch 'master' into SPARK-20427
wangyum Aug 16, 2017
e08ccbb
Merge remote-tracking branch 'origin/master' into SPARK-20427
wangyum Aug 22, 2017
247fc78
Refactor code.
wangyum Aug 23, 2017
b8b03e2
Revert wrong change
wangyum Aug 23, 2017
b040c72
Fix python style
wangyum Aug 23, 2017
2ea56fc
Fix test error
wangyum Aug 23, 2017
1e2c1d9
Let user specified all the columns.
wangyum Aug 24, 2017
b38a1a8
Change all SQL type to upper case.
wangyum Sep 8, 2017
0b67f0f
customDataFrameColumnTypes -> customSchema
wangyum Sep 10, 2017
7fc97b4
revert assertNoSpecifiedSchema
wangyum Sep 10, 2017
1fdf002
parseUserSpecifiedColumnTypes -> getCustomSchema
wangyum Sep 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Improve custom schema.
  • Loading branch information
@wangyum
wangyum committed Jun 23, 2017
commit a984f3b02f17a43c9f0333351a462f8722ece7fb
4 changes: 2 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ object JDBCRDD extends Logging {
* @return A Catalyst schema corresponding to columns in the given order.
*/
private def pruneSchema(schema: StructType, columns: Array[String]): StructType = {
val fieldMap = Map(schema.fields.map(x => x.name -> x): _*)
new StructType(columns.map(name => fieldMap(name)))
val fieldMap = Map(schema.fields.map(x => x.name -> x): _*)
Copy link
Member Author

@wangyum wangyum Jun 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

x.metadata.getString("name") always equals x.name:
https://github.com/apache/spark/blob/v2.2.0-rc5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L291

Copy link
Member

@gatorsmile gatorsmile Aug 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a related change. Could you revert it back?

Copy link
Member Author

@wangyum wangyum Aug 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CatalystSqlParser.parseTableSchema(columnTypes) constructed StructType without metadata, error message:

key not found: name
java.util.NoSuchElementException: key not found: name
	at scala.collection.MapLike$class.default(MapLike.scala:228)
	at scala.collection.AbstractMap.default(Map.scala:59)
	at scala.collection.MapLike$class.apply(MapLike.scala:141)
	at scala.collection.AbstractMap.apply(Map.scala:59)
	at org.apache.spark.sql.types.Metadata.get(Metadata.scala:111)
	at org.apache.spark.sql.types.Metadata.getString(Metadata.scala:60)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anonfun$1.apply(JDBCRDD.scala:83)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anonfun$1.apply(JDBCRDD.scala:83)

Copy link
Member

@gatorsmile gatorsmile Sep 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I did not get your point. Could you show me an example? Is it a behavior breaking change?

Copy link
Member Author

@wangyum wangyum Sep 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

scala> org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parseTableSchema("id int, name string").fields.map(x => x.metadata.getString("name") -> x)
java.util.NoSuchElementException: key not found: name
  at scala.collection.MapLike$class.default(MapLike.scala:228)
  at scala.collection.AbstractMap.default(Map.scala:59)
  at scala.collection.MapLike$class.apply(MapLike.scala:141)
  at scala.collection.AbstractMap.apply(Map.scala:59)
  at org.apache.spark.sql.types.Metadata.get(Metadata.scala:111)
  at org.apache.spark.sql.types.Metadata.getString(Metadata.scala:60)
  at $anonfun$1.apply(<console>:24)
  at $anonfun$1.apply(<console>:24)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
  at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
  ... 48 elided

Copy link
Member

@gatorsmile gatorsmile Sep 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Could we just get rid of the line where we put name in the metadata?

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L304

Copy link
Member Author

@wangyum wangyum Sep 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems safe to remove this line.

new StructType(columns.map(name => fieldMap(name)))
}

/**
Expand Down