Skip to content

[SPARK-20427][SQL] Read JDBC table use custom schema #18266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 20 commits into from
Closed

[SPARK-20427][SQL] Read JDBC table use custom schema #18266

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
871c303
Read JDBC table use custom schema
wangyum Jun 11, 2017
0444c4d
Fix test error.
wangyum Jun 11, 2017
06881e8
Revert api.
wangyum Jun 15, 2017
ffaee42
Improve custom schema.
wangyum Jun 23, 2017
a984f3b
Improve custom schema.
wangyum Jun 23, 2017
e0fc6b4
Throw exception if custom schema field names does not match table col…
wangyum Jul 4, 2017
9e6f7cf
Merge branch 'master' into SPARK-20427
wangyum Jul 4, 2017
5fdd2bb
Remove SPARK-16848: jdbc API throws an exception for user specified s…
wangyum Jul 4, 2017
87df014
Support DDL-like JDBC custom schema.
wangyum Aug 16, 2017
63d3244
Merge branch 'master' into SPARK-20427
wangyum Aug 16, 2017
e08ccbb
Merge remote-tracking branch 'origin/master' into SPARK-20427
wangyum Aug 22, 2017
247fc78
Refactor code.
wangyum Aug 23, 2017
b8b03e2
Revert wrong change
wangyum Aug 23, 2017
b040c72
Fix python style
wangyum Aug 23, 2017
2ea56fc
Fix test error
wangyum Aug 23, 2017
1e2c1d9
Let user specified all the columns.
wangyum Aug 24, 2017
b38a1a8
Change all SQL type to upper case.
wangyum Sep 8, 2017
0b67f0f
customDataFrameColumnTypes -> customSchema
wangyum Sep 10, 2017
7fc97b4
revert assertNoSpecifiedSchema
wangyum Sep 10, 2017
1fdf002
parseUserSpecifiedColumnTypes -> getCustomSchema
wangyum Sep 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
revert assertNoSpecifiedSchema
  • Loading branch information
@wangyum
wangyum committed Sep 10, 2017
commit 7fc97b49d21fa05bca819ee93b72be084b3e67c7
2 changes: 1 addition & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ class JdbcUtilsSuite extends SparkFunSuite {
StructType(Seq(StructField("c1", DateType, true), StructField("c1", StringType, true)))
}
assert(duplicate.getMessage.contains(
"Found duplicate column(s) in the createTableColumnTypes option value"))
"Found duplicate column(s) in the customSchema option value"))

val allColumns = intercept[AnalysisException]{
JdbcUtils.parseUserSpecifiedColumnTypes(schema, "C1 STRING", caseSensitive) ===
Expand Down
4 changes: 2 additions & 2 deletions sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -960,12 +960,12 @@ class JDBCSuite extends SparkFunSuite
val e1 = intercept[AnalysisException] {
spark.read.schema(schema).jdbc(urlWithUserAndPass, "TEST.PEOPLE", parts, new Properties())
}.getMessage
assert(e1.contains("Please use customSchema option to specified column types."))
assert(e1.contains("User specified schema not supported with `jdbc`"))

val e2 = intercept[AnalysisException] {
spark.read.schema(schema).jdbc(urlWithUserAndPass, "TEST.PEOPLE", new Properties())
}.getMessage
assert(e2.contains("Please use customSchema option to specified column types."))
assert(e2.contains("User specified schema not supported with `jdbc`"))
}

test("jdbc API support custom schema") {
Expand Down