Skip to content

[SPARK-20427][SQL] Read JDBC table use custom schema #18266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 20 commits into from
Closed

[SPARK-20427][SQL] Read JDBC table use custom schema #18266

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
871c303
Read JDBC table use custom schema
wangyum Jun 11, 2017
0444c4d
Fix test error.
wangyum Jun 11, 2017
06881e8
Revert api.
wangyum Jun 15, 2017
ffaee42
Improve custom schema.
wangyum Jun 23, 2017
a984f3b
Improve custom schema.
wangyum Jun 23, 2017
e0fc6b4
Throw exception if custom schema field names does not match table col…
wangyum Jul 4, 2017
9e6f7cf
Merge branch 'master' into SPARK-20427
wangyum Jul 4, 2017
5fdd2bb
Remove SPARK-16848: jdbc API throws an exception for user specified s…
wangyum Jul 4, 2017
87df014
Support DDL-like JDBC custom schema.
wangyum Aug 16, 2017
63d3244
Merge branch 'master' into SPARK-20427
wangyum Aug 16, 2017
e08ccbb
Merge remote-tracking branch 'origin/master' into SPARK-20427
wangyum Aug 22, 2017
247fc78
Refactor code.
wangyum Aug 23, 2017
b8b03e2
Revert wrong change
wangyum Aug 23, 2017
b040c72
Fix python style
wangyum Aug 23, 2017
2ea56fc
Fix test error
wangyum Aug 23, 2017
1e2c1d9
Let user specified all the columns.
wangyum Aug 24, 2017
b38a1a8
Change all SQL type to upper case.
wangyum Sep 8, 2017
0b67f0f
customDataFrameColumnTypes -> customSchema
wangyum Sep 10, 2017
7fc97b4
revert assertNoSpecifiedSchema
wangyum Sep 10, 2017
1fdf002
parseUserSpecifiedColumnTypes -> getCustomSchema
wangyum Sep 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert wrong change
  • Loading branch information
@wangyum
wangyum committed Aug 23, 2017
commit b8b03e2e022c9440a4778a7bed2c88d2ac6aaa11
40 changes: 20 additions & 20 deletions docs/sql-programming-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -741,24 +741,24 @@ columns, `gender` and `country` as partitioning columns:
{% highlight text %}

path
????????? to
????????? table
????????? gender=male
??????? ????????? ...
??????? ???
??????? ????????? country=US
??????? ??????? ????????? data.parquet
??????? ????????? country=CN
??????? ??????? ????????? data.parquet
??????? ????????? ...
????????? gender=female
???? ????????? ...
???? ???
???? ????????? country=US
???? ??????? ????????? data.parquet
???? ????????? country=CN
???? ??????? ????????? data.parquet
???? ????????? ...
└── to
└── table
├── gender=male
│   ├── ...
│   │
│   ├── country=US
│   │   └── data.parquet
│   ├── country=CN
│   │   └── data.parquet
│   └── ...
└── gender=female
   ├── ...
   │
   ├── country=US
   │   └── data.parquet
   ├── country=CN
   │   └── data.parquet
   └── ...

{% endhighlight %}

Expand Down Expand Up @@ -1578,7 +1578,7 @@ options.
`Dataset[Row]`, while Java API users must replace `DataFrame` with `Dataset<Row>`. Both the typed
transformations (e.g., `map`, `filter`, and `groupByKey`) and untyped transformations (e.g.,
`select` and `groupBy`) are available on the Dataset class. Since compile-time type-safety in
Python and R is not a language feature, the concept of Dataset does not apply to these languages???
Python and R is not a language feature, the concept of Dataset does not apply to these languages’
APIs. Instead, `DataFrame` remains the primary programing abstraction, which is analogous to the
single-node data frame notion in these languages.

Expand Down Expand Up @@ -1835,7 +1835,7 @@ Spark SQL supports the vast majority of Hive features, such as:
* `CLUSTER BY`
* `SORT BY`
* All Hive operators, including:
* Relational operators (`=`, `???`, `==`, `<>`, `<`, `>`, `>=`, `<=`, etc)
* Relational operators (`=`, `⇔`, `==`, `<>`, `<`, `>`, `>=`, `<=`, etc)
* Arithmetic operators (`+`, `-`, `*`, `/`, `%`, etc)
* Logical operators (`AND`, `&&`, `OR`, `||`, etc)
* Complex type constructors
Expand Down