Skip to content

[SPARK-32106][SQL] Implement script transform in sql/core #29414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 16 commits into from
Closed

[SPARK-32106][SQL] Implement script transform in sql/core #29414

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b4c4b40
[SPARK-32106][SQL] Implement script transform in sql/core
AngersZhuuuu Aug 12, 2020
c58729b
Update SparkSqlParser.scala
AngersZhuuuu Aug 12, 2020
fc5bbce
follow comment
AngersZhuuuu Aug 13, 2020
a812333
follow commnet
AngersZhuuuu Aug 13, 2020
fdfc987
follow comment
AngersZhuuuu Aug 13, 2020
5b17cd9
Merge branch 'master' into SPARK-32106-MINOR
AngersZhuuuu Aug 24, 2020
dabae9b
fix add UT
AngersZhuuuu Aug 24, 2020
61466f6
Merge branch 'master' into SPARK-32106-MINOR
AngersZhuuuu Oct 26, 2020
d68066e
Update SparkScriptTransformationSuite.scala
AngersZhuuuu Oct 26, 2020
57735bf
Merge branch 'master' into SPARK-32106-MINOR
AngersZhuuuu Nov 13, 2020
3ad44e3
Update SparkSqlParser.scala
AngersZhuuuu Nov 13, 2020
5b41fbc
Update transform.sql.out
AngersZhuuuu Nov 14, 2020
63bdb53
Merge branch 'master' into SPARK-32106-MINOR
AngersZhuuuu Dec 9, 2020
d3c65d4
Update SparkSqlParser.scala
AngersZhuuuu Dec 9, 2020
4e66417
Merge branch 'master' into SPARK-32106-MINOR
AngersZhuuuu Dec 15, 2020
a10c5a6
Update transform.sql.out
AngersZhuuuu Dec 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update SparkSqlParser.scala
  • Loading branch information
@AngersZhuuuu
AngersZhuuuu committed Nov 13, 2020
commit 3ad44e3a175ec3324781d6e3a2ff9e3739455c82
4 changes: 3 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -776,7 +776,9 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder(conf) {
// Use default (serde) format.
val name = conf.getConfString("hive.script.serde",
"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe")
val props = Seq("field.delim" -> "\t")
val props = Seq(
"field.delim" -> "\t",
"serialization.last.column.takes.rest" -> "true")
val recordHandler = Option(conf.getConfString(configKey, defaultConfigValue))
(Nil, Option(name), props, recordHandler)
}
Expand Down