[SPARK-12362][SQL][WIP] Inline Hive Parser #10525

hvanhovell · 2015-12-30T10:55:04Z

This PR inlines the Hive SQL parser in Spark SQL.

The previous (merged) incarnation of this PR passed all tests, but had and still has problems with the build. These problems are caused by a the fact that - for some reason - in some cases the ANTLR generated code is not included in the compilation fase.

This PR is a WIP and should not be merged until we have sorted out the build issues.

This is a WIP. This inlines the hive sql grammar parser into spark sql in the hive subproject. This should eventually be moved into the SQL core project once all the hive dependencies are removed. This patch does some of that by cleaning up the hive code to remove much of semantic analysis.

SparkQA · 2015-12-30T11:08:35Z

Test build #48476 has finished for PR 10525 at commit b8e76b2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-30T11:40:22Z

retest this please

SparkQA · 2015-12-30T13:59:03Z

Test build #48478 has finished for PR 10525 at commit b8e76b2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-30T14:03:40Z

retest this please

SparkQA · 2015-12-30T14:22:59Z

Test build #48481 has finished for PR 10525 at commit b8e76b2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-30T14:23:49Z

retest this please

SparkQA · 2015-12-30T14:32:44Z

Test build #48483 has finished for PR 10525 at commit b8e76b2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-30T14:36:43Z

retest this please

SparkQA · 2015-12-30T14:42:19Z

Test build #48485 has finished for PR 10525 at commit b8e76b2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-30T14:43:22Z

retest this please

hvanhovell · 2015-12-30T14:43:44Z

last one promise :)

SparkQA · 2015-12-30T14:52:21Z

Test build #48486 has finished for PR 10525 at commit b8e76b2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-30T19:14:32Z

retest this please

SparkQA · 2015-12-30T19:33:49Z

Test build #48501 has finished for PR 10525 at commit a7749a8.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-30T21:35:45Z

Test build #48509 has finished for PR 10525 at commit 90c11e8.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-30T23:53:18Z

Test build #48518 has finished for PR 10525 at commit 84db0f4.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-31T11:17:30Z

retest this please

SparkQA · 2015-12-31T11:35:27Z

Test build #48550 has finished for PR 10525 at commit e3e2f72.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-31T11:40:24Z

retest this please

SparkQA · 2015-12-31T11:56:00Z

Test build #48552 has finished for PR 10525 at commit e3e2f72.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-31T11:59:38Z

retest this please

SparkQA · 2015-12-31T12:15:08Z

Test build #48553 has finished for PR 10525 at commit e3e2f72.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-31T12:58:34Z

retest this please

hvanhovell · 2015-12-31T13:39:58Z

@JoshRosen these builds fail often and non-deterministically with a fails build dependency tests. The problem is that I cannot find the cause of this problem (the build fails with no helpfull message), these builds share this problem:

After the initial error I thought I messed up the dependencies (I am messing with the build). I changed them and I got a correct build failure with 'nice' error message:

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48521/consoleFull

Could you take a look?

SparkQA · 2015-12-31T15:01:05Z

Test build #48555 has finished for PR 10525 at commit e3e2f72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2015-12-31T15:09:57Z

retest this please

SparkQA · 2015-12-31T18:53:29Z

Test build #48557 has finished for PR 10525 at commit e3e2f72.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-12-31T20:05:35Z

I have a few hunches about what could be causing the dependency manifest checking problems. I'll keep investigating when I get home later tonight. I think I've spotted two potential problems: there might be some portability issues with how we're using sort and I think there might be a mistake in the exit-code-handling logic.

rxin · 2015-12-31T23:03:34Z

Should we revert that dependency check thing first?

JoshRosen · 2016-01-01T02:48:57Z

Aha! It appears that there is a portability issue in sort, or at least in how we're using it in the dependency-checking script.

When I run echo -e "breeze-macros\nbreeze_2.0" | sort on my Mac, I get

breeze-macros
breeze_2.0

but I get the opposite ordering when running on AMPLab Jenknis.

On my mac, sort --version outputs sort (GNU coreutils) 5.93, whereas AMP Jenkins has sort (GNU coreutils) 8.4. But it turns out that this isn't the problem! It's a locale-dependent issue: https://stackoverflow.com/questions/28881/why-doesnt-sort-sort-the-same-on-every-machine

We need to set LC_ALL=C in order to make things portable. I'm going to open a hotfix PR to address these issues, then will retest this PR once my fix is in.

JoshRosen · 2016-01-01T04:09:37Z

Patch to fix it is at #10543; planning to merge soon.

JoshRosen · 2016-01-01T04:25:26Z

Jenkins, retest this please.

SparkQA · 2016-01-01T06:32:25Z

Test build #48563 has finished for PR 10525 at commit e3e2f72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-01-01T09:00:56Z

I triggered 3 more runs.

SparkQA · 2016-01-01T09:04:04Z

Test build #2285 has finished for PR 10525 at commit e3e2f72.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-01-01T09:05:43Z

cc @JoshRosen dependency failed again

SparkQA · 2016-01-01T11:09:09Z

Test build #2284 has finished for PR 10525 at commit e3e2f72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-01T11:19:44Z

Test build #2286 has finished for PR 10525 at commit e3e2f72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-01T21:27:06Z

Test build #2289 has finished for PR 10525 at commit e3e2f72.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-01-01T21:32:05Z

@JoshRosen I pushed a commit directly to disable test-dependencies.

SparkQA · 2016-01-01T23:52:02Z

Test build #2290 has finished for PR 10525 at commit e3e2f72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-01-02T07:22:43Z

OK merging again.

gatorsmile · 2016-01-03T19:11:28Z

After this merge, I am unable to make a build in IntelliJ IDEA. I hit the following the errors:

Error:(1477, 5) not found: value SparkSqlParser
    SparkSqlParser.Number,
    ^
Error:(1478, 5) not found: value SparkSqlParser
    SparkSqlParser.TinyintLiteral,
    ^
Error:(1479, 5) not found: value SparkSqlParser
    SparkSqlParser.SmallintLiteral,
    ^
Error:(1480, 5) not found: value SparkSqlParser
    SparkSqlParser.BigintLiteral,
    ^
Error:(1481, 5) not found: value SparkSqlParser
    SparkSqlParser.DecimalLiteral)
    ^
Error:(1686, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.StringLiteral =>
                                        ^
Error:(1689, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_DATELITERAL =>
                                        ^
Error:(1692, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_CHARSETLITERAL =>
                                        ^
Error:(1695, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_INTERVAL_YEAR_MONTH_LITERAL =>
                                        ^
Error:(1698, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_INTERVAL_DAY_TIME_LITERAL =>
                                        ^
Error:(1701, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_INTERVAL_YEAR_LITERAL =>
                                        ^
Error:(1704, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_INTERVAL_MONTH_LITERAL =>
                                        ^
Error:(1707, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_INTERVAL_DAY_LITERAL =>
                                        ^
Error:(1710, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_INTERVAL_HOUR_LITERAL =>
                                        ^
Error:(1713, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_INTERVAL_MINUTE_LITERAL =>
                                        ^
Error:(1716, 41) not found: value SparkSqlParser
    case ast: ASTNode if ast.getType == SparkSqlParser.TOK_INTERVAL_SECOND_LITERAL =>
                                        ^

gatorsmile · 2016-01-03T19:16:41Z

nvm, I just need to reimport all maven projects. The problem has been fixed. Thanks!

hvanhovell · 2016-01-03T19:20:23Z

Good one, was looking for that myself :)...

nongli and others added 9 commits December 21, 2015 14:13

Updates to support antlr 3.5.2 and SBT build.

0cbf502

Add anti join to grammar as an example.

7e1a145

Remove dead code from the parser.

cd07d7f

Remove tests no longer supported by parser (HIVE-11145).

8ced9c0

Remove ASTNodeOrigin

cb60ba0

Merge remote-tracking branch 'spark/master' into SPARK-12362

feee2ba

Fix CTE and position errors.

2e30d92

Fix described errors.

b8e76b2

Change log level of the Hive Compilation phase to 'debug'.

a7749a8

Add the generated files to the build by hand

90c11e8

Embed ANTLR generation code in build.

84db0f4

hvanhovell added 2 commits December 31, 2015 00:56

Merge remote-tracking branch 'spark/master' into SPARK-12362

ad8fecd

Fix deps.

84b006b

asfgit closed this in 970635a Jan 2, 2016

[SPARK-12362][SQL][WIP] Inline Hive Parser #10525

[SPARK-12362][SQL][WIP] Inline Hive Parser #10525

Uh oh!

Conversation

hvanhovell commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

hvanhovell commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

hvanhovell commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

hvanhovell commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

hvanhovell commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

hvanhovell commented Dec 30, 2015

Uh oh!

hvanhovell commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

hvanhovell commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

SparkQA commented Dec 30, 2015

Uh oh!

hvanhovell commented Dec 31, 2015

Uh oh!

SparkQA commented Dec 31, 2015

Uh oh!

hvanhovell commented Dec 31, 2015

Uh oh!

SparkQA commented Dec 31, 2015

Uh oh!

hvanhovell commented Dec 31, 2015

Uh oh!

SparkQA commented Dec 31, 2015

Uh oh!

hvanhovell commented Dec 31, 2015

Uh oh!

hvanhovell commented Dec 31, 2015

Uh oh!

SparkQA commented Dec 31, 2015

Uh oh!

hvanhovell commented Dec 31, 2015

Uh oh!

SparkQA commented Dec 31, 2015

Uh oh!

JoshRosen commented Dec 31, 2015

Uh oh!

rxin commented Dec 31, 2015

Uh oh!

JoshRosen commented Jan 1, 2016

Uh oh!

JoshRosen commented Jan 1, 2016

Uh oh!

JoshRosen commented Jan 1, 2016

Uh oh!

SparkQA commented Jan 1, 2016

Uh oh!

rxin commented Jan 1, 2016

Uh oh!

SparkQA commented Jan 1, 2016

Uh oh!

rxin commented Jan 1, 2016

Uh oh!

SparkQA commented Jan 1, 2016

Uh oh!