Skip to content

[SPARK-5084][SQL]Replaces TestHiveContext.configure() with HiveContext.overrideHiveConf() #3895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
baishuo wants to merge 7 commits into from
Closed

[SPARK-5084][SQL]Replaces TestHiveContext.configure() with HiveContext.overrideHiveConf() #3895

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ecc1231
add if not exists after create database
baishuo Jan 5, 2015
23d118f
modify the hard code about jdbc ConnectionURL
baishuo Jan 5, 2015
6a756bf
add if not exists again
baishuo Jan 5, 2015
062c783
choose an other getConf function
baishuo Jan 6, 2015
cdf3237
restore TestHive.scala
baishuo Mar 13, 2015
8c3eae4
Replaces TestHiveContext.configure() with HiveContext.overrideHiveConf()
liancheng Mar 15, 2015
d6c29c1
Fixes comment of TestHiveContext.overrideHiveConf()
liancheng Mar 15, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add if not exists again
  • Loading branch information
@baishuo
baishuo committed Mar 17, 2015
commit 6a756bf88587eb6c317778beb23435d65eb46ba6
2 changes: 1 addition & 1 deletion sql/hive/v0.13.1/src/main/scala/org/apache/spark/sql/hive/Shim13.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,7 @@ private[hive] object HiveShim {
def getStatsSetupConstRawDataSize = StatsSetupConst.RAW_DATA_SIZE

def createDefaultDBIfNeeded(context: HiveContext) = {
context.runSqlHive("CREATE DATABASE default")
context.runSqlHive("CREATE DATABASE IF NOT EXISTS default")
Copy link
Contributor

@yhuai yhuai Jan 21, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add it in Shim12?

Copy link
Contributor Author

@baishuo baishuo Mar 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we do not need to modify Shim12, Since there is not the same function in Shim12

Copy link
Contributor

@chenghao-intel chenghao-intel Mar 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LTGM since it's much safer. But I am curious how did it pass the unit test previously. Seems the database default will not be created while initializing, unless we create it explicitly?

Copy link
Contributor

@liancheng liancheng Mar 15, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit tricky to explain. When initializing a TestHiveContext, the following things happen:

  1. HiveContext.hiveconf is initialized (notice that the metastore and warehouse paths point to the whatever configured in hive-site.xml or the default locations)
  2. HiveContext.sessionState is initialized
  3. TestHiveContext.configure() is called, metastore and warehouse paths now point to temporary directories used for testing purposes, no default database is defined there.
  4. HiveShim.createDefaultDbIfNeeded() is called to create the default database in the temproary directories.

As Michael commented, the createDefaultDbIfNeeded method is more like a hack to fix the initialization disorder. And that's why I opened baishuo#2 against this PR branch. In that PR, the root cause of this initialization disorder is fixed.

context.runSqlHive("USE default")
}

Expand Down