Skip to content

[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition #18900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
debugger87 wants to merge 13 commits into from
Closed

[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition #18900

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
71a660a
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
debugger87 Aug 10, 2017
f668ce8
added createTime and lastAccessTime into CatalogTablePartition
debugger87 Aug 11, 2017
2fb1dda
minor tweak
debugger87 Aug 11, 2017
c833ce7
fix type missmatch
debugger87 Aug 11, 2017
bf2a105
added createTime and lastAccessTime into partition map for display
debugger87 Aug 11, 2017
478e205
Merge branch 'master' into fix/set-create-time-for-hive-partition
debugger87 Jun 7, 2018
a00e943
bug fix
debugger87 Jun 7, 2018
e3a0cc4
added some comments
debugger87 Jun 7, 2018
b0846c3
fix missing comma
debugger87 Jun 7, 2018
0390e88
ignore createTime and lastAccessTime in catalogPartitionsEqual
debugger87 Jun 22, 2018
c843ef1
fix sql output
debugger87 Jun 22, 2018
715c7cc
regenerate output files for SQLQueryTestSuite after added Created Tim…
debugger87 Jun 27, 2018
18c85b6
fix comment
debugger87 Jun 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -986,6 +986,7 @@ private[hive] object HiveClientImpl {
tpart.setTableName(ht.getTableName)
tpart.setValues(partValues.asJava)
tpart.setSd(storageDesc)
tpart.setCreateTime((System.currentTimeMillis() / 1000).toInt)
Copy link
Member

@gatorsmile gatorsmile Aug 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to Hive, how about from Hive?

Copy link
Contributor Author

@debugger87 debugger87 Aug 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just need to use API of hive_metastore like get_partition to fetch related information of Hive Partition.

  • Partition in hive_metastore.thrift
struct Partition {
  1: list<string> values // string value is converted to appropriate partition key type
  2: string       dbName,
  3: string       tableName,
  4: i32          createTime,
  5: i32          lastAccessTime,
  6: StorageDescriptor   sd,
  7: map<string, string> parameters,
  8: optional PrincipalPrivilegeSet privileges
}
  • get_partition in hive_metastore.thrit
Partition get_partition(1:string db_name, 2:string tbl_name, 3:list<string> part_vals)
                       throws(1:MetaException o1, 2:NoSuchObjectException o2)

We will set TTL as kv into parameters of Table and fetch createTime from Partition to decide if we can drop a partition in Hive.

Copy link
Contributor Author

@debugger87 debugger87 Aug 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gatorsmile sorry for my misunderstanding of your point. toHivePartition and fromHivePartition should be symmetric, and I change my implementation. could you please review it again?

new HivePartition(ht, tpart)
}

Expand Down