Skip to content

[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition #18900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
debugger87 wants to merge 13 commits into from
Closed

[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition #18900

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
71a660a
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
debugger87 Aug 10, 2017
f668ce8
added createTime and lastAccessTime into CatalogTablePartition
debugger87 Aug 11, 2017
2fb1dda
minor tweak
debugger87 Aug 11, 2017
c833ce7
fix type missmatch
debugger87 Aug 11, 2017
bf2a105
added createTime and lastAccessTime into partition map for display
debugger87 Aug 11, 2017
478e205
Merge branch 'master' into fix/set-create-time-for-hive-partition
debugger87 Jun 7, 2018
a00e943
bug fix
debugger87 Jun 7, 2018
e3a0cc4
added some comments
debugger87 Jun 7, 2018
b0846c3
fix missing comma
debugger87 Jun 7, 2018
0390e88
ignore createTime and lastAccessTime in catalogPartitionsEqual
debugger87 Jun 22, 2018
c843ef1
fix sql output
debugger87 Jun 22, 2018
715c7cc
regenerate output files for SQLQueryTestSuite after added Created Tim…
debugger87 Jun 27, 2018
18c85b6
fix comment
debugger87 Jun 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix type missmatch
  • Loading branch information
@debugger87
debugger87 committed Aug 11, 2017
commit c833ce7aa5f2ba0b684494fd1b24b7995f1c09c9
4 changes: 2 additions & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,9 @@ object CatalogStorageFormat {
case class CatalogTablePartition(
spec: CatalogTypes.TablePartitionSpec,
storage: CatalogStorageFormat,
parameters: Map[String, String] = Map.empty,
createTime: Long = System.currentTimeMillis,
lastAccessTime: Long = -1,
parameters: Map[String, String] = Map.empty) {
lastAccessTime: Long = -1) {

def toLinkedHashMap: mutable.LinkedHashMap[String, String] = {
Copy link
Member

@gatorsmile gatorsmile Aug 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to add it to this map for display

Copy link
Contributor Author

@debugger87 debugger87 Aug 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gatorsmile Thanks for your reminding, i will add it.

val map = new mutable.LinkedHashMap[String, String]()
Expand Down
6 changes: 3 additions & 3 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1006,10 +1006,10 @@ private[hive] object HiveClientImpl {
compressed = apiPartition.getSd.isCompressed,
properties = Option(apiPartition.getSd.getSerdeInfo.getParameters)
.map(_.asScala.toMap).orNull),
createTime = apiPartition.getCreateTime.toLong * 1000,
lastAccessTime = apiPartition.getLastAccessTime.toLong * 1000,
parameters =
if (hp.getParameters() != null) hp.getParameters().asScala.toMap else Map.empty)
if (hp.getParameters() != null) hp.getParameters().asScala.toMap else Map.empty,
createTime = apiPartition.getCreateTime.toLong * 1000,
lastAccessTime = apiPartition.getLastAccessTime.toLong * 1000)
Copy link
Contributor

@cxzl25 cxzl25 Jun 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comma to the end?

Copy link
Contributor Author

@debugger87 debugger87 Jun 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cxzl25 yeah, it's my mistake, i will fix it

}

// Below is the key of table properties for storing Hive-generated statistics
Expand Down