Skip to content

[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition #18900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
debugger87 wants to merge 13 commits into from
Closed

[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition #18900

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
71a660a
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
debugger87 Aug 10, 2017
f668ce8
added createTime and lastAccessTime into CatalogTablePartition
debugger87 Aug 11, 2017
2fb1dda
minor tweak
debugger87 Aug 11, 2017
c833ce7
fix type missmatch
debugger87 Aug 11, 2017
bf2a105
added createTime and lastAccessTime into partition map for display
debugger87 Aug 11, 2017
478e205
Merge branch 'master' into fix/set-create-time-for-hive-partition
debugger87 Jun 7, 2018
a00e943
bug fix
debugger87 Jun 7, 2018
e3a0cc4
added some comments
debugger87 Jun 7, 2018
b0846c3
fix missing comma
debugger87 Jun 7, 2018
0390e88
ignore createTime and lastAccessTime in catalogPartitionsEqual
debugger87 Jun 22, 2018
c843ef1
fix sql output
debugger87 Jun 22, 2018
715c7cc
regenerate output files for SQLQueryTestSuite after added Created Tim…
debugger87 Jun 27, 2018
18c85b6
fix comment
debugger87 Jun 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,12 +93,16 @@ object CatalogStorageFormat {
* @param spec partition spec values indexed by column name
* @param storage storage format of the partition
* @param parameters some parameters for the partition
* @param createTime creation time of the partition
Copy link
Contributor

@cloud-fan cloud-fan Jun 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's mention the time unit, i.e. in milliseconds.

Copy link
Contributor Author

@debugger87 debugger87 Jun 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it's the same as CatalogTable, in milliseconds. I fill fix this comment.

* @param lastAccessTime last access time
* @param stats optional statistics (number of rows, total size, etc.)
*/
case class CatalogTablePartition(
spec: CatalogTypes.TablePartitionSpec,
storage: CatalogStorageFormat,
parameters: Map[String, String] = Map.empty,
createTime: Long = System.currentTimeMillis,
lastAccessTime: Long = -1,
stats: Option[CatalogStatistics] = None) {

def toLinkedHashMap: mutable.LinkedHashMap[String, String] = {
Copy link
Member

@gatorsmile gatorsmile Aug 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also need to add it to this map for display

Copy link
Contributor Author

@debugger87 debugger87 Aug 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gatorsmile Thanks for your reminding, i will add it.

Expand All @@ -109,6 +113,8 @@ case class CatalogTablePartition(
if (parameters.nonEmpty) {
map.put("Partition Parameters", s"{${parameters.map(p => p._1 + "=" + p._2).mkString(", ")}}")
}
map.put("Created Time", new Date(createTime).toString)
map.put("Last Access", new Date(lastAccessTime).toString)
stats.foreach(s => map.put("Partition Statistics", s.simpleString))
map
}
Expand Down
4 changes: 4 additions & 0 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -995,6 +995,8 @@ private[hive] object HiveClientImpl {
tpart.setTableName(ht.getTableName)
tpart.setValues(partValues.asJava)
tpart.setSd(storageDesc)
tpart.setCreateTime((p.createTime / 1000).toInt)
tpart.setLastAccessTime((p.lastAccessTime / 1000).toInt)
tpart.setParameters(mutable.Map(p.parameters.toSeq: _*).asJava)
new HivePartition(ht, tpart)
}
Expand All @@ -1019,6 +1021,8 @@ private[hive] object HiveClientImpl {
compressed = apiPartition.getSd.isCompressed,
properties = Option(apiPartition.getSd.getSerdeInfo.getParameters)
.map(_.asScala.toMap).orNull),
createTime = apiPartition.getCreateTime.toLong * 1000,
lastAccessTime = apiPartition.getLastAccessTime.toLong * 1000)
Copy link
Contributor

@cxzl25 cxzl25 Jun 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comma to the end?

Copy link
Contributor Author

@debugger87 debugger87 Jun 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cxzl25 yeah, it's my mistake, i will fix it

parameters = properties,
stats = readHiveStats(properties))
}
Expand Down