Skip to content

[SPARK-18572][SQL] Add a method listPartitionNames to ExternalCatalog #15998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mallman wants to merge 12 commits into from
Closed

[SPARK-18572][SQL] Add a method listPartitionNames to ExternalCatalog #15998

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
826dee6
[SPARK-18572][SQL] Add a method `listPartitionName` to `ExternalCatalog`
Nov 23, 2016
54171ad
Make `ExternalCatalog.listPartitionNames` abstract and implement that
Nov 25, 2016
93cee97
Insert a couple of cosmetic newlines
Nov 26, 2016
27dc672
Modify `HiveCommandSuite` to make it test partition columns with
Nov 26, 2016
2a7b062
Formatting and code commenting
Nov 28, 2016
d183946
Code documentation and partition name/value path name escaping
Nov 30, 2016
9c71521
Add unit tests for the new `ExternalCatalog.listPartitionNames` method
Dec 1, 2016
b9dd303
Revert modifications to the HiveCommandSuite
Dec 1, 2016
28563d4
Explicitly sort the results from
Dec 1, 2016
860d985
Build and use a map of lower-cased partition column names to exact
Dec 2, 2016
fc57f23
Add tests for `SessionCatalog.listPartitionNames` and enhance coverage
Dec 2, 2016
37fc595
Fix something in DataSourceStrategy.scala. Patch provided by @gatorsmile
Dec 5, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Fix something in DataSourceStrategy.scala. Patch provided by @gatorsmile
  • Loading branch information
Michael Allman committed Dec 5, 2016
commit 37fc5955b2a26caa5ef133d19e415172a26330a2
22 changes: 11 additions & 11 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,8 +161,8 @@ case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] {
insert.copy(partition = parts.map(p => (p._1, None)), child = Project(projectList, query))


case i @ logical.InsertIntoTable(
l @ LogicalRelation(t: HadoopFsRelation, _, table), part, query, overwrite, false)
case logical.InsertIntoTable(
l @ LogicalRelation(t: HadoopFsRelation, _, table), _, query, overwrite, false)
if query.resolved && t.schema.sameType(query.schema) =>

// Sanity checks
Expand Down Expand Up @@ -192,11 +192,19 @@ case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] {
var initialMatchingPartitions: Seq[TablePartitionSpec] = Nil
var customPartitionLocations: Map[TablePartitionSpec, String] = Map.empty

val staticPartitionKeys: TablePartitionSpec = if (overwrite.enabled) {
overwrite.staticPartitionKeys.map { case (k, v) =>
(partitionSchema.map(_.name).find(_.equalsIgnoreCase(k)).get, v)
}
} else {
Map.empty
}

// When partitions are tracked by the catalog, compute all custom partition locations that
// may be relevant to the insertion job.
if (partitionsTrackedByCatalog) {
val matchingPartitions = t.sparkSession.sessionState.catalog.listPartitions(
l.catalogTable.get.identifier, Some(overwrite.staticPartitionKeys))
l.catalogTable.get.identifier, Some(staticPartitionKeys))
initialMatchingPartitions = matchingPartitions.map(_.spec)
customPartitionLocations = getCustomPartitionLocations(
t.sparkSession, l.catalogTable.get, outputPath, matchingPartitions)
Expand Down Expand Up @@ -225,14 +233,6 @@ case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] {
t.location.refresh()
}

val staticPartitionKeys: TablePartitionSpec = if (overwrite.enabled) {
overwrite.staticPartitionKeys.map { case (k, v) =>
(partitionSchema.map(_.name).find(_.equalsIgnoreCase(k)).get, v)
}
} else {
Map.empty
}

val insertCmd = InsertIntoHadoopFsRelationCommand(
outputPath,
staticPartitionKeys,
Expand Down