[SPARK-29511][SQL] DataSourceV2: Support CREATE NAMESPACE #26166

imback82 · 2019-10-18T15:58:13Z

What changes were proposed in this pull request?

This PR adds CREATE NAMESPACE support for V2 catalogs.

Why are the changes needed?

Currently, you cannot explicitly create namespaces for v2 catalogs.

Does this PR introduce any user-facing change?

The user can now perform the following:

CREATE NAMESPACE mycatalog.ns

to create a namespace ns inside mycatalog V2 catalog.

How was this patch tested?

Added unit tests.

imback82 · 2019-10-18T15:58:25Z

cc: @cloud-fan / @rdblue

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateNamespaceExec.scala

imback82 · 2019-10-18T16:02:09Z

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala

+    testShowNamespaces("SHOW NAMESPACES IN testcat", Seq("ns1"))
+    testShowNamespaces("SHOW NAMESPACES IN testcat.ns1", Seq("ns1.ns2"))
+
+    // TODO: Add tests for validating namespace metadata when DESCRIBE NAMESPACE is available.


I will create a JIRA for this and follow up.

SparkQA · 2019-10-18T19:44:15Z

Test build #112284 has finished for PR 26166 at commit 08728d3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala

rdblue · 2019-10-18T23:57:55Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala

+  ifNotExists: Boolean,
+  comment: Option[String],
+  locationSpec: Option[String],
+  properties: Map[String, String]) extends ParsedStatement


@cloud-fan, what is the behavior of IF NOT EXISTS with DBPROPERTIES? If the namespace exists, are properties updated?

same as table/database, if table/database doesn't exist, the command is a noop.

SparkQA · 2019-10-20T16:33:49Z

Test build #112341 has finished for PR 26166 at commit c1e6687.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-10-20T18:59:49Z

Test build #112344 has finished for PR 26166 at commit 0fce441.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class ExtractIntervalPart(
case class ExtractIntervalMillenniums(child: Expression)
case class ExtractIntervalCenturies(child: Expression)
case class ExtractIntervalDecades(child: Expression)
case class ExtractIntervalYears(child: Expression)
case class ExtractIntervalQuarters(child: Expression)
case class ExtractIntervalMonths(child: Expression)
case class ExtractIntervalDays(child: Expression)
case class ExtractIntervalHours(child: Expression)
case class ExtractIntervalMinutes(child: Expression)
case class ExtractIntervalSeconds(child: Expression)
case class ExtractIntervalMilliseconds(child: Expression)
case class ExtractIntervalMicroseconds(child: Expression)
case class ExtractIntervalEpoch(child: Expression)
case class RepairTableStatement(tableName: Seq[String]) extends ParsedStatement

SparkQA · 2019-10-20T19:21:39Z

Test build #112345 has finished for PR 26166 at commit 6f80ff4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateNamespaceExec.scala

SparkQA · 2019-10-21T07:05:02Z

Test build #112360 has finished for PR 26166 at commit 970338c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-10-21T07:33:52Z

retest this please

SparkQA · 2019-10-21T11:15:03Z

Test build #112371 has finished for PR 26166 at commit 970338c.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-10-21T12:08:32Z

retest this please

SparkQA · 2019-10-21T16:01:19Z

Test build #112388 has finished for PR 26166 at commit 970338c.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2019-10-22T03:34:00Z

Can we retest this?

cloud-fan · 2019-10-22T11:03:10Z

retest this please

cloud-fan · 2019-10-22T11:03:30Z

@imback82 anyone can trigger retest by typing "retest this please"

SparkQA · 2019-10-22T14:52:47Z

Test build #112456 has finished for PR 26166 at commit 970338c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-10-22T15:19:19Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

        ((COMMENT comment=STRING) |
         locationSpec |
-         (WITH DBPROPERTIES tablePropertyList))*                       #createDatabase
+         (WITH DBPROPERTIES tablePropertyList))*                       #createNamespace


Ah I missed this. Shall we use PROPERTIES instead of DBPROPERTIES? We can write DBPROPERTIES | PROPERTIES for backward compatibility

thanks for catching this. updated.

cloud-fan · 2019-10-22T15:20:34Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala

+ * The logical plan of the CREATE NAMESPACE command that works for v2 catalogs.
+ */
+case class CreateNamespace(
+  catalog: SupportsNamespaces,


nit: 4 space identation

cloud-fan · 2019-10-22T15:21:42Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala

+  test("create namespace") {
+    val sql =
+      """
+        |CREATE DATABASE IF NOT EXISTS database_name


can we use a.b.c instead of database_name?

cloud-fan · 2019-10-22T15:22:43Z

LGTM except several minor comments

SparkQA · 2019-10-22T17:00:17Z

Test build #112465 has finished for PR 26166 at commit db5b1c9.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
skeleton_class = type_constructor(name, bases, type_kwargs)
enum_class = metacls.__new__(metacls, name, bases, classdict)
case class BitAndAgg(child: Expression) extends DeclarativeAggregate with ExpectsInputTypes
case class BitOrAgg(child: Expression) extends DeclarativeAggregate with ExpectsInputTypes
abstract class AbstractSqlParser(conf: SQLConf) extends ParserInterface with Logging
class CatalystSqlParser(conf: SQLConf) extends AbstractSqlParser(conf)
case class TruncateTableStatement(
class SparkSqlParser(conf: SQLConf) extends AbstractSqlParser(conf)

SparkQA · 2019-10-23T00:12:33Z

Test build #112486 has finished for PR 26166 at commit 132cd7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-23T00:20:08Z

Test build #112487 has finished for PR 26166 at commit 3fd447d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-23T03:17:01Z

Test build #112494 has finished for PR 26166 at commit f7fa8ed.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ShowPartitionsStatement(tableName: Seq[String],

cloud-fan · 2019-10-23T04:17:30Z

thanks, merging to master!

imback82 added 4 commits October 18, 2019 00:42

initial checkin

daa17c1

Finishing up

27066a5

Merge branch 'master' into create_namespace

1298b0f

add more tests

08728d3

imback82 commented Oct 18, 2019

View reviewed changes

dongjoon-hyun added the SQL label Oct 18, 2019

rdblue reviewed Oct 18, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala Outdated Show resolved Hide resolved

rdblue reviewed Oct 18, 2019

View reviewed changes

imback82 added 3 commits October 19, 2019 20:14

Address comments

c1e6687

Merge branch 'master' into create_namespace

0fce441

change ifNotExist logic.

6f80ff4

cloud-fan reviewed Oct 21, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Oct 21, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateNamespaceExec.scala Outdated Show resolved Hide resolved

address comments

970338c

cloud-fan mentioned this pull request Oct 21, 2019

[SPARK-29519][SQL] SHOW TBLPROPERTIES should do multi-catalog resolution. #26176

Closed

Merge branch 'master' into create_namespace

db5b1c9

cloud-fan reviewed Oct 22, 2019

View reviewed changes

imback82 added 3 commits October 22, 2019 13:24

Address PR comments

132cd7f

style change

3fd447d

Merge branch 'master' into create_namespace

f7fa8ed

cloud-fan closed this in c128ac5 Oct 23, 2019

imback82 deleted the create_namespace branch October 24, 2019 14:25

[SPARK-29511][SQL] DataSourceV2: Support CREATE NAMESPACE #26166

[SPARK-29511][SQL] DataSourceV2: Support CREATE NAMESPACE #26166

Uh oh!

Conversation

imback82 commented Oct 18, 2019

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

imback82 commented Oct 18, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 18, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 20, 2019

Uh oh!

SparkQA commented Oct 20, 2019

Uh oh!

SparkQA commented Oct 20, 2019

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Oct 21, 2019

Uh oh!

cloud-fan commented Oct 21, 2019

Uh oh!

SparkQA commented Oct 21, 2019

Uh oh!

cloud-fan commented Oct 21, 2019

Uh oh!

SparkQA commented Oct 21, 2019

Uh oh!

imback82 commented Oct 22, 2019

Uh oh!

cloud-fan commented Oct 22, 2019

Uh oh!

cloud-fan commented Oct 22, 2019

Uh oh!

SparkQA commented Oct 22, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Oct 22, 2019

Uh oh!

SparkQA commented Oct 22, 2019

Uh oh!

SparkQA commented Oct 23, 2019

Uh oh!

SparkQA commented Oct 23, 2019

Uh oh!

SparkQA commented Oct 23, 2019

Uh oh!

cloud-fan commented Oct 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants