[SPARK-48668][SQL] Support ALTER NAMESPACE ... UNSET PROPERTIES in v2 #47038

panbingkun · 2024-06-20T08:14:37Z

What changes were proposed in this pull request?

The pr aims to support ALTER NAMESPACE ... UNSET PROPERTIES in v2.

Why are the changes needed?

For table and view, we can add, update, or delete table's properties using the following SQL:

ALTER (TABLE | VIEW) ... SET TBLPROPERTIES ...
ALTER (TABLE | VIEW) ... UNSET TBLPROPERTIES (IF EXISTS)? ...

spark/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

Lines 148 to 151 in 3469ec6

    
           | ALTER (TABLE | VIEW) identifierReference 
        
               SET TBLPROPERTIES propertyList                                 #setTableProperties 
        
           | ALTER (TABLE | VIEW) identifierReference 
        
               UNSET TBLPROPERTIES (IF EXISTS)? propertyList                  #unsetTableProperties

But at the SQL level, there is only SET syntax for namespace, not UNSET syntax

ALTER namespace ... SET (DBPROPERTIES | PROPERTIES) ...

spark/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4

Lines 106 to 107 in 3469ec6

    
           | ALTER namespace identifierReference 
        
               SET (DBPROPERTIES | PROPERTIES) propertyList                   #setNamespaceProperties

In addition, the underlying SupportsNamespaces interface supports deleting properties. I propose adding SQL syntax to facilitate users to use SQL instead of relying solely on APIs to manipulate the properties of namespace

spark/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsNamespaces.java

Lines 127 to 137 in 3469ec6

    
             /** 
        
              * Apply a set of metadata changes to a namespace in the catalog. 
        
              * 
        
              * @param namespace a multi-part namespace 
        
              * @param changes a collection of changes to apply to the namespace 
        
              * @throws NoSuchNamespaceException If the namespace does not exist (optional) 
        
              * @throws UnsupportedOperationException If namespace properties are not supported 
        
              */ 
        
             void alterNamespace( 
        
                 String[] namespace, 
        
                 NamespaceChange... changes) throws NoSuchNamespaceException;

spark/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/NamespaceChange.java

Lines 59 to 61 in 3469ec6

    
           static NamespaceChange removeProperty(String property) { 
        
             return new RemoveProperty(property); 
        
           }

Does this PR introduce any user-facing change?

Yes, end users can delete the properties of namespace through SQL.

How was this patch tested?

Add new UT.

Was this patch authored or co-authored using generative AI tooling?

No.

panbingkun · 2024-06-21T07:05:55Z

cc @cloud-fan @viirya @beliefer @yaooqinn @HyukjinKwon

cloud-fan · 2024-06-21T07:49:48Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala

+case class UnsetNamespaceProperties(
+    namespace: LogicalPlan,
+    propertyKeys: Seq[String],
+    ifExists: Boolean) extends UnaryCommand {


We can probably use RunnableCommand to simplify the v2 command implementation. We can add a UnaryRunnableCommand, and then do

case class UnsetNamespacePropertiesCommand(...) extends UnaryRunnableCommand { ... def run(...) { val ResolvedIdentifier(catalog, ident) = child ... } }

Okay, let me give it a try, thanks.

@cloud-fan I have improved it in the way you suggested.

panbingkun · 2024-06-25T01:53:07Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala

 }

 trait LeafRunnableCommand extends RunnableCommand with LeafLike[LogicalPlan]
+trait UnaryRunnableCommand extends RunnableCommand with UnaryLike[LogicalPlan]


Because RunnableCommand is defined in the module sql/core, it seems that adding UnaryRunnableCommand here is more appropriate.

panbingkun · 2024-06-25T01:54:26Z

sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala


  def unsetNonExistentPropertiesError(
-      properties: Seq[String], table: TableIdentifier): Throwable = {
+      properties: Seq[String], nameParts: Seq[String]): Throwable = {


Reuse UNSET_NONEXISTENT_PROPERTIES

panbingkun · 2024-06-25T01:58:22Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala

 }
+
+object ResolveCatalogs {
+  def resolveNamespace(


Extract method resolveNamespace from class ResolveCatalogs to object ResolveCatalogs and reuse it in UnsetNamespacePropertiesCommand

panbingkun · 2024-06-25T01:59:20Z

common/utils/src/main/resources/error/error-conditions.json

  "UNSET_NONEXISTENT_PROPERTIES" : {
    "message" : [
-      "Attempted to unset non-existent properties [<properties>] in table <table>."
+      "Attempted to unset non-existent properties [<properties>] in relation <relationId>."


I want to reuse it in the namespace scene

but namespace is not a relation... maybe just table or namespace <name>

panbingkun · 2024-06-25T02:03:13Z

.../src/main/scala/org/apache/spark/sql/execution/command/UnsetNamespacePropertiesCommand.scala

+ *     ALTER (DATABASE|SCHEMA|NAMESPACE) ... UNSET (DBPROPERTIES|PROPERTIES) [IF EXISTS] ...;
+ * }}}
+ */
+case class UnsetNamespacePropertiesCommand(


I have to define UnsetNamespacePropertiesCommand in module sql/core because it inherits UnaryRunnableCommand extends RunnableCommand and cannot be accessed in the module sql/catalyst (sql/core depends on sql/catalyst)

panbingkun · 2024-06-25T02:07:23Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala


    (ctx.LOCAL != null, finalStorage, Some(DDLUtils.HIVE_PROVIDER))
  }
+


I have to put the parse process of UnsetNamespacePropertiesCommand here, because UnsetNamespacePropertiesCommand can only be defined within the module sql/core, and AstBuilder cannot access the class UnsetNamespacePropertiesCommand within the module

cloud-fan · 2024-06-25T05:54:31Z

.../src/main/scala/org/apache/spark/sql/execution/command/UnsetNamespacePropertiesCommand.scala

+  override def run(sparkSession: SparkSession): Seq[Row] = {
+    val ResolvedIdentifier(catalog, ident) = child
+    val ns = ResolveCatalogs.resolveNamespace(
+      catalog, ident.namespace.toSeq :+ ident.name, fetchMetadata = true)


oh, we should leave this to the analyzer. We should put UnresolvedNamespace inside UnsetNamespacePropertiesCommand, instead of UnresolvedIdentifier

Okay, let me fix it.

Also, I found an issue during the migration of Unify v1 and v2 ALTER TABLE .. UNSET TBLPROPERTIES IF EXISTS tests.
The command (AlterTableExec) for ALTER TABLE ... UNSET TBLPROPERTIES ... in v2 seems to have ignored whether the IF EXISTS parameter.
I think we should fix it. WDYH?

oh, we should leave this to the analyzer. We should put UnresolvedNamespace inside UnsetNamespacePropertiesCommand, instead of UnresolvedIdentifier

Thanks, updated.

I think we should fix it. WDYH?

Yea we should fix, but in a separated PR.

Okay, let me do it in another separate PR.

panbingkun · 2024-06-25T07:48:33Z

common/utils/src/main/resources/error/error-conditions.json

          "<functionName> with AES-<mode> does not support initialization vectors (IVs)."
        ]
      },
+      "ALTER_NAMESPACE_PROPERTY" : {


Rename SET_NAMESPACE_PROPERTY to ALTER_NAMESPACE_PROPERTY, We will reuse it in SparkSqlParser

In SQL, SET also means update, I think it's fine

Because in the following two cases, it is UNSUPPORTED_FEATURE.SET_NAMESPACE_PROPERTY, and I am concerned that it may cause misunderstandings.

test reserved properties

sql(s"CREATE NAMESPACE $ns") val sqlText = s"ALTER NAMESPACE $ns SET PROPERTIES ('$key'='dummyVal')"

sql(s"CREATE NAMESPACE $ns") val sqlText = s"ALTER NAMESPACE $ns UNSET PROPERTIES ('$key')"

Okay, let me restore it

what are the corresponding errors for table properties?

The corresponding errors for table properties: UNSUPPORTED_FEATURE.SET_TABLE_PROPERTY, eg:

ALTER TABLE ... SET PROPERTIES ('$reservedKey'='...')" ALTER TABLE ... UNSET PROPERTIES ('$reservedKey')"

panbingkun · 2024-06-25T07:51:31Z

...test/scala/org/apache/spark/sql/execution/command/AlterNamespaceSetPropertiesSuiteBase.scala

-            sql(s"ALTER NAMESPACE $ns SET PROPERTIES ('$key'='dummyVal')")
-          }
-          assert(exception.getMessage.contains(s"$key is a reserved namespace property"))
+          val sqlText = s"ALTER NAMESPACE $ns SET PROPERTIES ('$key'='dummyVal')"


Not related to this PR,
Only let's use checkErrorMatchPVals to check for error-conditions

cloud-fan · 2024-06-25T08:06:51Z

.../src/main/scala/org/apache/spark/sql/execution/command/UnsetNamespacePropertiesCommand.scala

+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * A command that ALTER NAMESPACE UNSET PROPERTIES command.


please fix the grammar.

.../src/main/scala/org/apache/spark/sql/execution/command/UnsetNamespacePropertiesCommand.scala

cloud-fan · 2024-06-27T07:39:08Z

docs/sql-ref-syntax-ddl-alter-database.md

 is issued if the database is not found in the system.

-### ALTER PROPERTIES
+### ALTER SET PROPERTIES


since the entire page is for ALTER DATABASE, I think SET PROPERTIES is a better section title.

cloud-fan · 2024-06-27T07:39:15Z

docs/sql-ref-syntax-ddl-alter-database.md


    Specifies the name of the database to be altered.

+### ALTER UNSET PROPERTIES


Can we also add examples for this new syntax, in the ### Examples section?

cloud-fan · 2024-06-27T07:39:40Z

docs/sql-ref-syntax-ddl-alter-database.md

+
+  Specifies the name of the database to be altered.
+
 ### ALTER LOCATION


cloud-fan · 2024-06-27T07:41:46Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

+  /**
+   * Create a [[UnsetNamespacePropertiesCommand]] command.
+   *
+   * For example:


Suggested change

* For example:

* Expected format:

cloud-fan · 2024-06-27T07:42:19Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

+   * For example:
+   * {{{
+   *   ALTER (DATABASE|SCHEMA|NAMESPACE) database
+   *   UNSET (DBPROPERTIES | PROPERTIES) ('comment', 'key');


Suggested change

* UNSET (DBPROPERTIES | PROPERTIES) ('comment', 'key');

* UNSET (DBPROPERTIES | PROPERTIES) ('key1', 'key2');

panbingkun · 2024-06-27T08:28:55Z

@cloud-fan
All suggestions have been updated.
Thank you for your patient review!

cloud-fan · 2024-06-28T02:36:42Z

thanks, merging to master!

cloud-fan · 2024-07-09T13:35:27Z

...st/scala/org/apache/spark/sql/execution/command/AlterNamespaceUnsetPropertiesSuiteBase.scala

+    }
+    checkError(e,
+      errorClass = "SCHEMA_NOT_FOUND",
+      parameters = Map("schemaName" -> s"`$ns`"))


just noticed one thing: why does the error message omit the catalog name? For table not found, the error message contains the user-given table name in the SQL statement.

Let me investigate it

v1 catalog

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

Line 316 in 4c99c4d

requireDbExists(dbName)

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

Line 253 in 4c99c4d

throw new NoSuchDatabaseException(db)

v2 catalog

spark/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableCatalog.scala

Lines 206 to 213 in 4c99c4d

override def loadNamespaceMetadata(namespace: Array[String]): util.Map[String, String] = {

Option(namespaces.get(namespace.toSeq)) match {

case Some(metadata) =>

metadata.asJava

case _ if namespaceExists(namespace) =>

util.Collections.emptyMap[String, String]

case _ =>

throw new NoSuchNamespaceException(namespace)

From the user's perspective (when he uses multiple catalogs for federated queries and encounters some tables that do not exist), we should fix it, WDYT?

Also, do we also need to fix similar scenarios (when table not exist) below together?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

Lines 257 to 267 in 4c99c4d

private def requireTableExists(name: TableIdentifier): Unit = {

if (!tableExists(name)) {

throw new NoSuchTableException(db = name.database.get, table = name.table)

}

}

private def requireTableNotExists(name: TableIdentifier): Unit = {

if (tableExists(name)) {

throw new TableAlreadyExistsException(db = name.database.get, table = name.table)

}

}

@cloud-fan Initial PR, just attach catalog name before the namespace. If this way is ok, I will continue to add table related as well.
#47276
Thanks!

…pace` to `catalog.namespace` ### What changes were proposed in this pull request? The pr aims to change the value of `SCHEMA_NOT_FOUND` from `namespace` to `catalog.namespace`. ### Why are the changes needed? As discussing #47038 (comment), we should provide more friendly and clear prompt error message. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Update existed UT & Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47276 from panbingkun/db_with_catalog. Authored-by: panbingkun <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…pace` to `catalog.namespace` ### What changes were proposed in this pull request? The pr aims to change the value of `SCHEMA_NOT_FOUND` from `namespace` to `catalog.namespace`. ### Why are the changes needed? As discussing apache#47038 (comment), we should provide more friendly and clear prompt error message. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Update existed UT & Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47276 from panbingkun/db_with_catalog. Authored-by: panbingkun <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

[SPARK-48668][SQL] Support ALTER NAMESPACE ... UNSET PROPERTIES in v2

12f8f71

github-actions bot added the SQL label Jun 20, 2024

panbingkun marked this pull request as ready for review June 21, 2024 07:05

cloud-fan reviewed Jun 21, 2024

View reviewed changes

panbingkun added 4 commits June 22, 2024 16:45

Merge branch 'master' into SPARK-48668

48e3b44

update

363ad49

Merge branch 'master' into SPARK-48668

7ddf703

fix

1f3ed06

panbingkun commented Jun 25, 2024

View reviewed changes

update

960f32a

panbingkun commented Jun 25, 2024

View reviewed changes

cloud-fan reviewed Jun 25, 2024

View reviewed changes

panbingkun added 2 commits June 25, 2024 14:19

Merge branch 'master' into SPARK-48668

0748ea4

update

53a9a25

panbingkun commented Jun 25, 2024

View reviewed changes

cloud-fan reviewed Jun 25, 2024

View reviewed changes

.../src/main/scala/org/apache/spark/sql/execution/command/UnsetNamespacePropertiesCommand.scala Show resolved Hide resolved

panbingkun added 4 commits June 25, 2024 17:25

update

33fbc7c

fix AlterNamespaceUnsetPropertiesParserSuite

0d09a65

Merge branch 'master' into SPARK-48668

b3b60ac

remove IF EXISTS

0e925c6

github-actions bot added the DOCS label Jun 27, 2024

cloud-fan reviewed Jun 27, 2024

View reviewed changes

docs/sql-ref-syntax-ddl-alter-database.md Outdated

Specifies the name of the database to be altered.

### ALTER LOCATION

Copy link

Contributor

cloud-fan Jun 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

cloud-fan reviewed Jun 27, 2024

View reviewed changes

cloud-fan approved these changes Jun 27, 2024

View reviewed changes

panbingkun added 2 commits June 27, 2024 15:53

Merge branch 'master' into SPARK-48668

87b0bcb

update docs

f5efc77

panbingkun mentioned this pull request Jun 27, 2024

[SPARK-48720][SQL] Align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2 #47097

Closed

cloud-fan closed this in 3bf7de0 Jun 28, 2024

cloud-fan reviewed Jul 9, 2024

View reviewed changes

panbingkun mentioned this pull request Jul 10, 2024

[SPARK-48851][SQL] Change the value of SCHEMA_NOT_FOUND from namespace to catalog.namespace #47276

Closed

panbingkun mentioned this pull request Sep 6, 2024

[SPARK-48906][SQL] Introduce SHOW COLLATIONS LIKE ... syntax to show all collations #47364

Closed

	\| ALTER (TABLE \| VIEW) identifierReference
	SET TBLPROPERTIES propertyList #setTableProperties
	\| ALTER (TABLE \| VIEW) identifierReference
	UNSET TBLPROPERTIES (IF EXISTS)? propertyList #unsetTableProperties

	\| ALTER namespace identifierReference
	SET (DBPROPERTIES \| PROPERTIES) propertyList #setNamespaceProperties

	/**
	* Apply a set of metadata changes to a namespace in the catalog.
	*
	* @param namespace a multi-part namespace
	* @param changes a collection of changes to apply to the namespace
	* @throws NoSuchNamespaceException If the namespace does not exist (optional)
	* @throws UnsupportedOperationException If namespace properties are not supported
	*/
	void alterNamespace(
	String[] namespace,
	NamespaceChange... changes) throws NoSuchNamespaceException;

	static NamespaceChange removeProperty(String property) {
	return new RemoveProperty(property);
	}


		(ctx.LOCAL != null, finalStorage, Some(DDLUtils.HIVE_PROVIDER))
		}


		Specifies the name of the database to be altered.

		### ALTER UNSET PROPERTIES


		Specifies the name of the database to be altered.

		### ALTER LOCATION

	* UNSET (DBPROPERTIES \| PROPERTIES) ('comment', 'key');
	* UNSET (DBPROPERTIES \| PROPERTIES) ('key1', 'key2');

	override def loadNamespaceMetadata(namespace: Array[String]): util.Map[String, String] = {
	Option(namespaces.get(namespace.toSeq)) match {
	case Some(metadata) =>
	metadata.asJava
	case _ if namespaceExists(namespace) =>
	util.Collections.emptyMap[String, String]
	case _ =>
	throw new NoSuchNamespaceException(namespace)

	private def requireTableExists(name: TableIdentifier): Unit = {
	if (!tableExists(name)) {
	throw new NoSuchTableException(db = name.database.get, table = name.table)
	}
	}

	private def requireTableNotExists(name: TableIdentifier): Unit = {
	if (tableExists(name)) {
	throw new TableAlreadyExistsException(db = name.database.get, table = name.table)
	}
	}

[SPARK-48668][SQL] Support ALTER NAMESPACE ... UNSET PROPERTIES in v2 #47038

[SPARK-48668][SQL] Support ALTER NAMESPACE ... UNSET PROPERTIES in v2 #47038

Uh oh!

Conversation

panbingkun commented Jun 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

panbingkun commented Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun Jun 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jun 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Jun 20, 2024 •

edited

Loading

panbingkun commented Jun 21, 2024 •

edited

Loading

panbingkun Jun 25, 2024 •

edited

Loading

cloud-fan Jun 25, 2024 •

edited

Loading

cloud-fan Jul 9, 2024 •

edited

Loading

panbingkun Jul 9, 2024 •

edited

Loading