Skip to content

[SPARK-29014][SQL] DataSourceV2: Fix current/default catalog usage #26120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
imback82 wants to merge 8 commits into from
Closed

[SPARK-29014][SQL] DataSourceV2: Fix current/default catalog usage #26120

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2fa927a
initial commit
imback82 Oct 14, 2019
8317d31
fix tests
imback82 Oct 15, 2019
46d2fa3
Make CatalogManager's defaultCatalog and *sessionCatalog private.
imback82 Oct 16, 2019
d12d4ce
Merge branch 'master' into cleanup_catalog
imback82 Oct 16, 2019
4131274
Address PR comments
imback82 Oct 17, 2019
9f1e574
fix tests
imback82 Oct 17, 2019
6d49017
adding back "private[sql]" that was removed by mistake.
imback82 Oct 18, 2019
b93c5e3
fix tests
imback82 Oct 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
initial commit
  • Loading branch information
@imback82
imback82 committed Oct 14, 2019
commit 2fa927a1dbe69ae6bb1b69917e026f66a363305b
5 changes: 2 additions & 3 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,9 +178,8 @@ class ResolveCatalogs(val catalogManager: CatalogManager)
case ShowTablesStatement(Some(NonSessionCatalog(catalog, nameParts)), pattern) =>
ShowTables(catalog.asTableCatalog, nameParts, pattern)

// TODO (SPARK-29014): we should check if the current catalog is not session catalog here.
case ShowTablesStatement(None, pattern) if defaultCatalog.isDefined =>
ShowTables(defaultCatalog.get.asTableCatalog, catalogManager.currentNamespace, pattern)
case ShowTablesStatement(None, pattern) if !isSessionCatalog(currentCatalog) =>
ShowTables(currentCatalog.asTableCatalog, catalogManager.currentNamespace, pattern)

case UseStatement(isNamespaceSet, nameParts) =>
if (isNamespaceSet) {
Expand Down
3 changes: 1 addition & 2 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,7 @@ private[sql] trait LookupCatalog extends Logging {
Some((catalogManager.catalog(nameParts.head), nameParts.tail))
} catch {
case _: CatalogNotFoundException =>
// TODO (SPARK-29014): use current catalog here.
Some((defaultCatalog.getOrElse(sessionCatalog), nameParts))
Some((currentCatalog, nameParts))
}
}
}
Expand Down
3 changes: 1 addition & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,8 +263,7 @@ class ResolveSessionCatalog(
}
ShowTablesCommand(Some(nameParts.head), pattern)

// TODO (SPARK-29014): we should check if the current catalog is session catalog here.
case ShowTablesStatement(None, pattern) if defaultCatalog.isEmpty =>
case ShowTablesStatement(None, pattern) if isSessionCatalog(currentCatalog) =>
ShowTablesCommand(None, pattern)
}

Expand Down
17 changes: 17 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -725,6 +725,23 @@ class DataSourceV2SQLSuite
expectV2Catalog = false)
}

test("ShowTables: change current catalog and namespace with USE statements") {
sql("CREATE TABLE testcat.ns1.ns2.table (id bigint) USING foo")

// Initially, the v2 session catalog (current catalog) is used.
runShowTablesSql(
"SHOW TABLES", Seq(Row("", "source", true), Row("", "source2", true)),
expectV2Catalog = false)

// Update the current catalog, and no table is matched since the current namespace is Array().
sql("USE testcat")
runShowTablesSql("SHOW TABLES", Seq())

// Update the current namespace to match ns1.ns2.table.
sql("USE testcat.ns1.ns2")
runShowTablesSql("SHOW TABLES", Seq(Row("ns1.ns2", "table")))
}

private def runShowTablesSql(
sqlText: String,
expected: Seq[Row],
Expand Down