Skip to content

[Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories #23108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kevinyu98 wants to merge 10 commits into from
+265 −3
Closed

[Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories #23108

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
4e45ef9
add test case for orc table location
kevinyu98 Nov 21, 2018
e238764
doc the change in migration-guide
kevinyu98 Nov 21, 2018
e731746
address comments
kevinyu98 Nov 25, 2018
d6e582b
fix typo
kevinyu98 Nov 26, 2018
39ebcf8
address comment
kevinyu98 Nov 28, 2018
d75b923
adding testcase for parquet
kevinyu98 Dec 3, 2018
fe472c8
change to three level testing
kevinyu98 Dec 5, 2018
51d1d78
move the test cases
kevinyu98 Dec 5, 2018
d851169
address comments
kevinyu98 Dec 7, 2018
fef8c68
address comments
kevinyu98 Dec 12, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
add test case for orc table location
  • Loading branch information
@kevinyu98
kevinyu98 committed Nov 21, 2018
commit 4e45ef90fba26b34bd4d9b575b6bf793d0500fdc
33 changes: 33 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -597,6 +597,39 @@ abstract class OrcQueryTest extends OrcTest {
assert(m4.contains("Malformed ORC file"))
}
}

test("SPARK-25993 Add test cases for resolution of ORC table location") {
Copy link
Member

@dongjoon-hyun dongjoon-hyun Nov 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HiveOrcSourceSuite.scala will be the better place. And, we had better have the following and cover both case behaviors; true and false.

    Seq(true, false).foreach { convertMetastore =>
      withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> s"$convertMetastore") {

Copy link
Contributor Author

@kevinyu98 kevinyu98 Nov 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I will move the test case to there. Thanks.

withTempDir { dir =>
val someDF1 = Seq((1, 1, "orc1"), (2, 2, "orc2")).toDF("c1", "c2", "c3").repartition(1)
val tableName1 = "orcTable1"
val tableName2 = "orcTable2"
withTable(tableName1, tableName2) {
val path1 = s"${dir.getCanonicalPath}/dir1/"
someDF1.write.orc(path1)
val path2 = s"${dir.getCanonicalPath}/"
val sqlStatement1 =
s"""
|CREATE EXTERNAL TABLE $tableName1(C1 INT, C2 INT, C3 STRING)
|STORED AS ORC LOCATION '${path2}'
""".stripMargin
sql(sqlStatement1)
checkAnswer(
sql(s"SELECT * FROM ${tableName1}"), Nil)

val path3 = s"${dir.getCanonicalPath}/*"
val sqlStatement2 =
s"""
|CREATE EXTERNAL TABLE $tableName2(C1 INT, C2 INT, C3 STRING)
|STORED AS ORC LOCATION '${path3}'
""".stripMargin
sql(sqlStatement2)
checkAnswer(
sql(s"SELECT * FROM ${tableName2}"),
(1 to 2).map(i => Row(i, i, s"orc$i")))

}
}
}
}

class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
Expand Down