Skip to content

[SPARK-22672][SQL][TEST] Refactor ORC Tests #19882

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 5 commits into from
Closed

[SPARK-22672][SQL][TEST] Refactor ORC Tests #19882

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
278f4ac
[SPARK-22672][SQL][TEST] Refactor ORC Tests
dongjoon-hyun Aug 18, 2017
fcb2ccb
Address comments
dongjoon-hyun Dec 6, 2017
1828571
Update comments and rename files.
dongjoon-hyun Dec 7, 2017
19f4fd1
Revert "Update comments and rename files."
dongjoon-hyun Dec 7, 2017
f56423d
fix
dongjoon-hyun Dec 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update comments and rename files.
  • Loading branch information
@dongjoon-hyun
dongjoon-hyun committed Dec 7, 2017
commit 18285717edcbefe9a79d51a6fdcb112c3d0bc5c4
9 changes: 4 additions & 5 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ import org.apache.spark.sql.types._

/**
* A test suite that tests Apache ORC filter API based filter pushdown optimization.
* OrcFilterSuite and HiveOrcFilterSuite is logically duplicated to provide the same test coverage.
* The difference are the packages containing 'Predicate' and 'SearchArgument' classes.
* - OrcFilterSuite uses 'org.apache.orc.storage.ql.io.sarg' package.
* - HiveOrcFilterSuite uses 'org.apache.hadoop.hive.ql.io.sarg' package.
*/
class OrcFilterSuite extends OrcTest with SharedSQLContext {
Copy link
Member

@gatorsmile gatorsmile Dec 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let HiveOrcFilterSuite extend OrcFilterSuite?

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Dec 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ur, it's impossible because of the reason I mentioned in PR description.

OrcFilterSuite and HiveOrcFilterSuite cannot reuse test cases due to the different function signatures using Hive 1.2.1 ORC classes and Apache ORC 1.4.1 classes.

Copy link
Member

@HyukjinKwon HyukjinKwon Dec 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we leave some comments to explain that reason?
Seems there are many duplications and I would wonder why.


Expand Down Expand Up @@ -287,11 +291,6 @@ class OrcFilterSuite extends OrcTest with SharedSQLContext {

test("filter pushdown - combinations with logical operators") {
withOrcDataFrame((1 to 4).map(i => Tuple1(Option(i)))) { implicit df =>
// Because `ExpressionTree` is not accessible at Hive 1.2.x, this should be checked
// in string form in order to check filter creation including logical operators
// such as `and`, `or` or `not`. So, this function uses `SearchArgument.toString()`
// to produce string expression and then compare it to given string expression below.
// This might have to be changed after Hive version is upgraded.
checkFilterPredicate(
'_1.isNotNull,
"leaf-0 = (IS_NULL _1), expr = (not leaf-0)"
Expand Down
16 changes: 8 additions & 8 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,16 @@ import org.apache.spark.sql.test.SQLTestUtils
/**
* OrcTest
* -> OrcSuite
* -> OrcSourceSuite
* -> HiveOrcSourceSuite
* -> org.apache.spark.sql.execution.datasources.orc.OrcSourceSuite
* -> org.apache.spark.sql.hive.orc.OrcSourceSuite
* -> OrcQueryTests
* -> OrcQuerySuite
* -> HiveOrcQuerySuite
* -> org.apache.spark.sql.execution.datasources.orc.OrcQuerySuite
* -> org.apache.spark.sql.hive.orc.OrcQuerySuite
* -> OrcPartitionDiscoveryTest
* -> OrcPartitionDiscoverySuite
* -> HiveOrcPartitionDiscoverySuite
* -> OrcFilterSuite
* -> HiveOrcFilterSuite
* -> org.apache.spark.sql.execution.datasources.orc.OrcPartitionDiscoverySuite
* -> org.apache.spark.sql.hive.orc.OrcPartitionDiscoverySuite
* -> org.apache.spark.sql.execution.datasources.orc.OrcFilterSuite
* -> org.apache.spark.sql.hive.orc.OrcFilterSuite
*/
abstract class OrcTest extends QueryTest with SQLTestUtils with BeforeAndAfterAll {
import testImplicits._
Expand Down
3 changes: 1 addition & 2 deletions ...ark/sql/hive/orc/HiveOrcFilterSuite.scala → ...e/spark/sql/hive/orc/OrcFilterSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,12 @@ import org.apache.spark.sql.catalyst.planning.PhysicalOperation
import org.apache.spark.sql.execution.datasources.{DataSourceStrategy, HadoopFsRelation, LogicalRelation}
import org.apache.spark.sql.execution.datasources.orc.OrcTest
import org.apache.spark.sql.hive.test.TestHiveSingleton
import org.apache.spark.sql.internal.SQLConf.ORC_IMPLEMENTATION
import org.apache.spark.sql.types._

/**
* A test suite that tests Hive ORC filter API based filter pushdown optimization.
*/
class HiveOrcFilterSuite extends OrcTest with TestHiveSingleton {
class OrcFilterSuite extends OrcTest with TestHiveSingleton {

override val orcImp: String = "hive"

Expand Down
2 changes: 1 addition & 1 deletion .../orc/HiveOrcPartitionDiscoverySuite.scala → ...hive/orc/OrcPartitionDiscoverySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,6 @@ package org.apache.spark.sql.hive.orc
import org.apache.spark.sql.execution.datasources.orc.OrcPartitionDiscoveryTest
import org.apache.spark.sql.hive.test.TestHiveSingleton

class HiveOrcPartitionDiscoverySuite extends OrcPartitionDiscoveryTest with TestHiveSingleton {
class OrcPartitionDiscoverySuite extends OrcPartitionDiscoveryTest with TestHiveSingleton {
override val orcImp: String = "hive"
}
2 changes: 1 addition & 1 deletion ...park/sql/hive/orc/HiveOrcQuerySuite.scala → ...he/spark/sql/hive/orc/OrcQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import org.apache.spark.sql.hive.HiveUtils
import org.apache.spark.sql.hive.test.TestHiveSingleton
import org.apache.spark.sql.internal.SQLConf

class HiveOrcQuerySuite extends OrcQueryTest with TestHiveSingleton {
class OrcQuerySuite extends OrcQueryTest with TestHiveSingleton {
import testImplicits._

override val orcImp: String = "hive"
Expand Down
2 changes: 1 addition & 1 deletion ...ark/sql/hive/orc/HiveOrcSourceSuite.scala → ...e/spark/sql/hive/orc/OrcSourceSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import org.apache.spark.sql.execution.datasources.orc.OrcSuite
import org.apache.spark.sql.hive.test.TestHiveSingleton
import org.apache.spark.util.Utils

class HiveOrcSourceSuite extends OrcSuite with TestHiveSingleton {
class OrcSourceSuite extends OrcSuite with TestHiveSingleton {

override val orcImp: String = "hive"

Expand Down