Skip to content

[SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileFormat based on ORC 1.4.1 #19651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 22 commits into from
Closed

[SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileFormat based on ORC 1.4.1 #19651

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b342196
[SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileFormat based o…
dongjoon-hyun May 15, 2017
ca78ac7
Address comments.
dongjoon-hyun Nov 7, 2017
9d18834
Recover OrcFileFormat back, avoid function serialization, add TODO.
dongjoon-hyun Nov 10, 2017
6971cdf
Simplify `isSearchableType`
dongjoon-hyun Nov 10, 2017
b373495
Avoid boxing for primitive types.
dongjoon-hyun Nov 10, 2017
de8b509
Use getMissingColumnNames instead of getMissingSchema
dongjoon-hyun Nov 12, 2017
726406f
Move withNullSafe to OrcUtils
dongjoon-hyun Nov 12, 2017
4097457
Handle top-level columns in a while-loop and split the logic for Stru…
dongjoon-hyun Nov 12, 2017
8e0d392
fix
dongjoon-hyun Nov 12, 2017
9e3ac1a
Address comments.
dongjoon-hyun Nov 13, 2017
a3ebfbf
fix
dongjoon-hyun Nov 13, 2017
cc40fba
Move out column name handling logic.
dongjoon-hyun Nov 22, 2017
f482179
Use Updater like Parquet.
dongjoon-hyun Nov 28, 2017
e13dfa3
fix
dongjoon-hyun Nov 29, 2017
fdab6a7
remove outdate comment.
dongjoon-hyun Nov 29, 2017
daef4ba
refactor
cloud-fan Dec 1, 2017
f143e17
address comment
cloud-fan Dec 1, 2017
8a34731
Revert the change on HiveInspectors.scala.
dongjoon-hyun Dec 1, 2017
74cb053
Merge branch 'PR-2' into SPARK-20682
dongjoon-hyun Dec 1, 2017
eae50b3
bug fix
cloud-fan Dec 1, 2017
520837f
Merge pull request #3 from cloud-fan/orc
dongjoon-hyun Dec 2, 2017
71be008
Restore old ORC implementation.
dongjoon-hyun Dec 3, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Simplify isSearchableType
  • Loading branch information
@dongjoon-hyun
dongjoon-hyun committed Nov 29, 2017
commit 6971cdf7d0330b933c09d11fe415f3aada10609d
7 changes: 4 additions & 3 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,12 @@ private[orc] object OrcFilters {

/**
* Return true if this is a searchable type in ORC.
* Both CharType and VarcharType are cleaned at AstBuilder.
*/
private def isSearchableType(dataType: DataType) = dataType match {
case ByteType | ShortType | FloatType | DoubleType => true
case IntegerType | LongType | StringType | BooleanType => true
case TimestampType | _: DecimalType => true
// TODO: SPARK-21787 Support for pushing down filters for DateType in ORC
case BinaryType | DateType => false
case _: AtomicType => true
case _ => false
}

Expand Down