Skip to content

[SPARK-5068] [SQL] Fix bug query data when path doesn't exist for HiveContext #4356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chenghao-intel wants to merge 6 commits into from
Closed

[SPARK-5068] [SQL] Fix bug query data when path doesn't exist for HiveContext #4356

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0033ed2
SPARK-5068: fix bug query data when path doesn't exists
jeanlyn Jan 4, 2015
1a65548
add the Licensed
jeanlyn Jan 28, 2015
76df33f
fix code style
jeanlyn Feb 2, 2015
6958312
Return empty row when table / partition path doesn't exist
chenghao-intel Feb 4, 2015
1f033cd
Move the FileSystem variable as class member
chenghao-intel Feb 4, 2015
d3a4d3c
Fix the potential bug of referencing FileSystem
chenghao-intel Feb 4, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move the FileSystem variable as class member
  • Loading branch information
@chenghao-intel
chenghao-intel committed Feb 4, 2015
commit 1f033cd8901bd97c8a4677e284847a2e975c6987
7 changes: 4 additions & 3 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
package org.apache.spark.sql.hive

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{Path, PathFilter}
import org.apache.hadoop.fs.{FileSystem, Path, PathFilter}
import org.apache.hadoop.hive.conf.HiveConf
import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants._
import org.apache.hadoop.hive.ql.exec.Utilities
Expand Down Expand Up @@ -68,6 +68,8 @@ class HadoopTableReader(
math.max(sc.hiveconf.getInt("mapred.map.tasks", 1), sc.sparkContext.defaultMinPartitions)
}

@transient private lazy val fs = FileSystem.get(sc.hiveconf)

// TODO: set aws s3 credentials.

private val _broadcastedHiveConf =
Expand Down Expand Up @@ -218,11 +220,10 @@ class HadoopTableReader(
* returned in a single, comma-separated string.
*/
private def applyFilterIfNeeded(path: Path, filterOpt: Option[PathFilter]): Option[String] = {
if (path.getFileSystem(sc.hiveconf).exists(path)) {
if (fs.exists(path)) {
Copy link
Contributor

@jeanlyn jeanlyn Feb 4, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd better get fs from the path,because in the hadoop namenode federation we may get some problems like Wrong FS exception if we use the FileSystem.get(sc.hiveconf) to get fs.

Copy link
Contributor

@liancheng liancheng Feb 5, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern is similar to what @marmbrus mentioned in #3981. It's pretty expensive to check each path in serial for tables with lots of partitions. Especially when the data reside on S3. Can we use listStatus or globStatus to retrieve all FileStatus objects under some path(s), and then do the filtering locally?

// if the file exists
filterOpt match {
case Some(filter) =>
val fs = path.getFileSystem(sc.hiveconf)
val filteredFiles = fs.listStatus(path, filter).map(_.getPath.toString)
if (filteredFiles.length > 0) {
Some(filteredFiles.mkString(","))
Expand Down