Skip to content

[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile #1959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ghost wants to merge 5 commits into from
Closed

[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile #1959

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f8fc32a
filter out tmp dir
josephsu Aug 13, 2014
c7e44f2
match syntax
josephsu Aug 14, 2014
40ea9bd
fix hadoop-0.23 compile error
josephsu Aug 15, 2014
8fe2398
add note to explain
josephsu Aug 15, 2014
be30793
remove .* and _* except _metadata
josephsu Aug 23, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add note to explain
  • Loading branch information
@josephsu
josephsu committed Aug 15, 2014
commit 8fe2398f0e460c65ae57def2198308c4fe2de0f1
2 changes: 2 additions & 0 deletions sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypes.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,8 @@ private[parquet] object ParquetTypesConverter extends Logging {
}
ParquetRelation.enableLogForwarding()

// NOTE: Explicitly list "_temporary" because hadoop 0.23 removed the variable TEMP_DIR_NAME
// from FileOutputCommitter. Check MAPREDUCE-5229 for the detail.
val children = fs.listStatus(path).filterNot { status =>
val name = status.getPath.getName
name(0) == '.' || name == FileOutputCommitter.SUCCEEDED_FILE_NAME || name == "_temporary"
Copy link
Member

@srowen srowen Aug 15, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about ignoring any file starting with _ ? Hadoop (also) uses this convention, for things like the _SUCCESS file.

Copy link
Author

@ghost ghost Aug 15, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, that would ignore the metadata file "_metadata" as well.

Copy link
Contributor

@chutium chutium Aug 21, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should rethink about why we use filterNot here? simple filter works fine here, something like:

val children = fs.listStatus(path).filter { status =>
  val name = status.getPath.getName
  name == ParquetFileWriter.PARQUET_METADATA_FILE || (name(0) != '.' && name(0) != '_')
}

so we can ignore all of hidden/tmp files without _metadata

Copy link
Contributor

@liancheng liancheng Aug 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with this. Just remove .* and _* except _metadata.

Expand Down