Skip to content

[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile #1924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ghost wants to merge 2 commits into from
Closed

[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile #1924

ghost wants to merge 2 commits into from

Conversation

@ghost
Copy link

@ghost ghost commented Aug 13, 2014

No description provided.

@AmplabJenkins
Copy link

AmplabJenkins commented Aug 13, 2014

Can one of the admins verify this patch?

@marmbrus
Copy link
Contributor

marmbrus commented Aug 13, 2014

test this please

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA tests have started for PR 1924. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18449/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 13, 2014

QA results for PR 1924:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18449/consoleFull

@marmbrus
Copy link
Contributor

marmbrus commented Aug 13, 2014

BTW you can check style on your local machine by running sbt/sbt scalastyle.

@ghost
Copy link
Author

ghost commented Aug 14, 2014

code fixed.

@marmbrus
Copy link
Contributor

marmbrus commented Aug 14, 2014

test this please

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA tests have started for PR 1924. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18526/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 14, 2014

QA results for PR 1924:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18526/consoleFull

@marmbrus
Copy link
Contributor

marmbrus commented Aug 14, 2014

The test failures look unrelated.

I've merge this to master and 1.1. Thanks!

asfgit pushed a commit that referenced this pull request Aug 14, 2014
@josephsu @marmbrus
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
221c84e 
…ontext.parquetFile

Author: Chia-Yung Su <[email protected]>

Closes #1924 from joesu/bugfix-spark3011 and squashes the following commits:

c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir

(cherry picked from commit 078f3fb)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in 078f3fb Aug 14, 2014
@tgravescs
Copy link
Contributor

tgravescs commented Aug 14, 2014

I'm seeing build failures from this change on hadoop 0.23. I haven't had time to dig into it.

sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypes.scala:382: value TEMP_DIR_NAME is not a member of object org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
[ERROR] name == FileOutputCommitter.TEMP_DIR_NAME

@marmbrus
Copy link
Contributor

marmbrus commented Aug 14, 2014

I'll revert this change.

@marmbrus marmbrus mentioned this pull request Aug 14, 2014
Closed
asfgit pushed a commit that referenced this pull request Aug 14, 2014
@marmbrus
Revert [SPARK-3011][SQL] _temporary directory should be filtered out …
c39a3f3 
…by sqlContext.parquetFile

Reverts #1924 due to build failures with hadoop 0.23.

Author: Michael Armbrust <[email protected]>

Closes #1949 from marmbrus/revert1924 and squashes the following commits:

6bff940 [Michael Armbrust] Revert "[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile"

(cherry picked from commit a7f8a4f)
Signed-off-by: Michael Armbrust <[email protected]>
asfgit pushed a commit that referenced this pull request Aug 14, 2014
@marmbrus
Revert [SPARK-3011][SQL] _temporary directory should be filtered out …
a7f8a4f 
…by sqlContext.parquetFile

Reverts #1924 due to build failures with hadoop 0.23.

Author: Michael Armbrust <[email protected]>

Closes #1949 from marmbrus/revert1924 and squashes the following commits:

6bff940 [Michael Armbrust] Revert "[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile"
@tgravescs
Copy link
Contributor

tgravescs commented Aug 14, 2014

Thanks, I took a quick look to see what the problem is and it appears that was removed in hadoop 0.23 and then put back in in hadoop 2.x to be compatible with hadoop 1.x:

https://issues.apache.org/jira/browse/MAPREDUCE-5229

So unfortunately we would need a way to have it not look at that on hadoop 0.23.

@marmbrus
Copy link
Contributor

marmbrus commented Aug 14, 2014

We could just hard code the string _temporary with a note about why.
On Aug 14, 2014 1:07 PM, "Tom Graves" [email protected] wrote:

Thanks, I took a quick look to see what the problem is and it appears that
was removed in hadoop 0.23 and then put back in in hadoop 2.x to be
compatible with hadoop 1.x:

https://issues.apache.org/jira/browse/MAPREDUCE-5229

So unfortunately we would need a way to have it not look at that on hadoop
0.23.

—
Reply to this email directly or view it on GitHub
#1924 (comment).

@ghost ghost mentioned this pull request Aug 15, 2014
Closed
@ghost
Copy link
Author

ghost commented Aug 15, 2014

Submitted another pull request: #1959

asfgit pushed a commit that referenced this pull request Aug 26, 2014
@josephsu @marmbrus
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
b5dc9b4 
…ontext.parquetFile

fix compile error on hadoop 0.23 for the pull request #1924.

Author: Chia-Yung Su <[email protected]>

Closes #1959 from joesu/bugfix-spark3011 and squashes the following commits:

be30793 [Chia-Yung Su] remove .* and _* except _metadata
8fe2398 [Chia-Yung Su] add note to explain
40ea9bd [Chia-Yung Su] fix hadoop-0.23 compile error
c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir

(cherry picked from commit 4243bb6)
Signed-off-by: Michael Armbrust <[email protected]>
asfgit pushed a commit that referenced this pull request Aug 26, 2014
@josephsu @marmbrus
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
4243bb6 
…ontext.parquetFile

fix compile error on hadoop 0.23 for the pull request #1924.

Author: Chia-Yung Su <[email protected]>

Closes #1959 from joesu/bugfix-spark3011 and squashes the following commits:

be30793 [Chia-Yung Su] remove .* and _* except _metadata
8fe2398 [Chia-Yung Su] add note to explain
40ea9bd [Chia-Yung Su] fix hadoop-0.23 compile error
c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@josephsu @conviva-zz
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
a869de8 
…ontext.parquetFile

Author: Chia-Yung Su <[email protected]>

Closes apache#1924 from joesu/bugfix-spark3011 and squashes the following commits:

c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@marmbrus @conviva-zz
Revert [SPARK-3011][SQL] _temporary directory should be filtered out …
91f59d7 
…by sqlContext.parquetFile

Reverts apache#1924 due to build failures with hadoop 0.23.

Author: Michael Armbrust <[email protected]>

Closes apache#1949 from marmbrus/revert1924 and squashes the following commits:

6bff940 [Michael Armbrust] Revert "[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile"
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@josephsu @conviva-zz
[SPARK-3011][SQL] _temporary directory should be filtered out by sqlC…
fbcdc39 
…ontext.parquetFile

fix compile error on hadoop 0.23 for the pull request apache#1924.

Author: Chia-Yung Su <[email protected]>

Closes apache#1959 from joesu/bugfix-spark3011 and squashes the following commits:

be30793 [Chia-Yung Su] remove .* and _* except _metadata
8fe2398 [Chia-Yung Su] add note to explain
40ea9bd [Chia-Yung Su] fix hadoop-0.23 compile error
c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@AmplabJenkins @marmbrus @SparkQA @tgravescs @josephsu