Skip to content

Spark 3754 spark streaming file system api callable from java #2703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
holdenk wants to merge 7 commits into from
Closed

Spark 3754 spark streaming file system api callable from java #2703

holdenk wants to merge 7 commits into from

Conversation

@holdenk
Copy link
Contributor

@holdenk holdenk commented Oct 8, 2014

This is a WIP of making the filesystem api callable for the Java version of Spark Streaming.

@AmplabJenkins
Copy link

AmplabJenkins commented Oct 8, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21433/Test FAILed.

@SparkQA
Copy link

SparkQA commented Oct 9, 2014

QA tests have started for PR 2703 at commit 47cfdd5.

  • This patch merges cleanly.

@holdenk holdenk changed the title WIP: Spark 3754 spark streaming file system api callable from java Spark 3754 spark streaming file system api callable from java Oct 9, 2014
@holdenk
Copy link
Contributor Author

holdenk commented Oct 9, 2014

@tdas this fixes the issue with the Java API I found while working on the book.

@SparkQA
Copy link

SparkQA commented Oct 9, 2014

QA tests have finished for PR 2703 at commit 47cfdd5.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Oct 9, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21496/Test FAILed.

@SparkQA
Copy link

SparkQA commented Oct 9, 2014

QA tests have started for PR 2703 at commit 80704c5.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Oct 9, 2014

QA tests have finished for PR 2703 at commit 80704c5.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Oct 9, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21507/Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented Oct 9, 2014

Can one of the admins verify this patch?

@holdenk
Copy link
Contributor Author

holdenk commented Oct 22, 2014

jenkins, retest this please

@tdas
Copy link
Contributor

tdas commented Oct 23, 2014

Jenkins, this is ok to test.

@SparkQA
Copy link

SparkQA commented Oct 23, 2014

QA tests have started for PR 2703 at commit 2bc3687.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Oct 23, 2014

QA tests have finished for PR 2703 at commit 2bc3687.

  • This patch fails MiMa tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Oct 23, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22090/
Test FAILed.

tdas
tdas reviewed Oct 23, 2014
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
Copy link
Contributor

@tdas tdas Oct 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be related to SPARK 3754

Copy link
Contributor

@tdas tdas Oct 23, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure whether there is any use case for exposing this.

Copy link
Contributor Author

@holdenk holdenk Oct 27, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the use case of exposing this was so that I could expose equivelent java functionality to what was available in Scala

Copy link
Contributor

@tdas tdas Nov 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @holdenk, although the underlying fileinputdstream supported it, it was not exposed in the API. I am not sure its worth exposing this kind of functionality. So lets not change this.

@tdas
Copy link
Contributor

tdas commented Oct 23, 2014

Thanks @holdenk for doing this. This is a great fix.
But since this is an API change which breaks binary compatibility (I am okay with it, as no one is probably using this method), you have to add exclusions in the mima.

Run the mima test with dev/mima, see the error and use that output to add to the exclusions in project/MimaExcludes

@holdenk
Copy link
Contributor Author

holdenk commented Nov 5, 2014

@tdas: Ok will do. No one should be using the old API because it ins't callable from Java.

@SparkQA
Copy link

SparkQA commented Nov 5, 2014

QA tests have started for PR 2703 at commit 3a2cd21.

  • This patch does not merge cleanly.

@SparkQA
Copy link

SparkQA commented Nov 6, 2014

QA tests have finished for PR 2703 at commit 3a2cd21.

  • This patch fails Spark unit tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 6, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22958/
Test FAILed.

@tdas
Copy link
Contributor

tdas commented Nov 7, 2014

Jenkins, test this again

@SparkQA
Copy link

SparkQA commented Nov 10, 2014

Test build #23146 has started for PR 2703 at commit efac1fc.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 10, 2014

Test build #23146 has finished for PR 2703 at commit efac1fc.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 10, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23146/
Test PASSed.

tdas
tdas reviewed Nov 11, 2014
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
Copy link
Contributor

@tdas tdas Nov 11, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same reason as above, no need to expose this functionality of newFilesOnly

Copy link
Contributor

@tdas tdas Dec 27, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction on this comment. newFilesOnly should be exposed as it is exposed in the Scala api.

@tdas
Copy link
Contributor

tdas commented Dec 31, 2014

Ping?

@holdenk
Copy link
Contributor Author

holdenk commented Dec 31, 2014

I'll take a look this weekend when I have some time.

@tdas
Copy link
Contributor

tdas commented Dec 31, 2014

Thanks!

@tdas
Copy link
Contributor

tdas commented Feb 10, 2015

This has already been fixed by #4101 Can you please close this PR?

@holdenk holdenk closed this Feb 11, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@holdenk @AmplabJenkins @SparkQA @tdas