Skip to content

[SPARK-7497] [PySpark] [Streaming] fix streaming flaky tests #6239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
davies wants to merge 3 commits into from
Closed

[SPARK-7497] [PySpark] [Streaming] fix streaming flaky tests #6239

davies wants to merge 3 commits into from

Conversation

@davies
Copy link
Contributor

@davies davies commented May 18, 2015

Increase the duration and timeout in streaming python tests.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #33012 has started for PR 6239 at commit 7947db6.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #33012 has finished for PR 6239 at commit 7947db6.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 18, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33012/
Test FAILed.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #822 has started for PR 6239 at commit 7947db6.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #822 has finished for PR 6239 at commit 7947db6.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #823 has started for PR 6239 at commit 7947db6.

@SparkQA
Copy link

SparkQA commented May 18, 2015

Test build #823 has finished for PR 6239 at commit 7947db6.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tdas
Copy link
Contributor

tdas commented May 18, 2015

test this again

@andrewor14
Copy link
Contributor

andrewor14 commented May 19, 2015

retest this please

@AmplabJenkins
Copy link

AmplabJenkins commented May 19, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 19, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 19, 2015

Test build #33034 has started for PR 6239 at commit 7947db6.

@tdas
Copy link
Contributor

tdas commented May 19, 2015

May be a good idea to merge this any way to reduce some test flakiness.

tdas
tdas reviewed May 19, 2015
View reviewed changes
python/pyspark/streaming/tests.py
Copy link
Contributor

@tdas tdas May 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the duration? Batch duration? Why increase this?

Copy link
Contributor Author

@davies davies Jun 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's batch duration. These tests became flaky when we change the duration from 1 to 0.2, I'd like to try to change it to 0.5. If they are still flaky, then move to 1 second.

@SparkQA
Copy link

SparkQA commented May 19, 2015

Test build #33034 has finished for PR 6239 at commit 7947db6.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 19, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 19, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33034/
Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 19, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 19, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 19, 2015

Test build #33049 has started for PR 6239 at commit d6aee8f.

@SparkQA
Copy link

SparkQA commented May 19, 2015

Test build #33049 has finished for PR 6239 at commit d6aee8f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 19, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 19, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33049/
Test PASSed.

@tdas
Copy link
Contributor

tdas commented May 19, 2015

@davies Why are you increasing the batch duration??

@davies
Copy link
Contributor Author

davies commented May 19, 2015

@tdas The tests become flaky with lower duration.

@SparkQA
Copy link

SparkQA commented May 19, 2015

Test build #833 has started for PR 6239 at commit d6aee8f.

@SparkQA
Copy link

SparkQA commented May 19, 2015

Test build #833 has finished for PR 6239 at commit d6aee8f.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tdas
Copy link
Contributor

tdas commented May 19, 2015

Could you explain why does it become flaky? I want to understand it so that
we can learn some lessons that may help on the Scala/Java side as well.

On Tue, May 19, 2015 at 1:03 PM, Apache Spark QA [email protected]
wrote:

Test build #833 has finished
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/833/consoleFull
for PR 6239 at commit d6aee8f
d6aee8f
.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

—
Reply to this email directly or view it on GitHub
#6239 (comment).

@davies
Copy link
Contributor Author

davies commented May 19, 2015

@tdas Can not reproduce this flakiness locally right now. Remembered that when the duration is low and the job can not finished within it, something weird happened few months ago.

@SparkQA
Copy link

SparkQA commented May 21, 2015

Test build #845 has started for PR 6239 at commit d6aee8f.

@SparkQA
Copy link

SparkQA commented May 21, 2015

Test build #845 has finished for PR 6239 at commit d6aee8f.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tdas
Copy link
Contributor

tdas commented May 21, 2015

@JoshRosen Flaky test?
org.apache.spark.SparkContextSuite.Cancelling job group should not cause SparkContext to shutdown (SPARK-6414)

@SparkQA
Copy link

SparkQA commented May 21, 2015

Test build #847 has started for PR 6239 at commit d6aee8f.

@SparkQA
Copy link

SparkQA commented May 21, 2015

Test build #847 has finished for PR 6239 at commit d6aee8f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 23, 2015

Test build #856 has started for PR 6239 at commit d6aee8f.

@SparkQA
Copy link

SparkQA commented May 23, 2015

Test build #856 has finished for PR 6239 at commit d6aee8f.

  • This patch fails to build.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 23, 2015

Test build #857 has started for PR 6239 at commit d6aee8f.

@SparkQA
Copy link

SparkQA commented May 23, 2015

Test build #857 has finished for PR 6239 at commit d6aee8f.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 26, 2015

Test build #858 has started for PR 6239 at commit d6aee8f.

@SparkQA
Copy link

SparkQA commented May 26, 2015

Test build #858 has finished for PR 6239 at commit d6aee8f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 1, 2015

Test build #865 has started for PR 6239 at commit d6aee8f.

@SparkQA
Copy link

SparkQA commented Jun 1, 2015

Test build #865 has finished for PR 6239 at commit d6aee8f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tdas
Copy link
Contributor

tdas commented Jun 1, 2015

The problem with this approach is that it will increase the run time of python streaming tests by a lot. And since these tests run 3 times for 3 different version of python, the increase will be multiple. But probably that is okay for now. The flakiness is causing problems for others, so I will merge this.

asfgit pushed a commit that referenced this pull request Jun 1, 2015
@tdas
[SPARK-7497] [PYSPARK] [STREAMING] fix streaming flaky tests
d023300 
Increase the duration and timeout in streaming python tests.

Author: Davies Liu <[email protected]>

Closes #6239 from davies/flaky_tests and squashes the following commits:

d6aee8f [Davies Liu] fix window tests
26317f7 [Davies Liu] Merge branch 'master' of github.com:apache/spark into flaky_tests
7947db6 [Davies Liu] fix streaming flaky tests

(cherry picked from commit b7ab029)
Signed-off-by: Tathagata Das <[email protected]>
@tdas
Copy link
Contributor

tdas commented Jun 1, 2015

I have merged it master and branch-1.4

@asfgit asfgit closed this in b7ab029 Jun 1, 2015
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@jeanlyn
[SPARK-7497] [PYSPARK] [STREAMING] fix streaming flaky tests
a724215 
Increase the duration and timeout in streaming python tests.

Author: Davies Liu <[email protected]>

Closes apache#6239 from davies/flaky_tests and squashes the following commits:

d6aee8f [Davies Liu] fix window tests
26317f7 [Davies Liu] Merge branch 'master' of github.com:apache/spark into flaky_tests
7947db6 [Davies Liu] fix streaming flaky tests
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
[SPARK-7497] [PYSPARK] [STREAMING] fix streaming flaky tests
a66b48e 
Increase the duration and timeout in streaming python tests.

Author: Davies Liu <[email protected]>

Closes apache#6239 from davies/flaky_tests and squashes the following commits:

d6aee8f [Davies Liu] fix window tests
26317f7 [Davies Liu] Merge branch 'master' of github.com:apache/spark into flaky_tests
7947db6 [Davies Liu] fix streaming flaky tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@davies @AmplabJenkins @SparkQA @tdas @andrewor14