Skip to content

[SPARK-32287][TESTS] Flaky Test: ExecutorAllocationManagerSuite.add executors default profile #29225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tgravescs wants to merge 1 commit into from
Closed

[SPARK-32287][TESTS] Flaky Test: ExecutorAllocationManagerSuite.add executors default profile #29225

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/test/scala/org/apache/spark/ExecutorAllocationManagerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1603,7 +1603,7 @@ class ExecutorAllocationManagerSuite extends SparkFunSuite {
.set(config.DYN_ALLOCATION_TESTING, true)
// SPARK-22864: effectively disable the allocation schedule by setting the period to a
// really long value.
.set(TEST_SCHEDULE_INTERVAL, 10000L)
.set(TEST_SCHEDULE_INTERVAL, 30000L)
Copy link
Member

@dongjoon-hyun dongjoon-hyun Jul 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, this is for the same reason with SPARK-22864 at line 1604?

Copy link
Member

@Ngone51 Ngone51 Aug 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tgravescs It happens again at: https://github.com/apache/spark/pull/29418/checks?check_run_id=975304054

BTW, this change only prevents the automatic invocation of schedule() for the second time but the first time invocation of schedule() always happens because the initialDelay is 0?

spark/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala

Line 250 in 8f0fef1

executor.scheduleWithFixedDelay(scheduleTask, 0, intervalMillis, TimeUnit.MILLISECONDS)

Copy link
Contributor Author

@tgravescs tgravescs Aug 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not seeing. a test report in that run?
Yes but it seemed like the failures we were seeing before we after an interation or two or the test starting, although it did vary. If we can see the logs from it we should be able to see timing and possibly some log messages to tell us if that is the problem.

Copy link
Member

@Ngone51 Ngone51 Aug 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not seeing. a test report in that run?

You need to scroll to the end of Run tests: core, unsafe, kvstore, avro section to see the failed tests.

Yes but it seemed like the failures we were seeing before we after an interation or two or the test starting, although it did vary.

Ok, not aware of that...

If we can see the logs from it we should be able to see timing and possibly some log messages to tell us if that is the problem.

@tgravescs A feasible way to debug it is to open a PR in your own fork repository and adds some println in the source code.

@HyukjinKwon I think we still can not see the complete logs in GithubActions like the unit-test.log in Jenkins. I mean, even we download the archive logs after checks finished.

Copy link
Member

@HyukjinKwon HyukjinKwon Aug 17, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I can try to upload that specific log file as an artifact (so we can download) Let me take a look in coming few days.

Copy link
Member

@Ngone51 Ngone51 Aug 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, the unit-tests.log file is generated by log4j:

spark/core/src/test/resources/log4j.properties

Line 23 in b33066f

log4j.appender.file.file=target/unit-tests.log

Copy link
Contributor Author

@tgravescs tgravescs Aug 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for updating and uploading the logs, let me know if you see a build that has this test failure and I'll look at the log

sparkConf
}

Expand Down