Skip to content

[SPARK-26288][CORE] Restore RegisteredExecutors information for External shuffle service in Standalone/Kubernetes backend when the service is restarted #23393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
weixiuli wants to merge 15 commits into from
Closed

[SPARK-26288][CORE] Restore RegisteredExecutors information for External shuffle service in Standalone/Kubernetes backend when the service is restarted #23393

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
105a321
add initRegisteredExecutorsDB
weixiuli Dec 28, 2018
b9ff5aa
Check all the dirs and use the dir which has be used
weixiuli Jan 24, 2019
628ffa5
Remove some registeredExecutors information of DB in external shuffle…
weixiuli Mar 15, 2019
d1b54fc
VisibleForTesting
weixiuli Mar 5, 2019
2ba3695
findRegisteredExecutorsDBFile
weixiuli Mar 6, 2019
8424852
fix ExternalShuffleServiceDbSuite
weixiuli Mar 6, 2019
d53c52a
fix style
weixiuli Mar 6, 2019
c681bfb
add WorkDirCleanup test
weixiuli Mar 7, 2019
29804eb
fix small
weixiuli Mar 15, 2019
7e87f64
add it to the docs
weixiuli Mar 12, 2019
7b8b31c
fix style
weixiuli Mar 15, 2019
cb78728
fix style
weixiuli Mar 14, 2019
ed9a842
change spark-standalone.md
weixiuli Mar 15, 2019
1f94b86
revert default
weixiuli Mar 15, 2019
475d278
fix small
weixiuli Mar 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add it to the docs
  • Loading branch information
@weixiuli
weixiuli committed Mar 15, 2019
commit 7e87f64c3ce878078728c7585d2590085c7440c8
14 changes: 13 additions & 1 deletion docs/spark-standalone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,10 +236,11 @@ SPARK_WORKER_OPTS supports the following system properties:
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
<td><code>spark.worker.cleanup.enabled</code></td>
<td>false</td>
<td>true</td>
Copy link
Contributor

@attilapiros attilapiros Mar 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have you changed this default value in the documentation? As I see it is still false.

spark/core/src/main/scala/org/apache/spark/internal/config/Worker.scala

Lines 31 to 33 in b154233

val WORKER_CLEANUP_ENABLED = ConfigBuilder("spark.worker.cleanup.enabled")
.booleanConf
.createWithDefault(false)

Copy link
Contributor Author

@weixiuli weixiuli Mar 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry,i missed it.

Copy link
Contributor

@squito squito Mar 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe attila meant to revert your change to the docs, not to change the default

<td>
Enable periodic cleanup of worker / application directories. Note that this only affects standalone
mode, as YARN works differently. Only the directories of stopped applications are cleaned up.
This must be enabled if spark.shuffle.service.db.enabled is "true"
</td>
</tr>
<tr>
Expand All @@ -260,6 +261,17 @@ SPARK_WORKER_OPTS supports the following system properties:
especially if you run jobs very frequently.
</td>
</tr>
<tr>
<td><spark.shuffle.service.db.enabled</code></td>
<td>true</td>
<td>
Enable record RegisteredExecutors information by leveldb, which can be reloaded and
used again when the external shuffle service is restarted. Note that this only affects standalone
mode, its has always on for yarn. We should Enable `spark.worker.cleanup.enabled` to remove the entry
(It will leave an entry in the DB forever when an application is stopped while the external shuffle
service is down) in the leveldb with WorkDirCleanup. It may be removed in the future.
Copy link
Contributor

@squito squito Mar 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor rewordings:

Store External Shuffle service state on local disk so that when the external shuffle service is restarted, it will
automatically reload info on current executors.  This only affects standalone mode (yarn always has this behavior
enabled).  You should also enable <code>spark.worker.cleanup.enabled</code>, to ensure that the state
eventually gets cleaned up.  This config may be removed in the future.

Copy link
Contributor Author

@weixiuli weixiuli Mar 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clear description, thank you.

</td>
</tr>
<tr>
<td><code>spark.storage.cleanupFilesAfterExecutorExit</code></td>
<td>true</td>
Expand Down