Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
b9d2ced
incomplete setup for external shuffle service tests
squito Jul 28, 2015
36127d3
wip
squito Jul 28, 2015
bb3ba49
minor cleanup
squito Jul 28, 2015
c69f46b
maybe working version, needs tests & cleanup ...
squito Jul 30, 2015
5e5a7c3
fix build
squito Jul 31, 2015
ad122ef
more fixes
squito Jul 31, 2015
0b588bd
more fixes ...
squito Jul 31, 2015
1136f44
test needs to have an actual shuffle
squito Jul 31, 2015
9eae119
cleanup lots of duplication
squito Jul 31, 2015
0e9d69b
better names
squito Jul 31, 2015
d596969
cleanup imports
squito Jul 31, 2015
dd93dc0
test for shuffle service w/ NM restarts
squito Aug 1, 2015
efb6195
proper unit test, and no longer leak if apps stop during NM restart
squito Aug 1, 2015
a36729c
cleanup
squito Aug 1, 2015
7504de8
style
squito Aug 1, 2015
23994ff
style
squito Aug 1, 2015
86e0cb9
for tests, shuffle service finds an open port
squito Aug 1, 2015
bdc4b32
rename
squito Aug 1, 2015
bb9d1e6
formatting
squito Aug 1, 2015
857331a
better tests & comments
squito Aug 1, 2015
bdbbf0d
comments, remove some unnecessary changes
squito Aug 1, 2015
62586a6
just serialize the whole executors map
squito Aug 3, 2015
245db19
style
squito Aug 3, 2015
55f49fc
make sure the service doesnt die if the registered executor file is c…
squito Aug 3, 2015
0a39b98
Merge branch 'master' into external_shuffle_service_NM_restart
squito Aug 3, 2015
4492835
lol, dont use a PrintWriter b/c of scalastyle checks
squito Aug 3, 2015
f729e2b
debugging
squito Aug 4, 2015
d7450f0
style
squito Aug 4, 2015
32fe5ae
Merge branch 'master' into external_shuffle_service_NM_restart
squito Aug 4, 2015
59800b7
Files.move in case renaming is unsupported
squito Aug 4, 2015
d596410
store executor data in leveldb
squito Aug 4, 2015
694934c
only open leveldb connection once per service
squito Aug 5, 2015
c878fbe
better explanation of shuffle service port handling
squito Aug 5, 2015
12b6a35
save registered executors when apps are removed; add tests
squito Aug 5, 2015
79922b7
rely on yarn to call stopApplication; assorted cleanup
squito Aug 6, 2015
acedb62
switch to writing out one record per executor
squito Aug 6, 2015
9378ba3
fail gracefully on corrupt leveldb files
squito Aug 6, 2015
e9f99e8
cleanup the handling of bad dbs a little
squito Aug 6, 2015
8267d2a
style
squito Aug 7, 2015
1a7980b
version
squito Aug 7, 2015
594d520
use json to serialize application executor info
squito Aug 7, 2015
81f80e2
Merge branch 'master' into leveldb_external_shuffle_service_NM_restart
squito Aug 12, 2015
795d28f
review feedback
squito Aug 12, 2015
2499c8c
explicit dependency on jackson-annotations
squito Aug 12, 2015
5c71c8c
save executor to db before registering; style
squito Aug 14, 2015
70951d6
Merge branch 'master' into leveldb_external_shuffle_service_NM_restart
squito Aug 17, 2015
0d285d3
review feedback
squito Aug 17, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
review feedback
  • Loading branch information
squito committed Aug 17, 2015
commit 0d285d3fac15afc77313255799a3392dcf74518f
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,10 @@ public ExternalShuffleBlockResolver(TransportConf conf, File registeredExecutorF
// one, so we can keep processing new apps
logger.error("error opening leveldb file {}. Creating new file, will not be able to " +
"recover state for existing applications", registeredExecutorFile, e);
for (File f: registeredExecutorFile.listFiles()) {
f.delete();
if (registeredExecutorFile.isDirectory()) {
for (File f : registeredExecutorFile.listFiles()) {
f.delete();
}
}
registeredExecutorFile.delete();
options.createIfMissing(true);
Expand Down Expand Up @@ -353,8 +355,10 @@ private static byte[] dbAppExecKey(AppExecId appExecId) throws IOException {
}

private static AppExecId parseDbAppExecKey(String s) throws IOException {
int p = s.indexOf(';');
String json = s.substring(p + 1);
if (!s.startsWith(APP_KEY_PREFIX)) {
throw new IllegalArgumentException("expected a string starting with " + APP_KEY_PREFIX);
}
String json = s.substring(APP_KEY_PREFIX.length() + 1);
AppExecId parsed = mapper.readValue(json, AppExecId.class);
return parsed;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,6 @@
package org.apache.spark.network.yarn

import java.io.File
import java.util.{List => JList, Map => JMap}
import java.util.Map.Entry


import org.apache.spark.network.shuffle.ExternalShuffleBlockResolver
import org.apache.spark.network.shuffle.protocol.ExecutorShuffleInfo

/**
* just a cheat to get package-visible members in tests
Expand Down