[WIP] experimenting with shuffle locations in scheduler...

[yifeih]

Trying to add ShuffleLocations into the scheduler as a first class concept

[yifeih]

@mccheah @ifilonenko I'm seeing that the scheduler treats the idea of "host" and "location" as two different things. There are parts where it deletes all map outputs from a particular location (host/port combo) and parts where it deletes only based on host. To preserve the current behavior, we should probably expose the idea of a host from the ShuffleLocation interface.

@squito also brought up a good point that we have no use case for a generic ShuffleLocation that does not follow a host/port combination. The only implementation that needs to support a more generic form of ShuffleLocation is the individual file servers (for a DFS, we can configure a namenode location and not use ShuffleLocation. For async upload, we would use a host/port combination to represent the local location, and the namenode can be configured globally. For local implementation, the ShuffleLocation is also a host/port combination since it's just a wrapper around the BlockManagerId, as demonstrated in our PRs.) And for the individual file servers, I also can't see that we'd use anything other than a list of host/port combos. To simplify the API, I think we can safely and explicitly define the shuffle locations metadata stored in the driver to be an array of host/port combinations. With explicit definitions of shuffle locations as an array of host/port combos, it becomes easier to blacklist by host since you're no longer dealing with this nebulous object that can be anything (i.e. for the individual file server case, you can remove a location from the array of host/port combos if you discover that a particular location is inaccessible)

[yifeih]

follow up here: palantir#548