Skip to content

[SPARK-27659][PYTHON] Allow PySpark to prefetch during toLocalIterator #25515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
holdenk wants to merge 11 commits into from
Closed

[SPARK-27659][PYTHON] Allow PySpark to prefetch during toLocalIterator #25515

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0937158
Start working on allowing toLocalIter to prefetch in Python
holdenk Aug 15, 2019
77cab47
Move the partitionArray blocking up above the peak at head so we are …
holdenk Aug 16, 2019
4fc6db9
Fix python long line
holdenk Aug 16, 2019
b39a83c
Pull the head off & peak at the head+1 elem while before we block on …
holdenk Aug 17, 2019
e0b3871
Add a micro benchmark in prefetch
holdenk Aug 17, 2019
a060214
accidental line change we don't need
holdenk Aug 17, 2019
c477fec
oops on \n
holdenk Aug 17, 2019
6dc4748
Fix missing call
holdenk Aug 21, 2019
e0327a2
Fix sphinx build issues
holdenk Aug 21, 2019
11d6688
Cleanup the tests and some typos
holdenk Sep 10, 2019
f8e67f3
Remove the prefetch example we used as a benchmark
holdenk Sep 10, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
accidental line change we don't need
  • Loading branch information
@holdenk
holdenk committed Aug 17, 2019
commit a060214580e49818de2b7613a119e2ddce3ef782
1 change: 1 addition & 0 deletions bin/spark-shell
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,4 @@ main "$@"
# then reenable echo and propagate the code.
exit_status=$?
onExit
\n