Skip to content

[SPARK-11419][STREAMING] Parallel recovery for FileBasedWriteAheadLog + minor recovery tweaks #9373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
brkyvz wants to merge 21 commits into from
Closed

[SPARK-11419][STREAMING] Parallel recovery for FileBasedWriteAheadLog + minor recovery tweaks #9373

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
573b657
progress
brkyvz Oct 29, 2015
655f4bf
ready for PR
brkyvz Oct 29, 2015
06da0d1
ready for PR
brkyvz Oct 30, 2015
be5a2ab
minor
brkyvz Oct 30, 2015
7f8cfe3
fix whitespace
brkyvz Oct 30, 2015
98da092
update
brkyvz Nov 10, 2015
0b7279f
minor
brkyvz Nov 10, 2015
c2cafe1
fix conflicts
brkyvz Nov 10, 2015
83aa28e
add check for openInputStream
brkyvz Nov 10, 2015
9162ed9
save change
brkyvz Nov 11, 2015
c9ea423
add reader limiting test
brkyvz Nov 11, 2015
c250d2e
minor
brkyvz Nov 11, 2015
285a8d1
Merge branch 'master' of github.com:apache/spark into par-recovery
brkyvz Nov 11, 2015
22fbaca
address comments
brkyvz Nov 11, 2015
f43ecbe
minor2
brkyvz Nov 11, 2015
1ba8340
minor grammar
brkyvz Nov 11, 2015
ccf7f5b
fix merge conflicts
brkyvz Nov 11, 2015
7e1829b
address 3
brkyvz Nov 12, 2015
dbb31e3
address minor
brkyvz Nov 12, 2015
a31822c
increase thread size for Jenkins
brkyvz Nov 12, 2015
79e9b03
reduce max thread limit
brkyvz Nov 12, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
minor
  • Loading branch information
@brkyvz
brkyvz committed Nov 11, 2015
commit c250d2e15e9435e8c7e9ac9c4927057e4c6cf9f7
3 changes: 2 additions & 1 deletion streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,9 +272,10 @@ private[streaming] object FileBasedWriteAheadLog {
threadpool: ThreadPoolExecutor,
source: Seq[I],
handler: I => Iterator[O]): Iterator[O] = {
val taskSupport = new ThreadPoolTaskSupport(threadpool)
source.grouped(threadpool.getCorePoolSize).flatMap { element =>
Copy link
Contributor

@tdas tdas Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, my bad. grouped creates groups of n, not n groups. We want n groups.

Copy link
Contributor

@tdas tdas Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nvm, i see you have implemented it differently than what we discussed. Seems to work. In fact this is more evenly load balanced i think. Just change element to group.

val parallelCollection = element.par
parallelCollection.tasksupport = new ThreadPoolTaskSupport(threadpool)
parallelCollection.tasksupport = taskSupport
parallelCollection.map(handler)
}.flatten
}
Expand Down