Skip to content

[SPARK-12429][Streaming][Doc]Add Accumulator and Broadcast example for Streaming #10385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zsxwing wants to merge 4 commits into from
Closed

[SPARK-12429][Streaming][Doc]Add Accumulator and Broadcast example for Streaming #10385

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9928ca5
Add Accumulator and Broadcast Scala example for Streaming
zsxwing Dec 18, 2015
78d15bd
Address
zsxwing Dec 18, 2015
9e241e7
Add Java and Python examples
zsxwing Dec 21, 2015
455968a
Address comments
zsxwing Dec 22, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address
  • Loading branch information
@zsxwing
zsxwing committed Dec 18, 2015
commit 78d15bd7e343c4d6d07dcf97df76a6a8e6e8fe61
2 changes: 1 addition & 1 deletion docs/streaming-programming-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1417,7 +1417,7 @@ Note that the connections in the pool should be lazily created on demand and tim

## Accumulator and Broadcast
Copy link
Contributor

@tdas tdas Dec 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accumulators and Broadcast variables


Accumulator and Broadcast cannot be recovered from checkpoint in Streaming. If you enable checkpoint and use Accumulator or Broadcast as well, you have to create lazily instantiated singleton instances for Accumulator and Broadcast so that they can be restarted on driver failures. This is shown in the following example.
Accumulator and Broadcast cannot be recovered from checkpoint in Spark Streaming. If you enable checkpointing and use Accumulator or Broadcast as well, you'll have to create lazily instantiated singleton instances for Accumulator and Broadcast so that they can be restarted on driver failures. This is shown in the following example.
Copy link
Contributor

@tdas tdas Dec 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe its called "Broadcast variable". see http://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables
Might be a good idea to put links to the Spark programming guide

"...so that they can be re-instantiated after the driver restarts on failure"


<div class="codetabs">
<div data-lang="scala" markdown="1">
Expand Down