Skip to content

Commit 43559f5

Browse files
zsxwingroygao94
authored andcommitted
[SPARK-13888][DOC] Remove Akka Receiver doc and refer to the DStream Akka project
## What changes were proposed in this pull request? I have copied the docs of Streaming Akka to https://github.com/spark-packages/dstream-akka/blob/master/README.md So we can remove them from Spark now. ## How was this patch tested? Only document changes. (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Author: Shixiong Zhu <[email protected]> Closes apache#11711 from zsxwing/remove-akka-doc.
1 parent 04e81e5 commit 43559f5

File tree

2 files changed

+7
-78
lines changed

2 files changed

+7
-78
lines changed

docs/streaming-custom-receivers.md

Lines changed: 0 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -256,64 +256,3 @@ The following table summarizes the characteristics of both types of receivers
256256
<td></td>
257257
</tr>
258258
</table>
259-
260-
## Implementing and Using a Custom Actor-based Receiver
261-
262-
Custom [Akka Actors](http://doc.akka.io/docs/akka/2.3.11/scala/actors.html) can also be used to
263-
receive data. Here are the instructions.
264-
265-
1. **Linking:** You need to add the following dependency to your SBT or Maven project (see [Linking section](streaming-programming-guide.html#linking) in the main programming guide for further information).
266-
267-
groupId = org.apache.spark
268-
artifactId = spark-streaming-akka_{{site.SCALA_BINARY_VERSION}}
269-
version = {{site.SPARK_VERSION_SHORT}}
270-
271-
2. **Programming:**
272-
273-
<div class="codetabs">
274-
<div data-lang="scala" markdown="1" >
275-
276-
You need to extend [`ActorReceiver`](api/scala/index.html#org.apache.spark.streaming.akka.ActorReceiver)
277-
so as to store received data into Spark using `store(...)` methods. The supervisor strategy of
278-
this actor can be configured to handle failures, etc.
279-
280-
class CustomActor extends ActorReceiver {
281-
def receive = {
282-
case data: String => store(data)
283-
}
284-
}
285-
286-
// A new input stream can be created with this custom actor as
287-
val ssc: StreamingContext = ...
288-
val lines = AkkaUtils.createStream[String](ssc, Props[CustomActor](), "CustomReceiver")
289-
290-
See [ActorWordCount.scala](https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/ActorWordCount.scala) for an end-to-end example.
291-
</div>
292-
<div data-lang="java" markdown="1">
293-
294-
You need to extend [`JavaActorReceiver`](api/scala/index.html#org.apache.spark.streaming.akka.JavaActorReceiver)
295-
so as to store received data into Spark using `store(...)` methods. The supervisor strategy of
296-
this actor can be configured to handle failures, etc.
297-
298-
class CustomActor extends JavaActorReceiver {
299-
@Override
300-
public void onReceive(Object msg) throws Exception {
301-
store((String) msg);
302-
}
303-
}
304-
305-
// A new input stream can be created with this custom actor as
306-
JavaStreamingContext jssc = ...;
307-
JavaDStream<String> lines = AkkaUtils.<String>createStream(jssc, Props.create(CustomActor.class), "CustomReceiver");
308-
309-
See [JavaActorWordCount.scala](https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/JavaActorWordCount.scala) for an end-to-end example.
310-
</div>
311-
</div>
312-
313-
3. **Deploying:** As with any Spark applications, `spark-submit` is used to launch your application.
314-
You need to package `spark-streaming-akka_{{site.SCALA_BINARY_VERSION}}` and its dependencies into
315-
the application JAR. Make sure `spark-core_{{site.SCALA_BINARY_VERSION}}` and `spark-streaming_{{site.SCALA_BINARY_VERSION}}`
316-
are marked as `provided` dependencies as those are already present in a Spark installation. Then
317-
use `spark-submit` to launch your application (see [Deploying section](streaming-programming-guide.html#deploying-applications) in the main programming guide).
318-
319-
<span class="badge" style="background-color: grey">Python API</span> Since actors are available only in the Java and Scala libraries, AkkaUtils is not available in the Python API.

docs/streaming-programming-guide.md

Lines changed: 7 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -594,7 +594,7 @@ data from a source and stores it in Spark's memory for processing.
594594
Spark Streaming provides two categories of built-in streaming sources.
595595

596596
- *Basic sources*: Sources directly available in the StreamingContext API.
597-
Examples: file systems, socket connections, and Akka actors.
597+
Examples: file systems, and socket connections.
598598
- *Advanced sources*: Sources like Kafka, Flume, Kinesis, Twitter, etc. are available through
599599
extra utility classes. These require linking against extra dependencies as discussed in the
600600
[linking](#linking) section.
@@ -631,7 +631,7 @@ as well as to run the receiver(s).
631631
We have already taken a look at the `ssc.socketTextStream(...)` in the [quick example](#a-quick-example)
632632
which creates a DStream from text
633633
data received over a TCP socket connection. Besides sockets, the StreamingContext API provides
634-
methods for creating DStreams from files and Akka actors as input sources.
634+
methods for creating DStreams from files as input sources.
635635

636636
- **File Streams:** For reading data from files on any file system compatible with the HDFS API (that is, HDFS, S3, NFS, etc.), a DStream can be created as:
637637

@@ -658,17 +658,12 @@ methods for creating DStreams from files and Akka actors as input sources.
658658

659659
<span class="badge" style="background-color: grey">Python API</span> `fileStream` is not available in the Python API, only `textFileStream` is available.
660660

661-
- **Streams based on Custom Actors:** DStreams can be created with data streams received through Akka
662-
actors by using `AkkaUtils.createStream(ssc, actorProps, actor-name)`. See the [Custom Receiver
663-
Guide](streaming-custom-receivers.html) for more details.
664-
665-
<span class="badge" style="background-color: grey">Python API</span> Since actors are available only in the Java and Scala
666-
libraries, `AkkaUtils.createStream` is not available in the Python API.
661+
- **Streams based on Custom Receivers:** DStreams can be created with data streams received through custom receivers. See the [Custom Receiver
662+
Guide](streaming-custom-receivers.html) and [DStream Akka](https://github.com/spark-packages/dstream-akka) for more details.
667663

668664
- **Queue of RDDs as a Stream:** For testing a Spark Streaming application with test data, one can also create a DStream based on a queue of RDDs, using `streamingContext.queueStream(queueOfRDDs)`. Each RDD pushed into the queue will be treated as a batch of data in the DStream, and processed like a stream.
669665

670-
For more details on streams from sockets, files, and actors,
671-
see the API documentations of the relevant functions in
666+
For more details on streams from sockets and files, see the API documentations of the relevant functions in
672667
[StreamingContext](api/scala/index.html#org.apache.spark.streaming.StreamingContext) for
673668
Scala, [JavaStreamingContext](api/java/index.html?org/apache/spark/streaming/api/java/JavaStreamingContext.html)
674669
for Java, and [StreamingContext](api/python/pyspark.streaming.html#pyspark.streaming.StreamingContext) for Python.
@@ -2439,13 +2434,8 @@ that can be called to store the data in Spark. So, to migrate your custom networ
24392434
BlockGenerator object (does not exist any more in Spark 1.0 anyway), and use `store(...)` methods on
24402435
received data.
24412436

2442-
**Actor-based Receivers**: Data could have been received using any Akka Actors by extending the actor class with
2443-
`org.apache.spark.streaming.receivers.Receiver` trait. This has been renamed to
2444-
[`org.apache.spark.streaming.receiver.ActorHelper`](api/scala/index.html#org.apache.spark.streaming.receiver.ActorHelper)
2445-
and the `pushBlock(...)` methods to store received data has been renamed to `store(...)`. Other helper classes in
2446-
the `org.apache.spark.streaming.receivers` package were also moved
2447-
to [`org.apache.spark.streaming.receiver`](api/scala/index.html#org.apache.spark.streaming.receiver.package)
2448-
package and renamed for better clarity.
2437+
**Actor-based Receivers**: The Actor-based Receiver APIs have been moved to [DStream Akka](https://github.com/spark-packages/dstream-akka).
2438+
Please refer to the project for more details.
24492439

24502440
***************************************************************************************************
24512441
***************************************************************************************************

0 commit comments

Comments
 (0)