You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-13888][DOC] Remove Akka Receiver doc and refer to the DStream Akka project
## What changes were proposed in this pull request?
I have copied the docs of Streaming Akka to https://github.com/spark-packages/dstream-akka/blob/master/README.md
So we can remove them from Spark now.
## How was this patch tested?
Only document changes.
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Author: Shixiong Zhu <[email protected]>
Closesapache#11711 from zsxwing/remove-akka-doc.
Copy file name to clipboardExpand all lines: docs/streaming-custom-receivers.md
-61Lines changed: 0 additions & 61 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -256,64 +256,3 @@ The following table summarizes the characteristics of both types of receivers
256
256
<td></td>
257
257
</tr>
258
258
</table>
259
-
260
-
## Implementing and Using a Custom Actor-based Receiver
261
-
262
-
Custom [Akka Actors](http://doc.akka.io/docs/akka/2.3.11/scala/actors.html) can also be used to
263
-
receive data. Here are the instructions.
264
-
265
-
1.**Linking:** You need to add the following dependency to your SBT or Maven project (see [Linking section](streaming-programming-guide.html#linking) in the main programming guide for further information).
You need to extend [`ActorReceiver`](api/scala/index.html#org.apache.spark.streaming.akka.ActorReceiver)
277
-
so as to store received data into Spark using `store(...)` methods. The supervisor strategy of
278
-
this actor can be configured to handle failures, etc.
279
-
280
-
class CustomActor extends ActorReceiver {
281
-
def receive = {
282
-
case data: String => store(data)
283
-
}
284
-
}
285
-
286
-
// A new input stream can be created with this custom actor as
287
-
val ssc: StreamingContext = ...
288
-
val lines = AkkaUtils.createStream[String](ssc, Props[CustomActor](), "CustomReceiver")
289
-
290
-
See [ActorWordCount.scala](https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/ActorWordCount.scala) for an end-to-end example.
291
-
</div>
292
-
<div data-lang="java" markdown="1">
293
-
294
-
You need to extend [`JavaActorReceiver`](api/scala/index.html#org.apache.spark.streaming.akka.JavaActorReceiver)
295
-
so as to store received data into Spark using `store(...)` methods. The supervisor strategy of
296
-
this actor can be configured to handle failures, etc.
297
-
298
-
class CustomActor extends JavaActorReceiver {
299
-
@Override
300
-
public void onReceive(Object msg) throws Exception {
301
-
store((String) msg);
302
-
}
303
-
}
304
-
305
-
// A new input stream can be created with this custom actor as
See [JavaActorWordCount.scala](https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/JavaActorWordCount.scala) for an end-to-end example.
310
-
</div>
311
-
</div>
312
-
313
-
3.**Deploying:** As with any Spark applications, `spark-submit` is used to launch your application.
314
-
You need to package `spark-streaming-akka_{{site.SCALA_BINARY_VERSION}}` and its dependencies into
315
-
the application JAR. Make sure `spark-core_{{site.SCALA_BINARY_VERSION}}` and `spark-streaming_{{site.SCALA_BINARY_VERSION}}`
316
-
are marked as `provided` dependencies as those are already present in a Spark installation. Then
317
-
use `spark-submit` to launch your application (see [Deploying section](streaming-programming-guide.html#deploying-applications) in the main programming guide).
318
-
319
-
<spanclass="badge"style="background-color: grey">Python API</span> Since actors are available only in the Java and Scala libraries, AkkaUtils is not available in the Python API.
Copy file name to clipboardExpand all lines: docs/streaming-programming-guide.md
+7-17Lines changed: 7 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -594,7 +594,7 @@ data from a source and stores it in Spark's memory for processing.
594
594
Spark Streaming provides two categories of built-in streaming sources.
595
595
596
596
-*Basic sources*: Sources directly available in the StreamingContext API.
597
-
Examples: file systems, socket connections, and Akka actors.
597
+
Examples: file systems, and socket connections.
598
598
-*Advanced sources*: Sources like Kafka, Flume, Kinesis, Twitter, etc. are available through
599
599
extra utility classes. These require linking against extra dependencies as discussed in the
600
600
[linking](#linking) section.
@@ -631,7 +631,7 @@ as well as to run the receiver(s).
631
631
We have already taken a look at the `ssc.socketTextStream(...)` in the [quick example](#a-quick-example)
632
632
which creates a DStream from text
633
633
data received over a TCP socket connection. Besides sockets, the StreamingContext API provides
634
-
methods for creating DStreams from files and Akka actors as input sources.
634
+
methods for creating DStreams from files as input sources.
635
635
636
636
-**File Streams:** For reading data from files on any file system compatible with the HDFS API (that is, HDFS, S3, NFS, etc.), a DStream can be created as:
637
637
@@ -658,17 +658,12 @@ methods for creating DStreams from files and Akka actors as input sources.
658
658
659
659
<span class="badge" style="background-color: grey">Python API</span> `fileStream` is not available in the Python API, only `textFileStream` is available.
660
660
661
-
-**Streams based on Custom Actors:** DStreams can be created with data streams received through Akka
662
-
actors by using `AkkaUtils.createStream(ssc, actorProps, actor-name)`. See the [Custom Receiver
663
-
Guide](streaming-custom-receivers.html) for more details.
664
-
665
-
<spanclass="badge"style="background-color: grey">Python API</span> Since actors are available only in the Java and Scala
666
-
libraries, `AkkaUtils.createStream` is not available in the Python API.
661
+
-**Streams based on Custom Receivers:** DStreams can be created with data streams received through custom receivers. See the [Custom Receiver
662
+
Guide](streaming-custom-receivers.html) and [DStream Akka](https://github.com/spark-packages/dstream-akka) for more details.
667
663
668
664
-**Queue of RDDs as a Stream:** For testing a Spark Streaming application with test data, one can also create a DStream based on a queue of RDDs, using `streamingContext.queueStream(queueOfRDDs)`. Each RDD pushed into the queue will be treated as a batch of data in the DStream, and processed like a stream.
669
665
670
-
For more details on streams from sockets, files, and actors,
671
-
see the API documentations of the relevant functions in
666
+
For more details on streams from sockets and files, see the API documentations of the relevant functions in
672
667
[StreamingContext](api/scala/index.html#org.apache.spark.streaming.StreamingContext) for
and the `pushBlock(...)` methods to store received data has been renamed to `store(...)`. Other helper classes in
2446
-
the `org.apache.spark.streaming.receivers` package were also moved
2447
-
to [`org.apache.spark.streaming.receiver`](api/scala/index.html#org.apache.spark.streaming.receiver.package)
2448
-
package and renamed for better clarity.
2437
+
**Actor-based Receivers**: The Actor-based Receiver APIs have been moved to [DStream Akka](https://github.com/spark-packages/dstream-akka).
0 commit comments