[SPARK-7786][STREAMING] Allow StreamingListener to be specified in SparkConf and loaded when creating StreamingContext #6380

Jeffrharr · 2015-05-23T19:03:57Z

Add SparkConf option spark.streamingListeners to implement similar functionality to spark.extraListeners defined in [SPARK-5411].

Also modifies StatsReportListener to have a zero-valued constructor and adds a simple test

spark.streamingListeners

argument constructor in order to make it compatible with config option.

…'s single-valued constructor. Added docs for config option spark.streamingListeners

JoshRosen · 2015-05-24T04:11:28Z

Jenkins, this is ok to test.

SparkQA · 2015-05-24T04:15:53Z

Test build #33425 has finished for PR 6380 at commit e8506d4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

Jeffrharr · 2015-05-24T15:00:12Z

How can I run Scala style checks? By the way, thanks for writing the code that I essentially lifted for this :P

SparkQA · 2015-05-24T16:55:06Z

Test build #33445 has finished for PR 6380 at commit 6453c90.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

tdas · 2015-05-26T08:22:19Z

streaming/src/main/scala/org/apache/spark/streaming/scheduler/StreamingListener.scala

Why was this needed?

You have not addressed this. Why was this needed?? I dont see anything that required this.

andrewor14 · 2015-05-26T22:22:05Z

@His-name-is-Joof you can do dev/scalastyle locally

andrewor14 · 2015-05-26T22:22:33Z

docs/configuration.md

this should be under the spark.streaming namespace instead, so a better name would be spark.streaming.listeners, or spark.streaming.extraListeners to be more consistent with the existing spark.extraListeners.

SparkQA · 2015-05-27T21:41:09Z

Test build #33612 has finished for PR 6380 at commit d92d55b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

SparkQA · 2015-05-28T01:02:30Z

Test build #33625 has finished for PR 6380 at commit 00c0409.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

SparkQA · 2015-05-28T02:22:54Z

Test build #33635 has finished for PR 6380 at commit c94982f.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

zsxwing · 2015-06-09T00:54:36Z

streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala

~~listenerClass.getConstructor(classOf[SparkConf]) is simpler.~~

I see. getConstructor will throw NoSuchMethodException and make the code complex.

zsxwing · 2015-06-09T01:04:57Z

retest this please

SparkQA · 2015-06-09T02:55:20Z

Test build #34479 has finished for PR 6380 at commit c94982f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

tdas · 2015-06-12T07:57:51Z

Hey so, I thought about this feature a little bit. I am not this is an essential feature.The spark.extraListeners was added so that listeners can get added before SparkContext is started even in environments where the user does not explicitly start the SparkContext, e.g. spark shell. Since the SparkContext in the spark-shell is started automatically before the user gets any opportunity to attach their own listener, the extraListener conf was added to make sure the system automatically attaches them before creating any listener event.

But StreamingContext is really never started automatically like SparkContext in spark-shell. One can always add listeners before starting the StreamingContext. So what is the usecase where this is really essential?

tdas · 2015-06-18T23:14:26Z

Any thoughts on this?

Jeffrharr · 2015-06-19T04:44:48Z

Sorry, been unusually busy for quite a while. You make an excellent point,
it doesn't seem essential. However it does provide a certain consistency
that may make it simpler to move from listeners to streamingListeners. A
sort of "expected functionality". That it was requested in the first place
(not by me) is some evidence of this. Perhaps ask on the JIRA the precise
motivation?

The downside being that streamingListeners must be formatted properly to
work (perhaps encouraging obsfucated zero-argument constructors) and extra
code maintained.

Any thoughts on this?

—
Reply to this email directly or view it on GitHub
#6380 (comment).

avoid problems with classloaders

SparkQA · 2015-07-10T04:41:05Z

Test build #36995 has finished for PR 6380 at commit 233335d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-07-10T08:39:44Z

Jenkins, test this please.

SparkQA · 2015-07-10T10:43:18Z

Test build #37027 has finished for PR 6380 at commit 233335d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

tdas · 2015-07-10T19:28:32Z

your test is failing :)

tdas · 2015-07-10T19:28:55Z

streaming/src/test/scala/org/apache/spark/streaming/StreamingListenerSuite.scala

This needs to be changed too to extraListeners.

SparkQA · 2015-07-10T23:58:44Z

Test build #37056 has finished for PR 6380 at commit 28e7f27.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

SparkQA · 2015-07-11T02:53:24Z

Test build #37069 has finished for PR 6380 at commit af41098.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

tdas · 2015-07-13T21:52:42Z

streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala

This stop() is not needed as the StreamingContext has not been started yet at this point. And in that case the whole try ... catch is not needed, and the whole things can be simplified to a single throw new SparkException("Exception when registering StreamingListener: $className did not have a zero-argument constructor or a ..."

tdas · 2015-07-13T23:29:07Z

docs/configuration.md

This is incorrect:
"when initializing SparkContext"? ==> StreamingContext

"Spark's streaming listener bus" ==> "StreamingContext's listener bus"

SparkQA · 2015-07-14T00:58:42Z

Test build #37157 has finished for PR 6380 at commit 9ec68c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

tdas · 2015-07-14T20:05:48Z

Couple of more comments to address and it is good to go.

tdas · 2015-07-16T01:06:31Z

@His-name-is-Joof ping!

Jeffrharr · 2015-07-16T20:33:31Z

Working on it now. In the middle of moving apartments. Thanks for your
patience!

On Wed, Jul 15, 2015 at 7:07 PM, Tathagata Das [email protected]
wrote:

@His-name-is-Joof https://github.com/His-name-is-Joof ping!

—
Reply to this email directly or view it on GitHub
#6380 (comment).

SparkQA · 2015-07-17T00:53:28Z

Test build #37548 has finished for PR 6380 at commit fa8c752.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- class StatsReportListener(numBatchInfos: Int) extends StreamingListener

tdas · 2015-07-17T07:45:57Z

@His-name-is-Joof I am not sure you realized, but the PR does not merge cleanly! It has conflicts, please update with the master branch

tdas · 2015-07-17T07:52:55Z

streaming/src/test/scala/org/apache/spark/streaming/StreamingListenerSuite.scala

Lets not be so specific, the exact text can change. Just verify whether the message has the name of the class inside it.

tdas · 2015-07-22T02:57:57Z

@His-name-is-Joof Mind updating this soon. Would be good to get this merged before the underlying code diverges further.

tdas · 2015-07-27T21:01:18Z

Hey @His-name-is-Joof if you are not able to update this PR, mind if I take over merging it?

andrewor14 · 2015-09-01T23:39:11Z

Let's close this PR for now and reopen it later if needed. @tdas can take over.

Jeffrharr added 6 commits May 21, 2015 11:45

Add func setupStreamingListeners to add listeners from

73549ff

spark.streamingListeners

Merge branch 'master' into SparkConf_Listeners

4eb6987

Use class API to add listener. Modify StatsReportListener to have a zero

f8ad629

argument constructor in order to make it compatible with config option.

Reduced ambiguity by removing default values from StatsReportListener…

3c1a19d

…'s single-valued constructor. Added docs for config option spark.streamingListeners

Add test

f3a3fee

Fix minor syntax derp

e8506d4

Jeffrharr changed the title ~~[Spark 7786] Allow StreamingListener to be specified in SparkConf and loaded when creating StreamingContext~~ [Spark 7786][STREAMING] Allow StreamingListener to be specified in SparkConf and loaded when creating StreamingContext May 24, 2015

Jeffrharr changed the title ~~[Spark 7786][STREAMING] Allow StreamingListener to be specified in SparkConf and loaded when creating StreamingContext~~ [SPARK-7786][STREAMING] Allow StreamingListener to be specified in SparkConf and loaded when creating StreamingContext May 24, 2015

Fix style issues and small type in StreamingListenerSuite

6453c90

tdas reviewed May 26, 2015
View reviewed changes

andrewor14 reviewed May 26, 2015
View reviewed changes

Maintain consistancy with property naming conventions

d92d55b

Jeffrharr added 2 commits May 27, 2015 17:16

Fix test for new format

00c0409

Merge remote-tracking branch 'upstream/master' into SparkConf_Listeners

186766f

Test renamed to spark.streaming.listeners

c94982f

zsxwing reviewed Jun 9, 2015
View reviewed changes

Update Class.forName to org.apache.spark.util.Utils.classForName to

233335d

avoid problems with classloaders

tdas reviewed Jul 10, 2015
View reviewed changes

Jeffrharr added 2 commits July 10, 2015 14:01

Use spark.streaming.extraListeners

28e7f27

checking

af41098

tdas reviewed Jul 13, 2015
View reviewed changes

Changes in clarity, removed redundant error handling

9ec68c1

tdas reviewed Jul 13, 2015
View reviewed changes

Jeffrharr added 3 commits July 16, 2015 15:13

Fix documentation, add more/clearer test cases

70c31e4

Accidental insertion removed

aff08ff

Correct test text

fa8c752

tdas reviewed Jul 17, 2015
View reviewed changes

asfgit closed this in 804a012 Sep 4, 2015

[SPARK-7786][STREAMING] Allow StreamingListener to be specified in SparkConf and loaded when creating StreamingContext #6380

[SPARK-7786][STREAMING] Allow StreamingListener to be specified in SparkConf and loaded when creating StreamingContext #6380

Uh oh!

Conversation

Jeffrharr commented May 23, 2015

Uh oh!

JoshRosen commented May 24, 2015

Uh oh!

SparkQA commented May 24, 2015

Uh oh!

Jeffrharr commented May 24, 2015

Uh oh!

SparkQA commented May 24, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented May 26, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 27, 2015

Uh oh!

SparkQA commented May 28, 2015

Uh oh!

SparkQA commented May 28, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zsxwing commented Jun 9, 2015

Uh oh!

SparkQA commented Jun 9, 2015

Uh oh!

tdas commented Jun 12, 2015

Uh oh!

tdas commented Jun 18, 2015

Uh oh!

Jeffrharr commented Jun 19, 2015

Uh oh!

SparkQA commented Jul 10, 2015

Uh oh!

tdas commented Jul 10, 2015

Uh oh!

SparkQA commented Jul 10, 2015

Uh oh!

tdas commented Jul 10, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 10, 2015

Uh oh!

SparkQA commented Jul 11, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 14, 2015

Uh oh!

tdas commented Jul 14, 2015

Uh oh!

tdas commented Jul 16, 2015

Uh oh!

Jeffrharr commented Jul 16, 2015

Uh oh!

SparkQA commented Jul 17, 2015

Uh oh!

tdas commented Jul 17, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdas commented Jul 22, 2015

Uh oh!

tdas commented Jul 27, 2015

Uh oh!

andrewor14 commented Sep 1, 2015

Uh oh!