Skip to content

[SPARK-11663][Streaming]Add Java API for trackStateByKey #9636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zsxwing wants to merge 7 commits into from
Closed

[SPARK-11663][Streaming]Add Java API for trackStateByKey #9636

zsxwing wants to merge 7 commits into from

Conversation

@zsxwing
Copy link
Member

@zsxwing zsxwing commented Nov 11, 2015

TODO

  • Add Java API
  • Add API tests
  • Add a function test

tdas
tdas reviewed Nov 11, 2015
View reviewed changes
examples/src/main/java/org/apache/spark/examples/streaming/JavaStatefulNetworkWordCount.java Outdated
Copy link
Contributor

@tdas tdas Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use the simple function API in this example?

Copy link
Member Author

@zsxwing zsxwing Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need the key to emit the word and its count

@zsxwing
Copy link
Member Author

zsxwing commented Nov 11, 2015

Removed JavaState and JavaStateSpec

tdas
tdas reviewed Nov 11, 2015
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala Outdated
Copy link
Contributor

@tdas tdas Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add JavaPairDStream back in the docs.

@SparkQA
Copy link

SparkQA commented Nov 11, 2015

Test build #45672 has finished for PR 9636 at commit ac6c491.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * final class JavaState[S](state: State[S]) extends Serializable\n * final class JavaStateSpec[K, V, S, T](\n * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

tdas
tdas reviewed Nov 11, 2015
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaTrackStateDStream.scala
Copy link
Contributor

@tdas tdas Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this automatically link to the Scala docs of JavaDStream? May be safer to do [[org.apache....JavaDStream JavaDStream]]

Copy link
Member Author

@zsxwing zsxwing Nov 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine for classes in the same package. Take a look at: http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrame

@SparkQA
Copy link

SparkQA commented Nov 12, 2015

Test build #45677 has finished for PR 9636 at commit 44cf5c2.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing zsxwing changed the title [SPARK-11663][Streaming][WIP]Add Java API for trackStateByKey [SPARK-11663][Streaming]Add Java API for trackStateByKey Nov 12, 2015
tdas
tdas reviewed Nov 12, 2015
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala Outdated
Copy link
Contributor

@tdas tdas Nov 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldnt these be JavaTrackStateDStream<Integer, Integer, Integer, String>??

Copy link
Contributor

@tdas tdas Nov 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure that the code would compile

Copy link
Member Author

@zsxwing zsxwing Nov 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just went through all examples and fixed them

@SparkQA
Copy link

SparkQA commented Nov 12, 2015

Test build #45692 has finished for PR 9636 at commit 7160786.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

@zsxwing
Copy link
Member Author

zsxwing commented Nov 12, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Nov 12, 2015

Test build #45697 has finished for PR 9636 at commit 04845c8.

  • This patch fails from timeout after a configured wait of 250m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 12, 2015

Test build #45704 has finished for PR 9636 at commit 04845c8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

@SparkQA
Copy link

SparkQA commented Nov 12, 2015

Test build #45711 has finished for PR 9636 at commit f6c45cb.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

@zsxwing
Copy link
Member Author

zsxwing commented Nov 12, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Nov 12, 2015

Test build #45751 has finished for PR 9636 at commit f6c45cb.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

@zsxwing
Copy link
Member Author

zsxwing commented Nov 12, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Nov 12, 2015

Test build #45761 has finished for PR 9636 at commit f6c45cb.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

@zsxwing
Copy link
Member Author

zsxwing commented Nov 12, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Nov 13, 2015

Test build #45780 has finished for PR 9636 at commit f6c45cb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class JavaTrackStateDStream[KeyType, ValueType, StateType, EmittedType](\n

@tdas
Copy link
Contributor

tdas commented Nov 13, 2015

Thanks @zsxwing I am merging this to master and 1.6

@asfgit asfgit closed this in 0f1d00a Nov 13, 2015
asfgit pushed a commit that referenced this pull request Nov 13, 2015
@zsxwing @tdas
[SPARK-11663][STREAMING] Add Java API for trackStateByKey
05666e0 
TODO
- [x] Add Java API
- [x] Add API tests
- [x] Add a function test

Author: Shixiong Zhu <[email protected]>

Closes #9636 from zsxwing/java-track.

(cherry picked from commit 0f1d00a)
Signed-off-by: Tathagata Das <[email protected]>
dskrvk pushed a commit to dskrvk/spark that referenced this pull request Nov 13, 2015
@zsxwing @dskrvk
[SPARK-11663][STREAMING] Add Java API for trackStateByKey
ec42361 
TODO
- [x] Add Java API
- [x] Add API tests
- [x] Add a function test

Author: Shixiong Zhu <[email protected]>

Closes apache#9636 from zsxwing/java-track.
@zsxwing zsxwing deleted the java-track branch December 23, 2015 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zsxwing @SparkQA @tdas