[SPARK-18852][SS]StreamingQuery.lastProgress should be null when recentProgress is empty #16273
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brkyvz
reviewed
Dec 13, 2016
| } | ||
|
|
||
| test("lastProgress should be null when recentProgress is empty") { | ||
| BlockingSource.latch = new CountDownLatch(1) |
Contributor
There was a problem hiding this comment.
Glad that you got to re-use BlockingSource!
brkyvz
reviewed
Dec 13, 2016
| ) | ||
| } | ||
|
|
||
| test("lastProgress should be null when recentProgress is empty") { |
Contributor
There was a problem hiding this comment.
should we add a similar test to python just to make sure it returns None? Do we need to update any docs?
brkyvz
reviewed
Dec 13, 2016
| progressBuffer.toArray | ||
| } | ||
|
|
||
| /** Returns the most recent query progress update. */ |
Contributor
There was a problem hiding this comment.
Maybe add returns null if there were no progress updates
Contributor
|
Left one comment on updating docs, otherwise LGTM |
|
Test build #70106 has finished for PR 16273 at commit
|
|
Test build #70105 has finished for PR 16273 at commit
|
|
Test build #70112 has finished for PR 16273 at commit
|
Member
Author
|
retest this please |
|
Test build #70124 has started for PR 16273 at commit |
Member
Author
|
retest this please |
Contributor
|
LGTM! |
|
Test build #70138 has finished for PR 16273 at commit
|
|
Test build #70144 has finished for PR 16273 at commit
|
asfgit
pushed a commit
that referenced
this pull request
Dec 14, 2016
…entProgress is empty
## What changes were proposed in this pull request?
Right now `StreamingQuery.lastProgress` throws NoSuchElementException and it's hard to be used in Python since Python user will just see Py4jError.
This PR just makes it return null instead.
## How was this patch tested?
`test("lastProgress should be null when recentProgress is empty")`
Author: Shixiong Zhu <[email protected]>
Closes #16273 from zsxwing/SPARK-18852.
(cherry picked from commit 1ac6567)
Signed-off-by: Shixiong Zhu <[email protected]>
robert3005
pushed a commit
to palantir/spark
that referenced
this pull request
Dec 15, 2016
…entProgress is empty
## What changes were proposed in this pull request?
Right now `StreamingQuery.lastProgress` throws NoSuchElementException and it's hard to be used in Python since Python user will just see Py4jError.
This PR just makes it return null instead.
## How was this patch tested?
`test("lastProgress should be null when recentProgress is empty")`
Author: Shixiong Zhu <[email protected]>
Closes apache#16273 from zsxwing/SPARK-18852.
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…entProgress is empty
## What changes were proposed in this pull request?
Right now `StreamingQuery.lastProgress` throws NoSuchElementException and it's hard to be used in Python since Python user will just see Py4jError.
This PR just makes it return null instead.
## How was this patch tested?
`test("lastProgress should be null when recentProgress is empty")`
Author: Shixiong Zhu <[email protected]>
Closes apache#16273 from zsxwing/SPARK-18852.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Right now
StreamingQuery.lastProgressthrows NoSuchElementException and it's hard to be used in Python since Python user will just see Py4jError.This PR just makes it return null instead.
How was this patch tested?
test("lastProgress should be null when recentProgress is empty")