[SPARK-18423][STREAMING] ReceiverTracker should close checkpoint dir when stopped even if it was not started #15867
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #68571 has finished for PR 15867 at commit
|
HyukjinKwon
changed the title
Member
Author
Member
Author
|
Build started: [Streaming] Build started: [Streaming] |
|
Test build #68616 has finished for PR 15867 at commit
|
HyukjinKwon
changed the title
Member
|
LGTM. Merging to master and 2.1. Thanks! |
asfgit
pushed a commit
that referenced
this pull request
Nov 15, 2016
…when stopped even if it was not started
## What changes were proposed in this pull request?
Several tests are being failed on Windows due to the failure of removing the checkpoint dir between each tests.
This is caused by not closed file in `ReceiverTracker`. When it is not started, it does not close it even if `stop()` is called.
```
Test org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery started
Test org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery failed: java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\1478983663710-0, took 3.828 sec
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)
at org.apache.spark.util.Utils.deleteRecursively(Utils.scala)
at org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery(JavaAPISuite.java:1809)
...
```
```
- mapWithState - basic operations with simple API (7 seconds, 640 milliseconds)
Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.MapWithStateSuite *** ABORTED *** (12 seconds, 688 milliseconds)
java.io.IOException: Failed to delete: C:\projects\spark\streaming\checkpoint\spark-b8486e2b-6468-4e6f-bb24-88277d2c033c
...
```
## How was this patch tested?
Tests in `JavaAPISuite` and `MapWithStateSuite`.
Manually tested via AppVeyor:
**Before**
- `org.apache.spark.streaming.JavaAPISuite`
Build: https://ci.appveyor.com/project/spark-test/spark/build/71-MapWithStateSuite-1
Diff: master...spark-test:188c828e682ec45b75d15c3dfc782bcdc8ce024c
- `org.apache.spark.streaming.MapWithStateSuite`
Build: https://ci.appveyor.com/project/spark-test/spark/build/72-MapWithStateSuite-1
Diff: master...spark-test:8f6945d0ccde022a23d3848f6b7fe6da1e7c902e
**After**
- `org.apache.spark.streaming.JavaAPISuite`
Build started: [Streaming] `org.apache.spark.streaming.JavaAPISuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/3D74F2D5-B0D5-4E1D-874C-685AE694FD37)
Diff: master...spark-test:3D74F2D5-B0D5-4E1D-874C-685AE694FD37
- `org.apache.spark.streaming.MapWithStateSuite`
Build started: [Streaming] `org.apache.spark.streaming.MapWithStateSuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/C8E88B64-49F0-4157-9AFA-FC3ACC442351)
Diff: master...spark-test:C8E88B64-49F0-4157-9AFA-FC3ACC442351
Author: hyukjinkwon <[email protected]>
Closes #15867 from HyukjinKwon/SPARK-18423.
(cherry picked from commit 503378f)
Signed-off-by: Shixiong Zhu <[email protected]>
Member
Author
|
Thank you!! |
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…when stopped even if it was not started
## What changes were proposed in this pull request?
Several tests are being failed on Windows due to the failure of removing the checkpoint dir between each tests.
This is caused by not closed file in `ReceiverTracker`. When it is not started, it does not close it even if `stop()` is called.
```
Test org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery started
Test org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery failed: java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\1478983663710-0, took 3.828 sec
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)
at org.apache.spark.util.Utils.deleteRecursively(Utils.scala)
at org.apache.spark.streaming.JavaAPISuite.testCheckpointMasterRecovery(JavaAPISuite.java:1809)
...
```
```
- mapWithState - basic operations with simple API (7 seconds, 640 milliseconds)
Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.MapWithStateSuite *** ABORTED *** (12 seconds, 688 milliseconds)
java.io.IOException: Failed to delete: C:\projects\spark\streaming\checkpoint\spark-b8486e2b-6468-4e6f-bb24-88277d2c033c
...
```
## How was this patch tested?
Tests in `JavaAPISuite` and `MapWithStateSuite`.
Manually tested via AppVeyor:
**Before**
- `org.apache.spark.streaming.JavaAPISuite`
Build: https://ci.appveyor.com/project/spark-test/spark/build/71-MapWithStateSuite-1
Diff: apache/spark@master...spark-test:188c828e682ec45b75d15c3dfc782bcdc8ce024c
- `org.apache.spark.streaming.MapWithStateSuite`
Build: https://ci.appveyor.com/project/spark-test/spark/build/72-MapWithStateSuite-1
Diff: apache/spark@master...spark-test:8f6945d0ccde022a23d3848f6b7fe6da1e7c902e
**After**
- `org.apache.spark.streaming.JavaAPISuite`
Build started: [Streaming] `org.apache.spark.streaming.JavaAPISuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/3D74F2D5-B0D5-4E1D-874C-685AE694FD37)
Diff: apache/spark@master...spark-test:3D74F2D5-B0D5-4E1D-874C-685AE694FD37
- `org.apache.spark.streaming.MapWithStateSuite`
Build started: [Streaming] `org.apache.spark.streaming.MapWithStateSuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/C8E88B64-49F0-4157-9AFA-FC3ACC442351)
Diff: apache/spark@master...spark-test:C8E88B64-49F0-4157-9AFA-FC3ACC442351
Author: hyukjinkwon <[email protected]>
Closes apache#15867 from HyukjinKwon/SPARK-18423.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Several tests are being failed on Windows due to the failure of removing the checkpoint dir between each tests.
This is caused by not closed file in
ReceiverTracker. When it is not started, it does not close it even ifstop()is called.How was this patch tested?
Tests in
JavaAPISuiteandMapWithStateSuite.Manually tested via AppVeyor:
Before
org.apache.spark.streaming.JavaAPISuiteBuild: https://ci.appveyor.com/project/spark-test/spark/build/71-MapWithStateSuite-1
Diff: master...spark-test:188c828e682ec45b75d15c3dfc782bcdc8ce024c
org.apache.spark.streaming.MapWithStateSuiteBuild: https://ci.appveyor.com/project/spark-test/spark/build/72-MapWithStateSuite-1
Diff: master...spark-test:8f6945d0ccde022a23d3848f6b7fe6da1e7c902e
After
org.apache.spark.streaming.JavaAPISuiteBuild started: [Streaming]
org.apache.spark.streaming.JavaAPISuiteDiff: master...spark-test:3D74F2D5-B0D5-4E1D-874C-685AE694FD37
org.apache.spark.streaming.MapWithStateSuiteBuild started: [Streaming]
org.apache.spark.streaming.MapWithStateSuiteDiff: master...spark-test:C8E88B64-49F0-4157-9AFA-FC3ACC442351