Skip to content

[SPARK-28903][STREAMING][PYSPARK][TESTS] Fix AWS JDK version conflict that breaks Pyspark Kinesis tests #25559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
srowen wants to merge 12 commits into from
Closed

[SPARK-28903][STREAMING][PYSPARK][TESTS] Fix AWS JDK version conflict that breaks Pyspark Kinesis tests #25559

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b283338
Test updating Kinesis deps and current state of Kinesis Python tests
srowen Aug 22, 2019
fdceda1
Undo Kinesis version change
srowen Aug 23, 2019
3cc5f64
More debugging
srowen Aug 26, 2019
0fbfdf7
Oops
srowen Aug 26, 2019
d8d0e59
Try without pypy
srowen Aug 26, 2019
361c90d
A few more guesses
srowen Aug 26, 2019
6ae413d
Revert many changes and try excluding aws-java-sdk
srowen Aug 26, 2019
e760ebb
Revert default
srowen Aug 26, 2019
101c4ce
Try just dependencyManagement
srowen Aug 27, 2019
78c4e62
Try managing version in dependencies
srowen Aug 27, 2019
24bea1e
Retry excluding the AWS SDK
srowen Aug 28, 2019
36830b4
Remove TODOs
srowen Aug 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import org.apache.spark.streaming.{Duration, StreamingContext}
import org.apache.spark.streaming.api.java.{JavaReceiverInputDStream, JavaStreamingContext}
import org.apache.spark.streaming.dstream.ReceiverInputDStream

// TODO remove this change
object KinesisUtils {
/**
* Create an input stream that pulls messages from a Kinesis stream.
Expand Down
6 changes: 6 additions & 0 deletions hadoop-cloud/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,12 @@
</exclusion>
</exclusions>
</dependency>
<!-- Manage the AWS SDK version to match Kinesis et al -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>${aws.java.sdk.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-openstack</artifactId>
Expand Down
1 change: 1 addition & 0 deletions python/pyspark/streaming/tests/test_kinesis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
PySparkStreamingTestCase


# TODO remove this change
@unittest.skipIf(not should_test_kinesis, kinesis_requirement_message)
class KinesisStreamTests(PySparkStreamingTestCase):

Expand Down