Skip to content

[SPARK-23695][PYTHON] Fix the error message for Kinesis streaming tests #20834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 1 commit into from
Closed

[SPARK-23695][PYTHON] Fix the error message for Kinesis streaming tests #20834

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Fix the error message for Kinesis streaming tests
  • Loading branch information
@HyukjinKwon
HyukjinKwon committed Mar 15, 2018
commit d2fe1e9a3dfdba9958b9ffa893fd903e902b4033
11 changes: 7 additions & 4 deletions python/pyspark/streaming/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1503,10 +1503,13 @@ def search_flume_assembly_jar():
return jars[0]


def search_kinesis_asl_assembly_jar():
def _kinesis_asl_assembly_dir():
SPARK_HOME = os.environ["SPARK_HOME"]
kinesis_asl_assembly_dir = os.path.join(SPARK_HOME, "external/kinesis-asl-assembly")
jars = search_jar(kinesis_asl_assembly_dir, "spark-streaming-kinesis-asl-assembly")
return os.path.join(SPARK_HOME, "external/kinesis-asl-assembly")


def search_kinesis_asl_assembly_jar():
jars = search_jar(_kinesis_asl_assembly_dir(), "spark-streaming-kinesis-asl-assembly")
if not jars:
return None
elif len(jars) > 1:
Expand Down Expand Up @@ -1569,7 +1572,7 @@ def search_kinesis_asl_assembly_jar():
else:
raise Exception(
("Failed to find Spark Streaming Kinesis assembly jar in %s. "
% kinesis_asl_assembly_dir) +
% _kinesis_asl_assembly_dir()) +
"You need to build Spark with 'build/sbt -Pkinesis-asl "
"assembly/package streaming-kinesis-asl-assembly/assembly'"
"or 'build/mvn -Pkinesis-asl package' before running this test.")
Expand Down