Skip to content

[SPARK-8422] [BUILD] [PROJECT INFRA] Add a module abstraction to dev/run-tests #6866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 14 commits into from
Closed

[SPARK-8422] [BUILD] [PROJECT INFRA] Add a module abstraction to dev/run-tests #6866

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
f0249bd
WIP
JoshRosen Jun 18, 2015
f53864b
Finish integrating module changes
JoshRosen Jun 18, 2015
37f3fb3
Remove doc profiles option, since it's not actually needed (see #6865)
JoshRosen Jun 18, 2015
3371441
Test everything if nothing has changed (needed for non-PRB builds)
JoshRosen Jun 18, 2015
43a0ced
Minor fixes
JoshRosen Jun 18, 2015
7092d3e
Skip SBT tests if no test goals are specified
JoshRosen Jun 18, 2015
dc6f1c6
Use changed files' extensions to decide whether to run style checks
JoshRosen Jun 19, 2015
3670d50
mllib should depend on streaming
JoshRosen Jun 19, 2015
df10e23
update to reflect fact that no module depends on root
JoshRosen Jun 19, 2015
35a3052
Enable Hive tests when running all tests
JoshRosen Jun 19, 2015
e46539f
Fix camel-cased endswith()
JoshRosen Jun 19, 2015
a86a953
Clean up modules; add new modules for streaming external projects
JoshRosen Jun 19, 2015
4224da5
Add documentation to Module.
JoshRosen Jun 20, 2015
75de450
Use module system to determine which build profiles to enable.
JoshRosen Jun 20, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clean up modules; add new modules for streaming external projects
  • Loading branch information
@JoshRosen
JoshRosen committed Jun 19, 2015
commit a86a953443411aac830b615cbfc06763520a6f2f
106 changes: 93 additions & 13 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,6 @@ def contains_file(self, filename):
source_file_regexes=[
"sql/(?!hive-thriftserver)",
"bin/spark-sql",
"examples/src/main/java/org/apache/spark/examples/sql/",
"examples/src/main/scala/org/apache/spark/examples/sql/",
],
sbt_test_goals=[
"catalyst/test",
Expand Down Expand Up @@ -114,35 +112,108 @@ def contains_file(self, filename):
name="streaming",
dependencies=[],
source_file_regexes=[
"external/",
"extras/java8-tests/",
"extras/kinesis-asl/",
"streaming",
Copy link
Contributor

@andrewor14 andrewor14 Jun 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is missing the example streaming files

Copy link
Contributor Author

@JoshRosen JoshRosen Jun 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these should be covered by the examples subproject instead.

],
sbt_test_goals=[
"streaming/test",
"streaming-flume/test",
"streaming-flume-sink/test",
"streaming-kafka/test",
"streaming-mqtt/test",
"streaming-twitter/test",
]
)


streaming_kinesis_asl = Module(
name="kinesis-asl",
dependencies=[streaming],
source_file_regexes=[
"extras/kinesis-asl/",
],
sbt_test_goals=[
"kinesis-asl/test",
]
)


streaming_zeromq = Module(
name="streaming-zeromq",
dependencies=[streaming],
source_file_regexes=[
"external/zeromq",
],
sbt_test_goals=[
"streaming-zeromq/test",
]
Copy link
Contributor

@davies davies Jun 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should run pyspark test, same to sql and mllib

Copy link
Contributor Author

@JoshRosen JoshRosen Jun 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent was to have this be covered via transitive dependencies: the pyspark module (on line 162) depends on mllib, sql, and streaming, so changes in any of those modules will cause the python tests to be run.

Copy link
Contributor

@davies davies Jun 18, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

O, I see, make sense.

)


streaming_twitter = Module(
name="streaming-twitter",
dependencies=[streaming],
source_file_regexes=[
"external/twitter",
],
sbt_test_goals=[
"streaming-twitter/test",
]
)


streaming_mqqt = Module(
name="streaming-mqqt",
dependencies=[streaming],
source_file_regexes=[
"external/mqqt",
],
sbt_test_goals=[
"streaming-mqqt/test",
]
)


streaming_kafka = Module(
name="streaming-kafka",
dependencies=[streaming],
source_file_regexes=[
"external/kafka",
"external/kafka-assembly",
],
sbt_test_goals=[
"streaming-kafka/test",
]
)


streaming_flume_sink = Module(
name="streaming-flume-sink",
dependencies=[streaming],
source_file_regexes=[
"external/flume-sink",
],
sbt_test_goals=[
"streaming-flume-sink/test",
]
)


streaming_flume = Module(
name="streaming_flume",
dependencies=[streaming],
source_file_regexes=[
"external/flume",
],
sbt_test_goals=[
"streaming-flume/test",
]
)


mllib = Module(
name="mllib",
dependencies=[streaming, sql],
source_file_regexes=[
"examples/src/main/java/org/apache/spark/examples/mllib/",
"examples/src/main/scala/org/apache/spark/examples/mllib",
"data/mllib/",
"mllib/",
],
sbt_test_goals=[
"mllib/test",
"examples/test",
]
)

Expand Down Expand Up @@ -188,6 +259,15 @@ def contains_file(self, filename):
)


ec2 = Module(
name="ec2",
dependencies=[],
source_file_regexes=[
"ec2/",
]
)


def determine_modules_for_files(filenames):
"""
Given a list of filenames, return the set of modules that contain those files.
Expand Down