Skip to content

Conversation

@JoshRosen
Copy link
Contributor

PySpark SQL's readerwriter and window doctests weren't being run by our test runner script; this patch re-enables them.

@JoshRosen JoshRosen changed the title [MINOR] Enable PySpark SQL readerwriter and window tests. [MINOR] Enable PySpark SQL readerwriter and window tests May 31, 2015
@rxin
Copy link
Contributor

rxin commented May 31, 2015

I wonder if we should create a simple linter rule in lint-python to check for stuff like this ...

@rxin
Copy link
Contributor

rxin commented May 31, 2015

or -- even better, should run-tests just run every python file?

@JoshRosen
Copy link
Contributor Author

In principle, we should probably be using a proper testrunner like nose to handle test discovery and execution. The reason why we didn't do this initially is because we need some custom code in __main__ to emulate a shared SparkContext fixture for the doctests, since putting SparkContext setup and teardown code into each doctest would be very messy and slow.

In the medium-term, we're going to want to refactor run-tests anyways in order to make it easier to run subsets of the tests and python versions (see #4269, for example).

@JoshRosen
Copy link
Contributor Author

Integrating my coverage reporting harness into the Jenkins builds would also help to catch this problem, since that makes it really obvious when test code isn't run.

@SparkQA
Copy link

SparkQA commented May 31, 2015

Test build #33864 has finished for PR 6542 at commit 9f46ce4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin
Copy link
Contributor

rxin commented May 31, 2015

Merging in master & branch-1.4.

@asfgit asfgit closed this in 9126ea4 May 31, 2015
asfgit pushed a commit that referenced this pull request Jun 2, 2015
PySpark SQL's `readerwriter` and `window` doctests weren't being run by our test runner script; this patch re-enables them.

Author: Josh Rosen <[email protected]>

Closes #6542 from JoshRosen/enable-more-pyspark-sql-tests and squashes the following commits:

9f46ce4 [Josh Rosen] Enable PySpark SQL readerwriter and window tests.
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
PySpark SQL's `readerwriter` and `window` doctests weren't being run by our test runner script; this patch re-enables them.

Author: Josh Rosen <[email protected]>

Closes apache#6542 from JoshRosen/enable-more-pyspark-sql-tests and squashes the following commits:

9f46ce4 [Josh Rosen] Enable PySpark SQL readerwriter and window tests.
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
PySpark SQL's `readerwriter` and `window` doctests weren't being run by our test runner script; this patch re-enables them.

Author: Josh Rosen <[email protected]>

Closes apache#6542 from JoshRosen/enable-more-pyspark-sql-tests and squashes the following commits:

9f46ce4 [Josh Rosen] Enable PySpark SQL readerwriter and window tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants