Skip to content

[SPARK-23754][Python] Re-raising StopIteration in client code #21383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
e-dorigatti wants to merge 14 commits into from
Closed

[SPARK-23754][Python] Re-raising StopIteration in client code #21383

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
ec7854a
re-raising StopIteration in client code
e-dorigatti May 21, 2018
fddd031
moved safe_iter to util module and more descriptive name
e-dorigatti May 22, 2018
ee54924
removed redundancy from tests
e-dorigatti May 22, 2018
d739eea
improved doc, error message and code style
e-dorigatti May 24, 2018
f0f80ed
improved tests
e-dorigatti May 24, 2018
d59f0d5
fixed style
e-dorigatti May 24, 2018
b0af18e
fixed udf and its test
e-dorigatti May 24, 2018
167a75b
preserving metadata of wrapped function
e-dorigatti May 24, 2018
90b064d
catching relevant exceptions only
e-dorigatti May 24, 2018
75316af
preserving argspecs of wrapped function
e-dorigatti May 26, 2018
026ecdd
style
e-dorigatti May 26, 2018
f7b53c2
saving argspec in udf
e-dorigatti May 29, 2018
8fac2a8
saving signature only for pandas udf, removed useless try/except
e-dorigatti May 29, 2018
5b5570b
comment explaining hack
e-dorigatti May 30, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
catching relevant exceptions only
  • Loading branch information
@e-dorigatti
e-dorigatti committed May 24, 2018
commit 90b064ddd2db562e90dd55846f1331779e795460
25 changes: 14 additions & 11 deletions python/pyspark/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1284,17 +1284,20 @@ def stopit(*x):

seq_rdd = self.sc.parallelize(range(10))
keyed_rdd = self.sc.parallelize((x % 2, x) for x in range(10))
exc = Py4JJavaError, RuntimeError

self.assertRaises(exc, seq_rdd.map(stopit).collect)
self.assertRaises(exc, seq_rdd.filter(stopit).collect)
self.assertRaises(exc, seq_rdd.cartesian(seq_rdd).flatMap(stopit).collect)
self.assertRaises(exc, seq_rdd.foreach, stopit)
self.assertRaises(exc, keyed_rdd.reduceByKeyLocally, stopit)
self.assertRaises(exc, seq_rdd.reduce, stopit)
self.assertRaises(exc, seq_rdd.fold, 0, stopit)
self.assertRaises(exc, seq_rdd.aggregate, 0, stopit, lambda *x: 1)
self.assertRaises(exc, seq_rdd.aggregate, 0, lambda *x: 1, stopit)

self.assertRaises(Py4JJavaError, seq_rdd.map(stopit).collect)
self.assertRaises(Py4JJavaError, seq_rdd.filter(stopit).collect)
self.assertRaises(Py4JJavaError, seq_rdd.cartesian(seq_rdd).flatMap(stopit).collect)
self.assertRaises(Py4JJavaError, seq_rdd.foreach, stopit)
self.assertRaises(Py4JJavaError, keyed_rdd.reduceByKeyLocally, stopit)
self.assertRaises(Py4JJavaError, seq_rdd.reduce, stopit)
self.assertRaises(Py4JJavaError, seq_rdd.fold, 0, stopit)

# the exception raised is non-deterministic
self.assertRaises((Py4JJavaError, RuntimeError),
seq_rdd.aggregate, 0, stopit, lambda *x: 1)
self.assertRaises((Py4JJavaError, RuntimeError),
seq_rdd.aggregate, 0, lambda *x: 1, stopit)


class ProfilerTests(PySparkTestCase):
Expand Down