Skip to content

Commit f54dd17

Browse files
committed
[SPARK-37252][PYTHON][TESTS] Ignore test_memory_limit on non-Linux environment
### What changes were proposed in this pull request? This PR aims to ignore `test_memory_limit` on non-Linux environment. ### Why are the changes needed? Like the documentation apache#23664, it fails on non-Linux environment like the following MacOS example. **BEFORE** ``` $ build/sbt -Phadoop-cloud -Phadoop-3.2 test:package $ python/run-tests --modules pyspark-core ... ====================================================================== FAIL: test_memory_limit (pyspark.tests.test_worker.WorkerMemoryTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dongjoon/APACHE/spark-merge/python/pyspark/tests/test_worker.py", line 212, in test_memory_limit self.assertEqual(soft_limit, 2 * 1024 * 1024 * 1024) AssertionError: 9223372036854775807 != 2147483648 ---------------------------------------------------------------------- ``` **AFTER** ``` ... Tests passed in 104 seconds Skipped tests in pyspark.tests.test_serializers with /Users/dongjoon/.pyenv/versions/3.8.12/bin/python3: test_serialize (pyspark.tests.test_serializers.SciPyTests) ... skipped 'SciPy not installed' Skipped tests in pyspark.tests.test_worker with /Users/dongjoon/.pyenv/versions/3.8.12/bin/python3: test_memory_limit (pyspark.tests.test_worker.WorkerMemoryTest) ... skipped "Memory limit feature in Python worker is dependent on Python's 'resource' module on Linux; however, not found or not on Linux." ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual. Closes apache#34527 from dongjoon-hyun/SPARK-37252. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 2c7f201) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 6eff896 commit f54dd17

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

python/pyspark/tests/test_worker.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
# limitations under the License.
1717
#
1818
import os
19+
import sys
1920
import tempfile
2021
import threading
2122
import time
@@ -188,9 +189,9 @@ def test_reuse_worker_of_parallelize_range(self):
188189

189190

190191
@unittest.skipIf(
191-
not has_resource_module,
192+
not has_resource_module or sys.platform != 'linux',
192193
"Memory limit feature in Python worker is dependent on "
193-
"Python's 'resource' module; however, not found.")
194+
"Python's 'resource' module on Linux; however, not found or not on Linux.")
194195
class WorkerMemoryTest(unittest.TestCase):
195196

196197
def setUp(self):

0 commit comments

Comments
 (0)