`test_ssl_in_multiple_threads` is flaky due to `with self.subTest` with threads

[colesbury]

Bug report

It's not safe to call with self.subTest from multiple threads concurrently. (See #131677)

======================================================================
ERROR: test_ssl_in_multiple_threads (test.test_ssl.ThreadedTests.test_ssl_in_multiple_threads)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/test_ssl.py", line 3036, in test_ssl_in_multiple_threads
    raise cm.exc_value
  File "/Users/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/threading.py", line 1075, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/threading.py", line 1017, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/test/test_ssl.py", line 2968, in test_echo
    with self.subTest(client=ssl.PROTOCOL_TLS_SERVER, server=ssl.PROTOCOL_TLS_CLIENT):
         ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/contextlib.py", line 148, in __exit__
    next(self.gen)
    ~~~~^^^^^^^^^^
  File "/Users/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/case.py", line 562, in subTest
    with self._outcome.testPartExecutor(self._subtest, subTest=True):
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/contextlib.py", line 148, in __exit__
    next(self.gen)
    ~~~~^^^^^^^^^^
  File "/Users/buildbot/buildarea/pull_request.itamaro-macos-arm64-aws.macos-with-brew.refleak.nogil/build/Lib/unittest/case.py", line 81, in testPartExecutor
    self.result.addSubTest(test_case.test_case, test_case, None)
                           ^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'test_case'

From https://buildbot.python.org/?#/builders/1367/builds/987/

cc @ZeroIntensity

Linked PRs

• gh-143443: Try to fix test_ssl_in_multiple_threads #143510

[ZeroIntensity]

I don't see any easy fix, so my only idea right now is to write dedicated stress tests there rather than just calling existing test functions concurrently. That might take me a little bit; did you want to skip that test in the meantime?

[Yashp002]

hey @ZeroIntensity, if you need a hand writing those dedicated stress tests, I'm happy to help. just lemme know what kind of coverage you're aiming for

[colesbury]

Alternatively, we could try to make with self.subTest thread-safe, but that could be tricky to do in a backwards compatible way.

[serhiy-storchaka]

Good point. There was other similar issue in this test: #126483.

I think that the simplest quick fix would be monkey-patching subTest in this test. For example, setting it temporary to contextlib.nullcontext.

BTW, subTest is misused in this test. I suggest to use thread_helper.start_threads() to run multiple threads.

[ZeroIntensity]

The subTest usage directly in test_ssl_in_multiple_threads isn't the problem; the issue is that the test runs other tests concurrently, and those tests use subTest.

[serhiy-storchaka]

I will try to fix the existing test, but writing dedicated stress tests is a better long term solution.

[serhiy-storchaka]

Another problem is that the test needs a large amount of address space (or memory), at least 12GB. The following command (only 11GB) hangs:

(ulimit -v 11000000; ./python -m test -vuall test_ssl -m test_ssl_in_multiple_threads; )

This may be the main cause of flakiness. I was not able to reproduce issue with unpatched test without limiting the address space.

[serhiy-storchaka]

Even if specify enough memory for one run, repeated runs of the test will finally hang. There is a leak.

(ulimit -v 12000000; ./python -m test -vuall test_ssl -m test_ssl_in_multiple_threads -F; )