Skip to content

Commit 07ff7ea

Browse files
committed
PYTHON-961 - Add warning to prevent deadlock during getaddrinfo.
1 parent 6b64405 commit 07ff7ea

File tree

3 files changed

+40
-0
lines changed

3 files changed

+40
-0
lines changed

doc/faq.rst

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -375,3 +375,20 @@ just that field::
375375

376376
>>> cur = coll.find({}, projection={'dt': False})
377377

378+
.. _multiprocessing:
379+
380+
Using PyMongo with Multiprocessing
381+
----------------------------------
382+
There are a few things to be aware of when using multiprocessing with PyMongo.
383+
On certain platforms (`defined here <https://hg.python.org/cpython/file/d2b8354e87f5/Modules/socketmodule.c#l187>`_)
384+
:class:`~pymongo.mongo_client.MongoClient` MUST be initialized with ``connect=False`` if a :class:`~pymongo.mongo_client.MongoClient` used in a
385+
child process is initialized before forking. If ``connect`` cannot be False,
386+
then :class:`~pymongo.mongo_client.MongoClient` must be initialized AFTER forking.
387+
388+
This is because CPython must acquire a lock before calling
389+
`getaddrinfo() <https://hg.python.org/cpython/file/d2b8354e87f5/Modules/socketmodule.c#l4203>`_.
390+
A deadlock will occur if the :class:`~pymongo.mongo_client.MongoClient`'s parent process forks (on the main
391+
thread) while its monitor thread is in the getaddrinfo() system call.
392+
393+
PyMongo will issue a warning if there is a chance of this deadlock occurring.
394+

pymongo/mongo_client.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,9 @@ def __init__(
9898
passwords reserved characters like ':', '/', '+' and '@' must be
9999
escaped following RFC 2396.
100100
101+
.. warning:: When using PyMongo in a multiprocessing context, please
102+
read :ref:`multiprocessing` first.
103+
101104
:Parameters:
102105
- `host` (optional): hostname or IP address of the
103106
instance to connect to, or a mongodb URI, or a list of

pymongo/topology.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,10 @@
1414

1515
"""Internal class to monitor a topology of one or more servers."""
1616

17+
import os
1718
import random
1819
import threading
20+
import warnings
1921

2022
from bson.py3compat import itervalues
2123
from pymongo import common
@@ -50,13 +52,31 @@ def __init__(self, topology_settings):
5052
self._lock = threading.Lock()
5153
self._condition = self._settings.condition_class(self._lock)
5254
self._servers = {}
55+
self._pid = None
5356

5457
def open(self):
5558
"""Start monitoring, or restart after a fork.
5659
5760
No effect if called multiple times.
61+
62+
.. warning:: To avoid a deadlock during Python's getaddrinfo call,
63+
will generate a warning if open() is called from a different
64+
process than the one that initialized the Topology. To prevent this
65+
from happening, MongoClient must be created after any forking OR
66+
MongoClient must be started with connect=False.
5867
"""
5968
with self._lock:
69+
if self._pid is None:
70+
self._pid = os.getpid()
71+
else:
72+
if os.getpid() != self._pid:
73+
warnings.warn(
74+
"MongoClient opened before fork. Create MongoClient "
75+
"with connect=False, or create client after forking. "
76+
"See PyMongo's documentation for details: http://api."
77+
"mongodb.org/python/current/faq.html#using-pymongo-"
78+
"with-multiprocessing>")
79+
6080
self._ensure_opened()
6181

6282
def select_servers(self,

0 commit comments

Comments
 (0)