CDLL does not use same paths as util.find_library

[shjohnsonpi]

BPO	18502
Nosy	@meadori, @vadmium

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-07-18.22:11:22.612>
labels = ['ctypes', 'type-bug']
title = 'CDLL does not use same paths as util.find_library'
updated_at = <Date 2016-04-25.01:53:50.735>
user = 'https://bugs.python.org/shjohnsonpi'

bugs.python.org fields:

activity = <Date 2016-04-25.01:53:50.735>
actor = 'martin.panter'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2013-07-18.22:11:22.612>
creator = 'shjohnson.pi'
dependencies = []
files = []
hgrepos = []
issue_num = 18502
keywords = []
message_count = 4.0
messages = ['193332', '193577', '193584', '264143']
nosy_count = 3.0
nosy_names = ['meador.inge', 'martin.panter', 'shjohnson.pi']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18502'
versions = ['Python 3.4']

[shjohnsonpi]

CDLL does not use the same paths as find_library and thus you can *find* a library, but you can't necessarily use it.

In my case, I had SDL2 in /usr/local/lib. find_library correctly gets
the name, but does not return the path. CDLL apparently does not search
/usr/local/lib and fails.

Python 3.4.0a0 (default:5a6cdc0d7de1, Jul 18 2013, 17:55:27) 
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> import ctypes.util
>>> ctypes.util.find_library("SDL2")
'libSDL2-2.0.so.0'
>>> CDLL(_)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/steven/Programming/cpython/Lib/ctypes/__init__.py", line 351, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libSDL2-2.0.so.0: cannot open shared object file: No such file or directory
>>> CDLL("/usr/local/lib/libSDL2.so")
<CDLL '/usr/local/lib/libSDL2.so', handle 22f89c0 at 7fc966da5ae8>

[meadori]

On Linux gcc and ld are used to implement 'find_library' and 'dlopen' is used to implement 'CDLL'. ld searches /usr/local/lib. 'dlopen' might not if the LD_LIBRARY_PATH isn't set up to do so. For example:

[meadori@li589-207 cpython]$ ls /usr/local/lib
libfoo.so  libfoo.so.1  libfoo.so.1.0  linode
[meadori@li589-207 cpython]$ ./python 
Python 3.4.0a0 (default:d6213012d87b, Jul 23 2013, 01:00:24) 
[GCC 4.7.2 20120921 (Red Hat 4.7.2-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> import ctypes.util
>>> ctypes.util.find_library('foo')
'libfoo.so.1'
>>> CDLL(_)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/meadori/src/cpython/Lib/ctypes/__init__.py", line 351, in __init__
    self._handle = _dlopen(self._name, mode)OSError: libfoo.so.1: cannot open shared object file: No such file or directory
>>> quit()
[meadori@li589-207 cpython]$ echo $LD_LIBRARY_PATH

[meadori@li589-207 cpython]$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
[meadori@li589-207 cpython]$ ./python 
Python 3.4.0a0 (default:d6213012d87b, Jul 23 2013, 01:00:24) 
[GCC 4.7.2 20120921 (Red Hat 4.7.2-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> import ctypes.util
>>> ctypes.util.find_library('foo')
'libfoo.so.1'
>>> CDLL(_)
<CDLL 'libfoo.so.1', handle 28e4b50 at 7f31e062b608>

I am OK with this behavior in ctypes since it matches the system behavior with 'gcc' and 'dlopen'. So, I would fixup LD_LIBRARY_PATH or /etc/ld.so.conf if you want /usr/local/lib to be searched at load time.

[shjohnsonpi]

Should the docs then be changed for find_library to inform that "It is not guaranteed that a found library can be opened" (at least for linux)?

Or would a feature request for full paths from find_library for linux be more appropriate?

It seems hackish to me, if a program can find a library, but can't use it without asking the user to change some environment variables or testing /usr/local/lib.

[vadmium]

Documenting this quirk seems reasonable to me if you want to suggest a patch (and we don’t agree to fix it).

See bpo-21042 for returning full paths on Linux. We tried to make this change, but it wasn’t easy to do it consistently for arbitrary ABIs, so I backed it out.

See bpo-9998 discussing changing find_library() to emulate a run-time linker, rather than a compile-time linker.