Fix ROOT-10484 by implementing dyld support for RPATH. #6969

vgvassilev · 2020-12-19T22:22:33Z

Some libraries are layered can depend on other libraries on a private paths. That is, libA can depend on libB which is neither on the LD_LIBRARY_PATH nor on a known system path. The posix linker injects "variables" such as @rpath which is expanded at link time to resolve the libraries on a relative path.

Prior to this patch, cling's Dyld-based symbol resolution could not trace down such cases causing failures in symbol resolution when a symbol is only defined in libB (a private library).

This patch implements the basic posix linker substitutions allowing cling's Dyld-based symbol resolution implementation to follow more closely the linker rules.

Performance

platform	test	master	PRv1	PRv2	mem master	mem PRv2
ubu20	roottest-python-pythonizations	6.5	6.55	6.65	342076	347168
ubu20	roottest-cling	112.8	114.53	181.26	411048	416036
ubu20	roottest-python	148.5	136.26	144.72	394292	397120
ubu20	roottest-root-hist	16.78	17.35	15.45	465156	235084
ubu20	roottest-root-math	28.28	27.89	27.40	424072	371256
ubu20	roottest-root-tree	462.84	510.59	465.79	796512	797992
ubu20	roottest-root-treeformula	117.53	129.16	114.44	261184	260168
ubu20	root-io-stdarray	66.90	67.89	65.71	263424	255120
ubu20	roottest-root-treeformula	117.55	129.10
osx 10.15	roottest-python-pythonizations	17.25	16.06
osx 10.15	roottest-cling	346.06	343.46
osx 10.15	roottest-root-hist	36.06	36.22
osx 10.15	roottest-root-math	49.78	52.42
osx 10.15	root-io-stdarray	119.32	118.73

That PR should reduce the number of stat calls improving #7774

Kudos Alexander Penev (@alexander-penev).

phsft-bot · 2020-12-19T22:22:43Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-12-19T23:28:26Z

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.roottest.python.JupyROOT.roottest_python_JupyROOT_tpython_notebook

vgvassilev · 2020-12-20T07:27:54Z

@phsft-bot build with flags -DCTEST_TEST_EXCLUDE_NONE=On

phsft-bot · 2020-12-20T07:28:01Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14 with flags -DCTEST_TEST_EXCLUDE_NONE=On
How to customize builds

phsft-bot · 2020-12-20T08:30:09Z

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.roottest.python.JupyROOT.roottest_python_JupyROOT_tpython_notebook

phsft-bot · 2020-12-21T14:52:29Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2020-12-21T15:56:05Z

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.roottest.python.JupyROOT.roottest_python_JupyROOT_tpython_notebook

oshadura

@vgvassilev wow! changes are so minimal, we always have this feature and never tested it? I afraid I am not sure if CI has R enabled (I see -Dr=Off is everywhere)

vgvassilev · 2021-01-05T11:15:29Z

The intent was that we only wanted to see what was failing. We are working on a bigger patch which will come in this PR.

vgvassilev · 2021-03-28T06:33:12Z

@phsft-bot build!

phsft-bot · 2021-03-28T06:33:20Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-04-01T19:54:53Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-04-01T19:59:45Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-04-01T20:18:17Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-04-01T20:43:44Z

Build failed on ROOT-fedora30/cxx14.
Running on root-fedora30-2.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.roottest.root.dataframe.roottest_root_dataframe_test_stringfiltercolumn

phsft-bot · 2021-04-02T06:48:37Z

Build failed on ROOT-debian10-i386/cxx14.
See console output.

knowall-bot

Please apply the code formatting changes without bloating the history:
Run:

git checkout fix-root10484"
wget https://raw.githubusercontent.com/llvm/llvm-project/main/clang/tools/clang-format/git-clang-format
chmod +x git-clang-format
git rebase -i -x "git-clang-format master && git commit -a --allow-empty --fixup=HEAD" --strategy-option=theirs origin/master
git log --oneline # to inspect the results
git rebase --autosquash -i master # squash without poluting the history

bindings/r/inc/RExports.h

interpreter/cling/lib/Interpreter/DynamicLibraryManager.cpp

interpreter/cling/lib/Interpreter/DynamicLibraryManagerSymbol.cpp

phsft-bot · 2021-04-09T20:13:05Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-04-09T20:16:51Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-fedora30/cxx14, ROOT-fedora31/noimt, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

@rpath

Some libraries are layered can depend on other libraries on a private paths. That is, libA can depend on libB which is neither on the LD_LIBRARY_PATH nor on a known system path. The posix linker injects "variables" such as @rpath which is expanded at link time to resolve the libraries on a relative path. Prior to this patch, cling's Dyld-based symbol resolution could not trace down such cases causing failures in symbol resolution when a symbol is only defined in libB (a private library). This patch implements the basic posix linker substitutions allowing cling's Dyld-based symbol resolution implementation to follow more closely the linker rules. Kudos Alexander Penev (@alexander-penev).

phsft-bot · 2021-08-05T19:56:34Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-08-05T20:29:19Z

Build failed on ROOT-ubuntu16/nortcxxmod.
Running on sft-ubuntu-1604-1.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Warnings:

[2021-08-05T19:59:31.071Z] /mnt/build/workspace/root-pullrequests-build/root/interpreter/cling/lib/Interpreter/DynamicLibraryManagerSymbol.cpp:281:45: warning: ignoring return value of ‘ssize_t readlink(const char*, char*, size_t)’, declared with attribute warn_unused_result [-Wunused-result]

phsft-bot · 2021-08-05T20:36:25Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

[2021-08-05T19:58:33.339Z] stderr: error: could not read '.git/rebase-apply/head-name': No such file or directory

Failing tests:

projectroot.roottest.python.cpp.roottest_python_cpp_cpp

phsft-bot · 2021-08-06T20:01:32Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-08-06T20:08:30Z

Build failed on ROOT-ubuntu16/nortcxxmod.
Running on sft-ubuntu-1604-1.cern.ch:/build/workspace/root-pullrequests-build
See console output.

phsft-bot · 2021-08-06T20:15:24Z

Build failed on ROOT-performance-centos8-multicore/default.
Running on olbdw-01.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See console output.

phsft-bot · 2021-08-06T20:21:37Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

[2021-08-06T20:02:38.872Z] stderr: error: could not read '.git/rebase-apply/head-name': No such file or directory

phsft-bot · 2021-08-06T20:30:48Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-08-06T20:34:30Z

Build failed on ROOT-ubuntu16/nortcxxmod.
Running on sft-ubuntu-1604-1.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

[2021-08-06T20:34:28.749Z] FAILED: /usr/bin/ccache /usr/bin/c++ -DCLING_VERSION="ROOT_1.0~dev" -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -Iinterpreter/cling/lib/Interpreter -I/mnt/build/workspace/root-pullrequests-build/root/interpreter/cling/lib/Interpreter -I/mnt/build/workspace/root-pullrequests-build/root/interpreter/cling/include -Iinterpreter/cling/include -isystem interpreter/llvm/src/tools/clang/include -isystem /mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/tools/clang/include -isystem interpreter/llvm/src/include -isystem /mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include -fdiagnostics-color=always -std=c++14 -pipe -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -fno-semantic-interposition -D_GNU_SOURCE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -fno-strict-aliasing -Wwrite-strings -Wno-shadow -Wno-unused-parameter -Wno-deprecated-declarations -fPIC -fno-rtti -O3 -fvisibility=hidden -fvisibility-inlines-hidden -std=c++14 -MD -MT interpreter/cling/lib/Interpreter/CMakeFiles/obj.clingInterpreter.dir/DynamicLibraryManagerSymbol.cpp.o -MF interpreter/cling/lib/Interpreter/CMakeFiles/obj.clingInterpreter.dir/DynamicLibraryManagerSymbol.cpp.o.d -o interpreter/cling/lib/Interpreter/CMakeFiles/obj.clingInterpreter.dir/DynamicLibraryManagerSymbol.cpp.o -c /mnt/build/workspace/root-pullrequests-build/root/interpreter/cling/lib/Interpreter/DynamicLibraryManagerSymbol.cpp
[2021-08-06T20:34:28.749Z] /mnt/build/workspace/root-pullrequests-build/root/interpreter/cling/lib/Interpreter/DynamicLibraryManagerSymbol.cpp:281:3: error: ‘ssizet_t’ was not declared in this scope
[2021-08-06T20:34:28.749Z] /mnt/build/workspace/root-pullrequests-build/root/interpreter/cling/lib/Interpreter/DynamicLibraryManagerSymbol.cpp:282:8: error: ‘len’ was not declared in this scope

phsft-bot · 2021-08-06T20:35:45Z

Build failed on ROOT-performance-centos8-multicore/default.
Running on olbdw-01.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See console output.

Errors:

[2021-08-06T20:35:25.039Z] /data/sftnight/workspace/root-pullrequests-build/root/interpreter/cling/lib/Interpreter/DynamicLibraryManagerSymbol.cpp:281:3: error: ‘ssizet_t’ was not declared in this scope
[2021-08-06T20:35:25.039Z] /data/sftnight/workspace/root-pullrequests-build/root/interpreter/cling/lib/Interpreter/DynamicLibraryManagerSymbol.cpp:282:8: error: ‘len’ was not declared in this scope

phsft-bot · 2021-08-06T20:39:35Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-08-06T21:18:05Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

[2021-08-06T20:40:03.506Z] stderr: error: could not read '.git/rebase-apply/head-name': No such file or directory

PR root-project#6969 allowed ROOT to build pcm files for the R package via the RInterface.pcm. However, various headers of R define macros which are defined by math.h and others (ERROR in RooFit). When the global module index is enabled, ROOT loads RInterface on a lookup for (Range). Then the exported macros "shadow" the ones which are still in the index and we do not load the correct corresponding module. This obsure problem seems to be coming from a current deficiency in the implementation/interaction between clang and the index. Pre-loading the RInterface, if enabled, seems to resolve the issue at a reasoanble compromise rather than trying to disable the module altoghether or undefine the fragile macros. as the global module index is not smart enough to figure it needs to offer

PR root-project#6969 allowed ROOT to build pcm files for the R package via the RInterface.pcm. However, various headers of R define macros which are defined by math.h and others (ERROR in RooFit). When the global module index is enabled, ROOT loads RInterface on a lookup for (Range). Then the exported macros "shadow" the ones which are still in the index and we do not load the correct corresponding module. This obscure problem seems to be coming from a current deficiency in the implementation/interaction between clang and the index. Pre-loading the RInterface, if enabled, seems to resolve the issue at a reasonable compromise rather than trying to disable the module altogether or undefine the fragile macros. as the global module index is not smart enough to figure it needs to offer

PR root-project#6969 allowed ROOT to build pcm files for the R package via the RInterface.pcm. However, various headers of R define macros which are defined by math.h and others (ERROR in RooFit). When the global module index is enabled, ROOT loads RInterface on a lookup for (Range). Then the exported macros "shadow" the ones which are still in the index and we do not load the correct corresponding module. This obscure problem seems to be coming from a current deficiency in the implementation/interaction between clang and the index. Pre-loading the RInterface, if enabled, seems to resolve the issue at a reasonable compromise rather than trying to disable the module altogether or undefine the fragile macros.

PR #6969 allowed ROOT to build pcm files for the R package via the RInterface.pcm. However, various headers of R define macros which are defined by math.h and others (ERROR in RooFit). When the global module index is enabled, ROOT loads RInterface on a lookup for (Range). Then the exported macros "shadow" the ones which are still in the index and we do not load the correct corresponding module. This obscure problem seems to be coming from a current deficiency in the implementation/interaction between clang and the index. Pre-loading the RInterface, if enabled, seems to resolve the issue at a reasonable compromise rather than trying to disable the module altogether or undefine the fragile macros.

PR root-project#6969 allowed ROOT to build pcm files for the R package via the RInterface.pcm. However, various headers of R define macros which are defined by math.h and others (ERROR in RooFit). When the global module index is enabled, ROOT loads RInterface on a lookup for (Range). Then the exported macros "shadow" the ones which are still in the index and we do not load the correct corresponding module. This obscure problem seems to be coming from a current deficiency in the implementation/interaction between clang and the index. Pre-loading the RInterface, if enabled, seems to resolve the issue at a reasonable compromise rather than trying to disable the module altogether or undefine the fragile macros.

vgvassilev requested a review from oshadura as a code owner December 19, 2020 22:22

vgvassilev marked this pull request as draft December 19, 2020 22:22

bellenot assigned vgvassilev Dec 21, 2020

oshadura reviewed Jan 5, 2021

View reviewed changes

vgvassilev force-pushed the fix-root10484 branch from 301c273 to f10cb97 Compare April 1, 2021 19:54

vgvassilev force-pushed the fix-root10484 branch from f10cb97 to d8cc34b Compare April 1, 2021 19:59

vgvassilev force-pushed the fix-root10484 branch from d8cc34b to badb317 Compare April 1, 2021 20:18

knowall-bot reviewed Apr 8, 2021

View reviewed changes

vgvassilev force-pushed the fix-root10484 branch from badb317 to d136c79 Compare April 9, 2021 20:12

vgvassilev force-pushed the fix-root10484 branch from d136c79 to 6872abc Compare April 9, 2021 20:16

vgvassilev marked this pull request as ready for review April 9, 2021 20:21

vgvassilev requested review from Axel-Naumann and omazapa as code owners April 9, 2021 20:21

vgvassilev added 2 commits August 5, 2021 19:56

Ask for the real path in any case.

6b52af7

vgvassilev force-pushed the fix-root10484 branch from 859a822 to 6b52af7 Compare August 5, 2021 19:56

vgvassilev force-pushed the fix-root10484 branch from d911414 to 673f76f Compare August 6, 2021 20:30

Silence warning ignoring return value of readlink.

ca11bbe

vgvassilev force-pushed the fix-root10484 branch from 673f76f to ca11bbe Compare August 6, 2021 20:39

vgvassilev merged commit 68c7ac9 into root-project:master Aug 7, 2021

vgvassilev deleted the fix-root10484 branch August 7, 2021 07:18

vgvassilev mentioned this pull request Aug 10, 2021

[urgent] [cxxmodules] Exclude the RInterface from the global module index. #8827

Merged

vgvassilev mentioned this pull request Aug 16, 2021

[dyld] Do iterate subfolders and other non-regular files. #8118

Closed

Axel-Naumann mentioned this pull request Oct 1, 2021

Unreasonably slow behaviour of CompileMacro #7774

Closed

1 task

Fix ROOT-10484 by implementing dyld support for RPATH. #6969

Fix ROOT-10484 by implementing dyld support for RPATH. #6969

Uh oh!

Conversation

vgvassilev commented Dec 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance

Uh oh!

phsft-bot commented Dec 19, 2020

Uh oh!

phsft-bot commented Dec 19, 2020

Failing tests:

Uh oh!

vgvassilev commented Dec 20, 2020

Uh oh!

phsft-bot commented Dec 20, 2020

Uh oh!

phsft-bot commented Dec 20, 2020

Failing tests:

Uh oh!

phsft-bot commented Dec 21, 2020

Uh oh!

phsft-bot commented Dec 21, 2020

Failing tests:

Uh oh!

oshadura left a comment

Choose a reason for hiding this comment

Uh oh!

vgvassilev commented Jan 5, 2021

Uh oh!

vgvassilev commented Mar 28, 2021

Uh oh!

phsft-bot commented Mar 28, 2021

Uh oh!

phsft-bot commented Apr 1, 2021

Uh oh!

phsft-bot commented Apr 1, 2021

Uh oh!

phsft-bot commented Apr 1, 2021

Uh oh!

phsft-bot commented Apr 1, 2021

Failing tests:

Uh oh!

phsft-bot commented Apr 2, 2021

Uh oh!

knowall-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phsft-bot commented Apr 9, 2021

Uh oh!

phsft-bot commented Apr 9, 2021

Uh oh!

phsft-bot commented Aug 5, 2021

Uh oh!

phsft-bot commented Aug 5, 2021

Warnings:

Uh oh!

phsft-bot commented Aug 5, 2021

Errors:

Failing tests:

Uh oh!

phsft-bot commented Aug 6, 2021

Uh oh!

phsft-bot commented Aug 6, 2021

Uh oh!

phsft-bot commented Aug 6, 2021

Uh oh!

phsft-bot commented Aug 6, 2021

Errors:

Uh oh!

vgvassilev commented Dec 19, 2020 •

edited

Loading