[tcling] Improve symbol resolution. #4717
Conversation
vgvassilev
force-pushed
the
cxxmodules-modules-bloomfilter
branch
from
fa0e485 to
ff3b307
Compare
|
@phsft-bot build! |
|
Starting build on |
|
Build failed on mac1014/cxx17. Failing tests: |
|
Starting build on |
|
Build failed on mac1014/cxx17. Warnings:
|
vgvassilev
force-pushed
the
cxxmodules-modules-bloomfilter
branch
from
a8fd3c8 to
337dc6c
Compare
|
Starting build on |
|
@phsft-bot build with flags -DCTEST_TEST_EXCLUDE_NONE=On |
|
Starting build on |
vgvassilev
changed the title
|
@phsft-bot build just on mac1014 with flags -Druntime_cxxmodules=On |
|
@phsft-bot build just on mac1014/cxx17 with flags -Druntime_cxxmodules=On |
|
Starting build on |
|
Build failed on mac1014/cxx17. |
vgvassilev
force-pushed
the
cxxmodules-modules-bloomfilter
branch
from
337dc6c to
35f29cd
Compare
|
Starting build on |
|
Starting build on |
|
Starting build on |
|
Build failed on mac1014/cxx17. Errors:
|
vgvassilev
force-pushed
the
cxxmodules-modules-bloomfilter
branch
from
13a7e7e to
b9f5014
Compare
|
Starting build on |
|
Starting build on |
This patch consolidates the symbol resolution facilities throughout TCling into
a new singleton class Dyld part of the cling's DynamicLibraryManager.
The new dyld is responsible for:
* Symlink resolution -- it implements a memory efficient representation of
the full path to shared objects allowing search at constant time O(1). This
also fixes issues when resolving symbols from OSX where the system libraries
contain multiple levels of symlinks.
* Bloom filter optimization -- it uses a stohastic data structure which gives
a definitive answer if a symbol is not in the set. The implementation checks
the .gnu.hash section in ELF which is the GNU implementation of a bloom
filter and uses it. If the symbol is not in the bloom filter, the
implementation builds its own and uses it. The measured performance of the
bloom filter is 30% speed up for 2mb more memory. The custom bloom filter on
top of the .gnu.hash filter gives 1-2% better performance.
The advantage for the custom bloom filter is that it works on all
implementations which do not support .gnu.hash (windows and osx). It is also
customizable if we want to further reduce the false positive rates
(currently at p=2%).
* Hash table optimization -- we build a hash table which contains all symbols
for a given library. This allows us to avoid the fallback symbol iteration
if multiple symbols from the same library are requested. The hash table
optimization targets to optimize the case where the bloom filter tells us
the symbol is *maybe* in the library.
Patch by Alexander Penev and me!
vgvassilev
force-pushed
the
cxxmodules-modules-bloomfilter
branch
from
ca88c9b to
ee856ff
Compare
|
Starting build on |
|
Build failed on ROOT-performance-centos7-multicore/default. Errors:
Failing tests: |
|
@phsft-bot build with flags -DCTEST_TEST_EXCLUDE_NONE=On |
|
Starting build on |
|
Build failed on ROOT-fedora29/python3. Failing tests: |
|
Build failed on mac1015/cxx17. Failing tests: |
Axel-Naumann
left a comment
Axel-Naumann
left a comment
There was a problem hiding this comment.
Please revisit the open points, if needed in a new PR
This is part of root-project#4717
This is part of root-project/root#4717
This addresses the rest of the comments of root-project#4717 Patch by Alexander Penev and me.
This addresses the rest of the comments of #4717 Patch by Alexander Penev and me.
This addresses the rest of the comments of root-project/root#4717 Patch by Alexander Penev and me.
This addresses the rest of the comments of root-project#4717 Patch by Alexander Penev and me.
This is part of root-project#4717
This addresses the rest of the comments of root-project#4717 Patch by Alexander Penev and me.
5 participants
This patch consolidates the symbol resolution facilities throughout TCling into
a new singleton class Dyld part of the cling's DynamicLibraryManager.
The new dyld is responsible for:
the full path to shared objects allowing search at constant time O(1). This
also fixes issues when resolving symbols from OSX where the system libraries
contain multiple levels of symlinks.
a definitive answer if a symbol is not in the set. The implementation checks
the .gnu.hash section in ELF which is the GNU implementation of a bloom
filter and uses it. If the symbol is not in the bloom filter, the
implementation builds its own and uses it. The measured performance of the
bloom filter is 30% speed up for 2mb more memory. The custom bloom filter on
top of the .gnu.hash filter gives 1-2% better performance.
The advantage for the custom bloom filter is that it works on all
implementations which do not support .gnu.hash (windows and osx). It is also
customizable if we want to further reduce the false positive rates
(currently at p=2%).
for a given library. This allows us to avoid the fallback symbol iteration
if multiple symbols from the same library are requested. The hash table
optimization targets to optimize the case where the bloom filter tells us
the symbol is maybe in the library.
Patch by Alexander Penev (@alexander-penev) and me!
Performance Report
The effect of running ctest -j8:
We think in
-j8we lose the advantage of the new PR because the PCH had the rootmaps read in memory and restarting the processes allowed the kernel efficiently reuse that memory. Whereas, the modules and this PR scans the libraries from disk and builds in-memory optimization data structures. Reading from disk seems to be the bottleneck (not verified) but if that's an issue in future we can write out the index making subsequent runs at almost zero cost.