[cxxmodules] Preload more modules #1697

vgvassilev · 2018-03-03T16:32:39Z

Rebase PR #1396.

Original PR comment:
"ROOT can't autoparse classes inside namespaces with the rootmap
system (as the loading callbacks don't correctly land where
they are supposed to land with our injected namespaces). As this
turns out to be a feature of some kind, let's preload
TMVA/TreePlayer/Graf to fix all failing tests that are related
to this feature/bug with modules enabled.

This commit can be dropped if we solve on of those problems:

figure out how to fix this bug in the rootmap-based loading
without regressin in performance.

replace the rootmap system with something else like attaching
all C++ modules on startup.

Note that we already do something like this in normal ROOT by
including these packages into the PCH which also makes those
decls available in the normal clang lookup."

phsft-bot · 2018-03-03T16:32:58Z

Starting build on centos7/gcc49, mac1013/native, slc6/gcc49, slc6/gcc62, slc6/gcc62, ubuntu16/native, ubuntu16/native, windows10/vc15 with flags -Dvc=OFF -Dimt=ON -Dccache=ON
How to customize builds

vgvassilev · 2018-03-03T16:34:09Z

@phsft-bot build just on slc6/gcc62 with flags -Druntime_cxxmodules=On -Dctest_test_exclude_none=on

phsft-bot · 2018-03-03T16:34:27Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

phsft-bot · 2018-03-03T16:44:40Z

Build failed on slc6/gcc62.
See console output.

Errors:

../root/core/metacling/src/TCling.cxx:1946:9: error: redeclaration of ‘bool ModuleWasSuccessfullyLoaded’

phsft-bot · 2018-03-03T16:52:15Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

phsft-bot · 2018-03-03T17:11:00Z

Build failed on slc6/gcc62.
See console output.

Errors:

Error in <TStreamerInfo::Build>: TH1, discarding: double* fBuffer, no [dimension]
Error in <TStreamerInfo::Build>: TH1, discarding: double* fIntegral, no [dimension]
Error in <TStreamerInfo::Build>: TArrayF, discarding: float* fArray, no [dimension]
Error in <TStreamerInfo::Build>: TString::Rep_t, unknown type: TString::Rep_t::(anonymous)
Error in <TStreamerInfo::Build>: __shared_ptr<TObjLink>: TObjLink* has no streamer or dictionary, data member _M_ptr will not be saved
Error in <TStreamerInfo::Build>: __weak_ptr<TObjLink,__gnu_cxx::_Lock_policy::_S_atomic>: TObjLink* has no streamer or dictionary, data member _M_ptr will not be saved
Error in <TStreamerInfo::Build>: __weak_count<__gnu_cxx::_Lock_policy::_S_atomic>: _Sp_counted_base<__gnu_cxx::_Lock_policy::_S_atomic>* has no streamer or dictionary, data member _M_pi will not be saved
Error in <TStreamerInfo::Build>: __shared_count<__gnu_cxx::_Lock_policy::_S_atomic>: _Sp_counted_base<__gnu_cxx::_Lock_policy::_S_atomic>* has no streamer or dictionary, data member _M_pi will not be saved
Error in <TStreamerInfo::Build>: TText, unknown type: void* fWcsTitle
Error in <TStreamerInfo::Build>: TLatex: TLatex::FormSize_t* has no streamer or dictionary, data member fTabSize will not be saved

And 23 more

ROOT can't autoparse classes inside namespaces with the rootmap system (as the loading callbacks don't correctly land where they are supposed to land with our injected namespaces). As this turns out to be a feature of some kind, let's preload TMVA/TreePlayer/Graf to fix all failing tests that are related to this feature/bug with modules enabled. This commit can be dropped if we solve on of those problems: 1. figure out how to fix this bug in the rootmap-based loading without regressin in performance. 2. replace the rootmap system with something else like attaching all C++ modules on startup. Note that we already do something like this in normal ROOT by including these packages into the PCH which also makes those decls available in the normal clang lookup.

pcanal · 2018-03-05T20:55:44Z

let's preload TMVA/TreePlayer/Graf to fix all failing tests that are related to this feature/bug with modules enabled.
... This commit can be dropped if we solve on of those problems:

Obviously the removal of this preload is a necessary precursor to making the use of module the default. Do we have a mechanism to 'not' forget to remove it?

vgvassilev · 2018-03-05T21:14:28Z

@pcanal, can you elaborate a little because I cannot understand both sentences ;)

phsft-bot · 2018-03-11T22:02:46Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

phsft-bot · 2018-03-11T22:24:18Z

Build failed on slc6/gcc62.
See console output.

Errors:

Error in <TStreamerInfo::Build>: TBaseClass: BaseClassInfo_t* has no streamer or dictionary, data member fInfo will not be saved
Error in <TBufferFile::CheckByteCount>: object of class TDictionary read too many bytes: 39 instead of 31
Error in <TExMap::Remove>: key 1186048 not found at 478
Error in <TExMap::Remove>: key 3 not found at 3
Error in <TExMap::Remove>: key 3045 not found at 27
Error in <TBufferFile::ReadClassBuffer>: class: TClassRef, attempting to access a wrong version: 4633, object skipped at offset 265
Error in <TExMap::Remove>: key 591700 not found at 173
Error in <TBufferFile::CheckByteCount>: object of class TBaseClass read too few bytes: 67 instead of 73
Error in <TExMap::Remove>: key 1074331651 not found at 107
Error in <TExMap::Remove>: key 156520290 not found at 272

And 352 more

Warnings:

ninja: warning: multiple rules generate tutorials/hsimple.root. builds involving this target will not be correct; continuing anyway [-w dupbuild=warn]

pcanal · 2018-03-12T18:39:47Z

Obviously the removal of this preload is a necessary precursor to making the use of module the
default. Do we have a mechanism to 'not' forget to remove it?

@pcanal, can you elaborate a little because I cannot understand both sentences ;)

@vgvassilev I was apparently mistakenly talking about 'preloading the library' rather than what you seem to have meant ('preloading the module'). Since it is 'only' the module and that we already do this pre-loading of the AST node via the PCH, removing this 'pre-attaching of the AST nodes via preloading the module' is indeed not a pre-requisite to making the use of module the default mechanism. [It is rather a question of optimization and/or implementation of the replacement for the rootmap forward declaration].

vgvassilev · 2018-03-21T15:24:02Z

@phsft-bot build just on slc6/gcc62 with flags -Druntime_cxxmodules=On -Dctest_test_exclude_none=on

phsft-bot · 2018-03-21T15:24:29Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

yamaguchi1024 · 2018-03-29T12:12:03Z

I think we can close this as it's out of date and we have a new patch for loading all the modules. Loading more modules have the negative effect on performance, so we're not going to this direction for now.

vgvassilev · 2018-03-29T12:14:29Z

Which is the new patch you have in mind?

yamaguchi1024 · 2018-03-29T12:19:23Z

https://gist.github.com/yamaguchi1024/56cc7a18638c9f581817e682b729e880
^ This one

vgvassilev · 2018-04-02T13:45:04Z

@phsft-bot build just on slc6/gcc62 with flags -Druntime_cxxmodules=On -Dctest_test_exclude_none=on

phsft-bot · 2018-04-02T13:46:10Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

phsft-bot · 2018-04-02T14:17:37Z

Build failed on slc6/gcc62.
See console output.

Errors:

ERROR: Timeout after 10 minutes
ERROR: Error cloning remote repo 'origin'
ERROR: Error cloning remote repo 'origin'
Error in <TStreamerInfo::Build>: TH1, discarding: double* fBuffer, no [dimension]
Error in <TStreamerInfo::Build>: TH1, discarding: double* fIntegral, no [dimension]
Error in <TStreamerInfo::Build>: TArrayF, discarding: float* fArray, no [dimension]
Error in <TStreamerInfo::Build>: TString::Rep_t, unknown type: TString::Rep_t::(anonymous)
Error in <TStreamerInfo::Build>: __shared_ptr<TObjLink>: TObjLink* has no streamer or dictionary, data member _M_ptr will not be saved
Error in <TStreamerInfo::Build>: __weak_ptr<TObjLink,__gnu_cxx::_Lock_policy::_S_atomic>: TObjLink* has no streamer or dictionary, data member _M_ptr will not be saved
Error in <TStreamerInfo::Build>: __weak_count<__gnu_cxx::_Lock_policy::_S_atomic>: _Sp_counted_base<__gnu_cxx::_Lock_policy::_S_atomic>* has no streamer or dictionary, data member _M_pi will not be saved

And 26 more

Warnings:

../root/core/metacling/src/TCling.cxx:1118:13: warning: ‘void loadModulePath(clang::HeaderSearch&, const char*)’ defined but not used [-Wunused-function]

phsft-bot · 2018-04-02T21:07:02Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

phsft-bot · 2018-04-02T21:31:03Z

Build failed on slc6/gcc62.
See console output.

Errors:

ERROR: Timeout after 10 minutes
ERROR: Error fetching remote repo 'origin'
ERROR: Error fetching remote repo 'origin'
Error: /mnt/build/jenkins/workspace/root-pullrequests-build/build/bin/rootcling: compilation failure (/mnt/build/jenkins/workspace/root-pullrequests-build/build/lib/libMathCore87e80712ed_dictUmbrella.h)

Warnings:

../root/core/metacling/src/TCling.cxx:1118:13: warning: ‘void loadModulePath(clang::HeaderSearch&, const char*)’ defined but not used [-Wunused-function]

phsft-bot · 2018-04-02T22:02:09Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

vgvassilev · 2018-04-02T22:03:09Z

@phsft-bot build just on slc6/gcc62 with flags -DROOT_PARALLEL_LINK_JOBS=2 -Druntime_cxxmodules=On -Dctest_test_exclude_none=on

phsft-bot · 2018-04-02T22:03:35Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -DROOT_PARALLEL_LINK_JOBS=2 -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

phsft-bot · 2018-04-02T22:24:20Z

Build failed on slc6/gcc62.
See console output.

Errors:

Error: /mnt/build/jenkins/workspace/root-pullrequests-build/build/bin/rootcling: compilation failure (/mnt/build/jenkins/workspace/root-pullrequests-build/build/lib/libMathCore1faeac90e9_dictUmbrella.h)

Warnings:

../root/core/metacling/src/TCling.cxx:1118:13: warning: ‘void loadModulePath(clang::HeaderSearch&, const char*)’ defined but not used [-Wunused-function]

vgvassilev · 2018-04-03T15:51:15Z

@phsft-bot build just on slc6/gcc62 with flags -DROOT_PARALLEL_LINK_JOBS=2 -Druntime_cxxmodules=On -Dctest_test_exclude_none=on

phsft-bot · 2018-04-03T15:51:22Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -DROOT_PARALLEL_LINK_JOBS=2 -Druntime_cxxmodules=On -Dctest_test_exclude_none=on
How to customize builds

phsft-bot · 2018-04-03T16:25:15Z

Build failed on slc6/gcc62.
See console output.

Errors:

Error: /mnt/build/jenkins/workspace/root-pullrequests-build/build/bin/rootcling: compilation failure (/mnt/build/jenkins/workspace/root-pullrequests-build/build/lib/libMathCorec2ea5ff842_dictUmbrella.h)

Warnings:

../root/core/metacling/src/TCling.cxx:1118:13: warning: ‘void loadModulePath(clang::HeaderSearch&, const char*)’ defined but not used [-Wunused-function]

vgvassilev · 2018-04-04T12:20:05Z

@phsft-bot build just on slc6/gcc62 with flags -DLLVM_BUILD_TYPE=Debug -Droofit=Off -Druntime_cxxmodules=On -Dctest_test_exclude_none=On

phsft-bot · 2018-04-04T12:20:24Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -DLLVM_BUILD_TYPE=Debug -Droofit=Off -Druntime_cxxmodules=On -Dctest_test_exclude_none=On
How to customize builds

phsft-bot · 2018-04-04T12:24:14Z

Build failed on slc6/gcc62.
See console output.

Errors:

ninja: error: '/mnt/build/jenkins/workspace/root-pullrequests-build/root/RooStats', needed by 'CMakeFiles/onepcm', missing and no known rule to make it

phsft-bot · 2018-04-04T15:51:48Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -DLLVM_BUILD_TYPE=Debug -Droofit=Off -Druntime_cxxmodules=On -Dctest_test_exclude_none=On
How to customize builds

phsft-bot · 2018-04-04T15:55:22Z

Build failed on slc6/gcc62.
See console output.

Errors:

ninja: error: '/mnt/build/jenkins/workspace/root-pullrequests-build/root/RooFit', needed by 'CMakeFiles/onepcm', missing and no known rule to make it

Currently RooFit has a bug and we cannot use it with modules builds.

phsft-bot · 2018-04-04T15:58:31Z

Starting build on slc6/gcc62 with flags -Dvc=OFF -Dimt=ON -Dccache=ON -DLLVM_BUILD_TYPE=Debug -Droofit=Off -Druntime_cxxmodules=On -Dctest_test_exclude_none=On
How to customize builds

phsft-bot · 2018-04-04T16:37:17Z

Build failed on slc6/gcc62.
See console output.

Errors:

Error in <TStreamerInfo::Build>: TH1, discarding: double* fBuffer, no [dimension]
Error in <TStreamerInfo::Build>: TH1, discarding: double* fIntegral, no [dimension]
Error in <TStreamerInfo::Build>: TArrayF, discarding: float* fArray, no [dimension]
Error in <TStreamerInfo::Build>: TString::Rep_t, unknown type: TString::Rep_t::(anonymous)
Error in <TStreamerInfo::Build>: __shared_ptr<TObjLink>: TObjLink* has no streamer or dictionary, data member _M_ptr will not be saved
Error in <TStreamerInfo::Build>: __weak_ptr<TObjLink,__gnu_cxx::_Lock_policy::_S_atomic>: TObjLink* has no streamer or dictionary, data member _M_ptr will not be saved
Error in <TStreamerInfo::Build>: __weak_count<__gnu_cxx::_Lock_policy::_S_atomic>: _Sp_counted_base<__gnu_cxx::_Lock_policy::_S_atomic>* has no streamer or dictionary, data member _M_pi will not be saved
Error in <TStreamerInfo::Build>: __shared_count<__gnu_cxx::_Lock_policy::_S_atomic>: _Sp_counted_base<__gnu_cxx::_Lock_policy::_S_atomic>* has no streamer or dictionary, data member _M_pi will not be saved
Error in <TStreamerInfo::Build>: TText, unknown type: void* fWcsTitle
Error in <TStreamerInfo::Build>: TLatex: TLatex::FormSize_t* has no streamer or dictionary, data member fTabSize will not be saved

And 23 more

Warnings:

../root/core/metacling/src/TCling.cxx:1118:13: warning: ‘void loadModulePath(clang::HeaderSearch&, const char*)’ defined but not used [-Wunused-function]

vgvassilev · 2018-05-12T21:53:18Z

@Teemperor, I'd like to close this PR but the change to the clang::SourceManager seems still relevant. Could you open a phab review for it?

vgvassilev · 2018-09-20T09:49:49Z

This is implemented in a different way in the master.

vgvassilev requested review from amadio and pcanal as code owners March 3, 2018 16:32

vgvassilev force-pushed the PreloadMoreModules branch from f2131ab to 7b3e73d Compare March 3, 2018 16:51

vgvassilev mentioned this pull request Mar 3, 2018

[cxxmodules] Also preload TMVA/TreePlayer/Graf #1396

Closed

vgvassilev changed the title ~~Preload more modules~~ [cxxmodules] Preload more modules Mar 3, 2018

Teemperor added 2 commits March 3, 2018 19:35

[cxxmodules] Add flag if module has C++ module alongside it

d0ca156

amadio assigned vgvassilev Mar 5, 2018

Teemperor added 8 commits March 11, 2018 22:53

Add LD_LIBRARY_PATH to module path (broken commit?)

ad48d79

Another attempt at loading from library path

6fe98eb

Fixed library loading

0650032

Re-enable header parsing on demand

3f6ba92

Also add cwd to prebuilt module path

e4df711

Preload TreeViewer, Net, Hist, GenVector, Gpad and RooStats.

44bb027

Preload EG, RGL, RooFit, TMVAGui and MathCore.

9757e12

Use sparse vector in LoadedSLocEntryTable

a900091

vgvassilev force-pushed the PreloadMoreModules branch from 7b3e73d to a900091 Compare March 11, 2018 22:02

yamaguchi1024 mentioned this pull request Mar 29, 2018

[cxxmodules] Preload tmva tree player graf #1365

Closed

vgvassilev force-pushed the PreloadMoreModules branch from 3ed20f2 to f5eed82 Compare April 2, 2018 21:06

vgvassilev force-pushed the PreloadMoreModules branch from 7d6f3d6 to 3b8336f Compare April 4, 2018 15:51

[cxxmodules] Remove RooFit and RooStats as they are optional.

390cc01

Currently RooFit has a bug and we cannot use it with modules builds.

vgvassilev force-pushed the PreloadMoreModules branch from 3b8336f to 390cc01 Compare April 4, 2018 15:58

vgvassilev closed this Sep 20, 2018

[cxxmodules] Preload more modules #1697

[cxxmodules] Preload more modules #1697

Uh oh!

Conversation

vgvassilev commented Mar 3, 2018

Uh oh!

phsft-bot commented Mar 3, 2018

Uh oh!

vgvassilev commented Mar 3, 2018

Uh oh!

phsft-bot commented Mar 3, 2018

Uh oh!

phsft-bot commented Mar 3, 2018

Errors:

Uh oh!

phsft-bot commented Mar 3, 2018

Uh oh!

phsft-bot commented Mar 3, 2018

Errors:

Uh oh!

pcanal commented Mar 5, 2018

Uh oh!

vgvassilev commented Mar 5, 2018

Uh oh!

phsft-bot commented Mar 11, 2018

Uh oh!

phsft-bot commented Mar 11, 2018

Errors:

Warnings:

Uh oh!

pcanal commented Mar 12, 2018

Uh oh!

vgvassilev commented Mar 21, 2018

Uh oh!

phsft-bot commented Mar 21, 2018

Uh oh!

yamaguchi1024 commented Mar 29, 2018

Uh oh!

vgvassilev commented Mar 29, 2018

Uh oh!

yamaguchi1024 commented Mar 29, 2018

Uh oh!

vgvassilev commented Apr 2, 2018

Uh oh!

phsft-bot commented Apr 2, 2018

Uh oh!

phsft-bot commented Apr 2, 2018

Errors:

Warnings:

Uh oh!

phsft-bot commented Apr 2, 2018

Uh oh!

phsft-bot commented Apr 2, 2018

Errors:

Warnings:

Uh oh!

phsft-bot commented Apr 2, 2018

Uh oh!

vgvassilev commented Apr 2, 2018

Uh oh!

phsft-bot commented Apr 2, 2018

Uh oh!

phsft-bot commented Apr 2, 2018

Errors:

Warnings:

Uh oh!

vgvassilev commented Apr 3, 2018

Uh oh!

phsft-bot commented Apr 3, 2018

Uh oh!

phsft-bot commented Apr 3, 2018

Errors:

Warnings:

Uh oh!

vgvassilev commented Apr 4, 2018

Uh oh!

phsft-bot commented Apr 4, 2018

Uh oh!

phsft-bot commented Apr 4, 2018

Errors: