[cxxmodules] Teach ACLiC to build modules. #3850

vgvassilev · 2019-05-23T08:45:58Z

ACLiC now synthesizes a modulemap with a suffix _ACLiC_dict.modulemap. The file contains the source file to be compiled and the corresponding library.

The modulemap is then passed to rootcling via -fmodule-map-file= flag to avoid naming clashes with possibly existing other modulemap files.

This patch teaches cling to work with the -fmodule-map-file= flag.

ACLiC supports automatic inclusion of Rtypes.h (making ClassDef macro available). Modules are built in isolation and are resilient to #include of Rtypes.h at rootcling startup time. We make module Core (containing Rtypes.h) visible via a newly implemented callback.

Depends on #3798

phsft-bot · 2019-05-23T08:46:08Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-05-23T08:51:14Z

Build failed on mac1014/cxx17.
See console output.

Errors:

FAILED: core/dictgen/CMakeFiles/Dictgen.dir/src/TModuleGenerator.cxx.o
/build/jenkins/workspace/root-pullrequests-build/root/core/dictgen/src/TModuleGenerator.cxx:500:20: error: use of undeclared identifier 'extraIncludes'
/build/jenkins/workspace/root-pullrequests-build/root/core/dictgen/src/TModuleGenerator.cxx:500:73: error: use of undeclared identifier 'extraIncludes'

phsft-bot · 2019-05-23T08:59:44Z

Build failed on ROOT-performance-centos7-multicore/default.
See console output.

Errors:

/data/sftnight/workspace/root-pullrequests-build/root/core/dictgen/src/TModuleGenerator.cxx:500:20: error: ‘extraIncludes’ was not declared in this scope

Warnings:

cc1plus: warning: unrecognized command line option "-Wno-class-memaccess" [enabled by default]

phsft-bot · 2019-05-23T09:03:22Z

Build failed on ROOT-ubuntu16/rtcxxmod.
See console output.

Errors:

FAILED: /usr/bin/ccache /usr/bin/c++ -Iinclude -I/mnt/build/workspace/root-pullrequests-build/root/core/dictgen/src -I/mnt/build/workspace/root-pullrequests-build/root/core/dictgen/res -I/mnt/build/workspace/root-pullrequests-build/root/core/dictgen/../foundation/res -I/mnt/build/workspace/root-pullrequests-build/root/core/dictgen/../clingutils/res -I/mnt/build/workspace/root-pullrequests-build/root/interpreter/cling/include -I/mnt/build/workspace/root-pullrequests-build/root/core/dictgen/SYSTEM -I/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/tools/clang/include -Iinterpreter/llvm/src/tools/clang/include -I/mnt/build/workspace/root-pullrequests-build/root/interpreter/llvm/src/include -Iinterpreter/llvm/src/include -std=c++11 -pipe -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -fvisibility=hidden -Wno-shadow -fno-strict-aliasing -Wno-unused-parameter -Wwrite-strings -Wno-long-long -Wno-missing-field-initializers -DLLVM_BUILD_GLOBAL_ISEL -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -O3 -DNDEBUG -fPIC -Wno-class-memaccess -std=c++11 -fno-rtti -MD -MT core/dictgen/CMakeFiles/Dictgen.dir/src/TModuleGenerator.cxx.o -MF core/dictgen/CMakeFiles/Dictgen.dir/src/TModuleGenerator.cxx.o.d -o core/dictgen/CMakeFiles/Dictgen.dir/src/TModuleGenerator.cxx.o -c /mnt/build/workspace/root-pullrequests-build/root/core/dictgen/src/TModuleGenerator.cxx
/mnt/build/workspace/root-pullrequests-build/root/core/dictgen/src/TModuleGenerator.cxx:500:20: error: ‘extraIncludes’ was not declared in this scope

Warnings:

cc1plus: warning: unrecognized command line option ‘-Wno-class-memaccess’

phsft-bot · 2019-05-23T09:05:11Z

Build failed on ROOT-fedora27/noimt.
See console output.

Errors:

FAILED: core/dictgen/CMakeFiles/Dictgen.dir/src/TModuleGenerator.cxx.o
/mnt/build/workspace/root-pullrequests-build/root/core/dictgen/src/TModuleGenerator.cxx:500:20: error: ‘extraIncludes’ was not declared in this scope

Warnings:

cc1plus: warning: unrecognized command line option ‘-Wno-class-memaccess’

phsft-bot · 2019-05-23T09:07:12Z

Build failed on windows10/default.
See console output.

phsft-bot · 2019-05-23T09:08:12Z

Build failed on ROOT-fedora29/python3.
See console output.

Errors:

FAILED: core/dictgen/CMakeFiles/Dictgen.dir/src/TModuleGenerator.cxx.o
/mnt/build/workspace/root-pullrequests-build/root/core/dictgen/src/TModuleGenerator.cxx:500:20: error: ‘extraIncludes’ was not declared in this scope

phsft-bot · 2019-05-24T07:09:04Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-05-24T07:12:38Z

Build failed on mac1014/cxx17.
See console output.

Errors:

FAILED: core/dictgen/CMakeFiles/Dictgen.dir/src/rootcling_impl.cxx.o
/build/jenkins/workspace/root-pullrequests-build/root/core/dictgen/src/rootcling_impl.cxx:4910:67: error: use of undeclared identifier 'isAclic'

phsft-bot · 2019-05-24T07:30:31Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-05-24T08:42:57Z

Build failed on ROOT-ubuntu16/rtcxxmod.
See console output.

Failing tests:

And 37 more

phsft-bot · 2019-06-15T18:56:04Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-06-15T19:30:41Z

Build failed on ROOT-ubuntu16/rtcxxmod.
See console output.

Failing tests:

And 47 more

phsft-bot · 2019-06-19T13:44:02Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-06-24T20:42:06Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-06-24T20:53:06Z

Build failed on mac1014/cxx17.
See console output.

Errors:

FAILED: core/foundation/CMakeFiles/Foundation_Stage1.dir/src/FoundationUtils.cxx.o
/build/jenkins/workspace/root-pullrequests-build/root/core/foundation/src/FoundationUtils.cxx:46:31: error: use of undeclared identifier 'errno'
/build/jenkins/workspace/root-pullrequests-build/root/core/foundation/src/FoundationUtils.cxx:46:40: error: use of undeclared identifier 'ERANGE'

phsft-bot · 2019-06-24T21:01:19Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-06-24T21:20:34Z

Build failed on windows10/default.
See console output.

phsft-bot · 2019-06-24T21:21:54Z

Build failed on ROOT-ubuntu16/rtcxxmod.
See console output.

Warnings:

/mnt/build/workspace/root-pullrequests-build/root/gui/gui/src/TGCommandPlugin.cxx:74:40: warning: ignoring return value of ‘char* fgets(char*, int, FILE*)’, declared with attribute warn_unused_result [-Wunused-result]

Failing tests:

And 11 more

phsft-bot · 2019-06-27T06:10:17Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-06-27T06:35:29Z

Build failed on windows10/default.
See console output.

phsft-bot · 2019-06-27T06:39:35Z

Build failed on ROOT-ubuntu16/rtcxxmod.
See console output.

Failing tests:

phsft-bot · 2019-06-27T08:24:53Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-06-27T08:48:55Z

Build failed on windows10/default.
See console output.

Warnings:

CMake Warning at C:/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:879 (message):

phsft-bot · 2019-06-27T20:02:50Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-06-27T20:08:50Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

vgvassilev · 2019-06-28T04:34:00Z

@phsft-bot build with flags -DCTEST_TEST_EXCLUDE_NONE=On

@Axel-Naumann could you take a look, I’d like to land it soon.

phsft-bot · 2019-06-28T04:34:35Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default with flags -DCTEST_TEST_EXCLUDE_NONE=On
How to customize builds

Axel-Naumann

Thanks, lot of good stuff here!

core/foundation/res/ROOT/FoundationUtils.hxx

core/metacling/src/TCling.cxx

interpreter/llvm/src/tools/clang/include/clang/Lex/PPCallbacks.h

core/dictgen/src/rootcling_impl.cxx

core/foundation/src/FoundationUtils.cxx

phsft-bot · 2019-06-28T20:05:25Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default with flags -DCTEST_TEST_EXCLUDE_NONE=On
How to customize builds

phsft-bot · 2019-06-28T20:39:30Z

Build failed on mac1014/cxx17.
See console output.

Warnings:

CMake Warning at /build/jenkins/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:879 (message):

Failing tests:

projectroot.runtutorials.tutorial_v7_line.cxx

Axel-Naumann

Please adjust the FIXME comment - otherwise fine, thanks!

Axel-Naumann · 2019-07-01T10:07:06Z

core/base/src/TSystem.cxx

-   rcling.Append(filename_fullpath).Append("\" \"").Append(linkdef).Append("\"");;
+
+   // Create a modulemap
+   // FIXME: Merge the generation from cmake and here in rootcling.


I cannot parse the comment. Do you mean "Merge this module file generation with that in cmake"?

The modulemap generation is what I meant. I will clarify.

ACLiC now synthesizes a modulemap with a suffix _ACLiC_dict.modulemap. The file contains the source file to be compiled and the corresponding library. The modulemap is then passed to rootcling via -fmodule-map-file= flag to avoid naming clashes with possibly existing other modulemap files. This patch teaches cling to work with the -fmodule-map-file= flag. ACLiC supports automatic inclusion of Rtypes.h (making ClassDef macro available). Modules are built in isolation and are resilient to #include of Rtypes.h at rootcling startup time. We make module Core (containing Rtypes.h) visible via a newly implemented callback.

If the user calls rootcling ... /full/path/a.h, and the current working directory is /full/path/ rootcling removes the full path and #includes "a.h" instead of what the user specified on the prompt. This odd feature caused problems for modules in ACLiC because the generated modulemap contained the full path to the header (i.e. header "/full/path/a.h") which confuses the modules system as it could not detect it is the same header in case where no include guards or #pragma once exist. This patch centralizes the rootcling make-path-relative logic into a function accessible from ACLiC in TSystem.

…ing. ACLiC has the following modes to autoload and select whether and how ACLiC tracks the dependency of the libraries: 0 - No tracking 1 - [Default] New libraries are explicitly linked to all currently loaded libraries 2 - Generate and use rootmap file for each library 3 - Generate rootmap file for each library _and_ explicitly link to the needed libraries. On Windows, the default is 3 This patch enables cxxmodules for mode 2 and 3. The initial implementation 'almost' allows running it by default except for the following tests: * roottest.cling.template.separateDict.roottest_cling_template_separateDict_make * roottest.root.io.interface.roottest_root_io_interface_make * roottest.root.meta.evolution.roottest_root_meta_evolution_execCheckSum * roottest.root.tree.selectorreader.roottest_root_tree_selectorreader_make * roottest.root.tree.selector.roottest_root_tree_selector_make * projectroot.roottest.cling.template.roottest_cling_template_runtemplatefriend We could fix them but enabling modules for 'bare' ACLiC might be too much. In many cases users have legacy headers or other not well structured C++ code and that might be problematic. The intent for the moment is to remove completely rootmap files when we use -Druntime_cxxmodules=On while keeping the dependency tracking information intact.

… true. NFC.

phsft-bot · 2019-07-01T10:55:28Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default with flags -DCTEST_TEST_EXCLUDE_NONE=On
How to customize builds

vgvassilev requested review from Axel-Naumann, amadio and pcanal as code owners May 23, 2019 08:45

vgvassilev force-pushed the cxxmodules-teach-aclic branch from b0c6cf9 to 70391d1 Compare May 24, 2019 07:08

vgvassilev force-pushed the cxxmodules-teach-aclic branch from 70391d1 to 810b3af Compare May 24, 2019 07:30

hageboeck assigned vgvassilev May 29, 2019

vgvassilev force-pushed the cxxmodules-teach-aclic branch from 810b3af to f9f9396 Compare June 15, 2019 18:55

vgvassilev force-pushed the cxxmodules-teach-aclic branch from f9f9396 to 8c1b72d Compare June 19, 2019 13:43

vgvassilev force-pushed the cxxmodules-teach-aclic branch from 8c1b72d to 9da8355 Compare June 24, 2019 20:41

vgvassilev force-pushed the cxxmodules-teach-aclic branch from 9da8355 to 669b288 Compare June 24, 2019 21:01

vgvassilev force-pushed the cxxmodules-teach-aclic branch from 669b288 to f6ba851 Compare June 26, 2019 19:53

vgvassilev force-pushed the cxxmodules-teach-aclic branch from 0d51f1a to c2986f4 Compare June 27, 2019 06:09

vgvassilev mentioned this pull request Jun 27, 2019

[cxxmodules] Filter out source locations. root-project/roottest#345

Merged

vgvassilev force-pushed the cxxmodules-teach-aclic branch from c2986f4 to b6e839d Compare June 27, 2019 08:24

vgvassilev force-pushed the cxxmodules-teach-aclic branch from b6e839d to fafed31 Compare June 27, 2019 20:02

vgvassilev force-pushed the cxxmodules-teach-aclic branch from fafed31 to 03f9162 Compare June 27, 2019 20:08

vgvassilev mentioned this pull request Jun 27, 2019

[cxxmodules] Do not generate a rootmap if a module exists. #4005

Merged

Axel-Naumann requested changes Jun 28, 2019

View reviewed changes

vgvassilev force-pushed the cxxmodules-teach-aclic branch from 03f9162 to 3a0b63d Compare June 28, 2019 20:05

Axel-Naumann approved these changes Jul 1, 2019

View reviewed changes

vgvassilev added 6 commits July 1, 2019 13:54

Extend the scope of the variable.

a959316

[cmake] Fix typo in docs.

9b69fc3

[cxxmodules] Remove the parameter. The function is called always with…

46939bc

… true. NFC.

vgvassilev force-pushed the cxxmodules-teach-aclic branch from 3a0b63d to 46939bc Compare July 1, 2019 10:55

vgvassilev merged commit 7665a83 into root-project:master Jul 1, 2019

vgvassilev deleted the cxxmodules-teach-aclic branch July 1, 2019 11:54

[cxxmodules] Teach ACLiC to build modules. #3850

[cxxmodules] Teach ACLiC to build modules. #3850

Uh oh!

Conversation

vgvassilev commented May 23, 2019

Uh oh!

phsft-bot commented May 23, 2019

Uh oh!

phsft-bot commented May 23, 2019

Errors:

Uh oh!

phsft-bot commented May 23, 2019

Errors:

Warnings:

Uh oh!

phsft-bot commented May 23, 2019

Errors:

Warnings:

Uh oh!

phsft-bot commented May 23, 2019

Errors:

Warnings:

Uh oh!

phsft-bot commented May 23, 2019

Uh oh!

phsft-bot commented May 23, 2019

Errors:

Uh oh!

phsft-bot commented May 24, 2019

Uh oh!

phsft-bot commented May 24, 2019

Errors:

Uh oh!

phsft-bot commented May 24, 2019

Uh oh!

phsft-bot commented May 24, 2019

Failing tests:

Uh oh!

phsft-bot commented Jun 15, 2019

Uh oh!

phsft-bot commented Jun 15, 2019

Failing tests:

Uh oh!

phsft-bot commented Jun 19, 2019

Uh oh!

phsft-bot commented Jun 24, 2019

Uh oh!

phsft-bot commented Jun 24, 2019

Errors:

Uh oh!

phsft-bot commented Jun 24, 2019

Uh oh!

phsft-bot commented Jun 24, 2019

Uh oh!

phsft-bot commented Jun 24, 2019

Warnings:

Failing tests:

Uh oh!

phsft-bot commented Jun 27, 2019

Uh oh!

phsft-bot commented Jun 27, 2019

Uh oh!

phsft-bot commented Jun 27, 2019

Failing tests:

Uh oh!

phsft-bot commented Jun 27, 2019

Uh oh!

phsft-bot commented Jun 27, 2019

Warnings:

Uh oh!

phsft-bot commented Jun 27, 2019

Uh oh!

phsft-bot commented Jun 27, 2019

Uh oh!

vgvassilev commented Jun 28, 2019

Uh oh!

phsft-bot commented Jun 28, 2019

Uh oh!

Axel-Naumann left a comment

Choose a reason for hiding this comment