[TMVA] Add multiprocessing to parallelise CV #1869

ashlaban · 2018-04-17T12:21:18Z

Based on PR #858. Implements only parts relevant for CV.

Rudimentary performance benchmark with ~60000 events with 2-fold CV

Multiproc: root -l -b -q TMVACrossValidation.C  15.82s user 0.46s system 152% cpu 10.679 total
Standard: root -l -b -q TMVACrossValidation.C  14.31s user 0.25s system 99% cpu 14.652 total

Which translates into a 4 second speed up.

A different example, with a deeper bdt forest and 10-fold CV, almost halves the required time on my machine :)

Single     : root -l -b -q TMVACrossValidation.C  33.66s user 1.99s system 98%  cpu 36.369 total
Multiproc 2: root -l -b -q TMVACrossValidation.C  39.14s user 2.68s system 167% cpu 25.016 total
Multiproc 4: root -l -b -q TMVACrossValidation.C  46.52s user 3.50s system 233% cpu 21.420 total

For the second example, these changes to TMVACrossValidation.C were used.

/* ...snip... */
TString cvOptions = Form("!V:NumWorkerProcs=4"
/* ...snip... */
cv.BookMethod(TMVA::Types::kBDT, "BDTG",
              "!H:!V:NTrees=1000:MinNodeSize=2.5%:BoostType=Grad"
              ":NegWeightTreatment=Pray:Shrinkage=0.10:nCuts=20"
              ":MaxDepth=6");
/* ...snip... */
// cv.BookMethod(TMVA::Types::kFisher, "Fisher",
//               "!H:!V:Fisher:VarTransform=None");
/* ...snip... */

phsft-bot · 2018-04-17T12:21:29Z

Starting build on centos7/gcc49, mac1013/native, slc6/gcc49, slc6/gcc62, slc6/gcc62, ubuntu16/native, ubuntu16/native, windows10/vc15 with flags -Dvc=OFF -Dimt=ON -Dccache=ON
How to customize builds

When doing CV, we reuse the same DataSet (and the corresponding list of results). Thus we need to empty the DataSet results between folds. A convenient way of doing this was missing.

In CV, there was a bug causing erroneous results to be added to the DataSet (where the size of those results where smaller than what was expected).

Sometimes, methods can add extra results to the DataSet. E.g. if `CreateMvaPDFs` is specified. This was not correctly picked up by CV.

- Variable renaming ce -> cv. - Remove extraneous text output from method logs.

phsft-bot · 2018-04-18T12:55:16Z

Starting build on centos7/gcc49, mac1013/native, slc6/gcc49, slc6/gcc62, slc6/gcc62, ubuntu16/native, ubuntu16/native, windows10/vc15 with flags -Dvc=OFF -Dimt=ON -Dccache=ON
How to customize builds

… process

phsft-bot · 2018-04-20T18:10:16Z

Starting build on centos7/gcc49, mac1013/native, slc6/gcc49, slc6/gcc62, slc6/gcc62, ubuntu16/native, ubuntu16/native, windows10/vc15 with flags -Dvc=OFF -Dimt=ON -Dccache=ON
How to customize builds

phsft-bot · 2018-04-20T18:37:53Z

Build failed on slc6/gcc62.
See console output.

Warnings:

include/TMVA/Config.h:145:26: warning: ‘TMVA::Config::fLogger’ will be initialized after [-Wreorder]
include/TMVA/Config.h:137:27: warning: ‘std::atomic<unsigned int> TMVA::Config::fNWorkers’ [-Wreorder]
../root/tmva/tmva/src/Config.cxx:56:1: warning: when initialized here [-Wreorder]
../root/tmva/tmva/test/crossvalidation/TestCrossValidationMultiProc.cxx:59:54: warning: declaration of ‘using clock_t = using high_resolution_clock = struct std::chrono::_V2::system_clock’ shadows a global declaration [-Wshadow]

Failing tests:

projectroot.tree.treeplayer.test.gtest_tree_treeplayer_test_dataframe_simple

phsft-bot · 2018-04-20T18:38:55Z

Build failed on ubuntu16/native.
See console output.

Warnings:

include/TMVA/Config.h:145:26: warning: ‘TMVA::Config::fLogger’ will be initialized after [-Wreorder]
include/TMVA/Config.h:137:27: warning: ‘std::atomic<unsigned int> TMVA::Config::fNWorkers’ [-Wreorder]
/mnt/build/workspace/root-pullrequests-build/root/tmva/tmva/src/Config.cxx:56:1: warning: when initialized here [-Wreorder]
/mnt/build/workspace/root-pullrequests-build/root/tmva/tmva/test/crossvalidation/TestCrossValidationMultiProc.cxx:59:54: warning: declaration of ‘using clock_t = using high_resolution_clock = struct std::chrono::_V2::system_clock’ shadows a global declaration [-Wshadow]

Failing tests:

projectroot.tree.treeplayer.test.gtest_tree_treeplayer_test_dataframe_simple

phsft-bot · 2018-04-20T18:41:09Z

Build failed on mac1013/native.
See console output.

Warnings:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__tree:1819:22: warning: the specified comparator type does not provide a const call operator [-Wuser-defined-warnings]
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__tree:1819:22: warning: the specified comparator type does not provide a const call operator [-Wuser-defined-warnings]
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__tree:1819:22: warning: the specified comparator type does not provide a const call operator [-Wuser-defined-warnings]
/Volumes/MacintoshHD/build/jenkins/workspace/root-pullrequests-build/root/tmva/tmva/src/Config.cxx:61:4: warning: field 'fLogger' will be initialized after field 'fNWorkers' [-Wreorder]
/Volumes/MacintoshHD/build/jenkins/workspace/root-pullrequests-build/root/tmva/tmva/test/crossvalidation/TestCrossValidationMultiProc.cxx:59:10: warning: declaration shadows a typedef in the global namespace [-Wshadow]

Failing tests:

projectroot.roottest.root.dataframe.roottest_root_dataframe_test_splitcoll_arrayview

lmoneta · 2018-04-25T09:11:53Z

@phsft-bot build

phsft-bot · 2018-04-25T09:12:03Z

Starting build on centos7/gcc49, mac1013/native, slc6/gcc49, slc6/gcc62, slc6/gcc62, ubuntu16/native, ubuntu16/native, windows10/vc15 with flags -Dvc=OFF -Dimt=ON -Dccache=ON
How to customize builds

phsft-bot · 2018-04-25T09:42:49Z

Build failed on centos7/gcc49.
See console output.

Warnings:

include/TMVA/Config.h:145:26: warning: ‘TMVA::Config::fLogger’ will be initialized after [-Wreorder]
include/TMVA/Config.h:137:27: warning: ‘std::atomic<unsigned int> TMVA::Config::fNWorkers’ [-Wreorder]
../root/tmva/tmva/src/Config.cxx:56:1: warning: when initialized here [-Wreorder]
../root/tmva/tmva/test/crossvalidation/TestCrossValidationMultiProc.cxx:59:54: warning: declaration of ‘using clock_t = using high_resolution_clock = struct std::chrono::_V2::system_clock’ shadows a global declaration [-Wshadow]

Failing tests:

projectroot.roottest.root.multicore.roottest_root_multicore_tp_process_imt

phsft-bot · 2018-04-25T10:19:19Z

Starting build on centos7/gcc49, mac1013/native, slc6/gcc49, slc6/gcc62, slc6/gcc62, ubuntu16/native, ubuntu16/native, windows10/vc15 with flags -Dvc=OFF -Dimt=ON -Dccache=ON
How to customize builds

phsft-bot · 2018-04-25T10:35:41Z

Build failed on mac1013/native.
See console output.

Warnings:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__tree:1819:22: warning: the specified comparator type does not provide a const call operator [-Wuser-defined-warnings]
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__tree:1819:22: warning: the specified comparator type does not provide a const call operator [-Wuser-defined-warnings]
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__tree:1819:22: warning: the specified comparator type does not provide a const call operator [-Wuser-defined-warnings]

phsft-bot · 2018-04-25T10:48:05Z

Build failed on centos7/gcc49.
See console output.

Failing tests:

projectroot.tmva.tmva.test.DNN.TMVA_DNN_Minimization_Cpu

lmoneta · 2018-04-27T07:50:02Z

@phsft-bot build

phsft-bot · 2018-04-27T07:50:12Z

Starting build on centos7/gcc49, mac1013/native, slc6/gcc49, slc6/gcc62, slc6/gcc62, ubuntu16/native, ubuntu16/native, windows10/vc15 with flags -Dvc=OFF -Dimt=ON -Dccache=ON
How to customize builds

phsft-bot · 2018-04-27T08:29:18Z

Build failed on slc6/gcc62.
See console output.

Failing tests:

projectroot.tree.treeplayer.test.gtest_tree_treeplayer_test_dataframe_simple

phsft-bot · 2018-04-27T08:29:46Z

Build failed on ubuntu16/native.
See console output.

Failing tests:

projectroot.tree.treeplayer.test.gtest_tree_treeplayer_test_dataframe_simple

[TMVA] Add multiprocessing to parallelise CV

c996bc5

ashlaban requested a review from lmoneta as a code owner April 17, 2018 12:21

amadio assigned lmoneta Apr 18, 2018

ashlaban added 5 commits April 18, 2018 14:40

[TMVA] DataSet - Add method DeleteAllResults

cf0eec5

When doing CV, we reuse the same DataSet (and the corresponding list of results). Thus we need to empty the DataSet results between folds. A convenient way of doing this was missing.

[TMVA] Sanity check for DataSet result size.

d2319c1

In CV, there was a bug causing erroneous results to be added to the DataSet (where the size of those results where smaller than what was expected).

[TMVA] Simplify GetTree copying of results

c9a13f0

[TMVA] CV bugfix, remove _all_ results for a fold from DataSet

93dc3d2

Sometimes, methods can add extra results to the DataSet. E.g. if `CreateMvaPDFs` is specified. This was not correctly picked up by CV.

[TMVA] Small fixes to TMVACrossValidation.C

f7a1e75

- Variable renaming ce -> cv. - Remove extraneous text output from method logs.

ashlaban requested a review from couet as a code owner April 18, 2018 12:55

ashlaban added 2 commits April 20, 2018 19:46

[TVMA] CV - Fix crash when not using an output file

693b51f

[TMVA] CV - Add test to verify equality of multiprocessing and single…

346d5e7

… process

ashlaban requested a review from amadio as a code owner April 20, 2018 18:10

ashlaban added 2 commits April 23, 2018 13:44

[TMVA] Reorder init order for Config (to supress warnings)

72bc7c1

[TMVA] CV - Fix warnings

92f063a

ashlaban mentioned this pull request Apr 24, 2018

[TMVA] CV - Evaluate single- + multiprocessing root-project/rootbench#61

Closed

lmoneta merged commit ad91ce2 into root-project:master Apr 27, 2018

[TMVA] Add multiprocessing to parallelise CV #1869

[TMVA] Add multiprocessing to parallelise CV #1869

Uh oh!

Conversation

ashlaban commented Apr 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phsft-bot commented Apr 17, 2018

Uh oh!

phsft-bot commented Apr 18, 2018

Uh oh!

phsft-bot commented Apr 20, 2018

Uh oh!

phsft-bot commented Apr 20, 2018

Warnings:

Failing tests:

Uh oh!

phsft-bot commented Apr 20, 2018

Warnings:

Failing tests:

Uh oh!

phsft-bot commented Apr 20, 2018

Warnings:

Failing tests:

Uh oh!

lmoneta commented Apr 25, 2018

Uh oh!

phsft-bot commented Apr 25, 2018

Uh oh!

phsft-bot commented Apr 25, 2018

Warnings:

Failing tests:

Uh oh!

phsft-bot commented Apr 25, 2018

Uh oh!

phsft-bot commented Apr 25, 2018

Warnings:

Uh oh!

phsft-bot commented Apr 25, 2018

Failing tests:

Uh oh!

lmoneta commented Apr 27, 2018

Uh oh!

phsft-bot commented Apr 27, 2018

Uh oh!

phsft-bot commented Apr 27, 2018

Failing tests:

Uh oh!

phsft-bot commented Apr 27, 2018

Failing tests:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ashlaban commented Apr 17, 2018 •

edited

Loading