[RF] Make it possible to store global observables in RooAbsData and consider these global observables in RooAbsPdf::fitTo

README/ReleaseNotes/v626/index.md

@@ -96,6 +96,48 @@ RooFit now contains two RDataFrame action helpers, `RooDataSetHelper` and `RooDa
 ```
 For more details, consult the tutorial [rf408_RDataFrameToRooFit](https://root.cern/doc/v626/rf408__RDataFrameToRooFit_8C.html).
 
+### Storing global observables in RooFit datasets
+
+RooFit groups model variables into *observables* and *parameters*, depending on if their values are stored in the dataset.
+For fits with parameter constraints, there is a third kind of variables, called *global observables*.
+These represent the results of auxiliary measurements that constrain the nuisance parameters.
+In the RooFit implementation, a likelihood is generally the sum of two terms:
+  * the likelihood of the data given the parameters, where the normalization set is the set of observables (implemented by `RooNLLVar`)
+  * the constraint term, where the normalization set is the set of *global observables* (implemented by `RooConstraintSum`)
+
+Before this release, the global observable values were always taken from the model/pdf.
+With this release, a mechanism is added to store a snapshot of global observables in any `RooDataSet` or `RooDataHist`.
+For toy studies where the global observables assume a different values for each toy, the bookkeeping of the set of global observables and in particular their values is much easier with this change.
+
+Usage example for a model with global observables `g1` and `g2`:
+```C++
+auto data = model.generate(x, 1000); // data has only the single observables x
+data->setGlobalObservables(g1, g2); // now, data also stores a snapshot of g1 and g2
+
+// If you fit the model to the data, the global observables and their values
+// are taken from the dataset:
+model.fitTo(*data);
+
+// You can still define the set of global observables yourself, but the values
+// will be takes from the dataset if available:
+model.fitTo(*data, GlobalObservables(g1, g2));
+
+// To force `fitTo` to take the global observable values from the model even
+// though they are in the dataset, you can use the new `GlobalObservablesSource`
+// command argument:
+model.fitTo(*data, GlobalObservables(g1, g2), GlobalObservablesSource("model"));
+// The only other allowed value for `GlobalObservablesSource` is "data", which
+// corresponds to the new default behavior explained above.
+```
+
+In case you create a RooFit dataset directly by calling its constructor, you can also pass the global observables in a command argument instead of calling `setGlobalObservables()` later:
+```C++
+RooDataSet data{"dataset", "dataset", x, RooFit::GlobalObservables(g1, g2)};
+```
+
+To access the set of global observables stored in a `RooAbsData`, call `RooAbsData::getGlobalObservables()`.
+It returns a `nullptr` if no global observable snapshots are stored in the dataset.
+
 ### Changes in `RooAbsPdf::fitTo` behaviour for multi-range fits
 
 The `RooAbsPdf::fitTo` and `RooAbsPdf::createNLL` functions accept a command argument to specify the fit range.
@@ -106,7 +148,7 @@ From now on, the likelihoods are normalized by the sum of integrals in each rang
 
 ### Deprecation of the `RooMinuit` class
 
-The `RooMinuit` class was the old interface between RooFit and minuit. With ROOT version 5.24, a the more general `RooMinimizer` adapter was introduced, which became the default with ROOT 6.08. 
+The `RooMinuit` class was the old interface between RooFit and minuit. With ROOT version 5.24, a the more general `RooMinimizer` adapter was introduced, which became the default with ROOT 6.08.
 
 Before 6.26, it was possible to still use the `RooMinuit` by passing the `Minimizer("OldMinuit", "minimizer")` command argument to `RooAbsPdf::fitTo()`. This option is now removed.
 

roofit/roofitcore/inc/RooAbsData.h

@@ -303,6 +303,13 @@ class RooAbsData : public TNamed, public RooPrintable {
 
   static StorageType getDefaultStorageType();
 
+  /// Returns snapshot of global observables stored in this data.
+  /// \return Pointer to a RooArgSet with the snapshot of global observables
+  ///         stored in the data. Can be `nullptr` if no global observales are
+  ///         stored.
+  RooArgSet const* getGlobalObservables() const { return _globalObservables.get(); }
+  void setGlobalObservables(RooArgSet const& globalObservables);
+
 protected:
 
   static StorageType defaultStorageType ;
@@ -350,8 +357,12 @@ class RooAbsData : public TNamed, public RooPrintable {
 
   std::map<std::string,RooAbsData*> _ownedComponents ; // Owned external components
 
+  std::unique_ptr<RooArgSet> _globalObservables; // Snapshot of global observables
+
 private:
-   ClassDef(RooAbsData, 5) // Abstract data collection
+  void copyGlobalObservables(const RooAbsData& other);
+
+   ClassDef(RooAbsData, 6) // Abstract data collection
 };
 
 #endif

roofit/roofitcore/inc/RooAbsPdf.h

@@ -159,6 +159,32 @@ class RooAbsPdf : public RooAbsReal {
                               const RooCmdArg& arg6=RooCmdArg::none(),  const RooCmdArg& arg7=RooCmdArg::none(), const RooCmdArg& arg8=RooCmdArg::none()) ;
   virtual RooFitResult* fitTo(RooAbsData& data, const RooLinkedList& cmdList) ;
 
+  /// Configuration struct for RooAbsPdf::minimizeNLL with all the default
+  //values that also should be taked as the default values for
+  //RooAbsPdf::fitTo.
+  struct MinimizerConfig {
+      double recoverFromNaN = 10.;
+      std::string fitOpt = "";
+      int optConst = 2;
+      int verbose = 0;
+      int doSave = 0;
+      int doTimer = 0;
+      int printLevel = 1;
+      int strat = 1;
+      int initHesse = 0;
+      int hesse = 1;
+      int minos = 0;
+      int numee = 10;
+      int doEEWall = 1;
+      int doWarn = 1;
+      int doSumW2 = -1;
+      int doAsymptotic = -1;
+      const RooArgSet* minosSet = nullptr;
+      std::string minType = "Minuit";
+      std::string minAlg = "minuit";
+  };
+  std::unique_ptr<RooFitResult> minimizeNLL(RooAbsReal & nll, RooAbsData const& data, MinimizerConfig const& cfg);
+
   virtual RooAbsReal* createNLL(RooAbsData& data, const RooLinkedList& cmdList) ;
   virtual RooAbsReal* createNLL(RooAbsData& data, const RooCmdArg& arg1=RooCmdArg::none(),  const RooCmdArg& arg2=RooCmdArg::none(),  
 				const RooCmdArg& arg3=RooCmdArg::none(),  const RooCmdArg& arg4=RooCmdArg::none(), const RooCmdArg& arg5=RooCmdArg::none(),  
@@ -359,8 +385,8 @@ class RooAbsPdf : public RooAbsReal {
   static TString _normRangeOverride ; 
 
 private:
-  int calculateAsymptoticCorrectedCovMatrix(RooMinimizer& minimizer, RooAbsData const& data);
-  int calculateSumW2CorrectedCovMatrix(RooMinimizer& minimizer, RooAbsReal const& nll) const;
+  int calcAsymptoticCorrectedCovariance(RooMinimizer& minimizer, RooAbsData const& data);
+  int calcSumW2CorrectedCovariance(RooMinimizer& minimizer, RooAbsReal const& nll) const;
 
   ClassDef(RooAbsPdf,5) // Abstract PDF with normalization support
 };

roofit/roofitcore/inc/RooAbsTestStatistic.h

@@ -51,6 +51,7 @@ class RooAbsTestStatistic : public RooAbsReal {
     bool cloneInputData = true;
     double integrateOverBinsPrecision = -1.;
     bool binnedL = false;
+    bool takeGlobalObservablesFromData = false;
   };
 
   // Constructors, assignment etc
@@ -155,10 +156,11 @@ class RooAbsTestStatistic : public RooAbsReal {
 
   RooFit::MPSplit _mpinterl = RooFit::BulkPartition; // Use interleaving strategy rather than N-wise split for partioning of dataset for multiprocessor-split
   Bool_t         _doOffset = false; // Apply interval value offset to control numeric precision?
+  const bool  _takeGlobalObservablesFromData = false; // If the global observable values are taken from data
   mutable ROOT::Math::KahanSum<double> _offset = 0.0; //! Offset as KahanSum to avoid loss of precision
   mutable Double_t _evalCarry = 0.0; //! carry of Kahan sum in evaluatePartition
 
-  ClassDef(RooAbsTestStatistic,2) // Abstract base class for real-valued test statistics
+  ClassDef(RooAbsTestStatistic,3) // Abstract base class for real-valued test statistics
 
 };
 

roofit/roofitcore/inc/RooConstraintSum.h

@@ -28,31 +28,39 @@ class RooConstraintSum : public RooAbsReal {
 public:
 
   RooConstraintSum() {}
-  RooConstraintSum(const char *name, const char *title, const RooArgSet& constraintSet, const RooArgSet& paramSet) ;
+  RooConstraintSum(const char *name, const char *title, const RooArgSet& constraintSet, const RooArgSet& paramSet, bool takeGlobalObservablesFromData=false) ;
 
   RooConstraintSum(const RooConstraintSum& other, const char* name = 0);
-  virtual TObject* clone(const char* newname) const { return new RooConstraintSum(*this, newname); }
+  virtual TObject* clone(const char* newname) const override { return new RooConstraintSum(*this, newname); }
 
   const RooArgList& list() { return _set1 ; }
 
   static std::unique_ptr<RooAbsReal> createConstraintTerm(
         std::string const& name,
         RooAbsPdf const& pdf,
-        RooArgSet const& observables,
+        RooAbsData const& data,
         RooArgSet const* constrainedParameters,
         RooArgSet const* externalConstraints,
         RooArgSet const* globalObservables,
         const char* globalObservablesTag,
+        bool takeGlobalObservablesFromData,
         RooWorkspace * workspace = nullptr);
 
+  bool setData(RooAbsData const& data, bool cloneData=true);
+  /// \copydoc setData(RooAbsData const&, bool)
+  bool setData(RooAbsData& data, bool cloneData=true) override {
+    return setData(static_cast<RooAbsData const&>(data), cloneData);
+  }
+
 protected:
 
   RooListProxy _set1 ;    // Set of constraint terms
   RooSetProxy _paramSet ; // Set of parameters to which constraints apply
+  const bool _takeGlobalObservablesFromData = false; // If the global observable values are taken from data
 
-  Double_t evaluate() const;
+  Double_t evaluate() const override;
 
-  ClassDef(RooConstraintSum,2) // sum of -log of set of RooAbsPdf representing parameter constraints
+  ClassDefOverride(RooConstraintSum,3) // sum of -log of set of RooAbsPdf representing parameter constraints
 };
 
 #endif

roofit/roofitcore/inc/RooGlobalFunc.h

@@ -239,10 +239,14 @@ RooCmdArg SplitRange(Bool_t flag=kTRUE) ;
 RooCmdArg SumCoefRange(const char* rangeName) ;
 RooCmdArg Constrain(const RooArgSet& params) ;
 RooCmdArg Constrain(RooArgSet && params) ;
-RooCmdArg GlobalObservables(const RooArgSet& globs) ;
-RooCmdArg GlobalObservables(RooArgSet && globs) ;
+
+template<class... Args_t>
+RooCmdArg GlobalObservables(Args_t &&... argsOrArgSet) {
+  return RooCmdArg("GlobalObservables",0,0,0,0,0,0,0,0,0,0,
+          &RooCmdArg::take(RooArgSet{std::forward<Args_t>(argsOrArgSet)...}));
+}
+RooCmdArg GlobalObservablesSource(const char* sourceName);
 RooCmdArg GlobalObservablesTag(const char* tagName) ;
-//RooCmdArg Constrained() ;
 RooCmdArg ExternalConstraints(const RooArgSet& constraintPdfs) ;
 RooCmdArg ExternalConstraints(RooArgSet && constraintPdfs) ;
 RooCmdArg PrintEvalErrors(Int_t numErrors) ;