update samples from Release-172 as a part of SDK release

Author: amlrelsa-ms

how-to-use-azureml/automated-machine-learning/forecasting-backtest-many-models/assets/score.py

@@ -122,7 +122,10 @@ def calculate_scores_and_build_plots(
     input_dir: str, output_dir: str, automl_settings: Dict[str, Any]
 ):
     os.makedirs(output_dir, exist_ok=True)
-    grains = automl_settings.get(constants.TimeSeries.TIME_SERIES_ID_COLUMN_NAMES)
+    grains = automl_settings.get(
+        constants.TimeSeries.TIME_SERIES_ID_COLUMN_NAMES,
+        automl_settings.get(constants.TimeSeries.GRAIN_COLUMN_NAMES, None),
+    )
     time_column_name = automl_settings.get(constants.TimeSeries.TIME_COLUMN_NAME)
     if grains is None:
         grains = []

how-to-use-azureml/automated-machine-learning/forecasting-backtest-many-models/auto-ml-forecasting-backtest-many-models.ipynb

@@ -40,7 +40,7 @@
       "metadata": {},
       "source": [
         "### Prerequisites\n",
-        "You'll need to create a compute Instance by following the instructions in the [EnvironmentSetup.md](../Setup_Resources/EnvironmentSetup.md)."
+        "You'll need to create a compute Instance by following [these](https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-create-manage-compute-instance?tabs=python) instructions."
       ]
     },
     {
@@ -259,6 +259,7 @@
         "| **forecast_horizon**               | The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly). Periods are inferred from your data. |\n",
         "| **time_column_name**               | The name of your time column. |\n",
         "| **time_series_id_column_names**    | The column names used to uniquely identify timeseries in data that has multiple rows with the same timestamp. |\n",
+        "| **cv_step_size**                   | Number of periods between two consecutive cross-validation folds. The default value is \\\"auto\\\", in which case AutoMl determines the cross-validation step size automatically, if a validation set is not provided. Or users could specify an integer value. |\n",
         "\n",
         "#### ``AutoMLConfig`` arguments\n",
         "| Property                           | Description|\n",
@@ -268,11 +269,10 @@
         "| **blocked_models**                 | Blocked models won't be used by AutoML. |\n",
         "| **iteration_timeout_minutes**      | Maximum amount of time in minutes that the model can train. This is optional but provides customers with greater control on exit criteria. |\n",
         "| **iterations**                     | Number of models to train. This is optional but provides customers with greater control on exit criteria. |\n",
-        "| **experiment_timeout_hours**       | Maximum amount of time in hours that the experiment can take before it terminates. This is optional but provides customers with greater control on exit criteria. |\n",
+        "| **experiment_timeout_hours**       | Maximum amount of time in hours that each experiment can take before it terminates. This is optional but provides customers with greater control on exit criteria. **It does not control the overall timeout for the pipeline run, instead controls the timeout for each training run per partitioned time series.** |\n",
         "| **label_column_name**              | The name of the label column. |\n",
-        "| **n_cross_validations**            | Number of cross-validation folds to use for model/pipeline selection. The default value is \\\"auto\\\", in which case AutoMl determines the number of cross-validations automatically, if a validation set is not provided. Or users could specify an integer value. |\n",
-        "| **cv_step_size**                   | Number of periods between two consecutive cross-validation folds. The default value is \\\"auto\\\", in which case AutoMl determines the cross-validation step size automatically, if a validation set is not provided. Or users could specify an integer value. |\n",
-        "| **enable_early_stopping**          | Flag to enable early termination if the score is not improving in the short term. |\n",
+        "| **n_cross_validations**            | Number of cross validation splits. The default value is \\\"auto\\\", in which case AutoMl determines the number of cross-validations automatically, if a validation set is not provided. Or users could specify an integer value. Rolling Origin Validation is used to split time-series in a temporally consistent way. |\n",
+        "| **enable_early_stopping**          | Flag to enable early termination if the primary metric is no longer improving. |\n",
         "| **enable_engineered_explanations** | Engineered feature explanations will be downloaded if enable_engineered_explanations flag is set to True. By default it is set to False to save storage space. |\n",
         "| **track_child_runs**               | Flag to disable tracking of child runs. Only best run is tracked if the flag is set to False (this includes the model and metrics of the run). |\n",
         "| **pipeline_fetch_max_batch_size**  | Determines how many pipelines (training algorithms) to fetch at a time for training, this helps reduce throttling when training at large scale. |\n",
@@ -281,7 +281,7 @@
         "#### ``HTSTrainParameters`` arguments\n",
         "| Property                           | Description|\n",
         "| :---------------                   | :------------------- |\n",
-        "| **automl_settings**                | ``AutoMLConfig`` object.\n",
+        "| **automl_settings**                | The ``AutoMLConfig`` object defined above. |\n",
         "| **hierarchy_column_names**         | The names of columns that define the hierarchical structure of the data from highest level to most granular. |\n",
         "| **training_level**                 | The level of the hierarchy to be used for training models. |\n",
         "| **enable_engineered_explanations** | The switch controls engineered explanations. |"
@@ -354,16 +354,25 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Parallel run step is leveraged to train the hierarchy. To configure the ParallelRunConfig you will need to determine the appropriate number of workers and nodes for your use case. The `process_count_per_node` is based off the number of cores of the compute VM. The node_count will determine the number of master nodes to use, increasing the node count will speed up the training process.\n",
+        "Parallel run step is leveraged to train multiple models at once. To configure the ParallelRunConfig you will need to determine the appropriate number of workers and nodes for your use case. The ``process_count_per_node`` is based off the number of cores of the compute VM. The node_count will determine the number of master nodes to use, increasing the node count will speed up the training process.\n",
+        "\n",
+        "| Property                           | Description|\n",
+        "| :---------------                   | :------------------- |\n",
+        "| **experiment**                     | The experiment used for training. |\n",
+        "| **train_data**                     | The file dataset to be used as input to the training run. |\n",
+        "| **node_count**                     | The number of compute nodes to be used for running the user script. We recommend to start with 3 and increase the node_count if the training time is taking too long. |\n",
+        "| **process_count_per_node**         | Process count per node, we recommend 2:1 ratio for number of cores: number of processes per node. eg. If node has 16 cores then configure 8 or less process count per node for optimal performance. |\n",
+        "| **train_pipeline_parameters**      | The set of configuration parameters defined in the previous section. |\n",
+        "| **run_invocation_timeout**         | Maximum amount of time in seconds that the ``ParallelRunStep`` class is allowed. This is optional but provides customers with greater control on exit criteria. This must be greater than ``experiment_timeout_hours`` by at least 300 seconds. |\n",
         "\n",
-        "* **experiment:** The experiment used for training.\n",
-        "* **train_data:** The tabular dataset to be used as input to the training run.\n",
-        "* **node_count:** The number of compute nodes to be used for running the user script. We recommend to start with 3 and increase the node_count if the training time is taking too long.\n",
-        "* **process_count_per_node:** Process count per node, we recommend 2:1 ratio for number of cores: number of processes per node. eg. If node has 16 cores then configure 8 or less process count per node or optimal performance.\n",
-        "* **train_pipeline_parameters:** The set of configuration parameters defined in the previous section. \n",
-        "* **run_invocation_timeout:** Maximum amount of time in seconds that the ``ParallelRunStep`` class is allowed. This is optional but provides customers with greater control on exit criteria. This must be greater than ``experiment_timeout_hours`` by at least 300 seconds.\n",
+        "Calling this method will create a new aggregated dataset which is generated dynamically on pipeline execution.\n",
         "\n",
-        "Calling this method will create a new aggregated dataset which is generated dynamically on pipeline execution."
+        "**Note**: Total time taken for the **training step** in the pipeline to complete = $ \\frac{t}{ p \\times n } \\times ts $\n",
+        "where,\n",
+        "- $ t $ is time taken for training one partition (can be viewed in the training logs)\n",
+        "- $ p $ is ``process_count_per_node``\n",
+        "- $ n $ is ``node_count``\n",
+        "- $ ts $ is total number of partitions in time series based on ``partition_column_names``"
       ]
     },
     {
@@ -527,19 +536,24 @@
       "source": [
         "## 5.0 Forecasting\n",
         "For hierarchical forecasting we need to provide the HTSInferenceParameters object.\n",
-        "#### HTSInferenceParameters arguments\n",
-        "* **hierarchy_forecast_level:** The default level of the hierarchy to produce prediction/forecast on.\n",
-        "* **allocation_method:** \\[Optional] The disaggregation method to use if the hierarchy forecast level specified is below the define hierarchy training level. <br><i>(average historical proportions) 'average_historical_proportions'</i><br><i>(proportions of the historical averages) 'proportions_of_historical_average'</i>\n",
-        "\n",
-        "#### get_many_models_batch_inference_steps arguments\n",
-        "* **experiment:** The experiment used for inference run.\n",
-        "* **inference_data:** The data to use for inferencing. It should be the same schema as used for training.\n",
-        "* **compute_target:** The compute target that runs the inference pipeline.\n",
-        "* **node_count:** The number of compute nodes to be used for running the user script. We recommend to start with the number of cores per node (varies by compute sku).\n",
-        "* **process_count_per_node:** The number of processes per node.\n",
-        "* **train_run_id:** \\[Optional] The run id of the hierarchy training, by default it is the latest successful training hts run in the experiment.\n",
-        "* **train_experiment_name:** \\[Optional] The train experiment that contains the train pipeline. This one is only needed when the train pipeline is not in the same experiement as the inference pipeline.\n",
-        "* **process_count_per_node:** \\[Optional] The number of processes per node, by default it's 4."
+        "#### ``HTSInferenceParameters`` arguments\n",
+        "| Property                           | Description|\n",
+        "| :---------------                   | :------------------- |\n",
+        "| **hierarchy_forecast_level:**      | The default level of the hierarchy to produce prediction/forecast on. |\n",
+        "| **allocation_method:**             | \\[Optional] The disaggregation method to use if the hierarchy forecast level specified is below the define hierarchy training level. <br><i>(average historical proportions) 'average_historical_proportions'</i><br><i>(proportions of the historical averages) 'proportions_of_historical_average'</i> |\n",
+        "\n",
+        "#### ``get_many_models_batch_inference_steps`` arguments\n",
+        "| Property                           | Description|\n",
+        "| :---------------                   | :------------------- |\n",
+        "| **experiment**                     | The experiment used for inference run. |\n",
+        "| **inference_data**                 | The data to use for inferencing. It should be the same schema as used for training.\n",
+        "| **compute_target**                 | The compute target that runs the inference pipeline. |\n",
+        "| **node_count**                     | The number of compute nodes to be used for running the user script. We recommend to start with the number of cores per node (varies by compute sku). |\n",
+        "| **process_count_per_node**         | \\[Optional] The number of processes per node. By default it's 2 (should be at most half of the number of cores in a single node of the compute cluster that will be used for the experiment).\n",
+        "| **inference_pipeline_parameters**  | \\[Optional] The ``HTSInferenceParameters`` object defined above. |\n",
+        "| **train_run_id**                   | \\[Optional] The run id of the **training pipeline**. By default it is the latest successful training pipeline run in the experiment. |\n",
+        "| **train_experiment_name**          | \\[Optional] The train experiment that contains the train pipeline. This one is only needed when the train pipeline is not in the same experiement as the inference pipeline. |\n",
+        "| **run_invocation_timeout**         | \\[Optional] Maximum amount of time in seconds that the ``ParallelRunStep`` class is allowed. This is optional but provides customers with greater control on exit criteria. |"
       ]
     },
     {