|
| 1 | +# Azure Machine Learning Batch Inference |
| 2 | + |
| 3 | +Azure Machine Learning Batch Inference targets large inference jobs that are not time-sensitive. Batch Inference provides cost-effective inference compute scaling, with unparalleled throughput for asynchronous applications. It is optimized for high-throughput, fire-and-forget inference over large collections of data. |
| 4 | + |
| 5 | +# Getting Started with Batch Inference Public Preview |
| 6 | + |
| 7 | +Batch inference public preview offers a platform in which to do large inference or generic parallel map-style operations. Below introduces the major steps to use this new functionality. For a quick try, please follow the prerequisites and simply run the sample notebooks provided in this directory. |
| 8 | + |
| 9 | +## Prerequisites |
| 10 | + |
| 11 | +### Python package installation |
| 12 | +Following the convention of most AzureML Public Preview features, Batch Inference SDK is currently available as a contrib package. |
| 13 | + |
| 14 | +If you're unfamiliar with creating a new Python environment, you may follow this example for [creating a conda environment](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-environment#local). Batch Inference package can be installed through the following pip command. |
| 15 | +``` |
| 16 | +pip install azureml-contrib-pipeline-steps |
| 17 | +``` |
| 18 | + |
| 19 | +### Creation of Azure Machine Learning Workspace |
| 20 | +If you do not already have a Azure ML Workspace, please run the [configuration Notebook](../../configuration.ipynb). |
| 21 | + |
| 22 | +## Configure a Batch Inference job |
| 23 | + |
| 24 | +To run a Batch Inference job, you will need to gather some configuration data. |
| 25 | + |
| 26 | +1. **ParallelRunConfig** |
| 27 | + - **entry_script**: the local file path to the scoring script. If source_directory is specified, use relative path, otherwise use any path accessible on machine. |
| 28 | + - **error_threshold**: the number of record failures for TabularDataset and file failures for FileDataset that should be ignored during processing. If the aggregated error count (across all workers) goes above this value, then the job will be aborted. Set to -1 to ignore all failures during processing. |
| 29 | + - **output_action**: one of the following values |
| 30 | + - **"append_row"**: all values output by run() method invocations will be aggregated into one unique file named parallel_run_step.txt that is created in the output location. |
| 31 | + - **"summary_only"** – scoring script will handle the output by itself. The script still needs to return one output row per successfully-processed input item. This is used for error threshold calculation (the actual value of the output row is ignored). |
| 32 | + - **source_directory**: supporting files for scoring (optional) |
| 33 | + - **compute_target**: only **AmlCompute** is supported currently |
| 34 | + - **node_count**: number of compute nodes to use. |
| 35 | + - **process_count_per_node**: number of processes per node (optional, default value is 1). |
| 36 | + - **mini_batch_size**: the approximate amount of input data passed to each run() invocation. For FileDataset input, this is number of files user script can process in one run() call. For TabularDataset input it is approximate size of data user script can process in one run() call. E.g. 1024, 1024KB, 10MB, 1GB (optional, default value 10 files for FileDataset and 1MB for TabularDataset.) |
| 37 | + - **logging_level**: log verbosity. Values in increasing verbosity are: 'WARNING', 'INFO', 'DEBUG' (optional, default value is 'INFO'). |
| 38 | + - **run_invocation_timeout**: run method invocation timeout period in seconds (optional, default value is 60). |
| 39 | + - **environment**: The environment definition. This field configures the Python environment. It can be configured to use an existing Python environment or to set up a temp environment for the experiment. The definition is also responsible for setting the required application dependencies. |
| 40 | + - **description**: name given to batch service. |
| 41 | + |
| 42 | +2. **Scoring (entry) script**: entry point for execution, scoring script should contain two functions: |
| 43 | + - **init()**: this function should be used for any costly or common preparation for subsequent inferences, e.g., deserializing and loading the model into a global object. |
| 44 | + - **run(mini_batch)**: The method to be parallelized. Each invocation will have one minibatch. |
| 45 | + - **mini_batch**: Batch inference will invoke run method and pass either a list or Pandas DataFrame as an argument to the method. Each entry in min_batch will be - a filepath if input is a FileDataset, a Pandas DataFrame if input is a TabularDataset. |
| 46 | + - **return value**: run() method should return a Pandas DataFrame or an array. For append_row output_action, these returned elements are appended into the common output file. For summary_only, the contents of the elements are ignored. For all output actions, each returned output element indicates one successful inference of input element in the input mini-batch. |
| 47 | + |
| 48 | +3. **Base image** (optional) |
| 49 | + - if GPU is required, use DEFAULT_GPU_IMAGE as base image in environment. [Example GPU environment](./file-dataset-image-inference-mnist.ipynb#specify-the-environment-to-run-the-script) |
| 50 | + |
| 51 | +Example image pull: |
| 52 | +```python |
| 53 | +from azureml.core.runconfig import ContainerRegistry |
| 54 | + |
| 55 | +# use an image available in public Container Registry without authentication |
| 56 | +public_base_image = "mcr.microsoft.com/azureml/o16n-sample-user-base/ubuntu-miniconda" |
| 57 | + |
| 58 | +# or use an image available in a private Container Registry |
| 59 | +base_image = "myregistry.azurecr.io/mycustomimage:1.0" |
| 60 | +base_image_registry = ContainerRegistry() |
| 61 | +base_image_registry.address = "myregistry.azurecr.io" |
| 62 | +base_image_registry.username = "username" |
| 63 | +base_image_registry.password = "password" |
| 64 | +``` |
| 65 | + |
| 66 | + |
| 67 | +## Create a batch inference job |
| 68 | + |
| 69 | +**ParallelRunStep** is a newly added step in the azureml.contrib.pipeline.steps package. You will use it to add a step to create a batch inference job with your Azure machine learning pipeline. (Use batch inference without an Azure machine learning pipeline is not supported yet). ParallelRunStep has all the following parameters: |
| 70 | + - **name**: this name will be used to register batch inference service, has the following naming restrictions: (unique, 3-32 chars and regex ^\[a-z\]([-a-z0-9]*[a-z0-9])?$) |
| 71 | + - **models**: zero or more model names already registered in Azure Machine Learning model registry. |
| 72 | + - **parallel_run_config**: ParallelRunConfig as defined above. |
| 73 | + - **inputs**: one or more Dataset objects. |
| 74 | + - **output**: this should be a PipelineData object encapsulating an Azure BLOB container path. |
| 75 | + - **arguments**: list of custom arguments passed to scoring script (optional) |
| 76 | + - **allow_reuse**: optional, default value is True. If the inputs remain the same as a previous run, it will make the previous run results immediately available (skips re-computing the step). |
| 77 | + |
| 78 | +## Passing arguments from pipeline submission to script |
| 79 | + |
| 80 | +Many tasks require arguments to be passed from job submission to the distributed runs. Below is an example to pass such information. |
| 81 | +``` |
| 82 | +# from script which creates pipeline job |
| 83 | +parallelrun_step = ParallelRunStep( |
| 84 | + ... |
| 85 | + arguments=["--model_name", "mosaic"] # name of the model we want to use, in case we have more than one option |
| 86 | +) |
| 87 | +``` |
| 88 | +``` |
| 89 | +# from driver.py/score.py/task.py |
| 90 | +import argparse |
| 91 | +
|
| 92 | +parser.add_argument('--model_name', dest="model_name") |
| 93 | +
|
| 94 | +args, unknown_args = parser.parse_known_args() |
| 95 | +
|
| 96 | +# to access values |
| 97 | +args.model_name # "mosaic" |
| 98 | +``` |
| 99 | + |
| 100 | +## Submit a batch inference job |
| 101 | + |
| 102 | +You can submit a batch inference job by pipeline_run, or through REST calls with a published pipeline. To control node count using REST API/experiment, please use aml_node_count(special) pipeline parameter. A typical use case follows: |
| 103 | + |
| 104 | +```python |
| 105 | +pipeline = Pipeline(workspace=ws, steps=[parallelrun_step]) |
| 106 | +pipeline_run = Experiment(ws, 'name_of_pipeline_run').submit(pipeline) |
| 107 | +``` |
| 108 | + |
| 109 | +## Monitor your batch inference job |
| 110 | + |
| 111 | +A batch inference job can take a long time to finish. You can monitor your job's progress from Azure portal, using Azure ML widgets, view console output through SDK, or check out overview.txt in log/azureml directory. |
| 112 | + |
| 113 | +```python |
| 114 | +# view with widgets (will display GUI inside a browser) |
| 115 | +from azureml.widgets import RunDetails |
| 116 | +RunDetails(pipeline_run).show() |
| 117 | + |
| 118 | +# simple console output |
| 119 | +pipeline_run.wait_for_completion(show_output=True) |
| 120 | +``` |
| 121 | + |
| 122 | +# Sample notebooks |
| 123 | + |
| 124 | +- [file-dataset-image-inference-mnist.ipynb](./file-dataset-image-inference-mnist.ipynb) demonstrates how to run batch inference on an MNIST dataset. |
| 125 | +- [tabular-dataset-inference-iris.ipynb](./tabular-dataset-inference-iris.ipynb) demonstrates how to run batch inference on an IRIS dataset. |
| 126 | + |
| 127 | + |
0 commit comments