Update distributed pytorch notebook

Author: rastala

how-to-use-azureml/training-with-deep-learning/distributed-pytorch-with-horovod/distributed-pytorch-with-horovod.ipynb

@@ -14,19 +14,16 @@
    "metadata": {},
    "source": [
     "# Distributed PyTorch with Horovod\n",
-    "In this tutorial, you will train a PyTorch model on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using distributed training via [Horovod](https://github.com/uber/horovod)."
+    "In this tutorial, you will train a PyTorch model on the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset using distributed training via [Horovod](https://github.com/uber/horovod) across a GPU cluster."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Prerequisites\n",
-    "* Understand the [architecture and terms](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture) introduced by Azure Machine Learning (AML)\n",
-    "* Go through the [00.configuration.ipynb](https://github.com/Azure/MachineLearningNotebooks/blob/master/00.configuration.ipynb) notebook to:\n",
-    "    * install the AML SDK\n",
-    "    * create a workspace and its configuration file (`config.json`)\n",
-    "* Review the [tutorial](https://aka.ms/aml-notebook-pytorch) on single-node PyTorch training using the SDK"
+    "* Go through the [Configuration](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) notebook to install the Azure Machine Learning Python SDK and create an Azure ML `Workspace`\n",
+    "* Review the [tutorial](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/training-with-deep-learning/train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.ipynb) on single-node PyTorch training using Azure Machine Learning"
    ]
   },
   {
@@ -92,10 +89,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Create a remote compute target\n",
-    "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) to execute your training script on. In this tutorial, you create an `AmlCompute` cluster as your training compute resource. This code creates a cluster for you if it does not already exist in your workspace.\n",
+    "## Create or attach existing AmlCompute\n",
+    "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, we use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training compute resource. Specifically, the below code creates an `STANDARD_NC6` GPU cluster that autoscales from `0` to `4` nodes.\n",
     "\n",
-    "**Creation of the cluster takes approximately 5 minutes.** If the cluster is already in your workspace this code will skip the cluster creation process."
+    "**Creation of AmlCompute takes approximately 5 minutes.** If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
+    "\n",
+    "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
    ]
   },
   {
@@ -115,31 +114,31 @@
     "    print('Found existing compute target.')\n",
     "except ComputeTargetException:\n",
     "    print('Creating a new compute target...')\n",
-    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
+    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6',\n",
     "                                                           max_nodes=4)\n",
     "\n",
     "    # create the cluster\n",
     "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
     "\n",
     "    compute_target.wait_for_completion(show_output=True)\n",
     "\n",
-    "# Use the 'status' property to get a detailed status for the current cluster. \n",
+    "# Use the 'status' property to get a detailed status for the current AmlCompute. \n",
     "print(compute_target.status.serialize())"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The above code creates a GPU cluster. If you instead want to create a CPU cluster, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
+    "The above code creates GPU compute. If you instead want to create CPU compute, provide a different VM size to the `vm_size` parameter, such as `STANDARD_D2_V2`."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Train model on the remote compute\n",
-    "Now that we have the cluster ready to go, let's run our distributed training job."
+    "Now that we have the AmlCompute ready to go, let's run our distributed training job."
    ]
   },
   {
@@ -166,7 +165,27 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Copy the training script `pytorch_horovod_mnist.py` into this project directory."
+    "### Prepare training script\n",
+    "Now you will need to create your training script. In this tutorial, the script for distributed training of MNIST is already provided for you at `pytorch_horovod_mnist.py`. In practice, you should be able to take any custom PyTorch training script as is and run it with Azure ML without having to modify your code.\n",
+    "\n",
+    "However, if you would like to use Azure ML's [metric logging](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#logging) capabilities, you will have to add a small amount of Azure ML logic inside your training script. In this example, at each logging interval, we will log the loss for that minibatch to our Azure ML run.\n",
+    "\n",
+    "To do so, in `pytorch_horovod_mnist.py`, we will first access the Azure ML `Run` object within the script:\n",
+    "```Python\n",
+    "from azureml.core.run import Run\n",
+    "run = Run.get_context()\n",
+    "```\n",
+    "Later within the script, we log the loss metric to our run:\n",
+    "```Python\n",
+    "run.log('loss', loss.item())\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Once your script is ready, copy the training script `pytorch_horovod_mnist.py` into the project directory."
    ]
   },
   {
@@ -205,7 +224,7 @@
    "metadata": {},
    "source": [
     "### Create a PyTorch estimator\n",
-    "The AML SDK's PyTorch estimator enables you to easily submit PyTorch training jobs for both single-node and distributed runs. For more information on the PyTorch estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-pytorch)."
+    "The Azure ML SDK's PyTorch estimator enables you to easily submit PyTorch training jobs for both single-node and distributed runs. For more information on the PyTorch estimator, refer [here](https://docs.microsoft.com/azure/machine-learning/service/how-to-train-pytorch)."
    ]
   },
   {
@@ -232,6 +251,26 @@
     "The above code specifies that we will run our training script on `2` nodes, with one worker per node. In order to execute a distributed run using MPI/Horovod, you must provide the argument `distributed_backend='mpi'`. Using this estimator with these settings, PyTorch, Horovod and their dependencies will be installed for you. However, if your script also uses other packages, make sure to install them via the `PyTorch` constructor's `pip_packages` or `conda_packages` parameters."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To use the latest version of PyTorch 1.0, run the following cell:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "estimator.conda_dependencies.remove_conda_package('pytorch=0.4.0')\n",
+    "estimator.conda_dependencies.remove_pip_package('horovod==0.13.11')\n",
+    "estimator.conda_dependencies.add_conda_package('pytorch-nightly')\n",
+    "estimator.conda_dependencies.add_channel('pytorch')\n",
+    "estimator.conda_dependencies.add_pip_package('horovod==0.15.2')"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -255,7 +294,7 @@
    "metadata": {},
    "source": [
     "### Monitor your run\n",
-    "You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes."
+    "You can monitor the progress of the run with a Jupyter widget. Like the run submission, the widget is asynchronous and provides live updates every 10-15 seconds until the job completes. You can see that the widget automatically plots and visualizes the loss metric that we logged to the Azure ML run."
    ]
   },
   {

how-to-use-azureml/training-with-deep-learning/distributed-pytorch-with-horovod/pytorch_horovod_mnist.py

@@ -1,17 +1,16 @@
-# Copyright 2017 Uber Technologies, Inc.
-# Licensed under the Apache License, Version 2.0
-# Script from horovod/examples: https://github.com/uber/horovod/blob/master/examples/pytorch_mnist.py
-
 from __future__ import print_function
 import argparse
 import torch.nn as nn
 import torch.nn.functional as F
 import torch.optim as optim
 from torchvision import datasets, transforms
-from torch.autograd import Variable
 import torch.utils.data.distributed
 import horovod.torch as hvd
 
+from azureml.core.run import Run
+# get the Azure ML run object
+run = Run.get_context()
+
 # Training settings
 parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
 parser.add_argument('--batch-size', type=int, default=64, metavar='N',
@@ -30,6 +29,8 @@
                     help='random seed (default: 42)')
 parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                     help='how many batches to wait before logging training status')
+parser.add_argument('--fp16-allreduce', action='store_true', default=False,
+                    help='use fp16 compression during allreduce')
 args = parser.parse_args()
 args.cuda = not args.no_cuda and torch.cuda.is_available()
 
@@ -97,9 +98,13 @@ def forward(self, x):
 optimizer = optim.SGD(model.parameters(), lr=args.lr * hvd.size(),
                       momentum=args.momentum)
 
+# Horovod: (optional) compression algorithm.
+compression = hvd.Compression.fp16 if args.fp16_allreduce else hvd.Compression.none
+
 # Horovod: wrap optimizer with DistributedOptimizer.
-optimizer = hvd.DistributedOptimizer(
-    optimizer, named_parameters=model.named_parameters())
+optimizer = hvd.DistributedOptimizer(optimizer,
+                                     named_parameters=model.named_parameters(),
+                                     compression=compression)
 
 
 def train(epoch):
@@ -108,7 +113,6 @@ def train(epoch):
     for batch_idx, (data, target) in enumerate(train_loader):
         if args.cuda:
             data, target = data.cuda(), target.cuda()
-        data, target = Variable(data), Variable(target)
         optimizer.zero_grad()
         output = model(data)
         loss = F.nll_loss(output, target)
@@ -117,13 +121,16 @@ def train(epoch):
         if batch_idx % args.log_interval == 0:
             print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                 epoch, batch_idx * len(data), len(train_sampler),
-                100. * batch_idx / len(train_loader), loss.data[0]))
+                100. * batch_idx / len(train_loader), loss.item()))
+
+            # log the loss to the Azure ML run
+            run.log('loss', loss.item())
 
 
 def metric_average(val, name):
-    tensor = torch.FloatTensor([val])
+    tensor = torch.tensor(val)
     avg_tensor = hvd.allreduce(tensor, name=name)
-    return avg_tensor[0]
+    return avg_tensor.item()
 
 
 def test():
@@ -133,10 +140,9 @@ def test():
     for data, target in test_loader:
         if args.cuda:
             data, target = data.cuda(), target.cuda()
-        data, target = Variable(data, volatile=True), Variable(target)
         output = model(data)
         # sum up batch loss
-        test_loss += F.nll_loss(output, target, size_average=False).data[0]
+        test_loss += F.nll_loss(output, target, size_average=False).item()
         # get the index of the max log-probability
         pred = output.data.max(1, keepdim=True)[1]
         test_accuracy += pred.eq(target.data.view_as(pred)).cpu().float().sum()