Merge pull request Azure#19 from rastala/master

Updates to onnx notebooks

Author: rastala

onnx/onnx-inference-emotion-recognition.ipynb

@@ -12,13 +12,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# 01. Handwritten Digit Classification (MNIST) using ONNX Runtime on AzureML\n",
+    "# Handwritten Digit Classification (MNIST) using ONNX Runtime on AzureML\n",
     "\n",
     "This example shows how to deploy an image classification neural network using the Modified National Institute of Standards and Technology ([MNIST](http://yann.lecun.com/exdb/mnist/)) dataset and Open Neural Network eXchange format ([ONNX](http://aka.ms/onnxdocarticle)) on the Azure Machine Learning platform. MNIST is a popular dataset consisting of 70,000 grayscale images. Each image is a handwritten digit of 28x28 pixels, representing number from 0 to 9. This tutorial will show you how to deploy a MNIST model from the [ONNX model zoo](https://github.com/onnx/models), use it to make predictions using ONNX Runtime Inference, and deploy it as a web service in Azure.\n",
     "\n",
     "Throughout this tutorial, we will be referring to ONNX, a neural network exchange format used to represent deep learning models. With ONNX, AI developers can more easily move models between state-of-the-art tools (CNTK, PyTorch, Caffe, MXNet, TensorFlow) and choose the combination that is best for them. ONNX is developed and supported by a community of partners including Microsoft AI, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai) and [open source files](https://github.com/onnx).\n",
     "\n",
-    "[ONNX Runtime](https://aka.ms/onnxruntime) is the runtime engine that enables evaluation of trained machine learning (Traditional ML and Deep Learning) models with high performance and low resource utilization.\n",
+    "[ONNX Runtime](https://aka.ms/onnxruntime-python) is the runtime engine that enables evaluation of trained machine learning (Traditional ML and Deep Learning) models with high performance and low resource utilization.\n",
     "\n",
     "#### Tutorial Objectives:\n",
     "\n",
@@ -36,12 +36,11 @@
     "### 1. Install Azure ML SDK and create a new workspace\n",
     "Please follow [00.configuration.ipynb](https://github.com/Azure/MachineLearningNotebooks/blob/master/00.configuration.ipynb) notebook.\n",
     "\n",
-    "\n",
     "### 2. Install additional packages needed for this Notebook\n",
-    "You need to install the popular plotting library `matplotlib` and the `onnx` library in the conda environment where Azure Maching Learning SDK is installed.\n",
+    "You need to install the popular plotting library `matplotlib`, the image manipulation library `opencv`, and the `onnx` library in the conda environment where Azure Maching Learning SDK is installed.\n",
     "\n",
     "```sh\n",
-    "(myenv) $ pip install matplotlib onnx\n",
+    "(myenv) $ pip install matplotlib onnx opencv-python\n",
     "```\n",
     "\n",
     "### 3. Download sample data and pre-trained ONNX model from ONNX Model Zoo.\n",
@@ -222,9 +221,11 @@
     "\n",
     "\n",
     "def init():\n",
-    "    global session\n",
+    "    global session, input_name, output_name\n",
     "    model = Model.get_model_path(model_name = 'mnist_1')\n",
     "    session = onnxruntime.InferenceSession(model, None)\n",
+    "    input_name = session.get_inputs()[0].name\n",
+    "    output_name = session.get_outputs()[0].name \n",
     "    \n",
     "def run(input_data):\n",
     "    '''Purpose: evaluate test input in Azure Cloud using onnxruntime.\n",
@@ -233,14 +234,14 @@
     "\n",
     "    try:\n",
     "        # load in our data, convert to readable format\n",
-    "        start = time.time()\n",
     "        data = np.array(json.loads(input_data)['data']).astype('float32')\n",
     "\n",
-    "        r = session.run([\"Plus214_Output_0\"], {\"Input3\": data})[0]\n",
-    "        result = choose_class(r[0])\n",
+    "        start = time.time()\n",
+    "        r = session.run([output_name], {input_name: data})[0]\n",
     "        end = time.time()\n",
-    "        result_dict = {\"result\": np.array(result).tolist(),\n",
-    "                      \"time\": np.array(end - start).tolist()}\n",
+    "        result = choose_class(r[0])\n",
+    "        result_dict = {\"result\": [result],\n",
+    "                      \"time_in_sec\": [end - start]}\n",
     "    except Exception as e:\n",
     "        result_dict = {\"error\": str(e)}\n",
     "    \n",
@@ -304,8 +305,7 @@
     "                                                  runtime = \"python\",\n",
     "                                                  conda_file = \"myenv.yml\",\n",
     "                                                  description = \"test\",\n",
-    "                                                  tags = {\"demo\": \"onnx\"} \n",
-    "                                                 )\n",
+    "                                                  tags = {\"demo\": \"onnx\"})                                                 )\n",
     "\n",
     "\n",
     "image = ContainerImage.create(name = \"onnxtest\",\n",
@@ -533,12 +533,12 @@
     "    # predict using the deployed model\n",
     "    r = json.loads(aci_service.run(input_data))\n",
     "    \n",
-    "    if len(r) == 1:\n",
+    "    if \"error\" in r:\n",
     "        print(r['error'])\n",
     "        break\n",
     "        \n",
-    "    result = r['result']\n",
-    "    time_ms = np.round(r['time'] * 1000, 2)\n",
+    "    result = r['result'][0]\n",
+    "    time_ms = np.round(r['time_in_sec'][0] * 1000, 2)\n",
     "    \n",
     "    ground_truth = int(np.argmax(test_outputs[i]))\n",
     "    \n",
@@ -570,7 +570,7 @@
    "source": [
     "### Try classifying your own images!\n",
     "\n",
-    "Create your own 28 pixel by 28 pixel handwritten image and pass it into the model."
+    "Create your own handwritten image and pass it into the model."
    ]
   },
   {
@@ -580,18 +580,22 @@
    "outputs": [],
    "source": [
     "# Preprocessing functions\n",
+    "import cv2\n",
     "\n",
     "def rgb2gray(rgb):\n",
     "    \"\"\"Convert the input image into grayscale\"\"\"\n",
     "    return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])\n",
     "\n",
+    "def resize_img(img):\n",
+    "    img = cv2.resize(img, dsize=(28, 28), interpolation=cv2.INTER_AREA)\n",
+    "    img.resize((1, 1, 28, 28))\n",
+    "    return img\n",
+    "\n",
     "def preprocess(img):\n",
     "    \"\"\"Resize input images and convert them to grayscale.\"\"\"\n",
-    "    if img.shape[0] != 28:\n",
-    "        print(\"Input image size is not 28 * 28 pixels. Please resize and try again.\")\n",
     "    grayscale = rgb2gray(img)\n",
-    "    grayscale.resize((1, 1, 28, 28))\n",
-    "    return grayscale"
+    "    processed_img = resize_img(grayscale)\n",
+    "    return processed_img"
    ]
   },
   {
@@ -601,7 +605,7 @@
    "outputs": [],
    "source": [
     "# Replace this string with your own path/test image\n",
-    "# Make sure the dimensions are 28 * 28 pixels\n",
+    "# Make sure your image is square and the dimensions are equal (i.e. 100 * 100 pixels or 28 * 28 pixels)\n",
     "\n",
     "# Any PNG or JPG image file should work\n",
     "# Make sure to include the entire path with // instead of /\n",
@@ -636,10 +640,10 @@
     "\n",
     "    try:\n",
     "        r = json.loads(aci_service.run(input_data))\n",
-    "        result = r['result']\n",
-    "        time_ms = np.round(r['time'] * 1000, 2)\n",
+    "        result = r['result'][0]\n",
+    "        time_ms = np.round(r['time_in_sec'][0] * 1000, 2)\n",
     "    except Exception as e:\n",
-    "        print(str(e), r['error'])\n",
+    "        print(str(e))\n",
     "\n",
     "    plt.figure(figsize = (16, 6))\n",
     "    plt.subplot(1, 15,1)\n",
@@ -657,7 +661,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Optional: How does our MNIST model work? \n",
+    "## Optional: How does our  ONNX MNIST model work? \n",
     "#### A brief explanation of Convolutional Neural Networks\n",
     "\n",
     "A [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN, or ConvNet) is a type of [feed-forward](https://en.wikipedia.org/wiki/Feedforward_neural_network) artificial neural network made up of neurons that have learnable weights and biases. The CNNs take advantage of the spatial nature of the data. In nature, we perceive different objects by their shapes, size and colors. For example, objects in a natural scene are typically edges, corners/vertices (defined by two of more edges), color patches etc. These primitives are often identified using different detectors (e.g., edge detection, color detector) or combination of detectors interacting to facilitate image interpretation (object classification, region of interest detection, scene description etc.) in real world vision related tasks. These detectors are also known as filters. Convolution is a mathematical operator that takes an image and a filter as input and produces a filtered output (representing say edges, corners, or colors in the input image).  \n",
@@ -709,103 +713,13 @@
     "![](http://www.cntk.ai/jup/conv103d_mnist-conv-mp.png)"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Try classifying your own images!\n",
-    "\n",
-    "Create your own 28 pixel by 28 pixel handwritten image and pass it into the model."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Preprocessing functions\n",
-    "\n",
-    "def rgb2gray(rgb):\n",
-    "    \"\"\"Convert the input image into grayscale\"\"\"\n",
-    "    return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])\n",
-    "\n",
-    "def preprocess(img):\n",
-    "    \"\"\"Resize input images and convert them to grayscale.\"\"\"\n",
-    "    if img.shape[0] != 28:\n",
-    "        print(\"Input image size is not 28 * 28 pixels. Please resize and try again.\")\n",
-    "    grayscale = rgb2gray(img)\n",
-    "    grayscale.resize((1, 1, 28, 28))\n",
-    "    return grayscale"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Replace this string with your own path/test image\n",
-    "# Make sure the dimensions are 28 * 28 pixels\n",
-    "\n",
-    "# Any PNG or JPG image file should work\n",
-    "# Make sure to include the entire path with // instead of /\n",
-    "\n",
-    "# e.g. your_test_image = \"C://Users//vinitra.swamy//Pictures//digit.png\"\n",
-    "\n",
-    "your_test_image = \"<path to file>\"\n",
-    "\n",
-    "import matplotlib.image as mpimg\n",
-    "\n",
-    "if your_test_image != \"<path to file>\":\n",
-    "    img = mpimg.imread(your_test_image)\n",
-    "    plt.subplot(1,3,1)\n",
-    "    plt.imshow(img, cmap = plt.cm.Greys)\n",
-    "    print(\"Old Dimensions: \", img.shape)\n",
-    "    img = preprocess(img)\n",
-    "    print(\"New Dimensions: \", img.shape)\n",
-    "else:\n",
-    "    img = None"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "if img is None:\n",
-    "    print(\"Add the path for your image data.\")\n",
-    "else:\n",
-    "    input_data = json.dumps({'data': img.tolist()})\n",
-    "\n",
-    "    try:\n",
-    "        r = json.loads(aci_service.run(input_data))\n",
-    "        result = r['result']\n",
-    "        time_ms = np.round(r['time'] * 1000, 2)\n",
-    "    except Exception as e:\n",
-    "        print(str(e), r['error'])\n",
-    "\n",
-    "    plt.figure(figsize = (16, 6))\n",
-    "    plt.subplot(1, 15,1)\n",
-    "    plt.axhline('')\n",
-    "    plt.axvline('')\n",
-    "    plt.text(x = -100, y = -20, s = \"Model prediction: \", fontsize = 14)\n",
-    "    plt.text(x = -100, y = -10, s = \"Inference time: \", fontsize = 14)\n",
-    "    plt.text(x = 0, y = -20, s = str(result), fontsize = 14)\n",
-    "    plt.text(x = 0, y = -10, s = str(time_ms) + \" ms\", fontsize = 14)\n",
-    "    plt.text(x = -100, y = 14, s = \"Input image: \", fontsize = 14)\n",
-    "    plt.imshow(img.reshape(28, 28), cmap = plt.cm.gray)    "
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "# remember to delete your service after you are done using it!\n",
-    "# uncomment the following line of code to delete your service\n",
     "\n",
     "# aci_service.delete()"
    ]
@@ -818,10 +732,10 @@
     "\n",
     "Congratulations!\n",
     "\n",
-    "In this tutorial, you have managed to:\n",
-    "- familiarize yourself with the ONNX model format, ONNX Runtime inference, and the pretrained models in the ONNX model zoo\n",
-    "- understand a state-of-the-art convolutional neural net image classification model (MNIST in ONNX) and deploy it in the Azure ML cloud\n",
-    "- ensure that your deep learning model is working perfectly (in the cloud) on test data, and check it against some of your own!\n",
+    "In this tutorial, you have:\n",
+    "- familiarized yourself with ONNX Runtime inference and the pretrained models in the ONNX model zoo\n",
+    "- understood a state-of-the-art convolutional neural net image classification model (MNIST in ONNX) and deployed it in Azure ML cloud\n",
+    "- ensured that your deep learning model is working perfectly (in the cloud) on test data, and checked it against some of your own!\n",
     "\n",
     "Next steps:\n",
     "- Check out another interesting application based on a Microsoft Research computer vision paper that lets you set up a [facial emotion recognition model](https://github.com/Azure/MachineLearningNotebooks/tree/master/onnx/onnx-inference-emotion-recognition.ipynb) in the cloud! This tutorial deploys a pre-trained ONNX Computer Vision model in an Azure ML virtual machine with GPU support.\n",
@@ -831,9 +745,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.6",
+   "display_name": "Python 3",
    "language": "python",
-   "name": "python36"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -845,7 +759,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.6"
+   "version": "3.6.5"
   },
   "msauthor": "vinitra.swamy"
  },