YOLOv5 CPU Export Benchmarks

[glenn-jocher]

Run YOLOv5 benchmarks on a PyTorch model for all supported export formats. Currently operates on CPU, future updates will implement GPU inference. For GPU benchmarks see #6963.

Format	export.py --include	Model
PyTorch	-	yolov5s.pt
TorchScript	torchscript	yolov5s.torchscript
ONNX	onnx	yolov5s.onnx
OpenVINO	openvino	yolov5s_openvino_model/
TensorRT	engine	yolov5s.engine
CoreML	coreml	yolov5s.mlmodel
TensorFlow SavedModel	saved_model	yolov5s_saved_model/
TensorFlow GraphDef	pb	yolov5s.pb
TensorFlow Lite	tflite	yolov5s.tflite
TensorFlow Edge TPU	edgetpu	yolov5s_edgetpu.tflite
TensorFlow.js	tfjs	yolov5s_web_model/

"""
Requirements:
    $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime openvino-dev tensorflow-cpu  # CPU
    $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime-gpu openvino-dev tensorflow  # GPU

Usage:
    $ python utils/benchmarks.py --weights yolov5s.pt --img 640
"""

Colab Pro+ V100 High-RAM CPU Results

benchmarks: weights=/content/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml, device=cpu, half=False, test=False
Checking setup...
YOLOv5 🚀 v6.1-135-g7926afc torch 1.10.0+cu111 CPU
Setup complete ✅ (8 CPUs, 51.0 GB RAM, 41.5/166.8 GB disk)

Benchmarks complete (241.20s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch        0.4623               127.61
1             TorchScript        0.4623               131.23
2                    ONNX        0.4623                69.34
3                OpenVINO        0.4623                66.52
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel        0.4623               123.79
7     TensorFlow GraphDef        0.4623               121.57
8         TensorFlow Lite        0.4623               316.61
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

MacOS Intel CPU Results (CoreML-capable)

iMac (Retina 5K, 27-inch, 2020) - 3.8 GHz 8-Core Intel Core i7

benchmarks: weights=yolov5s.pt, imgsz=640, batch_size=1, data=/Users/glennjocher/PycharmProjects/yolov5/data/coco128.yaml, device=, half=False, test=False, pt_only=False
Checking setup...
YOLOv5 🚀 v6.1-171-gb4f7fc5 torch 1.10.1 CPU
Setup complete ✅ (16 CPUs, 32.0 GB RAM, 213.4/465.6 GB disk)

Benchmarks complete (276.40s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch        0.4623               222.37
1             TorchScript        0.4623               231.01
2                    ONNX        0.4623                54.41
3                OpenVINO        0.4623                40.74
4                TensorRT           NaN                  NaN
5                  CoreML        0.4620                39.09
6   TensorFlow SavedModel        0.4623               153.32
7     TensorFlow GraphDef        0.4623               148.00
8         TensorFlow Lite        0.4623               162.60
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

iMac (Retina 5K, 27-inch, 2014)

benchmarks: weights=/Users/glennjocher/PycharmProjects/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/Users/glennjocher/PycharmProjects/yolov5/data/coco128.yaml, device=, half=False, test=False
Checking setup...
YOLOv5 🚀 v6.1-135-g7926afc torch 1.11.0 CPU
Setup complete ✅ (8 CPUs, 32.0 GB RAM, 793.4/931.6 GB disk)

Benchmarks complete (288.68s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch        0.4623               281.47
1             TorchScript        0.4623               262.97
2                    ONNX        0.4623                77.30
3                OpenVINO        0.4623                74.12
4                TensorRT           NaN                  NaN
5                  CoreML        0.4620                69.36
6   TensorFlow SavedModel        0.4623               123.12
7     TensorFlow GraphDef        0.4623               120.82
8         TensorFlow Lite        0.4623               249.62
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Ultralytics Hyperplane EPYC Milan AMD CPU Results

benchmarks: weights=yolov5s.pt, imgsz=640, batch_size=1, data=/usr/src/app/data/coco128.yaml, device=, half=False, test=False, pt_only=False
Checking setup...
YOLOv5 🚀 v6.1-174-gc4cb7c6 torch 1.11.0+cpu CPU
Setup complete ✅ (96 CPUs, 1007.7 GB RAM, 1948.4/3519.3 GB disk)

Benchmarks complete (128.39s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch        0.4623                56.52
1             TorchScript        0.4623                59.21
2                    ONNX        0.4623                60.95
3                OpenVINO        0.4623                27.44
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel        0.4623                70.84
7     TensorFlow GraphDef        0.4623                72.44
8         TensorFlow Lite        0.4623               130.43
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Resolves #6586

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

This PR introduces an enhanced export functionality, streamlines model type handling within the codebase, and adds a new benchmarking utility.

📊 Key Changes

• Added pandas library to export.py for dataframe support.
• Created export_formats function to list supported export formats in a dataframe.
• Simplified suffix checking in models/common.py by utilizing the model_type method, which relies on the new export formats dataframe.
• Ensured OpenVINO export handles directories correctly to find .xml files.
• Introduced benchmarks.py, a new utility script to benchmark YOLOv5 across various supported export formats.
• Updated val.py to support a 'benchmark' task, which affects image padding and inference shape handling.

🎯 Purpose & Impact

• The addition of export_formats function and dataframe usage aims to centralize and organize the different supported export formats, making it easier to manage them.
• Streamlining the model type determination process simplifies the way the code interprets and works with various model file types.
• Introducing benchmarks for YOLOv5 models offers users a means to measure and compare performance across different formats, which can help in choosing the right format for deployment.
• The option to alter image padding and the inference shape for benchmarks may provide more accurate and comparative speed measurements for the models.

Overall, these changes enhance the usability of the YOLOv5 export options and provide valuable insights into model performance, potentially benefiting developers and users focused on optimization and deployment of models in different environments. 🚀