How to deploy a PyTorch model in minutes

This tutorial will cover a fast and straightforward way to deploy a PyTorch model at scale without a docker and Kubernetes setup.

I've trained my model. What's next?

Have you ever wanted to deploy a Pytorch model to run it on a website or some mobile app? It is not an easy task as you probably noticed.

In the past, I was exactly in such a situation, trying different tools and resources to serve a sample Machine Learning model in the form of API. Meanwhile, I've experienced that most Machine Learning Deployment tutorials' primary focus is building a simple flask app, creating a docker, and setting up the cloud infrastructure. However, it is still time-consuming and challenging. Nevertheless, hundreds of hours spent building a simple one-page demo app that takes an image as an input and returns one word: Cat.

Since that time, I've managed to find a way to solve that problem in just three easy steps.

Let's look and deploy a PyTorch model (Check also How to deploy Keras model).

Step 1: Develop a model

In the first step, we need to have a trained model. For this purpose, we will use a pre-trained PyTorch YoloV5.

For those who don't know what, Yolo is a real-time object detection framework and stands for You Only Look Once, which means that the image passes only once through the Fully Convolutional Neural Network. Compared to previous versions, the V5 is much faster and smaller (only 27Mb).

In order to run the algorithm locally, you can copy and paste a script from PyTorch Hub. The Torch library will automatically download a pre-trained model and sample images to perform the inference of a model.

import cv2
import torch
from PIL import Image

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True).fuse().autoshape()  # for PIL/cv2/np inputs and NMS

# Images
for f in ['zidane.jpg', 'bus.jpg']:  # download 2 images
    print(f'Downloading {f}...')
    torch.hub.download_url_to_file('https://github.com/ultralytics/yolov5/releases/download/v1.0/' + f, f)
img1 = Image.open('zidane.jpg')  # PIL image
img2 = cv2.imread('bus.jpg')[:, :, ::-1]  # OpenCV image (BGR to RGB)
imgs = [img1, img2]  # batched list of images

# Inference
results = model(imgs, size=640)  # includes NMS

# Results
results.show()  # .show() results, .save() jpgs, or .print() to scree

Step 2: Deploy a Pytorch model

We have our basic script with the model, so now we can deploy it in the cloud.

For this purpose, we will use the Syndicai Platform, which helps developers deploy the AI model to production effortlessly without setting up an infrastructure. In addition, it takes care of security, scalability, monitoring. You don’t need to set up or configure anything. You need to prepare a model and connect your git repository and later the model will be deployed automatically. In the first place, we will prepare a model, then we will connect a git repository to the platform in order to finally deploy a pytorch model by creating the API.

Prepare a model

During the model preparation process, you need to ensure the following files are in the git repository:

1st file: requirements.txt - file with all libraries and frameworks required to recreate model's environment.

opencv-python==4.2.0.34
pillow
pyyaml
tqdm
torch
matplotlib
torchvision
scipy

2nd file: syndicai.py - main file with the PythonPredictor python class responsible for model prediction.

import torch
from PIL import Image 
from helpers import draw_box, url_to_img, img_to_bytes


class PythonPredictor:

    def __init__(self, config):
        """ Download pretrained model. """
        self.model = torch.hub.load('ultralytics/yolov5', 'yolov5s', pretrained=True).autoshape()

    def predict(self, payload):
        """ Run a model based on url input. """

        # Inference
        img = url_to_img(payload["url"])
        results = self.model(img)

        # Draw boxes
        boxes = results.xyxy[0].numpy()
        box_img = draw_box(img, boxes)

        # Save image
        #box_img.save("sample_data/output.png", "PNG")

        return img_to_bytes(box_img)

Those files are necessary to correctly build recreate the environment, build the infrastructure, and finally run the model in the form of a webservice. To illustrate that you can check with the Syndicai model repository.

Connect a repository

When the model is ready, it's time for deployment. Therefore, the only thing you need to do is log in to Syndicai Platform, go to the Models page and click New Model. Then, fill the form with the repository URL and path to the model (see the picture below).

Later click Add, and you will be redirected to Model Profile.

Create a deployment

Finally, in the Model Profile, click the Deploy button. Please fill the form with the name of the deployment (e.g., Production 1) and the name of the branch (in our case, it's master).

Later confirm the form by clicking Add and the Platform will create the new deployment with the first release. Afterward go to the Releases tab, and click on the recent one #1 to explore logs from the building and starting process.

In order to get the Running status of the deployment waits for a couple of minutes.

Step 3: Integrate

Great! We have our scalable API, so theoretically, the job is done. Now, you can run a model on the Platform or integrate the API with your app.

In order to test the deployment on the Platform, go to Deployment Profile Overview and scroll down to Validate & Integrate section. Edit the code editor with sample data by pasting the following code.

{
    "url": "https://images.pexels.com/photos/2083866/pexels-photo-2083866.jpeg?auto=compress&cs=tinysrgb&h=750&w=1260"
}

Later click Update to save and close the box. Click Send request to test the model. Remember that your deployment needs to have Running status to work!

In the further step, you can integrate the API of the model with some React app or a website to build a Showcase.

In order to do it, just fork the repository with the Showcase and explore yolov5 implementation on your own.

Summary

To summarize, the main goal of that tutorial was to deploy the PyTorch model in production quickly. No matter your specialty, whether you are a Data Scientist, AI Researcher, or Machine Learning Engineer, currently you can deliver an AI model without any configuration and infrastructure setup.

You can also explore How to deploy Keras model.