A Hands-On Coding Tutorial on Qualcomm AI Hub Models for Classification, Object Detection, and Hardware-Aware Deployment
In this tutorial, we work by way of an end-to-end workflow for Qualcomm AI Hub Models. We begin by organising the required package deal, discovering the out there mannequin assortment, and loading MobileNet-V2 for native PyTorch inference. We additionally deal with an essential input-shape problem by changing NHWC picture tensors into the NCHW format anticipated by the mannequin. From there, we run inference on each the mannequin’s built-in pattern enter and an actual picture, examine high predictions, execute the official Qualcomm AI Hub CLI demo, and prolong the workflow with a YOLOv7 object detection instance. Also, we embody an optionally available cloud-device part the place we compile, profile, and run the mannequin on an actual Qualcomm machine when an API token is accessible.
import subprocess, sys, os, glob, textwrap, traceback
import numpy as np, torch
from PIL import Image
import matplotlib.pyplot as plt
def pip_install(*pkgs):
subprocess.run([sys.executable, "-m", "pip", "install", "-q", *pkgs], examine=True)
pip_install("qai_hub_models")
OUT_DIR = "/content material/qaihm_out"; os.makedirs(OUT_DIR, exist_ok=True)
torch.set_grad_enabled(False)
def to_nchw(worth):
arr = worth[0] if isinstance(worth, (listing, tuple)) else worth
t = torch.from_numpy(np.asarray(arr, dtype=np.float32))
if t.ndim == 3:
t = t.unsqueeze(0)
if t.ndim == 4 and t.form[1] != 3 and t.form[-1] == 3:
t = t.permute(0, 3, 1, 2).contiguous()
return tWe start by importing libraries and organising a helper perform to put in packages instantly inside Colab. We set up qai_hub_models, create an output listing, and disable gradient monitoring since we solely want inference. We additionally outline the to_nchw() perform to transform any enter picture tensor to the channel-first format anticipated by the mannequin.
import pkgutil, qai_hub_models.fashions as _m
model_ids = sorted(n for _, n, p in pkgutil.iter_modules(_m.__path__)
if p and not n.startswith("_"))
print(f">>> {len(model_ids)} fashions out there. First 40:n")
print(textwrap.fill(", ".be a part of(model_ids[:40]), 100), "n")
from qai_hub_models.fashions.mobilenet_v2 import Model as MobileNetV2
mannequin = MobileNetV2.from_pretrained().eval()
spec = mannequin.get_input_spec()
input_name = listing(spec.keys())[0]
print(">>> Input:", input_name, spec[input_name].form, spec[input_name].dtype)
from torchvision.fashions import MobileNet_V2_Weights
IMAGENET_CLASSES = MobileNet_V2_Weights.IMAGENET1K_V1.meta["categories"]
def top5(logits):
if logits.ndim == 1: logits = logits.unsqueeze(0)
probs = torch.softmax(logits, dim=1)[0]
conf, idx = probs.topk(5)
return [(IMAGENET_CLASSES[i], float(c)) for c, i in zip(conf, idx)]We uncover the out there Qualcomm AI Hub mannequin packages and print the primary set of mannequin IDs to know what’s accessible. We then load the pretrained MobileNet-V2 mannequin, learn its enter specification, and establish the right enter title. We additionally put together the ImageInternet class labels and outline a top5() perform to transform mannequin logits into readable top-5 predictions.
pattern = mannequin.sample_inputs()
x = to_nchw(pattern[input_name])
print(">>> fed tensor form:", tuple(x.form))
print("n>>> Top-5 for the built-in pattern enter:")
for label, conf in top5(mannequin(x)):
print(f" {conf:6.2%} {label}")
from torchvision import transforms
preprocess = transforms.Compose([
transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(),
])
img = None
strive:
import urllib.request
p = os.path.be a part of(OUT_DIR, "enter.jpg")
urllib.request.urlretrieve(
"https://uncooked.githubusercontent.com/pytorch/hub/grasp/photos/canine.jpg", p)
img = Image.open(p).convert("RGB")
besides Exception as e:
print(">>> picture obtain skipped:", e)
if img will not be None:
preds = top5(mannequin(preprocess(img).unsqueeze(0)))
print("n>>> Top-5 for the downloaded picture:")
for label, conf in preds: print(f" {conf:6.2%} {label}")
plt.determine(figsize=(5,5)); plt.imshow(img); plt.axis("off")
plt.title(f"{preds[0][0]} ({preds[0][1]:.1%})"); plt.present()We first run inference utilizing the mannequin’s built-in pattern enter and use to_nchw() to repair the tensor form earlier than passing it to MobileNet-V2. We then obtain an actual picture, preprocess it utilizing customary resizing, cropping, and tensor conversion steps, and run one other prediction. We lastly show the picture with the highest predicted label to visually join the mannequin output to the enter picture.
def run_demo(module, additional=None, timeout=900):
cmd = [sys.executable, "-m", module, "--eval-mode", "fp",
"--output-dir", OUT_DIR] + (additional or [])
print(f"n>>> {' '.be a part of(cmd)}")
strive:
r = subprocess.run(cmd, capture_output=True, textual content=True, timeout=timeout)
print("n".be a part of((r.stdout + r.stderr).strip().splitlines()[-25:]))
besides Exception as e:
print(">>> demo skipped:", e)
run_demo("qai_hub_models.fashions.mobilenet_v2.demo")
strive:
pip_install("qai_hub_models[yolov7]")
run_demo("qai_hub_models.fashions.yolov7.demo")
imgs = sorted(glob.glob(OUT_DIR + "/*.png") + glob.glob(OUT_DIR + "/*.jpg"),
key=os.path.getmtime)
if imgs:
plt.determine(figsize=(9,9)); plt.imshow(Image.open(imgs[-1]).convert("RGB"))
plt.axis("off"); plt.title("YOLOv7 detections"); plt.present()
else:
print(">>> no output picture discovered (outcomes could have printed as a substitute).")
besides Exception:
print(">>> YOLOv7 part skipped:n", traceback.format_exc())We outline a reusable run_demo() perform that executes official Qualcomm AI Hub mannequin demos from the command line. We use it to run the MobileNet-V2 demo and then set up the YOLOv7 extras for object detection. We run the YOLOv7 demo, search for the generated output picture, and visualize the detections if a picture is created.
strive:
import qai_hub as hub
units = hub.get_devices()
print(f"n>>> Authenticated. {len(units)} cloud units out there.")
machine = hub.Device("Samsung Galaxy S24 (Family)")
pattern = mannequin.sample_inputs()
nchw = to_nchw(pattern[input_name])
traced = torch.jit.hint(mannequin, [nchw])
cloud_inputs = {input_name: [nchw.numpy()]}
cj = hub.submit_compile_job(mannequin=traced, machine=machine,
input_specs=mannequin.get_input_spec(),
choices="--target_runtime tflite")
goal = cj.get_target_model(); print(">>> compiled:", cj.url)
pj = hub.submit_profile_job(mannequin=goal, machine=machine); print(">>> profiling:", pj.url)
ij = hub.submit_inference_job(mannequin=goal, machine=machine, inputs=cloud_inputs)
out = ij.download_output_data()
dev_logits = torch.from_numpy(np.asarray(listing(out.values())[0][0]))
print(">>> Top-5 from the REAL machine:")
for label, conf in top5(dev_logits): print(f" {conf:6.2%} {label}")
goal.obtain(os.path.be a part of(OUT_DIR, "mobilenet_v2.tflite"))
print(">>> saved compiled .tflite to", OUT_DIR)
besides Exception as e:
print("n>>> Cloud (on-device) part skipped — no API token configured.")
print(" Get one at workbench.aihub.qualcomm.com, then:")
print(" !qai-hub configure --api_token YOUR_TOKEN")
print(" element:", (str(e).splitlines() or [type(e).__name__])[0])
print("n>>> Tutorial full. Outputs in:", OUT_DIR)We embody an optionally available Qualcomm AI Hub cloud workflow that runs solely when an API token is configured. We retrieve out there cloud units, hint the PyTorch mannequin, compile it for TFLite, profile it on a Qualcomm machine, and submit an inference job. We then obtain the machine output, print the highest predictions, save the compiled TFLite mannequin, and end by displaying the place all tutorial outputs are saved.
In conclusion, we’ve a whole sensible workflow for utilizing Qualcomm AI Hub Models inside Colab. We discovered the way to load pretrained fashions, put together inputs appropriately, run native inference, visualize classification and detection outcomes, and use the official demos as reproducible reference factors. We additionally noticed how the identical mannequin can transfer past native PyTorch execution into Qualcomm’s cloud-device pipeline for compilation, profiling, and real-device inference. It offers a path from easy experimentation to hardware-aware deployment with Qualcomm AI Hub.
Check out the Full Codes with Notebook here. Also, be at liberty to observe us on Twitter and don’t overlook to hitch our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so on.? Connect with us
The publish A Hands-On Coding Tutorial on Qualcomm AI Hub Models for Classification, Object Detection, and Hardware-Aware Deployment appeared first on MarkTechPost.
