Step-by-Step Model Deployment Examples

This guide walks through converting, compiling, and running ML models on a Torq device.

Install the compiler and runtime wheels first as described in the Quickstart pip setup.

Prepare Model (Convert to MLIR)

Before compiling, your model must be converted to MLIR. See Model Conversion for full instructions on each framework.

Here is a quick summary of conversion commands:

TFLite:

$ tosa-converter-for-tflite path/to/model.tflite --text -o model.mlir

ONNX:

$ python -m iree.compiler.tools.import_onnx path/to/model.onnx -o model.mlir --data-prop

Torch: See Convert Torch Model to MLIR for the Python-based conversion workflow.

Compile for Device

Use torq-compile to compile the MLIR model into a Torq runtime executable (.vmfb).

The key flags are:

Flag	Description
`--iree-input-type=<type>`	Specifies the MLIR dialect of the input file. See table below.
`-o <file>.vmfb`	Output path for the compiled model.

Input types by source framework:

Source Framework	Input Type	Typical File Extension
Auto (default)	`auto`	all
TFLite	`tosa-torq`	`.mlir`
ONNX	`onnx-torq`	`.mlir`
Torch	`torch-torq`	`.mlir`
Linalg	`linalg-torq`	`.mlir`

Note

Input type defaults to auto, which defers input type detection and conversion to the compiler. This behavior can be overridden by manually specifying --iree-input-type.

Example: TFLite model (MobileNetV2)

Model Source: The MobileNetV2 model is generated from tf.keras.applications using tf_model_generator.py. The dataset used for int8 quantization consists of random data. Download the TFLite model before running the conversion example.

# Download the TFLite model from Hugging Face
$ curl -L https://huggingface.co/Synaptics/MobileNetV2/resolve/main/MobileNetV2_int8.tflite?download=true -o MobileNetV2_int8.tflite

# Convert TFLite to TOSA
$ tosa-converter-for-tflite MobileNetV2_int8.tflite --text -o mobilenetv2.mlir

# Compile for device
$ torq-compile mobilenetv2.mlir -o mobilenetv2.vmfb

Example: ONNX model

# Convert ONNX to MLIR
$ python -m iree.compiler.tools.import_onnx path/to/model.onnx -o model.mlir --data-prop

# Compile for device
$ torq-compile model.mlir -o model.vmfb

Run Inference on Torq

Use the compiled model with torq-run-module to run inference:

$ torq-run-module \
    --module=mobilenetv2.vmfb \
    --function=main \
    --input="1x224x224x3xi8=1"

Note

To use an actual image as input, preprocess the image and save it as a NumPy array (.npy file). Then provide the path to the .npy file as input, as described in the image conversion section.

Using the Python Bindings

You can also run inference programmatically using the torq-runtime Python package:

from torq.runtime import VMFBInferenceRunner

runner = VMFBInferenceRunner("mobilenetv2.vmfb", device_uri="torq")
outputs = runner.infer(inputs)
print(f"Inference took {runner.infer_time_ms} ms")

The VMFBInferenceRunner class supports options such as function and load_method. See the Torq Runtime Python API page for the full API reference.

Compiling for Host (Simulation)

To compile for the current host machine (e.g., for testing without device hardware), add simulator-specific flags:

$ torq-compile mobilenetv2.mlir \
    --torq-target-host-triple=native \
    -o mobilenetv2.vmfb

Handling Unsupported Operations

For information about handling unsupported operations and CSS fallback, see Handling Unsupported Ops.