Model Conversion

Overview

The Torq compiler compiles models expressed in MLIR, which is a generic framework for intermediate representations, into Torq VMFB files. To deploy a source model, first convert it to MLIR, then compile the MLIR to VMFB, then run the VMFB with the Torq runtime.

This page describes how to convert models from TFLite, Torch, and ONNX to MLIR, compile them, and run them.

Model Selection and Verification

Before converting a model, check that both the model format and numeric type match the target path you plan to use.

  • Prefer an already quantized INT8 model when targeting NPU-heavy image models. Verify the input and output shapes, dtypes, scales, and zero points with a model viewer such as Netron or with the source framework tooling.

  • For FP32 models that need BF16 on NSS, use torq-compile --torq-convert-dtypes. Add --torq-convert-io-dtype when the public model inputs and outputs should also use converted dtypes; omit it when Torq should preserve the original public I/O dtypes with boundary casts.

  • Check the operator and dtype coverage in Supported Operations before spending time debugging a model that uses unsupported operators.

  • Keep a small set of representative inputs and reference outputs from the source framework. After conversion, compile the model, run the same inputs through Torq, and compare outputs with tolerances appropriate for INT8 or BF16 quantization.

For ONNX model exploration and layer-by-layer verification, use torq-gen-config. It can discover executor assignments, run layer and full-model checks, and report accuracy differences.

Convert Model to MLIR Format

Convert a TFLite model to TOSA and MLIR

A TFLite model must first be converted to TOSA MLIR before compilation. There are two approaches depending on which distribution you are using.

Using the Compiler Wheel

Note

Install the [tflite] extra to get the tosa-converter-for-tflite tool: pip install "torq_compiler-<version>-<platform>.whl[tflite]"

Convert the model to TOSA bytecode:

$ tosa-converter-for-tflite model.tflite --bytecode -o model.mlirbc

Use --text instead of --bytecode to produce a human-readable MLIR file:

$ tosa-converter-for-tflite model.tflite --text -o model.mlir

Using the Release Package

  • If not yet done, activate the Python environment as explained in Getting Started (skip this step if using the Docker container).

  • Navigate to the root directory of the Release Package, or run the Docker container. For the Docker container, the release package is located at:

    $ cd /opt/release

  • Convert the model to TOSA using the following command:

    Model Source: This model - MobileNetV2 is generated from tf.keras.applications using tf_model_generator.py. The dataset for int8 quantization is done using random data.

    $ tosa-converter-for-tflite tests/hf/Synaptics_MobileNetV2/MobileNetV2_int8.tflite --text -o mobilenetv2.mlir

Note

The tests/hf/ directory is only included in the Release Package and is not available in the compiler GitHub repository.

Convert Torch Model to MLIR

  • Torch models can be converted to MLIR using the torch-mlir toolchain.

  • This process involves exporting a PyTorch model and converting it to MLIR in various dialects such as TORCH, TOSA, LINALG_ON_TENSORS, or STABLEHLO.

  • The resulting MLIR file can then be used as input for the Torq compiler, depending on the supported dialects and features.

  • Export torch-mlir Python Packages

    $ export PYTHONPATH=`pwd`/build/tools/torch-mlir/python_packages/torch_mlir:`pwd`/test/python/fx_importer

  • Create Torch Test Model and Output to different MLIR Dialect

    import torch
from torch_mlir import torchscript

class SimpleModule(torch.nn.Module):

    def __init__(self):
        super().__init__()

    def forward(self, x):
        return torch.ops.aten.abs(x)


if __name__ == '__main__':
    test_input = torch.ones(2, 16)
    graph = SimpleModule()
    graph.eval()
    module = torchscript.compile(graph,
                                test_input,
                                torchscript.OutputType.TORCH,
                                use_tracing=False,
                                verbose=False)
    print(module.operation.get_asm())

    with open("./aten-torch.mlir", "w") as fp:
        fp.write(module.operation.get_asm())

  • output type could be

    • TORCH

    • LINALG_ON_TENSORS

    • TOSA

    • STABLEHLO

Convert ONNX Model to MLIR

IREE provides an ONNX importer that converts ONNX models into a text-based MLIR representation. The importer is available in the Torq compiler Python environment.

Note

Compiler wheel users: ONNX importing requires the onnx extra. Install with: pip install "torq_compiler-<version>-<platform>.whl[onnx]"

  • If not using the Docker container, activate the Python environment as explained in Getting Started.

  • Run the IREE importer to convert an ONNX model to MLIR:

    $ python -m iree.compiler.tools.import_onnx path/to/model.onnx -o path/to/model.mlir --data-prop

    Tip

    Use the -h flag to view all available importer options.

    $ python -m iree.compiler.tools.import_onnx -h

The output MLIR file can be compiled with Torq by specifying onnx-torq as the input dialect.

Compile and Run

After converting a source model to MLIR, use Step-by-Step Model Deployment Examples for the detailed torq-compile and torq-run-module flow. That page covers input dialect selection, compiling MLIR to VMFB, running on Torq devices, and simulator-oriented builds.

See the Input Guide for literal inputs, multiple inputs, .npy files, raw binary files, and image conversion.

References