Model Conversion

Overview

The Torq compiler compiles models expressed in MLIR, which is a generic framework for intermediate representations, into Torq VMFB files. To deploy a source model, first convert it to MLIR, then compile the MLIR to VMFB, then run the VMFB with the Torq runtime.

This page describes how to convert models from TFLite, Torch, and ONNX to MLIR, compile them, and run them.

Model Selection and Verification

Before converting a model, check that both the model format and numeric type match the target path you plan to use.

  • Prefer an already quantized INT8 model when targeting NPU-heavy image models. Verify the input and output shapes, dtypes, scales, and zero points with a model viewer such as Netron or with the source framework tooling.

  • For FP32 models that need BF16 on NSS, use torq-compile --torq-convert-dtypes. Add --torq-convert-io-dtype when the public model inputs and outputs should also use converted dtypes; omit it when Torq should preserve the original public I/O dtypes with boundary casts.

  • Check the operator and dtype coverage in Supported Operations before spending time debugging a model that uses unsupported operators.

  • Keep a small set of representative inputs and reference outputs from the source framework. After conversion, compile the model, run the same inputs through Torq, and compare outputs with tolerances appropriate for INT8 or BF16 quantization.

For ONNX model exploration and layer-by-layer verification, use torq-gen-config. It can discover executor assignments, run layer and full-model checks, and report accuracy differences.

Convert Model to MLIR Format

Convert a TFLite model to TOSA and MLIR

A TFLite model must first be converted to TOSA MLIR before compilation. There are two approaches depending on which distribution you are using.

Using the Compiler Wheel

Note

Install the [tflite] extra to get the tosa-converter-for-tflite tool: pip install "torq_compiler-<version>-<platform>.whl[tflite]"

Convert the model to TOSA bytecode:

$ tosa-converter-for-tflite model.tflite --bytecode -o model.mlirbc

Use --text instead of --bytecode to produce a human-readable MLIR file:

$ tosa-converter-for-tflite model.tflite --text -o model.mlir

Note

The TORQ compiler currently does not support dynamic shapes. If a TFLite model was exported with dynamic dimensions (for example a None batch size in Keras), convert it to static shapes first:

$ pip install "torq_compiler-<version>-<platform>.whl[tf]"   # one-time, for the tensorflow dependency
$ torq-convert-static tflite -i model.tflite -o model_static.tflite


#### MobileNetV2 TFLite Example

- If not yet done, activate the Python environment as explained in [Getting Started](./getting_started.md) (skip this step if using the Docker container).

- Convert the model to TOSA using the following command:

    **Model Source:** This MobileNetV2 model is generated from `tf.keras.applications` using [tf_model_generator.py](https://github.com/synaptics-torq/iree-synaptics-synpu/blob/main/tests/model_generator/tf_model_generator.py). The dataset for int8 quantization uses random data. Download the TFLite model from the [Synaptics Hugging Face repository](https://huggingface.co/Synaptics/MobileNetV2) before conversion.

    ```{code} shell
    $ curl -L https://huggingface.co/Synaptics/MobileNetV2/resolve/main/MobileNetV2_int8.tflite?download=true -o MobileNetV2_int8.tflite
    $ tosa-converter-for-tflite MobileNetV2_int8.tflite --text -o mobilenetv2.mlir
    ```

(convert-torch-model-to-mlir)=
### Convert Torch Model to MLIR

- Torch models can be converted to MLIR using the `torch-mlir` toolchain.  
- This process involves exporting a PyTorch model and converting it to MLIR in various dialects such as TORCH, TOSA, LINALG_ON_TENSORS, or STABLEHLO.  
- The resulting MLIR file can then be used as input for the Torq compiler, depending on the supported dialects and features.

- Export torch-mlir Python Packages

    ```{code} shell
    $ export PYTHONPATH=`pwd`/build/tools/torch-mlir/python_packages/torch_mlir:`pwd`/test/python/fx_importer
    ```

- Create Torch Test Model and Output to different MLIR Dialect

    ```{code} python
    import torch
    from torch_mlir import torchscript

    class SimpleModule(torch.nn.Module):

        def __init__(self):
            super().__init__()

        def forward(self, x):
            return torch.ops.aten.abs(x)


    if __name__ == '__main__':
        test_input = torch.ones(2, 16)
        graph = SimpleModule()
        graph.eval()
        module = torchscript.compile(graph,
                                    test_input,
                                    torchscript.OutputType.TORCH,
                                    use_tracing=False,
                                    verbose=False)
        print(module.operation.get_asm())

        with open("./aten-torch.mlir", "w") as fp:
            fp.write(module.operation.get_asm())
    ```

- output type could be
    - TORCH
    - LINALG_ON_TENSORS
    - TOSA
    - STABLEHLO

### Convert ONNX Model to MLIR

IREE provides an ONNX importer that converts ONNX models into a text-based MLIR representation. The importer is available in the Torq compiler Python environment.

```{note}
**Compiler wheel users:** ONNX importing requires the `onnx` extra. Install with: `pip install "torq_compiler-<version>-<platform>.whl[onnx]"`

  • If not using the Docker container, activate the Python environment as explained in Getting Started.

  • Run the IREE importer to convert an ONNX model to MLIR:

    $ python -m iree.compiler.tools.import_onnx path/to/model.onnx -o path/to/model.mlir --data-prop

    Tip

    Use the -h flag to view all available importer options.

    $ python -m iree.compiler.tools.import_onnx -h

The output MLIR file can be compiled with Torq by specifying onnx-torq as the input dialect.

Compile and Run

After converting a source model to MLIR, use Step-by-Step Model Deployment Examples for the detailed torq-compile and torq-run-module flow. That page covers input dialect selection, compiling MLIR to VMFB, running on Torq devices, and simulator-oriented builds.

See the Input Guide for literal inputs, multiple inputs, .npy files, raw binary files, and image conversion.

Advanced Options

Precomputing an i8 activation into a lookup table

An i8 activation takes 256 values, so any chain of elementwise ops that reads one i8 tensor and produces an i8 tensor has only 256 possible results. The compiler evaluates such a chain at compile time and stores the results in a single 256-entry table, replacing the chain with one lookup. This is on by default and affects i8 activations only.

Any function of a single i8 value qualifies, whatever ops the frontend used to express it. An op whose other operand varies does not qualify: a multiply by a per-channel weight, for instance, makes the result a function of the activation and the channel, and no single table covers it.

Pass --torq-disable-precalc-i8-table to keep the chain at runtime:

$ torq-compile model.mlir -o model.vmfb --torq-disable-precalc-i8-table

Important

The rewrite changes how LRAM is laid out, so error: failed to allocate LRAM addresses can appear on a model that used to compile. Disable it for that model rather than for the whole build.

References