Torq Compiler

• User Manual
  • Introduction
    • Technical Background
    • Distribution Contents
      • Release Package
      • Docker Image
      • Python Wheel (torq-compiler)
    • System Requirements
    • Getting Started
    • Support and Contact
  • Quickstart
    • Setup
      • Release Package Ubuntu 24.04
      • Docker Image
      • Python Wheel (pip)
    • Compile and Run the Model
  • Model Preparation
    • Overview
    • Convert a TFLite model to TOSA and MLIR
      • Using the Compiler Wheel
      • Using the Release Package
    • Convert Torch Model to MLIR
    • Convert ONNX Model to MLIR
    • References
  • Input Guide
    • Supported model formats for torq-compile
    • Input Structure for torq-run-module
      • Literal Inputs
      • Multiple Inputs
      • Feeding Input from Files
      • Converting Images to .npy or .bin
  • Step-by-Step Model Deployment Examples
    • Prerequisites
    • Prepare Model (Convert to MLIR)
    • Compile for Device
      • Example: TFLite model (MobileNetV2)
      • Example: ONNX model
    • Run Inference on Torq
      • Using the Python Bindings
    • Compiling for Host (Simulation)
    • Handling Unsupported Operations
  • Torq Runtime Python API (Beta)
    • Installation
    • Quick Start
    • Examples
      • Inspecting Model Inputs and Outputs
      • Profiling Inference Latency
      • Profiling System Resources
      • Running with Custom Inputs
      • Zero-Copy Device Outputs
    • API Reference
      • VMFBInferenceRunner
        • infer(inputs)
        • allocate_device_array(array)
      • profile_vmfb_inference_time
      • profile_vmfb_resources
      • ProfileStats
        • summary()
      • ResourceSampler
      • run_vmfb
      • TensorInfo
      • Utility Functions
        • random_inputs_from_info(inputs_info)
  • Supported Operators
    • Activation related Operators
    • Element-Wise Operators
    • Composite Operators
    • Other Operators
  • Profiling
    • Compile-Time Profiling
    • Runtime Profiling
    • Profiling with the Test Framework
      • Wall-time measurement
  • Inference Time Statistics
    • Sysfs Path
    • torq_inference_time
    • Clearing the Counter
    • Calculating NPU Load
    • Notes
  • Handling Unsupported Ops
    • CSS Fallback
    • Host Fallback
  • Compiling Models for Custom Synaptics SoCs
    • NSS Subsystem Overview
    • Custom Hardware Configuration
      • Example Configurations
  • Debug tips and tricks
  • torq-gen-config: a TORQ config generation tool
    • Table of Contents
    • 1. Overview
      • Why torq-gen-config?
      • How It Works
      • Example: SqueezeNet 1.0
      • Tested Models
      • Output Files
    • 2. Three-Step Workflow
      • Step 1: Discover
      • Step 2: Review
      • Step 3: Run Full Model
    • 3. Viewing Results
      • Using the Viewer Script
      • Viewing Compiler JSON
      • Understanding the JSON
      • How Recommended Executor is Determined
      • Changing the Recommended Executor
      • Timing-Based Executor Recommendation
    • 4. Handling Issues
      • Critical Failures (All Executors Failed)
      • Debugging a Specific Layer
      • Subgraph Debugging
      • Full Model Issues
      • Skipping Executors (Extra Debug Option)
      • Important: How --skip-mode and JSON Cache Work
        • 1. --skip-mode Behavior
        • 2. Layer Test vs Full Model Test
        • 3. Debugging Specific Layers - Common Pitfall
        • 4. Verifying Executor Assignment
        • 5. Summary Table: When to Use What
    • 5. Auto-Converting FP32 Models to BF16
      • Why Convert to BF16?
      • What is BF16?
      • The Conversion Process
      • Accuracy Evaluation Method
      • Inference-Level Accuracy Check (Optional)
      • Using BF16 with torq-gen-config
      • Batch Dimension Handling
      • Saving Converted Models
      • When to Use BF16 Conversion
    • 6. ONNX to MLIR Mapping Mechanism
      • torch-mlir Import Guarantees
      • Position-Based Matching
      • Verification
    • 7. Command Reference
      • torq-gen-config CLI
        • discover — Run executor discovery
        • run — Run full model test
        • view — View executor config
        • edit — Edit executor assignments
      • Advanced: raw pytest options
  • What is SyNAP?
    • SyNAP - Torq Backend
    • Torq Model to Synap model conversion
    • Runtime Integration
  • SyNAP - Torq Tests
    • synap_cli Examples
    • GStreamer Pipelines and AI Examples
    • OOBE AI Demo
  • References
  • Glossary
  • Release Notes
    • Version 1.5.1 (2026-04-20)
    • Version 1.5.0 (2026-04-03)
    • Version 1.1.0 to 1.4.0
    • Version 1.0.0
    • Version 0.9.0
      • Key Features & Enhancements
      • Fixed Issues
      • Known Issues
      • Breaking Changes in This Release
      • Deprecated Features
      • Documentation
• Developer Manual
  • Introduction
  • Build and setup
    • Prerequisites
    • Download the code
    • Install required system packages
    • Build compiler and runtime for host
    • Build runtime for target
    • Build the kernel module
  • Compile and run a model
  • Design
    • Torq compiler backend
      • Overview
      • Dialects
      • Passes
      • Data Layout
      • Memory Allocation and Planning
      • Strided Convolutions
      • Torq HAL driver
  • TOSA to Torq Conversion
    • Example
  • Kernel Development
    • NPU HW Overview
    • Programming a Slice
    • EasyKernel
    • Tutorial
      • Vector Multiplication (basic)
      • Vector Multiplication (vectorized)
      • Tensor multiplication (vectorized)
      • Tensor multiplication (optimized)
      • Matrix Reduce
    • Reference
      • Kernel Structure
      • ALU Instructions
      • Vectorization
      • Writing output results
  • Adding a New Pass
    • Example
  • The Tile and Fuse Pass
    • Fuse groups
    • Memory Footprint Check
    • Finding Optimal Tile Sizes
      • Tiling Order
      • Tiling Search (fitTileToMemory)
    • Producers Fuse Mode
    • Loop Reduction for Memory Simulation
  • Adding a new compute optimization
  • Recommended submission process
    • Coding Style
    • Updating this Documentation
  • Debug tips and tricks
  • Troubleshooting
  • References
    • Glossary
  • Testing
    • Running tests
    • Profiling of tests
    • Managing XFAIL tests
    • How test code works
      • Fixtures
      • Parametrization
      • Multiple parameters
      • Caching
  • Tools
    • Template Profiling Tool
      • Overview
      • Prerequisites
        • Hardware Setup for Remote Testing
      • Quick Start
        • Tool Output
      • Architecture
      • How It Works
        • Template MLIR Files
        • Shape Generation Algorithm
        • Compiler Variants
      • Features and Capabilities
        • What It Can Do
          • 1. Multi-Rank Testing
          • 2. Multi-Dtype Support
          • 3. Dynamic Shape Generation
          • 4. Performance Profiling
          • 5. Selective Test Execution
          • 6. Custom Compiler Options
      • Understanding Test Output
        • Profiling Output
      • Limitations and Current Constraints
        • What It Currently Cannot Do
          • 1. Arbitrary Shape Relationships
          • 3. Complex Dtype Interactions
          • 4. Memory Constraint Validation
          • 5. Multi-Input Shape Broadcasting
  • Model testing and debugging workflow
    • Running test suite for a model
      • Initial setup
    • Collect all the tests for a model
    • What the tests do
    • Executing the tests
    • Investigating ERRORs
    • Investigating FAILUREs
    • Debugging a test
    • Performance debugging
    • Executing tests using real hardware
      • Auto-deploying the runtime to the board
      • Deploying a custom NPU kernel module
      • Finding boards on the network
    • Using the results dashboard
    • Advanced: Test framework internals and PyTest fixtures
    • TFLite model testing
      • Initial setup
      • Collecting tests
      • What the tests do
      • Environment variables
      • Running specific subsets
      • How layer extraction works
      • Caching
      • Investigating errors and failures
      • Key TFLite fixtures