Developer Manual

• Introduction
• Build and setup
  • Prerequisites
  • Download the code
  • Install required system packages
  • Download pre-built or build mpact library
  • Build compiler and runtime for host
  • Build runtime for target
    • Flow A: reuse an existing host build
    • Flow B: runtime-only (without a full host compiler build)
  • Build the kernel module
• Compile and run a model
• Design
  • Torq compiler backend
    • Overview
    • Dialects
    • Passes
    • Data Layout
    • Memory Allocation and Planning
    • Strided Convolutions
    • Torq HAL driver
• TOSA to Torq Conversion
  • Example
• Kernel Development
  • NPU HW Overview
  • Programming a Slice
  • EasyKernel
  • Tutorial
    • Vector Multiplication (basic)
    • Vector Multiplication (vectorized)
    • Tensor multiplication (vectorized)
    • Tensor multiplication (optimized)
    • Matrix Reduce
  • Reference
    • Kernel Structure
    • ALU Instructions
    • Vectorization
    • Writing output results
• Adding a New Pass
  • Example
• Tile and fuse, and reduction tiling
  • The basic transformation
  • Where the passes sit
  • Fuse groups
  • How tile-and-fuse chooses work
    • Tile shape policies
    • The memory probe
  • Why reductions have a preparatory pass
    • What a split reduction computes
    • Keepdims output maps
    • Interaction with tile-and-fuse
  • Code map
• The prefetch list scheduler
  • 1. The problem
  • 2. The hardware model
  • 3. Where the pass runs and what it sees
  • 4. Step 1: the dependency graph (StreamGraph)
  • 5. Step 2: the scheduling loop (listSchedule)
    • State
    • One step: the decision order
  • 6. Worked example 1: overlap and both slices
    • The same example, picture by picture
  • 7. Worked example 2: the LRAM limits
  • 8. Step 3: choosing a schedule (runOnOperation)
  • 9. The tile budgets
  • 10. Limits and known gaps
  • 11. Flags
• Adding a new compute optimization
• Recommended submission process
  • Coding Style
  • Updating this Documentation
• Debug tips and tricks
• Troubleshooting
• References
  • Glossary
• Testing
  • Running tests
  • Profiling of tests
  • Managing XFAIL tests
  • How test code works
    • Fixtures
    • Parametrization
    • Multiple parameters
    • Caching
• More testing
  • Convolution sweep
• Tools
  • Template Profiling Tool
    • Overview
    • Prerequisites
      • Hardware Setup for Remote Testing
    • Quick Start
      • Tool Output
    • Architecture
    • How It Works
      • Template MLIR Files
      • Shape Generation Algorithm
      • Compiler Variants
    • Features and Capabilities
      • What It Can Do
        • 1. Multi-Rank Testing
        • 2. Multi-Dtype Support
        • 3. Dynamic Shape Generation
        • 4. Performance Profiling
        • 5. Selective Test Execution
        • 6. Custom Compiler Options
    • Understanding Test Output
      • Profiling Output
    • Limitations and Current Constraints
      • What It Currently Cannot Do
        • 1. Arbitrary Shape Relationships
        • 3. Complex Dtype Interactions
        • 4. Memory Constraint Validation
        • 5. Multi-Input Shape Broadcasting
  • torq-mlir-query
    • Building
    • Usage
      • Interactive use
      • Non-interactive use (file / stdin)
    • Registered matchers
    • Composing matchers
    • Extracting matches into a function
    • Tab completion
    • Registering a new matcher
  • torq.lab developer guide
    • Source layout
    • Core API
      • Driving the pipeline
      • Running the CLI in-process
    • Packaging
    • Adding a command
  • Gen-Config Architecture
    • Table of Contents
    • 1. Overview
    • 2. Module Map
      • Responsibility split
      • Key helpers in core.py
    • 3. Dependency Graph
    • 4. Two-JSON Design
      • Report JSON (ops format)
      • Compiler JSON (op_assignments format)
      • Generation rules
    • 5. Quantization Support
      • Discovery flow with quantization
      • Mapping quantized MLIR to original layers
      • Quantization options
    • 6. CLI Subcommand Flow
      • discover
      • run
      • view
      • edit
    • 7. How to Add a New Model Format
      • Step 1: Create _<format>.py extraction/conversion helpers
      • Step 2: Add the engine hooks in _runner.py (or _runner_<format>.py)
      • Step 3: Wire the format into cli.py
      • Key principle
    • 8. How to Add a New CLI Subcommand
      • Step 1: Add the handler in cli.py
      • Step 2: Register in main()
      • Step 3: Add tests
      • Convention
    • 9. Testing Architecture
      • Test files
      • CLI test pattern
      • Running tests
    • 10. Standalone CLI Options
    • 11. TFLite Discovery (Removed)
  • TORQ ONNX Quantization
    • Table of Contents
    • 1. Overview
      • High-level lowering flow
    • 2. ONNX Quantization Formats
      • QDQ
      • QOperator
      • Hybrid
    • 3. torq-gen-config Quantization Workflow
      • Supported Quantization Options
      • Per-Layer vs. Full-Model Quantization
      • CLI Usage
    • 4. Quantized Convolution Zero-Point Correction
      • 4.1 Operation definition
      • 4.2 Mechanical rewriting
      • 4.3 Sub-component definitions
        • 4.3.1 Weight zero-point folding
        • 4.3.2 Input zero-point correction
        • 4.3.3 Bias addition
        • 4.3.4 Output rescale and clamp
      • 4.4 Lowering each sub-component
        • 4.4.1 QConv2DPattern.cpp
        • 4.4.2 QuantizePattern.cpp (fallback)
          • Quantize generic
          • Dequantize generic
      • 4.5 Special cases
        • 4.5.1 weight_zp == 0
        • 4.5.2 input_zp == 0
        • 4.5.3 Both zero-points are zero
        • 4.5.4 Dynamic weights
      • 4.6 Why the other expanded terms disappear
      • 4.7 Generalization to other weighted ops
    • 5. References
      • ONNX Runtime quantization
      • Quantization theory and best-practice papers
• Model testing and debugging workflow
  • Running test suite for a model
    • Initial setup
  • Collect all the tests for a model
  • What the tests do
  • Executing the tests
  • Investigating ERRORs
  • Investigating FAILUREs
  • Debugging a test
  • Performance debugging
  • Executing tests using real hardware
    • Auto-deploying the runtime to the board
    • Deploying a custom NPU kernel module
    • Finding boards on the network
  • Using the results dashboard
  • Advanced: Test framework internals and PyTest fixtures
  • TFLite model testing
    • Initial setup
    • Collecting tests
    • What the tests do
    • Environment variables
    • Running specific subsets
    • How layer extraction works
    • Caching
    • Investigating errors and failures
    • Key TFLite fixtures
• CI/CD pipelines
• Development process
• Release process

This manual is structured step-by-step, starting from introduction and setup, through model conversion, compiling & running, supported operators, testing, and advanced topics.