Prism-MoE: Efficient Dense-to-MoE Conversion for Visual Autoregressive Generation

ICML 2026

Ying Li^*1, Zefang Wang^*2, Zhaode Wang³, Zhiwen Chen³, Chengfei Lv³, Huan Wang¹

¹ Westlake University ² Zhejiang University ³ Alibaba Group

* Equal contribution · Corresponding author: Huan Wang

TL;DR: Prism-MoE converts a pretrained visual autoregressive image generator into a sparse top-k2 Mixture-of-Experts model via a training-free two-stage initialization plus lightweight distillation, preserving image quality at a fraction of the compute.

Problem

Visual autoregressive image generation keeps scaling — from next-token models (LlamaGen, Janus-Pro) and next-scale models (VAR, Infinity) to tens-of-billions-parameter systems — which substantially increases inference cost. Mixture-of-Experts (MoE) is a natural way to add capacity through sparse activation and is already standard in large language models, but training MoE from scratch is prohibitively expensive, making low-cost dense-to-MoE conversion the appealing route. Yet dense-to-MoE conversion for autoregressive image generation remains underexplored: prior efforts mostly target general upcycling or diffusion with heavy fine-tuning, and the conversion is further complicated by step-dependent activation non-stationarity and spatially heterogeneous token uncertainty.

So we ask: can MoE be brought into visual autoregressive generation efficiently — converting a pretrained dense generator into a sparse MoE with only a fraction of the active parameters and training budget, while preserving generation quality?

Motivation

Visual AR representations exhibit strong step-dependent low-rank structures, indicating that feature complexity varies significantly across generation steps.

Expert activation patterns are highly non-uniform and evolve during fine-tuning, suggesting unstable routing behavior and lack of consistent expert specialization.

Generated images exhibit spatially heterogeneous entropy patterns, where high-uncertainty regions align with semantic structures, indicating a mismatch between routing behavior and visual complexity.

Method

Stage I: Trajectory-Consistent Initialization

Importance and stability analysis → Identify critical and stable neurons
Shared and routed expert decomposition → Separate shared and specialized features
Closed-form trajectory compensation → Recover the dense trajectory

Stage II: Condition-Adaptive Sparse Fine-Tuning

Dense-to-sparse distillation → Preserve dense representations
Normalized cosine router → Stabilize expert selection
Spatially structured routing regularization → Promote coherent expert specialization

Main Results

Strong initialization. At 75% active parameters, Prism-MoE achieves the best sparse initialization result, with FID 5.03.
Near dense quality. After efficient fine-tuning, Prism-MoE reaches FID closely matching the dense VAR-d20 baseline.

Ablation

Baseline exhibits scattered entropy with unstable uncertainty regions. Ours produces structured entropy aligned with semantic regions and more consistent routing behavior.

Visual Results

VAR-d20, 37.5% activation

LlamaGen, 37.5% activation

BibTeX

@inproceedings{li2026prismmoe,
  title     = {Prism-MoE: Efficient Dense-to-MoE Conversion for Visual Autoregressive Generation},
  author    = {Li, Ying and Wang, Zefang and Wang, Zhaode and Chen, Zhiwen and Lv, Chengfei and Wang, Huan},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026}
}