Failed to quantize Qwen3 VL MoE

[noobHappylife]

I'm having difficulty quantizing Qwen3 VL 30B-A3B MoE model. I'm using 4xA100 80GB VRAM. However, I'm encountering cuda out-of-memory error.

From the logs and while monitoring the GPU VRAM usage, I noticed the VRAM of 1 gpu is exhausted during the step "start to cache block inputs". Then later it hit the error CUDA oom and RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select).

During this stage, I notice only 1 GPU is really used, while the other 3 barely has any VRAM occupied. Was hoping someone can help to point to the right direction.

Command I used:

CUDA_VISIBLE_DEVICES=0,1,2,3 auto-round --model "Qwen/Qwen3-VL-30B-A3B-Instruct" --scheme "W4A16" --format auto_round --output_dir ./output_llmcompressor --device_map "auto" --enable_torch_compile

Logs:

2026-01-01 16:27:26 INFO __main__.py L529: `torch.compile` is enabled to reduce tuning costs. If it causes issues, you can disable it by removing `--enable_torch_compile` argument.
2026-01-01 16:27:26 INFO __main__.py L537: start to quantize Qwen/Qwen3-VL-30B-A3B-Instruct
2026-01-01 16:27:26 INFO autoround.py L158: using MLLM mode for multimodal model.
2026-01-01 16:27:27 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 128.74it/s]
2026-01-01 16:27:36 INFO base.py L388: using torch.bfloat16 for quantization tuning
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.0.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.0.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.1.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.1.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.2.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.2.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.3.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.3.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.4.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.4.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.5.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.5.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.6.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.6.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.7.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.7.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.8.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.8.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.9.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.9.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.10.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.10.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.11.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.11.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.12.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.12.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.13.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.13.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.14.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.14.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.15.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.15.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.16.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.16.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.17.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.17.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.18.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.18.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.19.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.19.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.20.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.20.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.21.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.21.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.22.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.22.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.23.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.23.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.24.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.24.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.25.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.25.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.26.mlp.linear_fc1 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 WARNING formats.py L142: model.visual.blocks.26.mlp.linear_fc2 skipped quantization (shape not divisible by 32).
2026-01-01 16:27:38 INFO formats.py L120: change `scale_dtype` to `torch.float32` for gguf format
2026-01-01 16:27:38 INFO replace_modules.py L164: Found 48 modules to replace
Replacing modules: 100%|███████████████████████████████████████████████████████████████████████████████████| 48/48 [02:54<00:00,  3.64s/it]
2026-01-01 16:30:33 INFO replace_modules.py L178: Replaced 48 modules
2026-01-01 16:30:36 INFO base.py L1451: start to cache block inputs
2026-01-01 16:31:33 INFO base.py L1953: switch to cpu to cache block inputs
cache block inputs:   0%|                                                                                          | 0/128 [00:00<?, ?it/s]
2026-01-01 16:31:53 ERROR base.py L1970: Traceback (most recent call last):
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/compressors/base.py", line 1929, in try_cache_inter_data_gpucpu
    self.model = dispatch_model(self.model, device_map=device_map)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/big_modeling.py", line 426, in dispatch_model
    attach_align_device_hook_on_blocks(
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  [Previous line repeated 1 more time]
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 634, in attach_align_device_hook_on_blocks
    add_hook_to_module(module, hook)
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 166, in add_hook_to_module
    module = hook.init_hook(module)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 288, in init_hook
    set_module_tensor_to_device(module, name, self.execution_device, tied_params_map=self.tied_params_map)
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/utils/modeling.py", line 335, in set_module_tensor_to_device
    new_value = old_value.to(device, non_blocking=non_blocking)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.14 GiB of which 10.75 MiB is free. Including non-PyTorch memory, this process has 79.12 GiB memory in use. Of the allocated memory 71.13 GiB is allocated by PyTorch, and 7.51 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Traceback (most recent call last):
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/compressors/base.py", line 1929, in try_cache_inter_data_gpucpu
    self.model = dispatch_model(self.model, device_map=device_map)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/big_modeling.py", line 426, in dispatch_model
    attach_align_device_hook_on_blocks(
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  [Previous line repeated 1 more time]
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 634, in attach_align_device_hook_on_blocks
    add_hook_to_module(module, hook)
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 166, in add_hook_to_module
    module = hook.init_hook(module)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 288, in init_hook
    set_module_tensor_to_device(module, name, self.execution_device, tied_params_map=self.tied_params_map)
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/utils/modeling.py", line 335, in set_module_tensor_to_device
    new_value = old_value.to(device, non_blocking=non_blocking)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.14 GiB of which 10.75 MiB is free. Including non-PyTorch memory, this process has 79.12 GiB memory in use. Of the allocated memory 71.13 GiB is allocated by PyTorch, and 7.51 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/akk/yh/uvenv/quantization/bin/auto-round", line 10, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/__main__.py", line 911, in run
    start()
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/__main__.py", line 485, in start
    tune(args)
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/__main__.py", line 680, in tune
    model, folders = autoround.quantize_and_save(export_dir, format=args.format)  # pylint: disable=E1101
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/compressors/base.py", line 818, in quantize_and_save
    model, _ = self.quantize()
               ^^^^^^^^^^^^^^^
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/compressors/base.py", line 1452, in quantize
    all_inputs = self.try_cache_inter_data_gpucpu(all_first_block_names, self.nsamples, layer_names=layer_names)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/compressors/base.py", line 1966, in try_cache_inter_data_gpucpu
    all_inputs = self.cache_inter_data(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/compressors/base.py", line 2014, in cache_inter_data
    self.calib(nsamples, calib_bs)
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/compressors/mllm/compressor.py", line 411, in calib
    raise error
  File "/home/akk/yh/quantize_autoround/auto-round/auto_round/compressors/mllm/compressor.py", line 407, in calib
    self.model(**data_new)
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 175, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/transformers/utils/generic.py", line 1072, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/transformers/models/qwen3_vl_moe/modeling_qwen3_vl_moe.py", line 1601, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/transformers/utils/generic.py", line 1072, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/transformers/models/qwen3_vl_moe/modeling_qwen3_vl_moe.py", line 1298, in forward
    inputs_embeds = self.get_input_embeddings()(input_ids)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/accelerate/hooks.py", line 175, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/nn/modules/sparse.py", line 192, in forward
    return F.embedding(
           ^^^^^^^^^^^^
  File "/home/akk/yh/uvenv/quantization/lib/python3.12/site-packages/torch/nn/functional.py", line 2542, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select)

[wenhuach21]

#1213 please use this branch for a workaround for now. Or you could use iters=0 and disable_opt_rtn without algorithm tuning. The ram usage is very high now, ~160GB

We will fix this issue after the holiday.

[wenhuach21]

you may also meet this issue, huggingface/transformers#41418

[noobHappylife]

Thank you for the suggestions. In the meantime, I tried using llm_compressor with autoroundmodifier to do the quantization and it seems like it's working.

However, there's 1 fix I need to apply to make it work. Now sure if I should report int the llmcompressor repo or here.

Here is the quantization script I used:

from auto_round.calib_dataset import get_dataset
from transformers import AutoTokenizer, Qwen3VLMoeForConditionalGeneration, AutoProcessor

from llmcompressor import oneshot
from llmcompressor.modifiers.autoround import AutoRoundModifier
from llmcompressor.utils import dispatch_for_generation

# Select model and load it.
model_id = "Qwen/Qwen3-VL-30B-A3B-Instruct"
model = Qwen3VLMoeForConditionalGeneration.from_pretrained(model_id, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Select calibration dataset.
NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 2048
# Get aligned calibration dataset.

ds = get_dataset(
    tokenizer=tokenizer,
    seqlen=MAX_SEQUENCE_LENGTH,
    nsamples=NUM_CALIBRATION_SAMPLES,
)


# Configure the quantization algorithm to run.
#   * quantize the weights to 4 bit with AutoRound with a group size 128
recipe = AutoRoundModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "re:.*lm_head",
        "re:visual.*",
        "re:model.visual.*",
        "re:.*mlp.gate$",
    ],
    iters=200
)


# Apply algorithms.
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    # disable shuffling to get slightly better mmlu score
    shuffle_calibration_samples=False,
)

# Confirm generations of the quantized model look sane.
print("\n\n")
print("========== SAMPLE GENERATION ==============")
dispatch_for_generation(model)
sample = tokenizer("Hello my name is", return_tensors="pt")
sample = {key: value.to(model.device) for key, value in sample.items()}
output = model.generate(**sample, max_new_tokens=100)
print(tokenizer.decode(output[0]))
print("==========================================\n\n")

# Save to disk compressed.
SAVE_DIR = model_id.rstrip("/").split("/")[-1] + "-W4A16-G128-AutoRound"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

The changes I made, in llmcompressor

modifiers/autoround/base.py
added a config attribute for _PretrainModelWrapper

class _PretrainModelWrapper(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = _LLModelWrapper()
        self.config = None # new

    def forward(self, *args, **kwargs):
        return self.model(*args, **kwargs)

then under the apply_autoround method

    def apply_autoround(self, state, subgraph):
        ...
        wrapped_model = _wrap_decoding_layer(decoding_layer)
        wrapped_model.config = state.model.config # new 
        wrapped_model.name_or_path = state.model.name_or_path

Without the above fix, I will encounter this error:

AttributeError: '_PretrainModelWrapper' object has no attribute 'config'

[yiliu30]

Thank you for the suggestions. In the meantime, I tried using llm_compressor with autoroundmodifier to do the quantization and it seems like it's working.

However, there's 1 fix I need to apply to make it work. Now sure if I should report int the llmcompressor repo or here.

Here is the quantization script I used:

from auto_round.calib_dataset import get_dataset
from transformers import AutoTokenizer, Qwen3VLMoeForConditionalGeneration, AutoProcessor

from llmcompressor import oneshot
from llmcompressor.modifiers.autoround import AutoRoundModifier
from llmcompressor.utils import dispatch_for_generation

# Select model and load it.
model_id = "Qwen/Qwen3-VL-30B-A3B-Instruct"
model = Qwen3VLMoeForConditionalGeneration.from_pretrained(model_id, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Select calibration dataset.
NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 2048
# Get aligned calibration dataset.

ds = get_dataset(
    tokenizer=tokenizer,
    seqlen=MAX_SEQUENCE_LENGTH,
    nsamples=NUM_CALIBRATION_SAMPLES,
)


# Configure the quantization algorithm to run.
#   * quantize the weights to 4 bit with AutoRound with a group size 128
recipe = AutoRoundModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "re:.*lm_head",
        "re:visual.*",
        "re:model.visual.*",
        "re:.*mlp.gate$",
    ],
    iters=200
)


# Apply algorithms.
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    # disable shuffling to get slightly better mmlu score
    shuffle_calibration_samples=False,
)

# Confirm generations of the quantized model look sane.
print("\n\n")
print("========== SAMPLE GENERATION ==============")
dispatch_for_generation(model)
sample = tokenizer("Hello my name is", return_tensors="pt")
sample = {key: value.to(model.device) for key, value in sample.items()}
output = model.generate(**sample, max_new_tokens=100)
print(tokenizer.decode(output[0]))
print("==========================================\n\n")

# Save to disk compressed.
SAVE_DIR = model_id.rstrip("/").split("/")[-1] + "-W4A16-G128-AutoRound"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

The changes I made, in llmcompressor

modifiers/autoround/base.py added a config attribute for _PretrainModelWrapper

class _PretrainModelWrapper(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = _LLModelWrapper()
        self.config = None # new

    def forward(self, *args, **kwargs):
        return self.model(*args, **kwargs)

then under the apply_autoround method

    def apply_autoround(self, state, subgraph):
        ...
        wrapped_model = _wrap_decoding_layer(decoding_layer)
        wrapped_model.config = state.model.config # new 
        wrapped_model.name_or_path = state.model.name_or_path

Without the above fix, I will encounter this error:

AttributeError: '_PretrainModelWrapper' object has no attribute 'config'

Hi @noobHappylife, that fix makes sense. Would you be willing to raise a PR on the LLMC side to address it?