Q: model support v0.9.3

[kalle07]

Jan
https://huggingface.co/janhq/Jan-v1-2509
works fine with no error export as gguf
(llama-version 11/25 - last month)

next two models i only changed the model path...

Mistral-Magistral-Devstral is supported?
https://huggingface.co/DavidAU/Mistral-Magistral-Devstral-Instruct-FUSED-CODER-Reasoning-36B
...
2025-12-27 16:20:38 WARNING modeling_utils.py L4670: torch_dtype is deprecated! Use dtype instead!
Loading checkpoint shards: 100%|██████████| 15/15 [00:00<00:00, 183.83it/s]
2025-12-27 16:20:39 INFO base.py L390: using torch.bfloat16 for quantization tuning
2025-12-27 16:20:39 WARNING base.py L419: using algorithm extension for quantization.
Traceback (most recent call last):
File "C:\Users\xxx\Documents\python\autoround\auto_next.py", line 66, in
autoround.quantize_and_save(output_dir, format='gguf:q3_k_s')
File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 1055, in quantize_and_save
format_list = self._parse_format_to_list(format)
File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 835, in _parse_format_to_list
gguf_args_check(self, formats, model_type=ModelType.TEXT)
File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\utils.py", line 535, in gguf_args_check
from auto_round.export.export_to_gguf.convert_hf_to_gguf import ( # pylint: disable=E0401
ImportError: cannot import name 'ModelBase' from 'auto_round.export.export_to_gguf.convert_hf_to_gguf' (C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\export\export_to_gguf\convert_hf_to_gguf.py)

Granite small is supported?
https://huggingface.co/ibm-granite/granite-4.0-h-small
...
2025-12-27 16:32:40 INFO main.py L524: start to quantize e:\granite_small
2025-12-27 16:32:40 WARNING modeling_utils.py L4670: torch_dtype is deprecated! Use dtype instead!
2025-12-27 16:32:40 WARNING modeling_granitemoehybrid.py L460: The fast path is not available because one of (selective_state_update, causal_conv1d_fn, causal_conv1d_update) is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d

Loading checkpoint shards: 0%| | 0/14 [00:00<?, ?it/s]
Loading checkpoint shards: 100%|██████████| 14/14 [00:00<00:00, 172.62it/s]
2025-12-27 16:35:34 INFO base.py L390: using torch.float16 for quantization tuning
Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "c:\Users\xxx\Documents\python\autoround\venv\Scripts\auto_round_best.exe_main.py", line 6, in
File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round_main.py", line 894, in run_best
tune(args)
File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round_main_.py", line 667, in tune
model, folders = autoround.quantize_and_save(export_dir, format=args.format) # pylint: disable=E1101
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 1055, in quantize_and_save
format_list = self._parse_format_to_list(format)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 835, in _parse_format_to_list
gguf_args_check(self, formats, model_type=ModelType.TEXT)
File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\utils.py", line 535, in gguf_args_check
from auto_round.export.export_to_gguf.convert_hf_to_gguf import ( # pylint: disable=E0401
ImportError: cannot import name 'ModelBase' from 'auto_round.export.export_to_gguf.convert_hf_to_gguf' (C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\export\export_to_gguf\convert_hf_to_gguf.py)

i can not give you more logging ... or you give me a python code for your needs

[n1ck-guo]

This error log shows that you use old version of convert_hf_to_gguf. Could you please try to uninstall the auto_round by pip uninstall auto_round && pip cache purge and re-install it by pip install auto_round. This should solve your problem. If there are still problems, please let me know. I will try my best to help you solve it.

[wenhuach21]

We should catch this exception and provide a more informative error message so users can quickly identify the root cause.

[n1ck-guo]

We should catch this exception and provide a more informative error message so users can quickly identify the root cause.

This doesn't seem to be a code issue, but rather an environment problem. It's possible that an older version of auto_round or cached files in the environment are causing this.

[kalle07]

i updated all incl llama.cpp

granite_small:

2025-12-29 15:20:11 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead!
2025-12-29 15:20:11 WARNING modeling_granitemoehybrid.py L460: The fast path is not available because one of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d
Loading checkpoint shards: 100%|██████████| 14/14 [00:00<00:00, 76.06it/s]
2025-12-29 15:20:12 INFO base.py L390: using torch.bfloat16 for quantization tuning
2025-12-29 15:20:12 WARNING base.py L419: using algorithm extension for quantization.
2025-12-29 15:20:12 INFO utils.py L619: format gguf:q3_k_s does not support for data_type=int, reset to data_type=int_sym_dq.
2025-12-29 15:20:12 INFO base.py L858: change `scale_dtype` to `torch.float32`
Quantizing model.layers.0.mamba.out_proj:   2%|▏         | 5/209 [00:01<00:58,  3.50it/s]2025-12-29 15:20:15 INFO convert.py L599: blk.0.ffn_gate_exps.weight,   torch.bfloat16 --> F32, shape = {4096, 768, 72}
2025-12-29 15:20:15 INFO convert.py L599: blk.0.ffn_up_exps.weight,     torch.bfloat16 --> F32, shape = {4096, 768, 72}
2025-12-29 15:20:15 INFO convert.py L599: blk.0.ffn_down_exps.weight,   torch.bfloat16 --> F32, shape = {768, 4096, 72}
2025-12-29 15:20:15 INFO convert.py L599: blk.0.ffn_gate_inp.weight,    torch.bfloat16 --> F32, shape = {4096, 72}
2025-12-29 15:20:15 INFO convert.py L599: blk.0.attn_norm.weight,       torch.bfloat16 --> F32, shape = {4096}
2025-12-29 15:20:15 INFO convert.py L599: blk.0.ffn_norm.weight,        torch.bfloat16 --> F32, shape = {4096}
Quantizing model.layers.0.mamba.out_proj:   2%|▏         | 5/209 [00:03<02:21,  1.45it/s]
Traceback (most recent call last):
  File "C:\Users\xxx\Documents\python\autoround\auto_next.py", line 66, in <module>
    autoround.quantize_and_save(output_dir, format='gguf:q3_k_s')
  File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 1076, in quantize_and_save
    model, _ = self.quantize()
  File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 1706, in quantize
    return self._quantize_rtn()
  File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\torch\utils\_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 1490, in _quantize_rtn
    self._quantize_layer_via_rtn(name)
  File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 1360, in _quantize_layer_via_rtn
    self._immediate_pack(name)
  File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 1391, in _immediate_pack
    pack_gguf_layer(
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\torch\utils\_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\export\export_to_gguf\export.py", line 198, in pack_gguf_layer
    gguf_model.prepare_tensors()
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\export\export_to_gguf\convert.py", line 591, in prepare_tensors
    data, data_qtype = _quant_data(cls, data_torch, data_qtype, name, modify_name, bid, device=device)
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\export\export_to_gguf\convert.py", line 233, in _quant_data
    attr_tensor = attr_tensor.reshape(ori_shape)
RuntimeError: shape '[3072, 256]' is invalid for input of size 393216

Mistral-Magistral-Devstral:

2025-12-29 16:00:54 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 100%|██████████| 15/15 [00:00<00:00, 74.93it/s]
2025-12-29 16:00:55 INFO base.py L390: using torch.bfloat16 for quantization tuning
2025-12-29 16:00:55 WARNING base.py L419: using algorithm extension for quantization.
2025-12-29 16:00:55 WARNING utils.py L549: Current version of gguf export does not support for MistralModel, will re-download dependency file. Please restart the task.
Traceback (most recent call last):
  File "C:\Users\xxx\Documents\python\autoround\auto_next.py", line 66, in <module>
    autoround.quantize_and_save(output_dir, format='gguf:q3_k_s')
  File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 1055, in quantize_and_save
    format_list = self._parse_format_to_list(format)
  File "auto_round/alg_ext.py", line 140, in alg_ext.wrapper_func
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\base.py", line 835, in _parse_format_to_list
    gguf_args_check(self, formats, model_type=ModelType.TEXT)
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\compressors\utils.py", line 564, in gguf_args_check
    download_convert_file(redownload)
  File "C:\Users\xxx\Documents\python\autoround\venv\Lib\site-packages\auto_round\export\export_to_gguf\convert.py", line 84, in download_convert_file
    f.write(response.text)
  File "C:\ProgramData\anaconda3\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\uff5c' in position 47231: character maps to <undefined>
>>> 

seems different errors

how i check if auto-round use the right gguf export ?
which file is it?
convert_hf_to_gguf.py ? (its from date today)

with this command i can simpel convert granite to gguf and it runs:
(venv) c:\Users\xxx\Documents\python\autoround\llama.cpp>c:\Users\xxx\Documents\python\autoround\venv\Scripts\python.exe convert_hf_to_gguf.py e:\granite_small --outfile d:\model-f16.gguf --outtype f16

[wenhuach21]

Most of us are on holiday, and we’ll get back to you next week.