Skip to content

Full finetuning with activation quantization #1108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
dxqb wants to merge 87 commits into
base: master
Choose a base branch
from
Draft

Full finetuning with activation quantization #1108

dxqb wants to merge 87 commits into from

Conversation

@dxqb
Copy link
Collaborator

@dxqb dxqb commented Nov 7, 2025 •
edited
Loading

based on #1034

image

dxqb and others added 30 commits October 4, 2025 17:42
@dxqb
Compile int svd
1e8fece
@dxqb
- fix cache dir
bbf6eb8 
 - increase offloading alignment to 16
 - disable grads for SVDQuant to save vram
@dxqb
hide checkpoints from LoRA saving
2676d33
@dxqb
fix buffer registration
dc289fa
@dxqb
fix buffer registration
73822b0
@dxqb
various
4821c9e
@dxqb
various
cb02a4c
@dxqb
various
efb9073
@dxqb
cleanup
35ba023
@dxqb
torch.compile bug workaround
5633b21
@dxqb
same workaround for Qwen
c37f805
@dxqb
gguf
d7532dc
@dxqb
merge
6883981
@dxqb
gguf
2de19c9
@dxqb
bugfix
a3cd936
@dxqb
requirements
bcf0b65
@dxqb
merge
0dcb3dc
@dxqb
name changes, axis wise
5a2e590
@dxqb
Merge branch 'compile_int8_svd' into compile_int8_svd_gguf
44a969b
@dxqb
merge
882154d
@dxqb
big type hint
e5317d3
@dxqb
Merge branch 'compile_int8_svd' into compile_int8_svd_gguf
7f07748
@dxqb
use axis-wise quantization for both forward and backward
c8cb33b
@dxqb
Merge branch 'upstream' into gguf
c6011d0
@dxqb
merge
b772d42
@dxqb
merge
3167b90
@dxqb
initial
2d4a0c3
@dxqb
merge Nerogar#1060
68f0f71
@dxqb
merge
0e282a2
@dxqb
ui fix
71af1f0
@dxqb
Copy link
Collaborator Author

dxqb commented Nov 11, 2025 •
edited
Loading

  • This PR works on recent models that are published in bf16, because then the original type remains BF16 and quantized to A8 during training. It fails on checkpoints that require conversion first, like SDXL checkpoints published in fp32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dxqb @O-J1