The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization

Chao Li · Shigeng Wang · Anbang Yao

Our work reveals and mitigates Optimization Imbalance in learning-based PTQ under sequential quantization:

• Optimization Imbalance arises because reconstruction loss magnitudes vary dramatically across quantization stages, and MSE translates these differences into highly uneven gradients and parameter updates.
• RMSE breaks this loss-dependent coupling through implicit gradient normalization, enabling more balanced optimization across stages.

Code and quantized model checkpoints are coming soon.