Making Knowledge Distillation Cheap Enough to Run at Scale
The paper introduces a fused chunked KL loss to reduce memory and computational costs in knowledge distillation for large language models, achieving lower VRAM usage and enabling long-context distillation on a single GPU. It compares four loss implementations, showing the fused chunked loss has the lowest memory usage, with peak usage of around 128GB, and improves training efficiency. The method is evaluated on a H200 GPU, demonstrating near-identical training loss across all four methods and enabling large-scale distillation campaigns with a smaller student model that retains most of the teacher's accuracy.
More from Huggingface
LFM2.5-VL-3B for Better and Faster Vision Capabilities for the Edge
Thinking of ACE? We Can Do It with Fewer Tokens
Meta is back with Muse Glimmer: local, agentic, multimodal, and open source
