AirLLM

AirLLM is an open source Python library that dramatically reduces the memory required to run large language models, letting a 70B parameter model run on a single 4GB GPU without quantization, distillation, or pruning. It extends this same approach to enormous models, including Llama 3.1 405B on 8GB, DeepSeek-V3 at 671B parameters on roughly 12GB, and even Kimi K3, a 2.8 trillion parameter sparse mixture of experts model, on under 4GB of VRAM.

The core technique keeps only one transformer layer on the GPU at any given time, streaming the rest from disk as needed, so VRAM usage depends on layer size rather than total model size. For sparse MoE architectures, AirLLM goes further by streaming only the specific experts a token actually routes to instead of loading an entire layer. An optional block-wise quantization based model compression mode, using 4bit or 8bit settings, can further speed up inference by up to 3x with minimal accuracy loss, since the bottleneck is primarily disk loading rather than compute.

Key features include:

• AutoModel — automatically detects the model architecture from a Hugging Face repo ID, removing the need to specify a model class
• Broad model support — works out of the box with Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, and InternLM families
• MacOS support — runs on Apple Silicon using MLX alongside the standard Linux workflow
• CPU inference and compression configs — supports non sharded models, CPU execution, and configurable compression, prefetching, and token based access

AirLLM is released under the Apache 2.0 license and is actively maintained, with recent updates adding FP8 model support and Kimi K3 compatibility. The project is used through a simple pip install and a single AutoModel.from_pretrained call, and it credits SimJeg's Kaggle competition work as a foundation for its layer streaming approach.