Transformers adds packed GGUF inference on Apple Silicon with ggml kernels

Hugging Face added an efficient packed-GGUF path to the main branch of Transformers on 22 September. On a supported Apple Silicon Mac, developers can select a GGUF checkpoint from the Hub, pass its filename to `from_pretrained`, and continue through the standard Transformers generation APIs instead of moving the model into a separate local-inference interface.

The same checkpoint can be exposed through `transformers serve`, which provides an OpenAI-compatible local API. The initial packed path targets Qwen3.5 dense and mixture-of-experts architectures, including compatible Qwen3.8 checkpoints. Hugging Face says users need Transformers main until the next stable release includes the work.

The implementation keeps supported quantised weights packed on Metal and calls ggml-derived kernels through Hugging Face’s `kernels` library. The integration includes packed matrix operations as well as ggml-based attention, normalisation and gated-delta-network kernels, while Hugging Face supplies a separate Metal top-k path for mixture-of-experts routing.

That distinction matters for memory use. Hugging Face warns that if a compatible quantisation kernel is unavailable, Transformers falls back to dequantising the model and consumes more memory. The model definition and generation loop still remain in Python and PyTorch, which lets developers use familiar hooks, evaluation code and decoding controls around the same GGUF checkpoint.

Hugging Face reports that its M2 Max measurements are close to llama.cpp across three selected GGUF checkpoints, but its own methodology says the two columns are not identical. The llama.cpp side uses `llama-bench` with prompt processing excluded, while the Transformers measurement generates from a 12-token prompt and therefore includes prefill; the Transformers result also uses the best of three warmed runs rather than the same averaging rule.

Those are author-run measurements rather than an independent performance comparison. Hugging Face continues to recommend llama.cpp when efficient local inference is the main priority, describing the Transformers path as useful when developers want GGUF checkpoints inside Python and PyTorch workflows for inspection, evaluation, conversion checks or experimentation.

Packed inference is MPS-only for now, and the initial target is a single interactive conversation on Apple Silicon. Hugging Face says padding and batching still need work, architecture coverage remains limited to the Qwen3.5 family and compatible Qwen3.8 checkpoints, and support for the GGUF file format should not be read as packed-kernel support on every device.

Hacker News surfaced the release but provides discovery provenance only. Hugging Face’s release post and inspectable source establish the feature itself, while broader claims about performance parity, quality under quantisation or behaviour on unsupported architectures remain outside what those primary sources establish.