Georgi Gerganov releases llama.cpp

Bulgarian developer Georgi Gerganov published llama.cpp, a from-scratch C/C++ implementation for running Meta’s LLaMA language models, days after the model’s weights leaked onto the open internet following Meta’s restricted research release. Where the original model required a Python stack, a CUDA-capable GPU and substantial memory to run, llama.cpp had no external dependencies and could run on ordinary CPUs — Apple Silicon Macs and standard x86 laptops among the first widely reported cases.

The project’s central technique was quantisation: compressing the model’s weights from 16- or 32-bit floating-point numbers down to as few as a handful of bits per parameter, trading a small amount of output quality for a large reduction in memory footprint and compute requirement. Combined with hand- written, hardware-specific optimisations, this let a 7-billion-parameter LLaMA model run at usable speed on consumer hardware that had no business running a large language model at all.

llama.cpp grew rapidly through community contribution rather than corporate backing. Its GGML file format, and the GGUF format that superseded it later in 2023, became a de facto standard for distributing quantised open-weight models. Tools such as Ollama and LM Studio were built directly on top of it, and llama.cpp became the substrate most “run an LLM on your own machine” tooling depended on for the next several years, independent of which lab’s weights a user wanted to run.

The project mattered less for any single technical innovation than for what it enabled: it decoupled running a capable language model from having datacentre-grade hardware or a cloud API key, at the exact moment open-weight models began proliferating, and made local inference a practical option rather than a theoretical one.