Moonshot AI releases Kimi Linear architecture model

Moonshot AI released Kimi Linear, a 48-billion-parameter mixture-of-experts model with 3 billion active parameters, built around a hybrid attention mechanism the company called Kimi Delta Attention (KDA). Rather than using standard “full” attention throughout, the architecture mixes KDA layers with ordinary global attention in a 3:1 ratio, aiming to keep most of full attention’s quality while cutting most of its memory cost.

Moonshot reported that the design reduced key-value cache memory use by up to 75% and increased decoding throughput by up to six times at a context length of one million tokens, compared with a full-attention baseline. The accompanying paper claimed the architecture was, under matched training conditions, the first linear-attention design to outperform full attention rather than merely approach it — across short-context, long-context and reinforcement-learning settings, on a model trained on 5.7 trillion tokens.

Moonshot released the model weights, training recipe and inference kernels openly on GitHub and Hugging Face rather than keeping the architecture proprietary, continuing the pattern of Chinese labs using open release to establish technical credibility against closed Western frontier models. Linear and hybrid-attention architectures had been proposed by multiple research groups as a route around the quadratic memory cost of standard transformer attention at long context lengths; Kimi Linear was among the first such designs released at frontier model scale with reported gains large enough for competing labs to take seriously.