AI21 Labs releases Jamba, a hybrid SSM-Transformer model

AI21 Labs released Jamba, which it described as the first production-grade language model to combine the Mamba state-space architecture with conventional transformer attention layers. Where the field’s dominant models were built entirely from transformer blocks, Jamba interleaved them with Mamba layers — roughly one attention layer for every eight total layers — and added a mixture-of-experts component, giving the model 52 billion total parameters of which only about 12 billion were active for any given input.

State-space models like Mamba had circulated in research as a possible answer to the transformer’s central weakness: attention’s memory and compute cost grows with the square of sequence length, making very long contexts expensive. Jamba’s hybrid design let it support a 256,000-token context window while fitting up to 140,000 tokens of context on a single 80GB GPU, and AI21 reported three times the throughput on long-context workloads compared with Mixtral 8x7B, a similarly sized transformer mixture-of-experts model. On standard benchmarks AI21 said Jamba matched or outperformed other models in its size class.

The model was released with open weights under the Apache 2.0 licence, available through Hugging Face and NVIDIA’s inference tooling. Jamba was one of the clearest commercial signals that state-space architectures, long a research curiosity relative to the transformer, had reached practical viability for long-context production use, and it set a precedent AI21 built on with subsequent Jamba releases.