olmoe

AI2 releases OLMoE mixture-of-experts model

1 billion active of 7 billion total parameters, trained on 5 trillion tokens, released with 244 intermediate checkpoints and full training data and logs.

Open weights & ecosystem · Models & capabilities