TII releases Falcon under Apache 2.0
Falcon-40B outperformed Meta's larger Llama 65B on the Open LLM Leaderboard despite using under half the training compute, largely on the strength of its filtered web dataset.
- Open weights & ecosystem
- Models & capabilities
- Notable
Abu Dhabi’s Technology Innovation Institute (TII) released Falcon, a family of open-weight language models at 7 billion and 40 billion parameters, plus instruction-tuned variants, distributed through Hugging Face under the Apache 2.0 licence — a fully permissive terms allowing unrestricted commercial use, without the usage restrictions some other nominally “open” releases carried at the time.
TII attributed Falcon’s performance chiefly to training data rather than architecture or scale. More than 80% of the training corpus came from RefinedWeb, a web dataset TII built by heavily filtering and deduplicating Common Crawl rather than relying primarily on curated sources such as books and code, the more conventional approach used by GPT-3 and Google’s PaLM. On Hugging Face’s Open LLM Leaderboard, the instruction-tuned Falcon-40B scored higher than Meta’s much larger LLaMA-65B, despite Falcon requiring under half the training compute — 2,800 petaflop-days against 6,300. TII released a 600-billion-token extract of RefinedWeb itself for others to use.
The release added a well-resourced, state-backed entrant to the small group of organisations shipping genuinely open, commercially usable frontier-scale weights that spring, alongside Meta’s more restricted LLaMA and Mistral’s forthcoming models, at a moment when most of the most capable systems — GPT-4, Claude, PaLM 2 — remained fully closed. Falcon’s showing was also read as evidence that careful data curation, not just parameter count or compute budget, could substitute for scale, a claim that shaped how competing open-weight labs, including Meta with Llama 2 the following month, described their own training-data choices.