Tencent open-sources Hunyuan-Large MoE model

Tencent open-sourced Hunyuan-Large (also called Hunyuan-A52B), publishing pretrained, instruction-tuned and FP8-quantised checkpoints on Hugging Face along with a technical report and training and inference documentation. The model used a mixture-of-experts architecture with 389 billion total parameters, of which 52 billion were active for any given inference pass, which Tencent said made it the largest openly released Transformer-based MoE model at the time. The pretrained checkpoint supported a 256K-token context window, with the instruction-tuned version supporting 128K.

Tencent reported that Hunyuan-Large’s instruct variant outperformed Meta’s much larger Llama 3.1 405B on several benchmarks despite activating a small fraction of the parameters per token — for example scoring higher on MATH (77.4 versus 73.8) and marginally higher on MMLU. The comparison illustrated the central argument for mixture-of-experts architectures at this point in the field: routing computation through a subset of specialised sub-networks per token could match or exceed a much larger dense model’s benchmark performance while activating far fewer parameters at inference time, lowering the compute cost of serving the model.

The release added Tencent to a widening group of Chinese labs — including Alibaba’s Qwen and, weeks later, DeepSeek’s V3 — publishing large open-weight MoE models through late 2024, at a moment when the largest Western labs (OpenAI, Anthropic, Google DeepMind) kept their most capable models closed. Hunyuan-Large’s release came with training scripts and model implementation code, but not the underlying training data, so the model could be run, fine-tuned and studied by outside researchers without being reproducible from scratch.