Arcee AI has published the technical report for Trinity Large, the largest model in its Trinity family of open-weight Mixture-of-Experts (MoE) language models. Trinity Large has 400B total parameters, with only 13B active per token. The report also covers two smaller models: Trinity Nano (6B total, 1B active) and Trinity Mini (26B total, 3B active). All checkpoints are available on Hugging Face.

Why it matters

Enterprises want models they can audit, host, and adapt inside their own environments, with clear data provenance and licensing. Agentic and reasoning workloads also produce long outputs over long inputs, so inference efficiency is a hard requirement. Trinity was designed for both.

Architecture built for efficiency

Trinity Large is extremely sparse: 256 routed experts plus one shared expert, with four routed experts active per token. Attention follows a 3:1 pattern of local sliding-window layers to global layers, which cuts cost at long sequence lengths. Gated attention, QK-normalization, depth-scaled sandwich norm, and sigmoid routing complete the design. The models were trained with the Muon optimizer, which supports larger batch sizes and better sample efficiency than AdamW.

Data and training at scale

DatologyAI curated the pretraining data, including more than 8 trillion tokens of synthetic web, code, STEM, and multilingual data, making this one of the largest documented synthetic data efforts for pretraining. Trinity Large trained on 17 trillion tokens across three phases that shifted progressively toward code, math, and science. Training ran on 2,048 NVIDIA B300 GPUs managed by Prime Intellect. All three models finished pretraining with zero loss spikes.

Two new stability techniques

Early Trinity Large runs hit router instability and expert collapse. The team introduced Soft-clamped Momentum Expert Bias Updates (SMEBU), a load-balancing method that replaces fixed-size bias steps with smooth, momentum-dampened updates that settle instead of oscillating. They also built the Random Sequential Document Buffer (RSDB), a data loader that reduces domain imbalance within each batch. In the final training phase, RSDB cut batch heterogeneity by 4.23x and step-to-step loss variance by 2.4x.

Long context that extrapolates

Trinity Large was extended to 256K tokens by adjusting only the global attention layers. It scored 0.994 on multi-key needle-in-a-haystack at 256K and 0.976 at 512K, a length it never trained on. At 1M tokens it scored 0.42, which signals headroom for future versions.

Results

Trinity Large Base is competitive with GLM-4.5 Base, despite 4x higher sparsity and roughly 2.5x fewer active parameters. It posted 88.62 on MBPP+, 82.58 on MMLU, and 90.11 on HellaSwag. The instruct model, Trinity Large Preview, scored 87.21 on MMLU, 75.25 on MMLU-Pro, and 63.32 on GPQA Diamond. In FP8 inference tests on 8xH200, Trinity Large delivered higher throughput, faster time to first token, and lower end-to-end latency than DeepSeek-V3 and GLM-4.7 across every tested input/output length.

What’s next

Post-training was intentionally light, so Trinity Large Preview is a preliminary release. The Arcee research team plans more extensive post-training and flags greater sparsity and larger-batch training as keys to efficient scaling.