We’re proud to share our collaboration with Prime Intellect on their INTELLECT-1 initiative, which has achieved the first-ever fully decentralized training of a 10B-parameter model. This milestone marks a shift in how AI is built and scaled, unlocking possibilities for greater collaboration and efficiency in LLM training.

What is INTELLECT-1?

Imagine training a cutting-edge AI model, not confined to a single high-tech data center, but distributed across computers worldwide. That’s the essence of INTELLECT-1. Using their innovative PRIME framework, Prime Intellect successfully coordinated the training of a 10-billion-parameter language model on 1 trillion tokens.

The project relied on 30 independent compute providers, spread across continents, to contribute their resources. Prime Intellect’s system managed challenges like varying internet speeds and occasional node dropouts, achieving 83% compute utilization across continents (96% on US-only nodes) and proving the viability of decentralized training.

How Does INTELLECT-1 Compare?

The results speak for themselves. INTELLECT-1 demonstrates competitive performance compared to many models trained using traditional centralized methods. INTELLECT-1 performs comparably to LLaMA-2-7B and LLaMA-2-13B, placing it roughly a year behind current state-of-the-art models of its size which have benefited from larger compute budgets, more data, and advanced post-training processes.

While INTELLECT-1’s numbers are impressive for the world's first fully decentralized training project, they also highlight the gap between decentralized approaches and current state-of-the-art methods. Bridging this gap is our next big goal.

How Arcee Helped

We contributed 8,200 hours of cutting-edge H100 GPU compute time to INTELLECT-1. Our contributions didn’t end with training; we brought our expertise to post-training, fine-tuning the model to maximize its real-world performance.

Here’s what we did to enhance INTELLECT-1:

1. Supervised Fine-Tuning (SFT): Teaching the model to handle specific instructions and complex tasks with greater accuracy.
2. Direct Preference Optimization (DPO): Aligning outputs with human preferences for more helpful, intuitive responses.
3. Model Merging: Combining the best aspects of multiple training runs into one unified model.
4. Logit Distillation: Transferring knowledge from larger models to INTELLECT-1 without increasing its size, boosting its abilities significantly.

Post-Training Results

INTELLECT-1-Instruct, the instruction-tuned version of INTELLECT-1, showed marked improvements following post-training. As seen in the table below, it competes with larger and more recent models like LLaMA-2-13B on specific benchmarks, demonstrating the power of careful post-training processes.

While INTELLECT-1-Instruct has made remarkable progress, we acknowledge that these results reflect a system still catching up to models built in centralized environments.

We've known that merging, distilling, and targeted training have healing effects, but we were blown away by the performance improvement compared to the base model. Current optimization techniques are inherently chaotic, especially the DiLoCo SGD algorithm. INTELLECT-1 shows how decentralized innovation and collaborative training can deliver unprecedented results, bridging the gap between experimentation and real-world applications.

All of our datasets are public and thoroughly checked for contamination. This reinforces that model merging, distilling, and future post-training improvements are key to unlocking the full potential of globally distributed trained models.

As part of the INTELLECT-1 release, we’re excited to share that we’re open-sourcing two of our proprietary datasets under the Apache-2.0 license. In the spirit of open science, we’re releasing logits extracted from Llama-3.1-405B during the development of SuperNova, as well as the complete EvolKit-75k dataset, a high-quality instruction tuning dataset generated using our in-house tools. Additionally, everything from the INTELLECT-1 project is being made open-source: data, checkpoints, and the code behind the PRIME framework. This marks a significant step toward making cutting-edge AI research more collaborative, accessible, and transparent. You can find them on HuggingFace.

Decentralized Training: The Future is Now

Prime Intellect’s technical ingenuity and vision made INTELLECT-1 a reality. They’ve proven that top-tier AI training no longer needs to depend on massive, centralized facilities. This breakthrough opens up new possibilities for collaborative, decentralized AI development that is more accessible to the global research community.

INTELLECT-1 is a glimpse into what decentralized training can achieve. While we acknowledge that it’s still catching up to state-of-the-art centralized models, the gap is closing. With the right innovations and partnerships, we’re excited to accelerate this progress, transforming AI development into a more inclusive, efficient, and scalable process.