Arcee Spark: A Compact, Efficient 7B Language Model

1 min read • Jun 24, 2024
Arcee Spark is a 7B model built on Qwen2, merged with MergeKit and refined with DPO. It beat GPT-3.5 and Claude 2.1 on MT-Bench at launch.
Arcee Spark is a 7B parameter language model designed to deliver high performance in a compact package. It demonstrates how smaller models can achieve competitive results when compared to their larger counterparts.
It is the highest-scoring model by far in the 7B-15B range, outperforming Mixtral-8x7B and Llama-3-8B-Instruct. Arcee Spark also outperforms bigger models including GPT 3.5 and Claude 2.1 on MT-Bench, which is a benchmark heavily correlated with performance on lmsys' chatbot arena.
Key Features of Arcee Spark
- 7B parameters
- Initialized from Qwen2
- Fine-tuned on 1.8 million samples
- Merged with Qwen2-7B-Instruct using Arcee's MergeKit
- Further refined using Direct Preference Optimization (DPO).
Performance
Arcee Spark has shown strong performance across key benchmarks:
- EQ-Bench: 71.4
- GPT4All Evaluation: Average of 69.37 across diverse language tasks.
Potential Applications
The model's combination of compact size and strong performance makes it suitable for:
- Real-time applications like chatbots and customer service automation
- Edge computing scenarios
- Cost-effective AI implementation across organizations
- Rapid prototyping of AI-powered features
- On-premise deployment for enhanced data privacy.
Efficiency and Flexibility
Arcee Spark offers:
- Faster inference times compared to larger models
- Lower computational requirements
- Adaptability for fine-tuning to specific domains or tasks.
Availability
The model is available in three main versions:
- GGUF quantized versions for efficiency and easy deployment
- BF16 version (main repository)
- FP32 version for maximum performance, scoring slightly higher on benchmarks.
Conclusion
Arcee Spark demonstrates the groundbreaking potential of optimized smaller models in the field of natural language processing. It offers a balance of performance and efficiency, making it a viable option for numerous AI applications.
Editor’s note: Llama-Spark is our successor to our original Arcee-Spark, incorporating advancements and learnings from our ongoing research and development.


