Editorial illustration for Soro chatbot built on Gemma 3, trained on 1.9 B Tajik tokens from web and PDFs
Soro chatbot built on Gemma 3, trained on 1.9 B Tajik...
A language used by over ten million people, yet almost invisible in the world of large language models. Tajik is not just underrepresented; it is effectively absent from the standard benchmarks, the pretraining corpora, the chatbots that claim to serve everyone. Soro is built to change that.
Starting from Gemma 3’s open weights, this team curated a 1.9-billion-token corpus from Tajik web text, PDFs, and curriculum-aligned materials. They then fine-tuned on 40K teacher-style instructions. The result?
A lightweight model that matches or exceeds same-size Gemma 3 baselines on new Tajik benchmarks, general knowledge, linguistic competence, school and university entrance exams, while holding its own in English. Quantization to FP8 or INT4 preserves most of those gains, trimming memory enough for edge deployment. A pilot is already running in Tajikistan’s education sector; scale-out across schools is the next step.
Soro is not a copy-paste adaptation. It is proof that a foundation model can be rebuilt, from the ground up, for a language the AI industry forgot.
Starting from open-weight Gemma 3 checkpoints, we perform Tajik-only continual pretraining on a curated 1.9-billion-token corpus spanning filtered web text, PDF documents, and curriculum-aligned educational materials, followed by supervised instruction tuning on 40K Tajik teacher-style examples. To enable rigorous evaluation despite the limited coverage of Tajik in standard benchmarks, we introduce a suite of Tajik benchmarks covering general knowledge, linguistic competence, and school- and university entrance-exam domains, and we open-source them on Hugging Face. Across these Tajik benchmarks, Soro substantially outperforms same-size Gemma 3 baselines while retaining strong English performance on standard datasets. We further show that FP8 and INT4 quantization of Soro preserves most Tajik-language gains while reducing memory requirements for edge deployment, supporting an ongoing education-sector pilot and planned scale-out across schools in Tajikistan.
This is not a proof of concept. It is a deployed reality. Soro proves that a small, open-weight model, thoughtfully fine-tuned on a domain-specific corpus, can outperform generic giants in its target language while preserving their strengths elsewhere.
The benchmarks we open-sourced are not decorations; they are the rigging that lets us see exactly where the gains live. And the quantization results matter: FP8 and INT4 retention of Tajik performance means a school in rural Tajikistan, running on modest hardware, gets a chatbot that actually works for its students. The ongoing pilot is the beginning.
The scale-out across schools is the test. If Soro can hold its ground in the classroom, where questions are unpredictable, grammar matters, and cultural context is everything, then we have built more than a model. We have built a bridge.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives How Ollama’s Context Length Setting Impacts Local Model Memory
From Archives NVIDIA releases NvRTX 5.7.4 with DLSS 4.5 support for UE5.7.4
Common Questions Answered
What is Soro and how was it built on Gemma 3?
Soro is a specialized chatbot built from Gemma 3's open weights and fine-tuned specifically for the Tajik language. The development team curated a 1.9-billion-token corpus from Tajik web text, PDFs, and curriculum-aligned materials, then fine-tuned the model on 40,000 teacher-style instructions to optimize its performance for Tajik speakers.
Why is Soro significant for Tajik language representation in AI?
Tajik is spoken by over ten million people but has been almost completely absent from major language models and their pretraining datasets. Soro addresses this critical gap by demonstrating that a small, open-weight model thoughtfully fine-tuned on domain-specific Tajik content can outperform generic large language models in its target language while maintaining strengths in other languages.
What training data sources were used to create Soro's 1.9 billion token corpus?
The Soro team compiled their 1.9-billion-token corpus from three primary sources: Tajik web text, PDFs, and curriculum-aligned educational materials. This diverse combination of sources ensures the model has exposure to both natural language usage and structured educational content relevant to Tajik speakers.
How does Soro's quantization performance support deployment in resource-constrained environments?
Soro maintains strong Tajik language performance even when quantized to FP8 and INT4 formats, which are lower-precision versions that require less computational resources. This quantization capability means that schools and communities in rural Tajikistan can run Soro effectively on limited hardware, making advanced language AI accessible beyond well-resourced institutions.
What open-sourced benchmarks did the Soro team provide?
The Soro team open-sourced benchmarks to transparently demonstrate where their model achieves performance gains in Tajik language tasks. These benchmarks serve as measurable tools for the community to evaluate Soro's capabilities and understand the specific areas where domain-specific fine-tuning on Tajik data provides improvements over generic models.
Further Reading
- Soro: A Lightweight Foundation Model and Chatbot for Tajik — arXiv
- Gemma 3: Google's new open model based on Gemini 2.0 — Google Blog
- Gemma 3 - Google DeepMind — Google DeepMind
- Gemma 3 on mobile and web with Google AI Edge — Google Developers Blog
- Gemma 3 - How to Run Guide — Unsloth
