AI researchers have made significant strides in training large language models on vast datasets, but until now, most of these efforts have focused on English and a handful of other widely spoken languages. In a new study published by Hugging Face, a team of scientists has announced a groundbreaking approach to pretraining models on a staggering 2.8 trillion tokens of data specifically in Indian languages, paving the way for more accessible and efficient AI development in these languages.
This open data generation pipeline, developed in collaboration with NVIDIA, marks a major milestone in the field of machine learning and natural language processing. By leveraging the power of NVIDIA GPUs and innovative data curation techniques, the researchers have created a scalable and reproducible method for generating high-quality training data in 14 Indian languages, including Hindi, Bengali, and Marathi.
What happened
On October 16, 2026, Hugging Face published a blog post titled "Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline." The post discussed the company's efforts to scale up pretraining for Indic languages using 2.8 trillion tokens of data. Hugging Face also announced an open data generation pipeline to help advance research and development in this area.
As part of their efforts, Hugging Face released a new model called "Indic-750B" which was trained on 750 billion tokens of data. This model is designed to help improve natural language processing tasks for Indic languages.
In addition to the model release, Hugging Face also made available 2.8 trillion tokens of open data for use by the research community. This data was generated using a new data generation pipeline that Hugging Face has developed.
The post also discussed some of the challenges that Hugging Face faced during the development of the new models and data generation pipeline. These challenges included dealing with the lack of available data for Indic languages and the need to develop new techniques for generating high-quality data.
phpWhat to do
- Explore the open data generation pipeline and consider contributing your own datasets to help expand the available resources for Indic language models.
- Experiment with the pre-trained Indic models released by Hugging Face, and share your findings to help advance the state-of-the-art in Indic natural language processing.
- Engage with the Indic NLP community through forums, social media, and conferences to stay up-to-date on the latest research and developments.
- Contribute to open-source projects related to Indic NLP, such as translating documentation or improving existing models.
- Consider incorporating Indic language models into your own applications or products to improve accessibility and user experience for the millions of people who speak these languages.
2TI lens
The Hugging Face blog post describes a significant milestone in the field of Indian language AI with a pretraining dataset of 2.8 trillion tokens, an impressive feat made possible by an open data generation pipeline. This achievement aligns with our approach at Spatial Digital Agency of leveraging cutting-edge AI to enhance the living office experience and provide best-in-class PaaS solutions. As we continue to innovate and push the boundaries of what's possible with AI, developments like this pretraining dataset will undoubtedly play a key role in enabling us to deliver even more impactful and personalized experiences to our clients.
Sources
Nvidia's Hugging Face announced the release of a new open-source data generation pipeline for Indian languages, containing 2.8 trillion tokens. The Scaling Indic Pretraining model was published on October 16, 2023. Hugging Face has been a leader in creating large language models and made this announcement to further advance the state of the art in AI for Indian languages.