Krutrim-1

India is one of the most vibrant and culturally diverse societies. Developing a general-purpose artificial intelligence system tailored for the Indian market presents unique challenges. These include accounting for the nation's cultural nuances, accommodating its linguistic diversity with numerous regional languages, adapting to the prominence of oral traditions, ensuring accessibility to relevant data sets, and achieving scalability to serve the vast population effectively. Careful consideration and innovative approaches are necessary to navigate these complexities successfully.

Existing foundation models for natural language tasks are predominantly trained on English data, limiting their effectiveness for languages native to India's over 1 billion citizens. Thousands of regional languages, dialects, language or code mixing pose representation challenges, exacerbated by sparse training data; Indic languages comprise just 1% of Common Crawl corpora despite India representing 18% of the global population. Consequently, lack of Indic language relevance and context representation leads current models to exhibit cultural and linguistic biases oriented towards Western contexts

We present Krutrim Large Language Model (LLM), a 2 trillion token multilingual foundation model designed to serve Indian demographic needs through equitable representation of the country's array of native tongues. Training data incorporates the largest known Indic language dataset, mitigating associated data scarcity obstacles that encumber model parity across dialects. Evaluations demonstrate Krutrim’s strong performance on Indic language benchmarks, surpassing or at par with state-of-the-art models despite being significantly smaller in training flops. Krutrim LLM also matches or exceeds standards set on English benchmarks by models trained on comparable flops (e.g. vs LLAMA-2 on 10 out of 16 tasks with average score of 0.57 vs 0.55 of LLAMA-2), evidencing flexible multilingual fluency. Through intentional design choices that redress endemic data imbalances, Krutrim LLM signifies meaningful progress in the pursuit of ethical, globally representative AI foundation models.

Multilingual Language Understanding and Generation

Krutrim-1 is natively multilingual and delivers promising performance on Indic language understanding and generation.

Backbone for Indic Multimodal Models

The model has been used as the backbone for our vision-language model - Chitrarth-1, as well as our speech-language model - Dhwani-1.

Cultural Sensitivity: Krutrim's ability to respect cultural practices. Krutrim is unbiased and does not favour one religion or caste.

Translation: Krutrim-1 demonstrates proficiency in translation tasks on Indic languages

The Krutrim model architecture draws from the standard decoder only transformer framework. We enlist key parameters of the model in the table below.

We trained the 7B parameter model on the context length of 4096 tokens. We use the ALiBi positional encoding method which helps in expanding the context length. We also leverage GQA for faster inference and lower KV cache memory footprint. We use clipping of QKV matrix values for stable training. The standard ReLU activation function is used.

Existing open source tokenizers do not perform well on Indic languages leading to high token to word ratio. A sub-optimal tokenizer leads to inferior training and inference performance in terms of speed and accuracy. In order to address that, we trained the tokenizer from scratch optimized for English and Indic languages.

We pre-trained Krutrim-1 LLM on a dataset of 2 Trillion tokens. This was followed by supervised finetuning for instruction following on various tasks covering translation, summarization, general knowledge, coding, safety and others.

En Benchmarks

We evaluate the Krutrim-1 fine-tuned model on various English benchmark tasks and compare against Llama-2 7B chat SFT.

Indic Benchmarks

API Integration

Run locally