AI’s Language Gap Is Closing – But Performance Shifts Between Model Releases, Warns RWS’s TrainAI Study

Findings underscore that successful enterprise AI strategies require continuous validation built on high-quality, culturally nuanced data

RWS (RWS.L), a global AI solutions company, announced findings from its latest TrainAI Multilingual LLM Synthetic Data Generation Study, revealing that while leading large language models (LLMs) are closing the global language gap, their performance from one model generation to the next can be unpredictable. The findings underscore the need for continuous, expert-led evaluation to ensure that enterprises select the right model for their specific business needs.

Also Read: AiThority Interview with Glenn Jocher, Founder & CEO, Ultralytics

A shrinking global language gap is one of the study’s most significant findings. The research shows the performance gap between well-supported languages like English and underrepresented ones has narrowed significantly. While noting an industry-wide trend of language improvements with models like GPT and Claude Sonnet showing meaningful gains, the research highlighted standout performance from Google’s Gemini Pro. It achieved high-quality scores (above 4.5 out of 5) in Kinyarwanda – a language where previous model generations struggled to produce coherent text.

“This study signals a transformative moment that’s not about replacing human expertise, but about elevating it with the right technology,” said Vasagi Kothandapani, CEO, TrainAI by RWS. “As AI becomes more capable across languages, the need for deep cultural intelligence and human validation is more critical than ever. This is why RWS is guiding enterprises into this new reality by integrating these powerful technologies into content workflows with experts in the loop to ensure accuracy, cultural resonance, and brand consistency on a global scale.”

The study also uncovered a significant caveat for enterprises: AI progress is not necessarily linear. The research identified a “benchmark drift,” where LLM capabilities can unexpectedly shift from one version to the next. For instance, the study found that the latest version of GPT fell behind smaller models on several content generation tasks, where its predecessor had been competitive. Core metrics like tokenizer efficiency, which impacts cost, also varied significantly between model generations. The study shows that model upgrades reshuffle strengths and weaknesses unpredictably, reinforcing the need to re-evaluate even familiar model families with each new release.

“A model’s real-world value often comes down to specific, frequently overlooked metrics,” noted Tomáš Burkert, Head of Innovation, TrainAI by RWS. “Factors like tokenizer efficiency, which can make one model 3.5 times more cost-effective than another in certain languages, are critical. The foundation of a successful AI strategy is a continuous validation process, rooted in high-quality, culturally-nuanced AI data, to ensure you’re not just adopting a model, but rather the optimal model to address your unique enterprise requirements.”

The study concludes that as the AI landscape continues to rapidly evolve, enterprises must move beyond public leaderboards and perform continuous, independent evaluation with each new model release to ensure it’s still the right fit for their specific AI use case.

Also Read: ​​The Infrastructure War Behind the AI Boom

[To share your insights with us, please write to psen@itechseries.com]