Imagine building a giant digital library with trillions of words and then teaching a computer to read and translate it all. This effort uses some of the world's most powerful supercomputers to clean up messy internet data and turn it into smart translation tools. It's like creating a universal translator that works for many different languages, including those rarely used in tech.
Companies struggle to find high-quality, cleaned, and ethically compliant training data for AI in non-English languages. This leads to poor translation accuracy and high costs when scaling to diverse European markets.
A massive collection of cleaned monolingual and bilingual corpora (v1.2) and a suite of open-source language and translation models hosted on HuggingFace.
If you are a translation agency dealing with high costs for rare language pairs — this project developed translation models for 18 low-resource languages paired with English that reduce the need for expensive manual drafting.
If you are a support provider dealing with poor accuracy in automated bots for EU languages — this project developed encoder-only language models for 75 languages that improve response quality and regulatory compliance.
If you are a research firm dealing with the difficulty of cleaning massive web-scraped data — this project developed a pipeline that processed 1.85 PB of data from the Internet Archive and Common Crawl.
The resulting models and data sets are open, free, and available from established language repositories for anyone interested in research or innovation.
Yes, the project utilized industrial-scale compute, securing 10 million GPU and 10 million CPU hours on HPC facilities like the LUMI supercomputer.
Based on available project data, the models are open and released through the HPLT website and HuggingFace platform for public use.
The project applies cleaning and privacy protecting services, including anonymization for bilingual datasets and a blacklist to remove adult content.
The project complements the European Language Grid, which is intended to be used for the actual deployment of these models.
The consortium is heavily weighted toward academic and research expertise, with 5 universities and 1 research center. However, it maintains a 25% industry ratio with 2 industrial partners, including a private NLP company, ensuring that the high-performance computing outputs are aligned with market needs.
Contact UNIVERZITA KARLOVA in Prague for technical specifications on the 75 language models.
Contact SciTransfer to find the specific HPLT model that fits your target language market.
