If you are a hospital network dealing with communication breakdowns between staff and patients who don't speak the local language — this project developed a multimodal AI agent running on mobile devices that understands speech, facial expressions, and gestures to bridge language and cultural gaps in care settings. It was validated in 2 use cases with prolonged trials involving real migrant communities across Germany and Spain.
AI-Powered Multilingual Health Assistant for Migrants on Mobile Devices
Imagine moving to a country where you don't speak the language and then trying to navigate their healthcare system — finding doctors, understanding prescriptions, dealing with insurance forms. Now imagine having a smart assistant on your phone that not only translates but actually understands your facial expressions, tone of voice, and cultural background to help you communicate with care providers. That's what a team of 9 partners across 5 European countries built and tested with real migrant communities in Germany and Spain over three years.
What needed solving
Millions of migrants across Europe struggle to access healthcare because they cannot communicate effectively with care providers — they don't speak the local language, they're unfamiliar with the healthcare system, and cultural differences make them reluctant to seek help. For care providers and health systems, this leads to worse patient outcomes, higher costs from miscommunication, and social exclusion of vulnerable populations like elderly migrants in care homes and temporary care workers without adequate training.
What was built
The project built a socially competent AI agent for mobile devices that combines dialogue management with multimodal communication analysis — understanding speech, facial expressions, and gestures — to serve as a trusted mediator between migrants and healthcare systems. The final system (D7.5) was demonstrated and validated through prolonged trials with real migrant users across 2 use cases in Germany and Spain, with 16 deliverables produced in total.
Who needs this
Who can put this to work
If you are an elder care provider struggling to communicate with residents from migrant backgrounds — this project built a socially competent digital agent that acts as a trusted mediator for elderly migrants in care homes. The system was specifically tested with elderly Turkish migrants and their relatives in Germany, addressing social exclusion caused by language and cultural barriers.
If you are a health-tech company looking to add culturally aware communication features to your platform — this project advanced the state of the art in dialogue management and multimodal communication analysis (vocal, facial, gestural). The technology was developed by a consortium of 9 partners including 3 SMEs, making it a strong candidate for licensing or integration into existing digital health products.
Quick answers
What would it cost to license or integrate this technology?
The project was funded with EUR 3,633,801 in EU contribution across 9 partners over 3 years. Licensing terms would need to be negotiated with the coordinator, Universidad Pompeu Fabra. As EU-funded research, results may be available under favorable academic licensing conditions.
Can this scale to serve thousands of users across multiple languages?
The system was designed to run on standard mobile communication devices, which supports broad deployment. It was validated in 2 use cases covering Turkish, Polish, and North African migrant communities, demonstrating multilingual capability. Scaling to additional languages and larger user bases would require further development and testing.
Who owns the intellectual property?
IP from EU-funded RIA projects typically stays with the consortium partners who generated it. The 9-partner consortium spans 5 countries (Germany, Greece, Spain, France, Netherlands), so licensing discussions would likely involve the coordinator at Universidad Pompeu Fabra and relevant technology-developing partners.
Does this meet healthcare data regulations like GDPR?
The project ran from 2015 to 2018, overlapping with GDPR implementation. As an EU-funded project processing sensitive health and personal data from migrant trial participants, compliance measures would have been required. Specific GDPR compliance details should be confirmed directly with the consortium.
How long would it take to adapt this for our specific use case?
The project completed its final system demonstration (D7.5) after 3 years of development. With 16 deliverables produced and prolonged user trials completed, the core technology is mature. Adaptation timeline depends on your target language pairs, care context, and integration requirements.
Can this integrate with our existing hospital or care management systems?
The system was built to run on mobile devices as a standalone agent. Based on available project data, integration with hospital information systems or electronic health records would require additional development work. The multimodal input capabilities (voice, face, gesture) rely on device sensors.
Is there ongoing support or a development team behind this?
The project ended in February 2018. The consortium included 4 universities, 1 research organization, and 2 industry partners with 3 SMEs. Ongoing support would depend on whether consortium members have continued development. The coordinator at Universidad Pompeu Fabra in Spain is the best starting point.
Who built it
The 9-partner consortium across 5 countries (Germany, Greece, Spain, France, Netherlands) has a balanced mix with 4 universities providing research depth and 2 industry partners plus 3 SMEs bringing commercial perspective — though the 22% industry ratio is on the lower side for near-market solutions. The coordinator, Universidad Pompeu Fabra in Spain, is a strong research university but not an industry player, which means commercialization would likely need an external business partner or one of the SME consortium members to drive market entry. The geographic spread across Southern and Northern Europe suggests the technology was designed with multiple cultural and linguistic contexts in mind.
- UNIVERSIDAD POMPEU FABRACoordinator · ES
- ETHNIKO KENTRO EREVNAS KAI TECHNOLOGIKIS ANAPTYXISparticipant · EL
- ALMENDE BVparticipant · NL
- UNIVERSITAET ULMparticipant · DE
- VOCAPIA RESEARCHparticipant · FR
- EBERHARD KARLS UNIVERSITAET TUEBINGENparticipant · DE
- UNIVERSITAET AUGSBURGparticipant · DE
Universidad Pompeu Fabra, Barcelona, Spain — reach out to the ICT/computational linguistics department
Talk to the team behind this work.
Want an introduction to the KRISTINA research team? SciTransfer can connect you with the right people and help you evaluate fit for your specific use case.