If you are an EdTech company offering music lessons online and struggling with high student dropout rates — TELMI developed a multi-modal learning platform combining audio, video, and motion analysis that provides real-time augmented feedback to students practicing alone. The system was built with 5 partners across 3 countries, iterated through multiple prototype versions, and includes a public multimodal database of expert performances for benchmarking student progress.
AI-Powered Music Learning Platform Using Motion Sensors and Video Feedback
Imagine learning violin from a world-class teacher, but instead of meeting them once a week, you have a smart system that watches your posture, listens to your playing, and tracks your bow movements — giving you real-time tips just like having a teacher looking over your shoulder. TELMI built exactly that: a multi-sensor platform combining cameras, motion trackers, and audio analysis that captures how master violinists play and then coaches students to improve on their own. They also created a public database of expert performances that anyone can study, like a YouTube for violin technique but with full 3D motion data.
What needed solving
Music education still relies heavily on infrequent one-on-one lessons followed by long unsupervised practice sessions, leading to high student dropout. Students practicing alone have no way to get objective feedback on their technique, posture, or bowing — they essentially practice blind until the next lesson. Current digital tools rarely go beyond simple audio and video recording, missing the rich movement data that teachers observe in person.
What was built
TELMI built a complete multi-modal music learning platform with sensor hardware (EMF sensors, IR cameras, audio capture), a system platform iterated through 3 versions (first prototype, improved second version, final platform), and a public multimodal database with an HTML5 web visualizer for browsing and downloading expert performance recordings.
Who needs this
Who can put this to work
If you are a musical instrument manufacturer looking to add digital learning tools to differentiate your products — TELMI created data acquisition hardware using EMF sensors, IR cameras, and audio capture that can be integrated into practice environments. The final prototypes were fully integrated with database, user interface, and feedback systems, ready for product adaptation beyond the original violin use case.
If you are a conservatory or music school looking to improve student retention and scale your teaching capacity — TELMI built a complete system platform covering student-teacher, student-only, and collaborative learning scenarios. The platform was tested across 3 countries with pedagogical evaluation, and includes a publicly available reference database of multimodal recordings that institutions can use for curriculum integration.
Quick answers
What would it cost to license or adopt this technology?
The project had a total EU contribution of EUR 2,617,425 shared across 5 partners over 3 years. Licensing terms would need to be negotiated with the coordinator (Universidad Pompeu Fabra). The public database component is freely available, but the integrated platform and sensor systems would likely require a commercial agreement.
Can this scale beyond violin to other instruments or disciplines?
The project used violin as a case study, but the underlying multi-modal technology (audio, video, motion capture, EMF sensors, IR cameras) is instrument-agnostic. The platform architecture supports student-teacher, student-only, and collaborative learning scenarios, making it adaptable to other performance-based disciplines like dance, sports training, or physical rehabilitation.
What is the IP situation — who owns the technology?
The consortium includes 2 industry partners and 3 universities across Spain, Italy, and the UK. IP ownership would follow the Horizon 2020 grant agreement, typically shared among partners based on contribution. Contact the coordinator at Universidad Pompeu Fabra to clarify licensing availability for specific components.
How mature is the technology — is it ready to deploy?
The project delivered multiple prototype iterations: first working prototypes, second improved versions based on user feedback, and final platforms of data acquisition systems fully integrated with the database and feedback systems. A final version of the TELMI System Platform was delivered, suggesting the technology reached demonstration level.
What data and infrastructure does this require?
The system uses EMF sensors, IR cameras, and audio capture for data acquisition. A public multimodal database was created with an HTML5 web-based visualizer for browsing recordings. The platform requires the sensor hardware plus server infrastructure for the database and feedback engine.
Has this been validated with real users?
Based on available project data, the second version of prototypes was explicitly improved based on user feedback about usability and practical concerns during recordings. The project included pedagogical evaluation of effectiveness, and the consortium included both technical and pedagogical partners working in tight collaboration.
Are there regulatory concerns for use in educational settings?
Based on available project data, the system captures multimodal recordings (audio, video, motion) of students, which would require compliance with GDPR and educational data protection regulations. The public database component was designed for research use, so commercial deployment would need additional privacy and consent measures.
Who built it
The TELMI consortium is a compact group of 5 partners across Spain, Italy, and the UK, with a healthy 40% industry ratio (2 industry partners, both SMEs) alongside 3 universities. This mix signals genuine market interest — the SME involvement means commercial players saw enough potential to commit resources. The coordinator, Universidad Pompeu Fabra in Barcelona, is a well-known research university with strong music technology expertise. For a business looking to adopt or license this technology, the small consortium size means fewer parties at the negotiation table, and the SME partners likely have practical insights into commercialization pathways.
- UNIVERSIDAD POMPEU FABRACoordinator · ES
- ROYAL COLLEGE OF MUSICparticipant · UK
- UNIVERSITA DEGLI STUDI DI GENOVAparticipant · IT
- HIGHSKILLZ LIMITEDparticipant · UK
Universidad Pompeu Fabra (Barcelona, Spain) — reach the music technology research group through their institutional website or the TELMI project page.
Talk to the team behind this work.
Want an introduction to the TELMI team to discuss licensing or integration? SciTransfer can arrange a direct connection with the coordinator and relevant consortium partners.