ESCAPE · Project

Open Data Lake and ML Archive Tools for Managing Massive Scientific Datasets

digitalTestedTRL 5Thin data (2/5)

Imagine the world's biggest telescopes and particle accelerators all generating mountains of data every second — but none of them can easily share or search each other's files. ESCAPE built a shared cloud-based "data lake" so that all these instruments can pool their data in one place, find what they need, and run analysis tools on it. They also created machine learning tools that automatically tag and organize archive data, like a smart librarian for petabytes of scientific records. The goal was to make all this data openly available and searchable, following the same standards used across European open science.

By the numbers

consortium partners across Europe

countries represented in the consortium

project deliverables produced

industry partners including SMEs

14%

industry participation ratio in consortium

The business problem

What needed solving

Organizations managing massive, distributed datasets — whether from scientific instruments, IoT sensors, or industrial systems — struggle to make that data findable, accessible, and usable across teams and sites. Data sits in silos, formats clash, and searching across archives is manual and slow. Without interoperable tools, valuable insights get buried in petabytes of unstructured records.

The solution

What was built

ESCAPE produced 32 deliverables including a prototype machine learning-enabled archive service that automatically enriches and classifies stored data, a federated data lake infrastructure connecting distributed storage into one searchable cloud facility, an open-source software catalogue integrated with the European Open Science Cloud, and a pilot implementation plan for FAIR-compliant data stewardship across institutions.

Audience

Who needs this

Cloud infrastructure providers building federated data lake solutions for multi-site clientsML/AI companies developing intelligent archive search and classification productsResearch data management companies helping institutions meet FAIR and open science mandatesTechnology integrators connecting scientific instruments to cloud analysis platformsData compliance firms advising on EU open data regulations for publicly funded research

Business applications

Who can put this to work

Cloud Infrastructure & Data Management

enterprise

Target: Cloud service providers and data platform companies managing large-scale scientific or industrial data

If you are a cloud platform provider dealing with clients who need to store, catalog, and analyze petabyte-scale datasets from multiple sources — ESCAPE developed a federated data lake architecture that connects distributed storage into one searchable, FAIR-compliant system. Their approach was tested across 36 partner institutions in 8 countries, handling data from telescopes and particle accelerators. The open-source tools and interoperability standards could be adapted for any industry facing multi-source big data integration.

AI & Machine Learning Services

any

Target: ML companies building intelligent search, classification, or archive management products

If you are an ML services company looking for proven approaches to automatically classify and enrich massive data archives — ESCAPE released a prototype machine learning-enabled archive service that adds value to stored scientific records. This was tested on real astronomy and particle physics datasets with 32 deliverables produced across the project. The approach could be repurposed for any domain where large archives need smart tagging, search, and content discovery.

Research Data Services

mid-size

Target: Companies providing data management, compliance, or open-data solutions to research institutions

If you are a research data services company helping universities and labs meet FAIR data requirements and open science mandates — ESCAPE built a working catalogue of open-source analysis software integrated with the European Open Science Cloud (EOSC). Their 36-partner consortium validated interoperability standards across astronomy and particle physics. These standards and tools represent a ready-made compliance toolkit for institutions needing to meet EU open data policies.

Frequently asked

Quick answers

What would it cost to adopt ESCAPE's data lake tools?

The software tools and catalogues developed by ESCAPE are open-source, meaning there are no licensing fees for the core technology. However, deployment costs would depend on your infrastructure scale — setting up a federated data lake across multiple sites requires cloud storage, compute resources, and integration work. Based on available project data, specific pricing was not published.

Can this scale to industrial data volumes?

ESCAPE was designed specifically for the extreme data volumes produced by instruments like the Large Hadron Collider and the Square Kilometre Array — among the largest data generators in the world. The data lake architecture was tested across 36 institutions in 8 countries. This suggests the technology is built for petabyte-scale operations, though adapting it to non-scientific industrial use cases would require additional validation.

What is the IP situation — can we use this commercially?

ESCAPE was funded as an RIA (Research and Innovation Action) and focused on open science principles, with outputs including an open-source software catalogue integrated into the EOSC. Most tools are likely available under open-source licenses. For specific licensing terms on individual components, you would need to check with the coordinator (CNRS) or the relevant consortium partner.

Is this production-ready or still experimental?

The project delivered 32 outputs including a prototype machine learning-enabled archive service and a pilot implementation plan. The tools were tested in real research environments across the consortium. This puts the technology at a late prototype or early pilot stage — functional in controlled settings but not yet deployed as a commercial product.

How does this integrate with existing data systems?

ESCAPE was built specifically for interoperability — connecting existing infrastructure like the astronomy Virtual Observatory with the European Open Science Cloud. The federated data lake approach means it layers on top of existing distributed storage rather than replacing it. Integration with commercial data platforms would require adapting the EOSC-oriented connectors to your specific stack.

Who built this — is the team credible?

The consortium includes 36 partners across 8 countries, led by CNRS (France's national research center). The team includes 21 research organizations, 9 universities, and 5 industry partners, with direct involvement from CERN, ESO, and major ESFRI research infrastructure projects. This is one of the strongest possible teams for large-scale scientific data management in Europe.

Are there regulatory advantages to using FAIR-compliant tools?

Yes — EU research funding increasingly mandates FAIR data principles (Findable, Accessible, Interoperable, Reusable) and open science compliance. Organizations using ESCAPE's FAIR-aligned tools would have an easier path to meeting these requirements. This is especially relevant for companies providing data services to publicly funded research institutions.

Consortium

Who built it

ESCAPE's 36-partner consortium is heavily research-dominated, with 21 research organizations and 9 universities making up the bulk of the team. Only 5 partners (14%) come from industry, all of them SMEs. The consortium spans 8 countries (Belgium, Switzerland, Germany, Spain, France, Italy, Netherlands, UK) and is led by CNRS, France's largest public research body. The presence of heavyweights like CERN and ESO signals deep technical credibility in large-scale data management, but the low industry ratio means commercial translation will require active business partnership beyond the existing consortium. For a company interested in this technology, the entry point would likely be through one of the 5 SME partners who bridge the research-to-market gap.

CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE CNRSCoordinator · FR
JOINT INSTITUTE FOR VERY LONG BASELINE INTERFEROMETRY AS A EUROPEAN RESEARCH INFRASTRUCTURE CONSORTIUM (JIV-ERIC)participant · NL
UNIVERSITA DEGLI STUDI DI ROMA TOR VERGATAparticipant · IT
FRIEDRICH-ALEXANDER-UNIVERSITAET ERLANGEN-NUERNBERGparticipant · DE
SKA ORGANISATIONparticipant · UK
TRUST-IT SERVICES SRLthirdparty · IT
STIFTUNG INSTITUT FUR SONNENPHYSIK (KIS)participant · DE
RUPRECHT-KARLS-UNIVERSITAET HEIDELBERGparticipant · DE
GSI HELMHOLTZZENTRUM FUR SCHWERIONENFORSCHUNG GMBHparticipant · DE
UNIVERSITE DE STRASBOURGthirdparty · FR
EUROPEAN SOUTHERN OBSERVATORY - ESO EUROPEAN ORGANISATION FOR ASTRONOMICAL RESEARCH IN THE SOUTHERN HEMISPHEREparticipant · DE
HITS GGMBHparticipant · DE
LEIBNIZ-INSTITUT FUR ASTROPHYSIK POTSDAM (AIP)participant · DE
AGENCIA ESTATAL CONSEJO SUPERIOR DE INVESTIGACIONES CIENTIFICASparticipant · ES
FACILITY FOR ANTIPROTON AND ION RESEARCH IN EUROPE GMBHparticipant · DE
SURF BVparticipant · NL
UNIVERSIDAD COMPLUTENSE DE MADRIDparticipant · ES
ISTITUTO NAZIONALE DI ASTROFISICAparticipant · IT
INSTITUTO DE FISICA DE ALTAS ENERGIASparticipant · ES
COMMPLA SRLthirdparty · IT
KONINKLIJKE STERRENWACHT VAN BELGIEparticipant · BE
ORGANISATION EUROPEENNE POUR LA RECHERCHE NUCLEAIREparticipant · CH
TRUST-IT SERVICES LIMITEDparticipant · UK
CHERENKOV TELESCOPE ARRAY OBSERVATORY GEMEINNUTZIGE GMBHparticipant · DE
RIJKSUNIVERSITEIT GRONINGENparticipant · NL
MAX-PLANCK-GESELLSCHAFT ZUR FORDERUNG DER WISSENSCHAFTEN EVparticipant · DE
INSTITUTO NACIONAL DE TECNICA AEROESPACIAL ESTEBAN TERRADASparticipant · ES
THE OPEN UNIVERSITYparticipant · UK
DEUTSCHES ELEKTRONEN-SYNCHROTRON DESYparticipant · DE
STICHTING NEDERLANDSE WETENSCHAPPELIJK ONDERZOEK INSTITUTENparticipant · NL
OROBIX SRLparticipant · IT
Universita' degli Studi di Urbino Carlo Bothirdparty · IT
THE UNIVERSITY OF EDINBURGHparticipant · UK
EUROPEAN GRAVITATIONAL OBSERVATORY(EGO) (OSSERVATORIO GRAVITAZIO NALEEUROPEO)participant · IT
ISTITUTO NAZIONALE DI FISICA NUCLEAREparticipant · IT

How to reach the team

CNRS (Centre National de la Recherche Scientifique), France — contact through project website or CORDIS portal

Order a report on this consortium See CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE CNRS profile →

Next steps

Talk to the team behind this work.

Want to explore how ESCAPE's open data lake and ML archive tools could solve your data management challenges? SciTransfer can connect you directly with the right consortium partner for your use case.

Order a report on this topic More in Digital & ICT