Imagine if a few giant companies decided which books you could find in a library based on who paid them the most. This project builds a public, open-source library index for the entire internet that anyone in Europe can use. It lets businesses and researchers find information without being forced to follow the rules of a few non-European gatekeepers.
European businesses are dependent on a few non-European search gatekeepers, leading to biased information access and high costs for those wanting to use web data for innovation.
An Open Web Index (OWI) and an Open Web Search and Analysis Infrastructure (OWSAI), including an open-source software stack for coordinating distributed crawlers.
If you are an AI developer dealing with high costs for training data — this project developed an Open Web Index that provides a way to tap into the web as an innovation resource without paying huge upfront costs.
If you are a data firm dealing with biased search results from dominant gatekeepers — this project developed a search and analysis infrastructure that allows for the creation of specialized search verticals based on European values.
If you are a public agency dealing with a lack of digital sovereignty over information access — this project developed a cooperative open web search infrastructure to ensure information access aligns with European legislation and ethics.
Based on available project data, the infrastructure is designed to allow users to tap into the web as a resource without paying huge upfront costs.
The project is piloting the core of an Open Web Index and utilizing a network of European HPC-infrastructure to ensure scalability.
The project specifically delivers an Open Source Software Stack for coordinating distributed crawlers, implying an open licensing model.
The infrastructure is explicitly built to be based on Europe's values, principles, legislation, ethics, and standards.
The project period runs from 2022-09-01 to 2026-02-28, with deliverables including the launch of the pilot infrastructure.
The consortium is heavily research-oriented, consisting of 6 universities and 4 research organizations, which suggests a strong theoretical foundation. However, it includes 1 industry partner and 1 SME, and leverages 14 partners across 7 countries to integrate HPC-infrastructure, indicating a push toward practical, large-scale technical implementation.
Contact Universitat Passau regarding the Open Web Index pilot.
Contact us to explore how to integrate the Open Web Index into your AI training pipeline.
