Safe AI made in the EU

We propose an EU Safe Generative AI Innovation Program to address a market failure in generative AI development. While developers can capture significant value from generative AI capability improvements, they bear only a fraction of potential safety failure costs, which leads to underinvestment in the technological breakthroughs necessary to make generative AI safe. The EU should establish explicit incentives for the necessary technological breakthroughs, complementing its existing policy responses to the rapid proliferation of generative AI. We propose a milestone-based incentive scheme where pre-specified payments would reward the achievement of verifiable safety milestones. This "pull" funding mechanism would aim to create predictable development paths for safety improvements, similar to how scaling laws have guided capability advances. The scheme would use robust safety metrics and competitive evaluation to prevent gaming while ensuring meaningful progress. Success would be measured through a combination of specific safety dimensions (like factual accuracy and harm prevention) and broader performance metrics, validated through adversarial testing and public comparative evaluation. The program's design would be technology-neutral and it could be open to all qualified institutions, with rewards calibrated through incentive-compatible elicitation mechanisms. This approach mirrors other applications of outcome-based funding, such as advance market commitments in vaccine development. It might also provide the breeding ground for "Safe AI made in the EU".