Frontier AI Risk Benchmark Database

Authors: Weibing Wang

Published: July 11, 2026

The Frontier AI Risk Benchmark Database is a public database developed by Concordia AI that collects metadata on benchmarks for evaluating frontier AI risks.

Why We Built It

For AI safety evaluation researchers, finding suitable benchmarks has always been a fundamental but time-consuming task. Determining whether a benchmark is suitable often requires considering many factors:

Relevance: Which risk domain does the benchmark focus on? What does it evaluate, such as a kind of capability, propensity, or safeguard? Does it evaluate models or agents?
Impact: How many citations has the benchmark paper received? How many stars does its code repository have? Which organizations use it for frontier risk evaluations?
Availability: Are the dataset and code publicly available? Which evaluation frameworks, such as Inspect, does it support?
Recency: When was the benchmark developed or last updated?

However, this information is usually scattered across papers, GitHub repositories, Hugging Face datasets, model system cards, and standalone project websites. Finding it is time-consuming, comparing benchmarks is inconvenient, and important benchmarks can easily be missed.

We frequently encountered the same problems while developing the Frontier AI Risk Monitoring Platform. We initially maintained an Excel spreadsheet to record benchmark information, but as the project grew, its limitations became clear:

As the number of industry benchmarks grew, manually maintaining the spreadsheet became increasingly costly.
Without a dynamic update mechanism, information such as citation counts and GitHub star counts quickly became outdated.
Without an effective structure, finding and comparing benchmarks became increasingly difficult.

We therefore created the Frontier AI Risk Benchmark Database (the "Database"). It combines an AI-assisted collection and human-review workflow with a risk-model-based structure and dynamic information updates, substantially improving our own efficiency. We are now making the Database publicly available so that other researchers can benefit from it as well.

What It Contains

The Database currently includes more than 200 benchmarks for frontier AI risks published since 2023. It organizes them by risk domain, evaluation type, evaluating feature, target type, open-source status, and other attributes, helping researchers find, filter, and compare benchmarks more efficiently.

The Database currently covers risk domains including cyber offense, biological risks, chemical risks, harmful manipulation, loss-of-control, nuclear risks, and embodied harm. Evaluation subjects include language models, vision-language models, AI agents, embodied models, and biological models. Benchmarks are grouped into three evaluation types:

Capability benchmarks: Evaluate key capabilities that may amplify risk, such as vulnerability exploitation, bioinformatics tasks, frontier scientific reasoning, self-replication, and machine learning engineering.
Propensity benchmarks: Evaluate dangerous behavioral propensities, such as deception, covertly influencing users, resisting shutdown, and agentic misalignment.
Safeguard benchmarks: Evaluate model safeguards, such as safeguards against jailbreak attacks and prompt injection, or defenses against cyber offense and harmful manipulation.

The Database records many useful attributes for each benchmark:

Evaluating feature: The specific capabilities, propensities, or safeguards evaluated by the benchmark, which helps users determine whether it meets their needs.
Adopters: Organizations that have used the benchmark for frontier AI risk evaluations. Benchmarks selected by these organizations are often of higher quality. The Database currently covers model system cards from OpenAI, Anthropic, and Google, as well as Concordia AI's own Risk Monitoring Platform, but does not claim to cover every adopter.
Impact: A score calculated from the benchmark paper's citation count, the code repository's GitHub stars, and the number of adopting organizations. This is the default sort field and helps users identify influential benchmarks more quickly.
Open-source status: Whether the dataset and code are open source, along with their license types, making it easier for researchers to find accessible datasets and code.
Inspect support: Whether the benchmark supports Inspect and, if so, a link to its Inspect implementation. Inspect is a widely used automated evaluation framework, and support makes a benchmark easier to integrate into Inspect-based evaluation systems.
Number of samples: Helps estimate the cost of running the full benchmark.
Release date: Helps users judge whether a benchmark may be outdated or has not yet stood the test of time.

How to Use It

The Database has two main entry points.

The first is the benchmark list. Users can filter benchmarks by risk domain, evaluation type, evaluating feature, target type, adopter, open-source status, Inspect support, release date, and keywords. For example, a researcher can quickly find benchmarks that focus on cyber offense, evaluate agents, and provide both open-source code and data. They can also find safeguard benchmarks that address biological risks and have already been adopted by an organization.

This approach is designed for the question, "I need a usable benchmark now." It consolidates information previously scattered across different sources into one structured table, allowing users to eliminate clearly unsuitable options before reviewing papers or code repositories in depth.

The second entry point is the risk model, the key feature that distinguishes the Database from a conventional table.

Drawing on the Frontier AI Risk Management Framework v2.0, the risk-model view first lists risk scenarios in domains such as cyber offense, biological risks, chemical risks, harmful manipulation, and loss-of-control. It then breaks each scenario down into relevant capabilities, propensities, and safeguards, before linking these evaluating features to specific benchmarks and subsets.

In other words, users can start from the risk itself rather than from a particular benchmark:

To assess whether AI could autonomously execute a high-risk cyberattack chain, users can review the cyber capabilities, dangerous propensities, and safeguards involved in that scenario, then find the corresponding benchmarks.
To assess whether a model might assist with high-risk biological tasks, users can see which benchmarks cover the relevant biological capabilities, dangerous propensities, and safety guardrails.
To examine self-improvement, covert action, situational awareness, or poor honesty in loss-of-control scenarios, users can search for capability or propensity evaluations by these evaluating features.

The goal of this structure is to help researchers move beyond simple benchmark scores toward risk assessment based on evaluation results. A score on a single benchmark only describes model performance on a particular task. Once benchmarks are mapped to risk scenarios, capabilities, propensities, and safeguards, their results become easier to interpret as evidence within a specific risk pathway.

How It Works

We maintain the Database through the following process:

Import high-quality seeds. We begin with high-quality benchmarks accumulated through our manual research over the past year.
Expand with AI. AI agents search online for benchmarks based on our selection criteria and seed examples, then compile candidate benchmarks and their metadata.
Review every candidate. Human reviewers check and correct all AI-generated candidates before adding them to the Database.
Update status regularly. Frequently changing data, such as citation and GitHub star counts, is updated programmatically and used to recalculate Impact Scores.

The current Impact Score $I$ is calculated as follows:

$I = \frac{0.75 \times ln(1 + C) + 0.25 \times ln(1 + S)}{\sqrt{Y}} + 0.5 \times A$

where:

$C$ is the Semantic Scholar citation count, or 0 if unavailable;
$S$ is the number of GitHub repository stars, or 0 if unavailable;
$A$ is the number of organizations that have adopted the benchmark (each organization counted once);
$Y = max(0.5, m / 12)$ is the benchmark's age in years, with a minimum of six months. $m$ is the number of full months from the benchmark's release date to the scoring date, with a minimum of 1.

The rationale is as follows:

Logarithmic compression reduces the contribution of citation and GitHub star counts for the most popular benchmarks.
Citations and GitHub stars are weighted 3:1. Citations receive more weight because academic citations are a stronger signal of recognition of a project's value.
Time decay is introduced by dividing by $\sqrt{Y}$ . This increases the weight of newer projects and reduces that of older ones, helping fast-growing new projects surface sooner while the square-root term prevents very young projects from receiving excessive weight.
The number of adopting organizations is added as a separate additive term, worth 0.5 points per organization: organizational adoption is a fact that does not fade over time, so it is not age-adjusted.

Ongoing Updates and Collaboration

Frontier AI risk benchmarks are still evolving rapidly. Concordia AI will continue to maintain the Frontier AI Risk Benchmark Database, track newly released benchmarks, update existing benchmark records, and improve the mappings from risk scenarios to benchmarks.

We welcome collaboration with researchers and organizations:

If an important benchmark is missing from the Database, please let us know.
If you find missing, incorrect, or outdated information, please let us know.
If you would like a benchmark to be integrated into the Frontier AI Risk Monitoring Platform's continuous evaluations, we also welcome collaboration.

We hope the Database will become infrastructure for the AI safety evaluation community, helping researchers not only find benchmarks faster, but also understand more accurately how benchmarks relate to real-world risks.

Table of Contents