CVE-Bench, in NAACL-2025

Introduction

Implementation

First please download and preprocess the CVEFixes database:

mkdir data
cd data
wget https://zenodo.org/records/13118970/files/CVEfixes_v1.0.8.zip
unzip CVEfixes_v1.0.8.zip
gzip -d  CVEfixes_v1.0.8/Data/CVEfixes_v1.0.8.sql.gz
sqlite3 CVEfixes.db < CVEfixes_v1.0.8/Data/CVEfixes_v1.0.8.sql

Run

python 1-prepare_repo.py
python 2-run-agent.py
python 3-verify.py