Collinear AI released CWE-bench v2, a new public benchmark designed to evaluate the cyber capabilities of frontier AI models. The evaluation setup requires an agent to fix security weaknesses in a real open-source repository. The benchmark includes 100 tasks covering 91 different Common Weakness Enumerations (CWEs).
Results are fully programmatic and deterministic, allowing anyone to reproduce them. Gemini 4 Argon leads the leaderboard with 67.8% pass@1 on the public set. DeepSeek V4.1 Flash is the most cost-efficient model, achieving 57.0% pass@1 at $0.04 per rollout.
This benchmark provides a standardized way to measure how well AI agents can handle real-world code security. Developers building autonomous security tools can use these results to compare different models. Researchers can also verify the safety claims of new systems against a rigorous, open standard.
