RomanNum3ral
/
cai
mirror of https://github.com/aliasrobotics/cai.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Added titles

This commit is contained in:
Francesco Balassone 2025-10-27 10:42:07 +01:00 committed by GitHub
parent 1470c40870
commit 96028ef3c3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 18 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
benchmarks/README.md
View File

@ -28,10 +28,24 @@ AutoPenBench │
*Categories marked with asterisk are available in CAI PRO version [^8].
| ![](utils/stackplot.png) | ![](utils/base_1col.png) |
|------------------------------------------------|---------------------------------|
| ![](utils/cyberpii_benchmark.png) | ![](utils/caibench_spider.png) |
<table>
<tr>
<th style="text-align:center;"><b>Best performance in Agent vs Agent A&amp;D</b></th>
<th style="text-align:center;"><b>Model performance in Jeopardy CTFs Base Benchmark</b></th>
</tr>
<tr>
<td align="center"><img src="utils/stackplot.png" alt="stackplot" /></td>
<td align="center"><img src="utils/base_1col.png" alt="base_1col" /></td>
</tr>
<tr>
<th style="text-align:center;"><b>Model performance in CyberPII Privacy Benchmark</b></th>
<th style="text-align:center;"><b>Model performance overall</b></th>
</tr>
<tr>
<td align="center"><img src="utils/cyberpii_benchmark.png" alt="cyberpii" /></td>
<td align="center"><img src="utils/caibench_spider.png" alt="caibench" /></td>
</tr>
</table>
Cybersecurity AI Benchmark or `CAIBench` for short is a meta-benchmark (*benchmark of benchmarks*) [^6] designed to evaluate the security capabilities (both offensive and defensive) of cybersecurity AI agents and their associated models. It is built as a composition of individual benchmarks, most represented by a Docker container for reproducibility. Each container scenario can contain multiple challenges or tasks. The system is designed to be modular and extensible, allowing for the addition of new benchmarks and challenges.