-
- 
-
-
-
-
-
-
-
+ Professional Edition with unlimited alias1 tokens | 📊 View Benchmarks | 🚀 Learn More
- Professional Edition with unlimited alias1 tokens | 📊 View Benchmarks | 🚀 Learn More
+
+
+
+
+
+
+
+
+
+
+
-
- 
- |
-
- 
- |
-
](https://arxiv.org/pdf/2504.06017) | [
](https://www.arxiv.org/pdf/2506.23592) | [
](https://arxiv.org/pdf/2508.13588) | [
](https://arxiv.org/pdf/2508.21669) |
+| [](https://arxiv.org/pdf/2504.06017) | [](https://www.arxiv.org/pdf/2506.23592) | [](https://arxiv.org/pdf/2508.13588) | [](https://arxiv.org/pdf/2508.21669) |
-
- | Humanoid Robots as Attack Vectors [](https://arxiv.org/abs/2509.14139) | The Cybersecurity of a Humanoid Robot [](https://arxiv.org/abs/2509.14096) | Evaluating Agentic Cybersecurity in Attack/Defense CTFs [](https://arxiv.org/abs/2510.17521) | CAIBench: A Meta-Benchmark for Evaluating Cybersecurity AI Agents [](https://arxiv.org/abs/2510.24317) |
+ | Humanoid Robots as Attack Vectors [](https://arxiv.org/abs/2509.14139) | The Cybersecurity of a Humanoid Robot [](https://arxiv.org/abs/2509.14096) | Evaluating Agentic Cybersecurity in Attack/Defense CTFs [](https://arxiv.org/abs/2510.17521) | CAIBench: A Meta-Benchmark for Evaluating Cybersecurity AI Agents [](https://arxiv.org/abs/2510.24317) |
|---|---|---|---|
-| [
](https://arxiv.org/pdf/2509.14139) | [
](https://arxiv.org/pdf/2509.14096) | [
](https://arxiv.org/pdf/2510.17521) | [
](https://arxiv.org/pdf/2510.24317) |
-
-
+| [](https://arxiv.org/pdf/2509.14139) | [](https://arxiv.org/pdf/2509.14096) | [](https://arxiv.org/pdf/2510.17521) | [](https://arxiv.org/pdf/2510.24317) |
## PoCs
| CAI with `alias0` on ROS message injection attacks in MiR-100 robot | CAI with `alias0` on API vulnerability discovery at Mercado Libre |
|-----------------------------------------------|---------------------------------|
| [](https://asciinema.org/a/dNv705hZel2Rzrw0cju9HBGPh) | [](https://asciinema.org/a/9Hc9z1uFcdNjqP3bY5y7wO1Ww) |
-
| CAI on JWT@PortSwigger CTF — Cybersecurity AI | CAI on HackableII Boot2Root CTF — Cybersecurity AI |
|-----------------------------------------------|---------------------------------|
| [](https://asciinema.org/a/713487) | [](https://asciinema.org/a/713485) |
@@ -273,11 +278,10 @@ CAI is built on the following core principles:
- **Tool-integration**: CAI integrates already built-in tools, and allows the user to integrate their own tools with their own logic easily.
- **Logging and tracing integrated**: using [`phoenix`](https://github.com/Arize-ai/phoenix), the open source tracing and logging tool for LLMs. This provides the user with a detailed traceability of the agents and their execution.
- **Multi-Model Support**: more than 300 supported and empowered by [LiteLLM](https://github.com/BerriAI/litellm). The most popular providers:
- - **Anthropic**: `Claude 3.7`, `Claude 3.5`, `Claude 3`, `Claude 3 Opus`
- - **OpenAI**: `O1`, `O1 Mini`, `O3 Mini`, `GPT-4o`, `GPT-4.5 Preview`
- - **DeepSeek**: `DeepSeek V3`, `DeepSeek R1`
- - **Ollama**: `Qwen2.5 72B`, `Qwen2.5 14B`, etc
-
+ - **Anthropic**: `Claude 3.7`, `Claude 3.5`, `Claude 3`, `Claude 3 Opus`
+ - **OpenAI**: `O1`, `O1 Mini`, `O3 Mini`, `GPT-4o`, `GPT-4.5 Preview`
+ - **DeepSeek**: `DeepSeek V3`, `DeepSeek R1`
+ - **Ollama**: `Qwen2.5 72B`, `Qwen2.5 14B`, etc
### Closed-source alternatives
Cybersecurity AI is a critical field, yet many groups are misguidedly pursuing it through closed-source methods for pure economic return, leveraging similar techniques and building upon existing closed-source (*often third-party owned*) models. This approach not only squanders valuable engineering resources but also represents an economic waste and results in redundant efforts, as they often end up reinventing the wheel. Here are some of the closed-source initiatives we keep track of and attempting to leverage genAI and agentic frameworks in cybersecurity AI:
@@ -307,44 +311,35 @@ Cybersecurity AI is a critical field, yet many groups are misguidedly pursuing i
- [Zynap](https://www.zynap.com)
- [7ai](https://7ai.com)
-
## Learn - `CAI` Fluency
-
-
- 
-
-
| Best performance in Agent vs Agent A&D | @@ -58,16 +61,13 @@ Cybersecurity AI Benchmark or `CAIBench` for short is a meta-benchmark (*benchma - [📁 Dataset: `memory01_80/`](#-dataset-memory01_80) - [🔍 Entity Coverage](#-entity-coverage) - [📐 Metrics](#-metrics) - - [📊 Evaluation](#--evaluation) + - [📊 Evaluation](#-evaluation) - [About `Attack-Defense CTF`](#about-attack-defense-ctf) - [Game Structure](#game-structure) - [Rules and Scoring](#rules-and-scoring) - [Architecture](#architecture) - [Technical Features](#technical-features) - [About challenges in benchmarks](#about-challenges-in-benchmarks) - - [`Jeopardy CTF`](#jeopardy-ctf) - - [`A&D`](#ad) - - [Cyber Ranges](#cyber-ranges) ## Difficulty classification @@ -123,6 +123,8 @@ Cybersecurity AI Benchmark or `CAIBench` for short is a meta-benchmark (*benchma :five: **Privacy** (`benchmarks/eval.py` :book:) - Assess AI models' ability to handle sensitive information appropriately, maintain privacy standards, and properly manage Personally Identifiable Information (PII) in cybersecurity contexts. +> **Note:** Categories :one: **Jeopardy-style CTFs**, :two: **Attack–Defense CTF**, and :three: **Cyber Range Exercises** are available in the **CAI PRO** version. Learn more at https://aliasrobotics.com/cybersecurityai.php + ## Benchmarks @@ -130,11 +132,11 @@ Currently, supporting the following benchmarks, refer to [`ctf_configs.jsonl`](. | Category | Benchmark | Difficulty | Description | |----------|-----------|------------|-------------| -| :one: `jeopardy` | Base | 🚩 - 🚩🚩🚩 | `21` curated CTFs that measures initial penetration testing capabilities across challenges in rev, misc, pwn, web, crypto and forensics. *This benchmark has been saturated and frontier Cybersecurity models are able to conquer most*. | -| :one: `jeopardy` | [Cybench](https://github.com/andyzorigin/cybench) | 🚩 - 🚩🚩🚩🚩🚩 | A curated list with `35` CTFs stemming from the popular *`Cybench` Framework for Evaluating Cybersecurity Capabilities and Risk*[^7]. | -| :one: `jeopardy` | RCTF2 | 🚩 - 🚩🚩🚩🚩🚩 | `27` Robotics CTFs challenges to attack and defend robots and robotic frameworks. Robots and robotics-related technologies considered include ROS, ROS 2, manipulators, AGVs and AMRs, collaborative robots, legged robots, humanoids and more. | -| :two: `A&D` | `A&D` | 🚩 - 🚩🚩🚩🚩 | A compilation of `10` **n** vs **n** attack and defense challenges wherein each team defends their own vulnerable assets while simultaneously attacking others'. Includes IT and OT/ICS themed challenges across multiple difficulty levels. | -| :three: `cyber-range` | Cyber Ranges | 🚩🚩 - 🚩🚩🚩🚩| 12 Cyber Ranges with 16 challenges to practice and test cybersecurity skills in realistic simulated environments. | +| :one: `jeopardy` [^8] | Base | 🚩 - 🚩🚩🚩 | `21` curated CTFs that measures initial penetration testing capabilities across challenges in rev, misc, pwn, web, crypto and forensics. *This benchmark has been saturated and frontier Cybersecurity models are able to conquer most*. | +| :one: `jeopardy` [^8] | [Cybench](https://github.com/andyzorigin/cybench) | 🚩 - 🚩🚩🚩🚩🚩 | A curated list with `35` CTFs stemming from the popular *`Cybench` Framework for Evaluating Cybersecurity Capabilities and Risk*[^7]. | +| :one: `jeopardy` [^8] | RCTF2 | 🚩 - 🚩🚩🚩🚩🚩 | `27` Robotics CTFs challenges to attack and defend robots and robotic frameworks. Robots and robotics-related technologies considered include ROS, ROS 2, manipulators, AGVs and AMRs, collaborative robots, legged robots, humanoids and more. | +| :two: `A&D` [^8] | `A&D` | 🚩 - 🚩🚩🚩🚩 | A compilation of `10` **n** vs **n** attack and defense challenges wherein each team defends their own vulnerable assets while simultaneously attacking others'. Includes IT and OT/ICS themed challenges across multiple difficulty levels. | +| :three: `cyber-range` [^8] | Cyber Ranges | 🚩🚩 - 🚩🚩🚩🚩| 15 Cyber Ranges with 28 challenges to practice and test cybersecurity skills in realistic simulated environments. | | :four: `knowledge` | [SecEval](https://github.com/XuanwuAI/SecEval) | N/A | Benchmark designed to evaluate large language models (LLMs) on security-related tasks. It includes various real-world scenarios such as phishing email analysis, vulnerability classification, and response generation. | | :four: `knowledge` | [CyberMetric](https://github.com/CyberMetric) | N/A | Benchmark framework that focuses on measuring the performance of AI systems in cybersecurity-specific question answering, knowledge extraction, and contextual understanding. It emphasizes both domain knowledge and reasoning ability. | | :four: `knowledge` | [CTIBench](https://github.com/xashru/cti-bench) | N/A | Benchmark focused on evaluating LLM models' capabilities in understanding and processing Cyber Threat Intelligence (CTI) information. | @@ -155,6 +157,8 @@ Currently, supporting the following benchmarks, refer to [`ctf_configs.jsonl`](. [^7]: CAIBench integrates only 35 (out of 40) curated Cybench scenarios for evaluation purposes. This reduction comes mainly down to restrictions in our testing infrastructure as well as reproducibility issues. +[^8]: Internal exercises related to Jeopardy-style CTFs, Attack–Defense CTF, and Cyber Range Exercises are available upon request to [CAI PRO](https://aliasrobotics.com/cybersecurityai.php) subscribers on a use case basis. Learn more at https://aliasrobotics.com/cybersecurityai.php + ## About `Cybersecurity Knowledge` benchmarks @@ -307,8 +311,8 @@ This is an example of how a text sould be sanitized: Some annotation rules: - Each detected entity should be sanitized using the **format: [ENTITY_TYPE]** -- Priorize IP_ADDRESS to URL: `https://192.168.2.100` is anonymized like this: `https://[IP_ADDRESS]:5050` instead of [URL] -- DATE_TIME is used for dates and for times, in this case `2025-03-11 11:41 UTC` it should be anonymized like this: ` [DATE_TIME] [DATE_TIME]` +- Priorize IP_ADDRESS to URL: `https://192.168.2.100` is anonimized like this: `https://[IP_ADDRESS]:5050` instead of [URL] +- DATE_TIME is used for dates and for times, in this case `2025-03-11 11:41 UTC` it sould be anonimized like this: ` [DATE_TIME] [DATE_TIME]` If you have any questions about the annotation, please write to us. @@ -451,9 +455,7 @@ The A&D benchmark includes **10 machines** spanning IT and OT/ICS domains across ## About challenges in benchmarks -**NOTE**: Various of the challenges used in the benchmarks have been sourced from public repositories. Proper credit is given to the original authors and repositories in the tables below. The challenges have been containerized to ensure reproducibility and ease of deployment within the CAIBench framework. Some images contain confidential information and are thereby unavailable. For an exemplary image, please refer to the https://github.com/NickNameInvalid/LLM_CTF/tree/main/database/crypto/lottery. - -### `Jeopardy CTF` +### `Jeopardy CTF` [^8]
|---|