From c73446b847e39d7fc1a25c4d453ed7153870f7e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?V=C3=ADctor=20Mayoral=20Vilches?= Date: Tue, 3 Jun 2025 08:22:21 +0000 Subject: [PATCH] Align CyberPII-Bench with the rest of the benchmarks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Víctor Mayoral Vilches --- benchmarks/README.md | 3 ++- {cyberPII-bench => benchmarks/cyberPII-bench}/README.md | 2 +- .../cyberPII-bench}/annotation_metrics.py | 0 .../cyberPII-bench}/memory01_gold.csv | 0 4 files changed, 3 insertions(+), 2 deletions(-) rename {cyberPII-bench => benchmarks/cyberPII-bench}/README.md (96%) rename {cyberPII-bench => benchmarks/cyberPII-bench}/annotation_metrics.py (100%) rename {cyberPII-bench => benchmarks/cyberPII-bench}/memory01_gold.csv (100%) diff --git a/benchmarks/README.md b/benchmarks/README.md index 40603f28..ece66cf6 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -11,7 +11,8 @@ Currently, this are the benchmarks included: | [SecEval](https://github.com/XuanwuAI/SecEval) | Benchmark designed to evaluate large language models (LLMs) on security-related tasks. It includes various real-world scenarios such as phishing email analysis, vulnerability classification, and response generation. | | [CyberMetric](https://github.com/CyberMetric) | Benchmark framework that focuses on measuring the performance of AI systems in cybersecurity-specific question answering, knowledge extraction, and contextual understanding. It emphasizes both domain knowledge and reasoning ability. | | [CTIBench](https://github.com/xashru/cti-bench) | Benchmark focused on evaluating LLM models' capabilities in understanding and processing Cyber Threat Intelligence (CTI) information. | -| [PentestPerf](https://gitlab.com/aliasrobotics/alias_research/caiextensions/pentestperf) | Internal Benchmark to evaluate the security capabilities. It is built as a composition of individual benchmarks, each represented by a Docker container. Each container scenario (CTF) can contain multiple challenges and tasks. The system is designed to be modular and extensible, allowing for the addition of new benchmarks and challenges. | +| [PentestPerf](https://gitlab.com/aliasrobotics/alias_research/caiextensions/pentestperf) | An internal benchmarking framework that measures penetration testing capabilities of LLM models in a proprietary set of IT, OT and robotics scenarios. Reach out if you wish to cooperate in this direction. | +| [CyberPII-Bench](https://github.com/aliasrobotics/cai/tree/main/benchmarks/cyberPII-bench/) | Benchmark designed to evaluate the ability of LLM models to maintain privacy and handle **Personally Identifiable Information (PII)** in cybersecurity contexts. Built from real-world data generated during offensive hands-on exercises conducted with **CAI (Cybersecurity AI)**. | The goal is to consolidate diverse evaluation tasks under a single framework to support rigorous, standardized testing. diff --git a/cyberPII-bench/README.md b/benchmarks/cyberPII-bench/README.md similarity index 96% rename from cyberPII-bench/README.md rename to benchmarks/cyberPII-bench/README.md index 6fd1ca47..6b603b65 100644 --- a/cyberPII-bench/README.md +++ b/benchmarks/cyberPII-bench/README.md @@ -1,6 +1,6 @@ ## 🧠 CyberPII-Bench -**CyberPII-Bench** is a benchmark designed to evaluate the ability of language models and automated systems to detect, analyze, and handle **Personally Identifiable Information (PII)** in cybersecurity contexts. The benchmark is built from real-world data generated during hands-on exercises conducted with **CAI (Cybersecurity AI)**, focusing on attack simulation, security testing, and reconnaissance scenarios. +**CyberPII-Bench** is a benchmark designed to evaluate the ability of LLM models to maintain privacy and handle **Personally Identifiable Information (PII)** in cybersecurity contexts. Built from real-world data generated during offensive hands-on exercises conducted with **CAI (Cybersecurity AI)**, focusing on attack simulation, security testing, and reconnaissance scenarios. ### 📁 Dataset: `memory01_80/` diff --git a/cyberPII-bench/annotation_metrics.py b/benchmarks/cyberPII-bench/annotation_metrics.py similarity index 100% rename from cyberPII-bench/annotation_metrics.py rename to benchmarks/cyberPII-bench/annotation_metrics.py diff --git a/cyberPII-bench/memory01_gold.csv b/benchmarks/cyberPII-bench/memory01_gold.csv similarity index 100% rename from cyberPII-bench/memory01_gold.csv rename to benchmarks/cyberPII-bench/memory01_gold.csv