cai/docs/benchmarking/jeopardy_ctfs.md

Jeopardy-style CTF Benchmarks

Jeopardy-style Capture The Flag (CTF) challenges evaluate AI agents on independent security tasks across multiple domains: cryptography, web exploitation, binary reversing, forensics, and pwn.

---

📊 Available Benchmarks

Base Benchmark

21 curated CTF challenges measuring initial penetration testing capabilities.

• Difficulty: 🚩 Very Easy to 🚩🚩🚩 Medium
• Categories: Reversing, Miscellaneous, Pwn, Web, Crypto, Forensics
• Status: ⚠️ Saturated - Frontier cybersecurity models (like alias1) conquer most challenges

Model Performance in Jeopardy CTFs Base Benchmark

!!! success "alias1 Performance" alias1 achieves near-perfect scores on the Base benchmark, demonstrating mastery of fundamental cybersecurity concepts and techniques.

Cybench Framework

35 CTF challenges from the comprehensive Cybench evaluation framework.

• Difficulty: 🚩 Very Easy to 🚩🚩🚩🚩🚩 Very Hard
• Source: Cybench Framework for Evaluating Cybersecurity Capabilities and Risk
• Coverage: Broad spectrum of real-world security scenarios
• Note: 35 of 40 Cybench scenarios (reduced for infrastructure/reproducibility)

RCTF2 - Robotics CTF

27 robotics-focused challenges for attacking and defending robots and robotic frameworks.

• Difficulty: 🚩 Very Easy to 🚩🚩🚩🚩🚩 Very Hard
• Systems Covered: ROS, ROS 2, manipulators, AGVs, AMRs, collaborative robots, legged robots, humanoids
• Unique Focus: Only benchmark evaluating AI capabilities against robotic systems

---

🎯 Challenge Categories

Web Exploitation

Vulnerabilities in web applications and services:

• SQL Injection
• Cross-Site Scripting (XSS)
• Server-Side Template Injection (SSTI)
• Authentication bypasses
• API vulnerabilities

Binary Exploitation (Pwn)

Memory corruption and exploitation:

• Buffer overflows
• Format string vulnerabilities
• Return-oriented programming (ROP)
• Heap exploitation
• Use-after-free

Cryptography

Breaking or exploiting cryptographic implementations:

• Weak encryption algorithms
• Poor key management
• Custom cryptography flaws
• Hash collisions
• Padding oracle attacks

Reverse Engineering

Analyzing and understanding compiled binaries:

• Assembly code analysis
• Decompilation and deobfuscation
• Anti-debugging techniques
• Packed/encrypted binaries
• Firmware analysis

Forensics

Investigating and extracting information from data:

• File carving
• Steganography
• Memory forensics
• Network traffic analysis
• Log analysis

Miscellaneous

Challenges that don't fit standard categories:

• OSINT (Open Source Intelligence)
• Scripting and automation
• Logic puzzles
• Unconventional attack vectors

---

🏆 alias1 Performance

!!! success "Superior Jeopardy CTF Performance" alias1 consistently outperforms all other AI models in Jeopardy-style CTF benchmarks:

- 🥇 **Highest solve rate** across all difficulty levels
- 🥇 **Fastest time to solve** for timed challenges
- 🥇 **Best multi-category performance** - Excels in web, pwn, crypto, forensics, and reversing
- 🥇 **Zero refusals** - Unrestricted responses for all CTF challenges

**General-purpose models** (GPT-4o, Claude 3.5) show:
- ❌ High refusal rates on pwn/exploitation challenges
- ❌ Inconsistent performance across categories
- ❌ Limited success on medium+ difficulty challenges

Get alias1 with CAI PRO →

---

🚀 Running Jeopardy CTF Benchmarks

!!! warning "CAI PRO Exclusive" Jeopardy-style CTF benchmarks are available exclusively with CAI PRO subscriptions.

General users can access:
- [Knowledge benchmarks](knowledge_benchmarks.md)
- [Privacy benchmarks](privacy_benchmarks.md)

For CAI PRO Subscribers

Docker-based CTF environments can be launched individually or in batches:

# Run single CTF challenge
docker run -it cai-ctf/base:challenge-01

# Run full Base benchmark suite
python benchmarks/eval_ctf.py --benchmark base --model alias1

# Run Cybench evaluation
python benchmarks/eval_ctf.py --benchmark cybench --model alias1

# Run RCTF2 robotics challenges
python benchmarks/eval_ctf.py --benchmark rctf2 --model alias1

Contact research@aliasrobotics.com for detailed setup instructions.

---

📊 Benchmark Configuration

CTF configurations are defined in ctf_configs.jsonl:

{
  "name": "example-ctf",
  "category": "web",
  "difficulty": "medium",
  "points": 100,
  "flag_format": "CTF{...}",
  "docker_image": "cai-ctf/web-01:latest",
  "timeout": 3600
}

---

🎓 Why Jeopardy CTFs Matter

Jeopardy-style CTFs are essential for evaluating cybersecurity AI because:

1. Diverse Skillset - Tests wide range of security knowledge and techniques
2. Independent Challenges - Isolates specific capabilities without dependencies
3. Scalable Difficulty - From beginner to elite-level challenges
4. Real-world Relevance - Based on actual vulnerabilities and attack patterns
5. Objective Measurement - Clear success criteria (flag captured or not)

Unlike traditional benchmarks that test general knowledge, CTFs require active exploitation and problem-solving - skills critical for real-world penetration testing.

---

📚 Research Papers

• 📊 CAIBench: Cybersecurity AI Benchmark (2025) Meta-benchmark framework including Jeopardy CTF evaluation methodology.

• 🚀 Cybersecurity AI (CAI) Framework (2025) Core framework demonstrating 3,600× performance improvement using CTF scenarios.

View all research →

---

🔗 Related Benchmarks

• Attack & Defense CTFs - Real-time competitive environments
• Cyber Ranges - Complex multi-system scenarios
• Running Benchmarks - Setup and usage guide

---

🚀 Upgrade to CAI PRO for access to Jeopardy CTF benchmarks →