mirror of https://github.com/aliasrobotics/cai.git
229 lines
8.3 KiB
Markdown
229 lines
8.3 KiB
Markdown
# Cyber Range Benchmarks
|
|
|
|
Cyber Range exercises provide realistic training environments with complex multi-system scenarios involving incident response, network defense, and operational security decision-making.
|
|
|
|
---
|
|
|
|
## 📊 Overview
|
|
|
|
**12 Cyber Ranges** with **16 challenges** designed to test cybersecurity skills in simulated real-world environments.
|
|
|
|
- **Difficulty**: 🚩🚩 Easy to 🚩🚩🚩🚩 Hard
|
|
- **Focus**: Realistic scenarios beyond isolated CTF challenges
|
|
- **Scope**: Multi-host networks, incident handling, policy decisions, operational context
|
|
|
|
---
|
|
|
|
## 🎯 Cyber Range Categories
|
|
|
|
### Incident Response Scenarios
|
|
Realistic security incidents requiring detection, analysis, and remediation:
|
|
- Malware outbreak investigation
|
|
- Insider threat detection
|
|
- Data breach response
|
|
- Ransomware attacks
|
|
- APT (Advanced Persistent Threat) campaigns
|
|
|
|
### Network Defense Operations
|
|
Defending enterprise networks against ongoing attacks:
|
|
- Firewall configuration and tuning
|
|
- IDS/IPS rule management
|
|
- Network segmentation
|
|
- Traffic analysis and monitoring
|
|
- Security policy enforcement
|
|
|
|
### Operational Security Exercises
|
|
Making security decisions in complex environments:
|
|
- Risk assessment and prioritization
|
|
- Business continuity planning
|
|
- Compliance and regulatory requirements
|
|
- Security architecture decisions
|
|
- Resource allocation under constraints
|
|
|
|
---
|
|
|
|
## 🏆 alias1 Performance in Cyber Ranges
|
|
|
|
!!! success "Real-world Environment Excellence"
|
|
**`alias1` excels in complex cyber range scenarios** that require:
|
|
|
|
- 🥇 **Multi-system coordination** - Managing security across interconnected environments
|
|
- 🥇 **Contextual decision-making** - Understanding business impact and priorities
|
|
- 🥇 **Incident response** - Rapid detection, analysis, and remediation
|
|
- 🥇 **Operational awareness** - Balancing security with service availability
|
|
- 🥇 **Strategic thinking** - Long-term security posture improvements
|
|
|
|
**General-purpose models struggle with**:
|
|
- ❌ Complex multi-step scenarios requiring coordination
|
|
- ❌ Understanding operational context and business priorities
|
|
- ❌ Making trade-offs between security and functionality
|
|
- ❌ Sustained engagement over long scenarios
|
|
|
|
**[Get alias1 with CAI PRO →](../cai_pro.md)**
|
|
|
|
---
|
|
|
|
## 🏗️ Cyber Range Architecture
|
|
|
|
### Typical Range Components
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Cyber Range Environment │
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ │
|
|
│ │ Corporate │────│ Firewall │────│ Internet │ │
|
|
│ │ Network │ │ │ │ (DMZ) │ │
|
|
│ └──────┬───────┘ └──────────────┘ └──────────┘ │
|
|
│ │ │
|
|
│ ┌────┴────┐ │
|
|
│ │ │ │
|
|
│ ┌─▼──┐ ┌─▼──┐ ┌──────┐ ┌──────┐ ┌────────┐ │
|
|
│ │ Web │ │ DB │ │ SIEM │ │ AD │ │ Backup │ │
|
|
│ └────┘ └────┘ └──────┘ └──────┘ └────────┘ │
|
|
│ │
|
|
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
|
|
│ │ Workst. │ │ Workst. │ │ Workst. │ (Users) │
|
|
│ └─────────┘ └─────────┘ └─────────┘ │
|
|
└─────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Docker-based Isolation
|
|
|
|
Each cyber range runs in isolated Docker containers:
|
|
- Multiple networked hosts
|
|
- Realistic services and applications
|
|
- Pre-configured vulnerabilities
|
|
- Monitoring and logging infrastructure
|
|
- Scoring and validation mechanisms
|
|
|
|
---
|
|
|
|
## 🎮 Challenge Types
|
|
|
|
### 1. Blue Team Defense
|
|
Protect networks against simulated attacks:
|
|
- Monitor for suspicious activity
|
|
- Implement security controls
|
|
- Patch vulnerabilities
|
|
- Maintain service availability
|
|
- Respond to incidents
|
|
|
|
### 2. Purple Team Exercises
|
|
Combine offensive and defensive perspectives:
|
|
- Identify weaknesses through testing
|
|
- Implement defensive measures
|
|
- Validate security controls
|
|
- Improve detection capabilities
|
|
|
|
### 3. Security Operations
|
|
Day-to-day security operations tasks:
|
|
- Log analysis and correlation
|
|
- Alert triage and investigation
|
|
- Threat hunting
|
|
- Vulnerability management
|
|
- Configuration management
|
|
|
|
### 4. Incident Investigation
|
|
Forensic analysis and incident response:
|
|
- Evidence collection and preservation
|
|
- Timeline reconstruction
|
|
- Root cause analysis
|
|
- Impact assessment
|
|
- Remediation recommendations
|
|
|
|
---
|
|
|
|
## 📊 Scoring and Evaluation
|
|
|
|
Cyber range performance is evaluated across multiple dimensions:
|
|
|
|
### Technical Metrics
|
|
- Threats detected and blocked
|
|
- Vulnerabilities patched
|
|
- Services maintained (uptime)
|
|
- Incident response time
|
|
- Correct configuration changes
|
|
|
|
### Operational Metrics
|
|
- Decision quality and rationale
|
|
- Resource allocation efficiency
|
|
- Business impact minimization
|
|
- Compliance adherence
|
|
- Documentation quality
|
|
|
|
### Strategic Metrics
|
|
- Security posture improvement
|
|
- Risk reduction achieved
|
|
- Cost-effectiveness
|
|
- Long-term sustainability
|
|
|
|
---
|
|
|
|
## 🚀 Running Cyber Range Benchmarks
|
|
|
|
!!! warning "CAI PRO Exclusive"
|
|
Cyber Range benchmarks are available exclusively with **[CAI PRO](../cai_pro.md)** subscriptions.
|
|
|
|
General users can access:
|
|
- [Knowledge benchmarks](knowledge_benchmarks.md)
|
|
- [Privacy benchmarks](privacy_benchmarks.md)
|
|
|
|
### For CAI PRO Subscribers
|
|
|
|
```bash
|
|
# Launch cyber range environment
|
|
python benchmarks/eval_cyberrange.py --range range-01 --model alias1
|
|
|
|
# Run full cyber range benchmark suite
|
|
python benchmarks/eval_cyberrange.py --benchmark all --model alias1
|
|
```
|
|
|
|
Contact research@aliasrobotics.com for detailed setup instructions and access.
|
|
|
|
---
|
|
|
|
## 🎓 Why Cyber Ranges Matter
|
|
|
|
Cyber ranges provide the most comprehensive evaluation of cybersecurity AI because:
|
|
|
|
1. **Realism** - Simulates actual enterprise environments and scenarios
|
|
2. **Complexity** - Tests ability to handle interconnected systems and dependencies
|
|
3. **Context** - Requires understanding business priorities and operational constraints
|
|
4. **Sustained Engagement** - Multi-hour or multi-day scenarios test endurance
|
|
5. **Decision Quality** - Evaluates strategic thinking beyond technical skills
|
|
|
|
Unlike isolated CTF challenges, cyber ranges assess **complete security operations capabilities** including:
|
|
- Technical skills (exploitation, hardening, monitoring)
|
|
- Operational thinking (prioritization, trade-offs, risk management)
|
|
- Strategic planning (long-term improvements, architecture decisions)
|
|
|
|
This makes cyber ranges the gold standard for evaluating production-ready cybersecurity AI for SOC and security engineering roles.
|
|
|
|
---
|
|
|
|
## 📚 Research Papers
|
|
|
|
- 📊 [**CAIBench: Cybersecurity AI Benchmark**](https://arxiv.org/pdf/2510.24317) (2025)
|
|
Includes cyber range evaluation methodology and results.
|
|
|
|
- 🚀 [**Cybersecurity AI (CAI) Framework**](https://arxiv.org/pdf/2504.06017) (2025)
|
|
Demonstrates multi-system coordination capabilities.
|
|
|
|
- 🤖 [**Automation vs Autonomy**](https://www.arxiv.org/pdf/2506.23592) (2025)
|
|
6-level taxonomy applicable to cyber range operations.
|
|
|
|
**[View all research →](https://aliasrobotics.com/research-security.php#papers)**
|
|
|
|
---
|
|
|
|
## 🔗 Related Benchmarks
|
|
|
|
- **[Jeopardy CTFs](jeopardy_ctfs.md)** - Independent skill-based challenges
|
|
- **[Attack & Defense CTFs](attack_defense.md)** - Real-time competitive environments
|
|
- **[Running Benchmarks](running_benchmarks.md)** - Setup and usage guide
|
|
|
|
---
|
|
|
|
🚀 **[Upgrade to CAI PRO for access to Cyber Range benchmarks →](../cai_pro.md)**
|