mirror of https://github.com/aliasrobotics/cai.git
80 lines
3.0 KiB
Markdown
80 lines
3.0 KiB
Markdown
# AI Model Evaluation Benchmarks
|
|
|
|
This chapter is a curated collection of benchmark datasets and evaluation tools designed to assess the capabilities of custom AI models, particularly in domains related to cybersecurity.
|
|
|
|
The collection is intended to support researchers and developers who are evaluating their own models using reliable, task-specific benchmarks.
|
|
|
|
Currently, this are the benchmark included:
|
|
|
|
- [SecEval](https://github.com/XuanwuAI/SecEval)
|
|
- [CyberMetric](https://github.com/CyberMetric)
|
|
|
|
The goal is to consolidate diverse evaluation tasks under a single framework to support rigorous, standardized testing.
|
|
|
|
## 🏆 General Summary Table
|
|
|
|
| Model | SecEval | CyberMetric | Total Value |
|
|
|-------------|-----------|--------------|-------------|
|
|
| model_name | `XX.X%` | `XX.X%` | `XX.X%` |
|
|
|
|
|
|
|
|
## 🔐 SecEval: [https://github.com/XuanwuAI/SecEval](https://github.com/XuanwuAI/SecEval)
|
|
|
|
### 📄 Description
|
|
|
|
SecEval is a benchmark designed to evaluate large language models (LLMs) on security-related tasks. It includes various real-world scenarios such as phishing email analysis, vulnerability classification, and response generation.
|
|
|
|
### 📥 Installation
|
|
|
|
```bash
|
|
git clone https://github.com/XuanwuAI/SecEval.git
|
|
cd SecEval
|
|
pip install -r requirements.txt
|
|
```
|
|
### ▶️ Usage
|
|
```bash
|
|
python evaluate.py --model your_model_name --task all
|
|
```
|
|
### 📊 Evaluation Results
|
|
|
|
| Model Name | Accuracy | F1 Score | ROUGE | Notes |
|
|
|----------------|----------|----------|-------|---------------------|
|
|
| GPT-4 | 87.5% | 84.2% | 0.61 | Zero-shot |
|
|
| LLaMA2-13B | 75.4% | 71.8% | 0.52 | Fine-tuned |
|
|
| Claude 3 Opus | 79.2% | 76.5% | 0.58 | Few-shot setup |
|
|
| Falcon-40B | 70.1% | 68.0% | 0.47 | Baseline |
|
|
| YourModel | XX.X% | XX.X% | XX.X | Custom results here |
|
|
|
|
📂 Source: results/seceval/scores.csv
|
|
|
|
---
|
|
|
|
## 🧠 CyberMetric: [https://github.com/CyberMetric](https://github.com/CyberMetric)
|
|
|
|
### 📄 Description
|
|
CyberMetric is a benchmark framework that focuses on measuring the performance of AI systems in cybersecurity-specific question answering, knowledge extraction, and contextual understanding. It emphasizes both domain knowledge and reasoning ability.
|
|
|
|
### 📥 Installation
|
|
```bash
|
|
git clone https://github.com/CyberMetric/CyberMetric.git
|
|
cd CyberMetric
|
|
pip install -r requirements.txt
|
|
```
|
|
### ▶️ Usage
|
|
```bash
|
|
python run.py --model your_model_name --task qa
|
|
```
|
|
|
|
### 📊 Evaluation Results
|
|
|
|
| Model Name | Accuracy | F1 Score | ROUGE | Notes |
|
|
|----------------|----------|----------|-------|---------------------|
|
|
| GPT-4 | 87.5% | 84.2% | 0.61 | Zero-shot |
|
|
| LLaMA2-13B | 75.4% | 71.8% | 0.52 | Fine-tuned |
|
|
| Claude 3 Opus | 79.2% | 76.5% | 0.58 | Few-shot setup |
|
|
| Falcon-40B | 70.1% | 68.0% | 0.47 | Baseline |
|
|
| YourModel | XX.X% | XX.X% | XX.X | Custom results here |
|
|
📂 Source: results/cybermetric/scores.csv
|
|
|