RomanNum3ral
/
cai
mirror of https://github.com/aliasrobotics/cai.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

ubdate readme

This commit is contained in:
Mery-Sanz 2025-05-08 15:05:39 +02:00
parent 8c1f6707cb
commit 1e66c916f2
1 changed files with 5 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
benchmarks/README.md
View File

@ -11,7 +11,7 @@ Currently, this are the benchmarks included:
The goal is to consolidate diverse evaluation tasks under a single framework to support rigorous, standardized testing.
## 🏆 General Summary Table
## 📊 General Summary Table
| Model | SecEval | CyberMetric | Total Value |
|-------------|-----------|--------------|-------------|
@ -35,16 +35,6 @@ pip install -r requirements.txt
```bash
python3 eval.py --dataset_file datasets/questions.json --output_dir outputs --backend ollama --model ollama/qwen2.5:14b
```
#### 📊 Evaluation Results
| Model Name | Accuracy | F1 Score | ROUGE | Notes |
|----------------|----------|----------|-------|---------------------|
| GPT-4 | 87.5% | 84.2% | 0.61 | Zero-shot |
| LLaMA2-13B | 75.4% | 71.8% | 0.52 | Fine-tuned |
| Claude 3 Opus | 79.2% | 76.5% | 0.58 | Few-shot setup |
| Falcon-40B | 70.1% | 68.0% | 0.47 | Baseline |
| YourModel | XX.X% | XX.X% | XX.X | Custom results here |
---
@ -53,22 +43,15 @@ python3 eval.py --dataset_file datasets/questions.json --output_dir outputs --ba
#### 📄 Description
CyberMetric is a benchmark framework that focuses on measuring the performance of AI systems in cybersecurity-specific question answering, knowledge extraction, and contextual understanding. It emphasizes both domain knowledge and reasoning ability.
#### ▶️ Usage
#### ▶️ Usage
```bash
cd benchmarks/cybermetric
```
```bash
python CyberMetric_evaluator.py --model_name ollama/qwen2.5:14b --file_path CyberMetric-2-v1.json
```
#### 📊 Evaluation Results
| Model Name | Accuracy | F1 Score | ROUGE | Notes |
|----------------|----------|----------|-------|---------------------|
| GPT-4 | 87.5% | 84.2% | 0.61 | Zero-shot |
| LLaMA2-13B | 75.4% | 71.8% | 0.52 | Fine-tuned |
| Claude 3 Opus | 79.2% | 76.5% | 0.58 | Few-shot setup |
| Falcon-40B | 70.1% | 68.0% | 0.47 | Baseline |
| YourModel | XX.X% | XX.X% | XX.X | Custom results here |
📂 Source: results/cybermetric/scores.csv