Merge remote changes

benchmarks/README.md

@@ -6,8 +6,13 @@ The collection is intended to support researchers and developers who are evaluat
 
 Currently, this are the benchmarks included:
 
-- [SecEval](https://github.com/XuanwuAI/SecEval)
-- [CyberMetric](https://github.com/CyberMetric)
+| Benchmark | Description |
+|-----------|-------------|
+| [SecEval](https://github.com/XuanwuAI/SecEval) | Benchmark designed to evaluate large language models (LLMs) on security-related tasks. It includes various real-world scenarios such as phishing email analysis, vulnerability classification, and response generation. |
+| [CyberMetric](https://github.com/CyberMetric) | Benchmark framework that focuses on measuring the performance of AI systems in cybersecurity-specific question answering, knowledge extraction, and contextual understanding. It emphasizes both domain knowledge and reasoning ability. |
+| [CTIBench](https://github.com/xashru/cti-bench) | Benchmark focused on evaluating LLM models' capabilities in understanding and processing Cyber Threat Intelligence (CTI) information. |
+
+
 
 The goal is to consolidate diverse evaluation tasks under a single framework to support rigorous, standardized testing.
 
@@ -19,39 +24,44 @@ The goal is to consolidate diverse evaluation tasks under a single framework to
 
 
 
-## 🔐 [SecEval](https://github.com/XuanwuAI/SecEval)   
-
-#### 📄 Description
-
-SecEval is a benchmark designed to evaluate large language models (LLMs) on security-related tasks. It includes various real-world scenarios such as phishing email analysis, vulnerability classification, and response generation.
-
-#### ▶️ Usage
-
-
-```bash
-git submodule update --init --recursive  # init submodules
-cd benchmarks/seceval/eval
-pip install -r requirements.txt
-```
-
-```bash
-python3 eval.py --dataset_file datasets/questions.json --output_dir outputs --backend ollama --model ollama/qwen2.5:14b
-```
-
----
-
-## 🧠 [CyberMetric](https://github.com/CyberMetric)
-
-#### 📄 Description 
-CyberMetric is a benchmark framework that focuses on measuring the performance of AI systems in cybersecurity-specific question answering, knowledge extraction, and contextual understanding. It emphasizes both domain knowledge and reasoning ability.
-
-
 #### ▶️ Usage
 
 ```bash
 git submodule update --init --recursive  # init submodules
-cd benchmarks/cybermetric
-python3 CyberMetric_evaluator.py --model_name ollama/qwen2.5:14b --file_path CyberMetric-2-v1.json
 ```
 
+```bash
+python benchmarks/eval.py --model MODEL_NAME --dataset_file INPUT_FILE --eval EVAL_TYPE --backend BACKEND
+```
+```bash
+Arguments:
+    -m, --model         # Specify the model to evaluate (e.g., "gpt-4", "ollama/qwen2.5:14b")
+    -d, --dataset_file  # IMPORTANT! By default: small test data of 2 samples 
+    -B, --backend       # Backend to use: "openai", "openrouter", "ollama" (required)
+    -e, --eval          # Specify the evaluation benchmark
 
+Output:
+   outputs/
+   └── benchmark_name/
+       └── model_date_random-num/
+           ├── answers.json       # the whole test with LLM answers
+           └── information.txt    # report of that precise run (e.g. model_name, benchmark_name, metrics, date)
+
+```
+#### 🔍 Examples
+
+```bash
+python benchmarks/eval.py --model ollama/qwen2.5:14b --dataset_file benchmarks/cybermetric/CyberMetric-2-v1.json --eval cybermetric --backend ollama
+````
+```bash
+python benchmarks/eval.py --model ollama/qwen2.5:14b --dataset_file benchmarks/seceval/eval/datasets/questions-2.json --eval seceval --backend ollama
+````
+```bash
+python benchmarks/eval.py --model ollama/qwen2.5:14b --dataset_file benchmarks/cti_bench/data/cti-mcq1.tsv --eval cti_bench --backend ollama
+````
+```bash
+python benchmarks/eval.py --model ollama/qwen2.5:14b --dataset_file benchmarks/cti_bench/data/cti-ate2.tsv --eval cti_bench --backend ollama
+````
+```bash
+python benchmarks/eval.py --model qwen/qwen3-32b:free --dataset_file benchmarks/cybermetric/CyberMetric-2-v1.json --eval cybermetric --backend openrouter
+```