diff --git a/benchmarks/README.md b/benchmarks/README.md index b59f975a..39c51fe4 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -6,8 +6,13 @@ The collection is intended to support researchers and developers who are evaluat Currently, this are the benchmarks included: -- [SecEval](https://github.com/XuanwuAI/SecEval) -- [CyberMetric](https://github.com/CyberMetric) +| Benchmark | Description | +|-----------|-------------| +| [SecEval](https://github.com/XuanwuAI/SecEval) | Benchmark designed to evaluate large language models (LLMs) on security-related tasks. It includes various real-world scenarios such as phishing email analysis, vulnerability classification, and response generation. | +| [CyberMetric](https://github.com/CyberMetric) | Benchmark framework that focuses on measuring the performance of AI systems in cybersecurity-specific question answering, knowledge extraction, and contextual understanding. It emphasizes both domain knowledge and reasoning ability. | +| [CTIBench](https://github.com/xashru/cti-bench) | Benchmark focused on evaluating LLM models' capabilities in understanding and processing Cyber Threat Intelligence (CTI) information. | + + The goal is to consolidate diverse evaluation tasks under a single framework to support rigorous, standardized testing. @@ -19,39 +24,44 @@ The goal is to consolidate diverse evaluation tasks under a single framework to -## πŸ” [SecEval](https://github.com/XuanwuAI/SecEval) - -#### πŸ“„ Description - -SecEval is a benchmark designed to evaluate large language models (LLMs) on security-related tasks. It includes various real-world scenarios such as phishing email analysis, vulnerability classification, and response generation. - -#### ▢️ Usage - - -```bash -git submodule update --init --recursive #Β init submodules -cd benchmarks/seceval/eval -pip install -r requirements.txt -``` - -```bash -python3 eval.py --dataset_file datasets/questions.json --output_dir outputs --backend ollama --model ollama/qwen2.5:14b -``` - ---- - -## 🧠 [CyberMetric](https://github.com/CyberMetric) - -#### πŸ“„ Description -CyberMetric is a benchmark framework that focuses on measuring the performance of AI systems in cybersecurity-specific question answering, knowledge extraction, and contextual understanding. It emphasizes both domain knowledge and reasoning ability. - - #### ▢️ Usage ```bash git submodule update --init --recursive #Β init submodules -cd benchmarks/cybermetric -python3 CyberMetric_evaluator.py --model_name ollama/qwen2.5:14b --file_path CyberMetric-2-v1.json ``` +```bash +python benchmarks/eval.py --model MODEL_NAME --dataset_file INPUT_FILE --eval EVAL_TYPE --backend BACKEND +``` +```bash +Arguments: + -m, --model # Specify the model to evaluate (e.g., "gpt-4", "ollama/qwen2.5:14b") + -d, --dataset_file # IMPORTANT! By default: small test data of 2 samples + -B, --backend # Backend to use: "openai", "openrouter", "ollama" (required) + -e, --eval # Specify the evaluation benchmark +Output: + outputs/ + └── benchmark_name/ + └── model_date_random-num/ + β”œβ”€β”€ answers.json # the whole test with LLM answers + └── information.txt # report of that precise run (e.g. model_name, benchmark_name, metrics, date) + +``` +#### πŸ” Examples + +```bash +python benchmarks/eval.py --model ollama/qwen2.5:14b --dataset_file benchmarks/cybermetric/CyberMetric-2-v1.json --eval cybermetric --backend ollama +```` +```bash +python benchmarks/eval.py --model ollama/qwen2.5:14b --dataset_file benchmarks/seceval/eval/datasets/questions-2.json --eval seceval --backend ollama +```` +```bash +python benchmarks/eval.py --model ollama/qwen2.5:14b --dataset_file benchmarks/cti_bench/data/cti-mcq1.tsv --eval cti_bench --backend ollama +```` +```bash +python benchmarks/eval.py --model ollama/qwen2.5:14b --dataset_file benchmarks/cti_bench/data/cti-ate2.tsv --eval cti_bench --backend ollama +```` +```bash +python benchmarks/eval.py --model qwen/qwen3-32b:free --dataset_file benchmarks/cybermetric/CyberMetric-2-v1.json --eval cybermetric --backend openrouter +``` \ No newline at end of file