mirror of https://github.com/aliasrobotics/cai.git
add final submodules
This commit is contained in:
Mery-Sanz
parent
49856c351c
commit
77f9c3b220
|
|
@ -0,0 +1,12 @@
|
|||
CyberMetric Evaluation
|
||||
=====================
|
||||
|
||||
Model: ollama/qwen2.5:14b
|
||||
Dataset: CyberMetric-2-v1.json
|
||||
Start Time: 2025-05-08 12:31:39
|
||||
Status: Completed
|
||||
Questions Processed: 2
|
||||
Correct Answers: 1
|
||||
Accuracy: 50.00%
|
||||
End Time: 2025-05-08 12:31:42
|
||||
Duration: 0:00:02.434703
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"questions": [
|
||||
{
|
||||
"question": "Which of the following is a desirable property of a biometric system?",
|
||||
"answers": {
|
||||
"A": "Permanent",
|
||||
"B": "Transferability",
|
||||
"C": "Uniformity",
|
||||
"D": "Forgiveness"
|
||||
},
|
||||
"solution": "A",
|
||||
"llm_answer": "C"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1 +0,0 @@
|
|||
Subproject commit 2d82e92dd6171a3fd5b40f8306af3b8c366179e6
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,3 +1,23 @@
|
|||
"""
|
||||
SecEval Evaluation Script
|
||||
|
||||
This script evaluates language models on cybersecurity multiple-choice questions.
|
||||
It supports various LLM backends including HuggingFace, Azure OpenAI, TextGen, and Ollama.
|
||||
The script processes questions in batches and calculates accuracy scores by topic.
|
||||
|
||||
Usage:
|
||||
python3 eval.py -d dataset.json -B backend_type -m model_name [options]
|
||||
Example:
|
||||
python3 eval.py --dataset_file datasets/questions.json --output_dir outputs --backend ollama --model ollama/qwen2.5:14b
|
||||
|
||||
Environment Variables:
|
||||
- OPENAI_API_ENDPOINT: Azure OpenAI endpoint URL
|
||||
- OPENAI_API_KEY: Azure OpenAI API key
|
||||
- TEXTGEN_MODEL_URL: URL for TextGen model
|
||||
- LOCAL_HF_MODEL_DIR: Directory containing local HuggingFace models
|
||||
- OLLAMA_API_BASE: Base URL for Ollama API (default: http://localhost:8000)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
from typing import Any, Dict, List
|
||||
from dotenv import load_dotenv
|
||||
|
|
@ -19,60 +39,59 @@ from pathlib import Path
|
|||
import os
|
||||
import logging
|
||||
|
||||
# Configure logging
|
||||
logger = logging.getLogger(__name__)
|
||||
import time
|
||||
from langchain.globals import set_llm_cache
|
||||
from langchain.cache import SQLiteCache
|
||||
import litellm
|
||||
|
||||
# Set up caching for LLM responses
|
||||
set_llm_cache(
|
||||
SQLiteCache(
|
||||
database_path=str(Path(__file__).parent.parent / ".langchain.db")
|
||||
)
|
||||
)
|
||||
|
||||
# Configure logging settings
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
logger.addHandler(logging.FileHandler(f"./eval-{int(time.time())}.log", "w"))
|
||||
instruction = "Below are multiple-choice questions concerning cybersecurity. Please select the correct answers and respond with the letters ABCD only."
|
||||
chat_few_shot = [
|
||||
HumanMessage(
|
||||
content="Question: Which mitigation prevent stack overflow bug? A: Stack Canary. B: ALSR. C: CFI. D: Code Signing.",
|
||||
),
|
||||
AIMessage(
|
||||
content="Answer: ABC",
|
||||
),
|
||||
]
|
||||
few_shot = """
|
||||
Question: You are reviewing a piece of code to identify potential vulnerabilities related to NULL pointer dereferences. Which of the following strategies should you consider implementing to mitigate the risk? Choose all that apply. A: Use a programming language that inherently manages pointers and prevents direct memory access. B: Apply input validation to all variables and data stores that receive data from external sources to ensure they are initialized to expected values. C: Explicitly initialize all variables and other data stores, either during declaration or just before the first usage. D: Implement a custom memory management system that automatically assigns a unique identifier to each pointer.
|
||||
Answer: ABC
|
||||
Question: Analyze the code snippet provided. Which of the following practices should be implemented in a multithreaded application to prevent NULL pointer dereference vulnerabilities? Select all that apply.A: Ensure that proper locking APIs are used to lock the pointer before the if statement and unlock when it has finished to prevent race conditions. B: Check the return value of all functions that return a value, including network-related functions, and verify that it is non-null before using it.C: Use automated static analysis tools that target this type of weakness, understanding that while not perfect, they can still be effective.D: Verify that a non-nil 'response' is present before deferring 'response.Body.Close()' to handle cases where the 'Do' method returns an error.
|
||||
Answer: ABCD
|
||||
Question: In a bank management application, a method called `createBankAccount` allows the creation of bank accounts without user identity verification or authority checks. Based on common security practices, which of the following measures would improve security?A: Implement a centralized authentication mechanism for operations that require a proven user identity.B: Duplicate client-side authentication checks on the server side to prevent bypassing of security checks.C: Use a vetted library or framework that offers robust authentication capabilities to prevent security weaknesses.D: Apply strong authentication requirements for all users accessing data stored in the cloud.
|
||||
Answer: ABCD
|
||||
Question: What are the potential consequences if an application handling sensitive data does not require authentication for data access, and what are the appropriate mitigation strategies?A: Attackers could easily access sensitive data from any location; require strong authentication for data access using cloud provider controls.B: Security checks might be bypassed if only performed on the client side; implement checks on both client and server sides.C: Unauthenticated users could alter product functionality; do not use authentication for critical functionality in products.D: Sensitive data may be accessed without proper credentials; utilize authentication capabilities provided by the framework or operating system.
|
||||
Answer: ABD
|
||||
Question: To prevent security vulnerabilities related to deserialization of untrusted data in a Java application, which of the following practices should a developer implement?A: Use the signing/sealing features of the programming language to assure that deserialized data has not been tainted.B: Explicitly define a final readObject() method to throw an exception and prevent deserialization.C: Populate a new object by deserializing data to ensure data flows through safe input validation functions.D: Make fields transient to protect them from deserialization and prevent carrying over sensitive variables.
|
||||
Answer: ABCD
|
||||
|
||||
# Instruction template for the LLM
|
||||
instruction = """Below are multiple-choice questions concerning cybersecurity.
|
||||
Please select the correct answers and respond with the correct letters A, B, C, or D.
|
||||
You could select more than one letter.
|
||||
"""
|
||||
|
||||
|
||||
def init_hf_llm(model_id: str):
|
||||
# check transformers and torch installation
|
||||
def init_hf_llm(model_id: str) -> HuggingFacePipeline:
|
||||
"""
|
||||
Initialize a HuggingFace language model.
|
||||
|
||||
Args:
|
||||
model_id: The model identifier from HuggingFace
|
||||
|
||||
Returns:
|
||||
HuggingFacePipeline: Initialized model pipeline
|
||||
|
||||
Raises:
|
||||
ImportError: If required dependencies are not installed
|
||||
"""
|
||||
# Check transformers and torch installation
|
||||
try:
|
||||
import transformers
|
||||
except ImportError:
|
||||
raise ImportError("Please install transformers with `pip install transformers`")
|
||||
try:
|
||||
import torch
|
||||
|
||||
flash_attn_enable = torch.cuda.get_device_capability()[0] >= 8
|
||||
except ImportError:
|
||||
raise ImportError("Please install torch with `pip install torch`")
|
||||
|
||||
# todo: add flash_attn_enable to the model_kwargs
|
||||
|
||||
# Initialize HuggingFace pipeline with specified parameters
|
||||
llm = HuggingFacePipeline.from_model_id(
|
||||
model_id=model_id,
|
||||
task="text-generation",
|
||||
|
|
@ -83,18 +102,45 @@ def init_hf_llm(model_id: str):
|
|||
return llm
|
||||
|
||||
|
||||
def init_textgen_llm(model_id: str):
|
||||
def init_textgen_llm(model_id: str) -> TextGen:
|
||||
"""
|
||||
Initialize a TextGen language model.
|
||||
|
||||
Args:
|
||||
model_id: The model identifier
|
||||
|
||||
Returns:
|
||||
TextGen: Initialized model
|
||||
|
||||
Raises:
|
||||
RuntimeError: If TEXTGEN_MODEL_URL is not set
|
||||
"""
|
||||
# Check for required environment variable
|
||||
if os.environ.get("TEXTGEN_MODEL_URL") is None:
|
||||
raise RuntimeError("Please set TEXTGEN_MODEL_URL")
|
||||
llm = TextGen(model_url=os.environ["TEXTGEN_MODEL_URL"]) # type: ignore
|
||||
return llm
|
||||
|
||||
|
||||
def init_azure_openai_llm(model_id: str):
|
||||
def init_azure_openai_llm(model_id: str) -> AzureChatOpenAI:
|
||||
"""
|
||||
Initialize an Azure OpenAI language model.
|
||||
|
||||
Args:
|
||||
model_id: The model identifier
|
||||
|
||||
Returns:
|
||||
AzureChatOpenAI: Initialized model
|
||||
|
||||
Raises:
|
||||
RuntimeError: If required environment variables are not set
|
||||
"""
|
||||
if os.environ.get("OPENAI_API_ENDPOINT") is None:
|
||||
raise RuntimeError("Please set OPENAI_API_ENDPOINT")
|
||||
if os.environ.get("OPENAI_API_KEY") is None:
|
||||
raise RuntimeError("Please set OPENAI_API_KEY")
|
||||
|
||||
# Configure Azure OpenAI parameters
|
||||
azure_params = {
|
||||
"model": model_id,
|
||||
"openai_api_base": os.environ["OPENAI_API_ENDPOINT"],
|
||||
|
|
@ -105,15 +151,116 @@ def init_azure_openai_llm(model_id: str):
|
|||
return AzureChatOpenAI(**azure_params) # type: ignore
|
||||
|
||||
|
||||
def load_dataset(dataset_path: str):
|
||||
def init_ollama_llm(model_id: str) -> 'OllamaChat':
|
||||
"""
|
||||
Initialize an Ollama language model.
|
||||
|
||||
Args:
|
||||
model_id: The model identifier
|
||||
|
||||
Returns:
|
||||
OllamaChat: Initialized model wrapper
|
||||
"""
|
||||
class OllamaChat:
|
||||
async def abatch(self, prompts: List[str]) -> List[str]:
|
||||
responses = []
|
||||
for prompt in prompts:
|
||||
try:
|
||||
ollama_api_base = os.getenv("OLLAMA_API_BASE", "http://localhost:8000")
|
||||
api_base = ollama_api_base.rstrip('/v1')
|
||||
completion = litellm.completion(
|
||||
model=model_id,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
api_base=api_base,
|
||||
custom_llm_provider="ollama"
|
||||
)
|
||||
if hasattr(completion, "choices") and completion.choices:
|
||||
content = completion.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
responses.append(result)
|
||||
else:
|
||||
print("Incorrect answer format detected.")
|
||||
responses.append("Error: No result parsed")
|
||||
except Exception as e:
|
||||
logging.error(f"Ollama error: {e}")
|
||||
responses.append(f"Error: {e}")
|
||||
return responses
|
||||
|
||||
def extract_answer(self, text: str) -> str:
|
||||
match = re.findall(r"[A-D]", text.upper())
|
||||
return "".join(sorted(set(match))) if match else ""
|
||||
|
||||
return OllamaChat()
|
||||
|
||||
|
||||
def init_openrouter_llm(model_id: str):
|
||||
class OpenRouterChat:
|
||||
async def abatch(self, prompts: List[str]):
|
||||
responses = []
|
||||
for prompt in prompts:
|
||||
try:
|
||||
api_base = os.getenv("OPENROUTER_API_BASE", "https://openrouter.ai/api/v1/chat/completions")
|
||||
api_key = os.getenv("OPENROUTER_API_KEY")
|
||||
|
||||
if not api_key:
|
||||
raise ValueError("OPENROUTER_API_KEY is not defined in the environment variables.")
|
||||
|
||||
completion = litellm.completion(
|
||||
model=model_id,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
custom_llm_provider="openrouter"
|
||||
)
|
||||
|
||||
if hasattr(completion, "choices") and completion.choices:
|
||||
content = completion.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
responses.append(result)
|
||||
else:
|
||||
print("Formato de respuesta incorrecto.")
|
||||
responses.append("Error: No se pudo extraer resultado")
|
||||
except Exception as e:
|
||||
logging.error(f"OpenRouter error: {e}")
|
||||
responses.append(f"Error: {e}")
|
||||
return responses
|
||||
|
||||
def extract_answer(self, text: str):
|
||||
match = re.findall(r"[A-D]", text.upper())
|
||||
return "".join(sorted(set(match))) if match else ""
|
||||
|
||||
return OpenRouterChat()
|
||||
|
||||
|
||||
def load_dataset(dataset_path: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Load evaluation dataset from JSON file.
|
||||
|
||||
Args:
|
||||
dataset_path: Path to dataset file
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: Loaded dataset
|
||||
"""
|
||||
with open(dataset_path, "r") as f:
|
||||
dataset = json.load(f)
|
||||
return dataset
|
||||
|
||||
|
||||
async def batch_inference_dataset(
|
||||
llm: BaseLanguageModel, batch: List[Dict[str, Any]], chat=False
|
||||
):
|
||||
async def batch_inference_dataset(llm: BaseLanguageModel, batch: List[Dict[str, Any]], chat: bool = False) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Process a batch of questions through the language model.
|
||||
|
||||
Args:
|
||||
llm: Language model to use
|
||||
batch: List of questions to process
|
||||
chat: Whether to use chat format
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: Processed results with scores
|
||||
"""
|
||||
results = []
|
||||
llm_inputs = []
|
||||
for dataset_row in batch:
|
||||
|
|
@ -122,13 +269,9 @@ async def batch_inference_dataset(
|
|||
)
|
||||
question_text = question_text.replace("\n", " ")
|
||||
if chat:
|
||||
llm_input = (
|
||||
[SystemMessage(content=instruction)]
|
||||
+ chat_few_shot
|
||||
+ [HumanMessage(content=question_text)]
|
||||
)
|
||||
llm_input = [SystemMessage(content=instruction)]
|
||||
else:
|
||||
llm_input = instruction + few_shot + question_text + "\n"
|
||||
llm_input = instruction + "\n"
|
||||
|
||||
llm_inputs.append(llm_input)
|
||||
try:
|
||||
|
|
@ -155,6 +298,14 @@ async def batch_inference_dataset(
|
|||
logging.info(
|
||||
f'llm_output: {llm_output}, parsed answer: {batch[idx]["llm_answer"]}, answer: {batch[idx]["answer"]}'
|
||||
)
|
||||
|
||||
print("Question:", batch[idx]["question"])
|
||||
print("Correct Answer:", batch[idx]["answer"])
|
||||
print("LLM Answer:", batch[idx]["llm_answer"])
|
||||
print("LLM Output:", llm_output)
|
||||
print("Score:", batch[idx]["score"])
|
||||
print("--------------------------------")
|
||||
|
||||
results.append(batch[idx])
|
||||
return results
|
||||
|
||||
|
|
@ -207,55 +358,17 @@ def count_score_by_topic(dataset: List[Dict[str, Any]]):
|
|||
def main():
|
||||
parser = argparse.ArgumentParser(description="SecEval Evaluation CLI")
|
||||
|
||||
parser.add_argument(
|
||||
"-o",
|
||||
"--output_dir",
|
||||
type=str,
|
||||
default="/tmp",
|
||||
help="Specify the output directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-d",
|
||||
"--dataset_file",
|
||||
type=str,
|
||||
required=True,
|
||||
help="Specify the dataset file to evaluate on.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-c",
|
||||
"--chat",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Evaluate on chat model.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-b",
|
||||
"--batch_size",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Specify the batch size.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-B",
|
||||
"--backend",
|
||||
type=str,
|
||||
choices=["remote_hf", "azure", "textgen", "local_hf"],
|
||||
required=True,
|
||||
help="Specify the llm type. remote_hf: remote huggingface model backed, azure: azure openai model, textgen: textgen backend, local_hf: local huggingface model backed",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-m",
|
||||
"--models",
|
||||
type=str,
|
||||
nargs="+",
|
||||
required=True,
|
||||
help="Specify the models.",
|
||||
)
|
||||
|
||||
parser.add_argument("-o", "--output_dir", type=str, default="/tmp", help="Specify the output directory.")
|
||||
parser.add_argument("-d", "--dataset_file", type=str, required=True, help="Specify the dataset file to evaluate on.")
|
||||
parser.add_argument("-c", "--chat", action="store_true", default=False, help="Evaluate on chat model.")
|
||||
parser.add_argument("-b", "--batch_size", type=int, default=1, help="Specify the batch size.")
|
||||
parser.add_argument("-B", "--backend", type=str, choices=["remote_hf", "azure", "textgen", "local_hf", "ollama", "openrouter"], required=True, help="Specify the llm type. remote_hf: remote huggingface model backed, azure: azure openai model, textgen: textgen backend, local_hf: local huggingface model backed, ollama: ollama model, openrouter: openrouter model")
|
||||
parser.add_argument("-m", "--models", type=str, nargs="+", required=True, help="Specify the models.")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
models = list(args.models)
|
||||
logging.info(f"evaluating models: {models}")
|
||||
|
||||
logging.info(f"Evaluating models: {models}")
|
||||
for model_id in models:
|
||||
if args.backend == "remote_hf":
|
||||
llm = init_hf_llm(model_id)
|
||||
|
|
@ -271,28 +384,31 @@ def main():
|
|||
llm = init_textgen_llm(model_id)
|
||||
elif args.backend == "azure":
|
||||
llm = init_azure_openai_llm(model_id)
|
||||
elif args.backend == "ollama":
|
||||
llm = init_ollama_llm(model_id)
|
||||
elif args.backend == "openrouter":
|
||||
llm = init_openrouter_llm(model_id)
|
||||
else:
|
||||
raise RuntimeError("Unknown backend")
|
||||
|
||||
dataset = load_dataset(args.dataset_file)
|
||||
result = inference_dataset(
|
||||
llm, dataset, batch_size=args.batch_size, chat=args.chat
|
||||
)
|
||||
result = inference_dataset(llm, dataset, batch_size=args.batch_size, chat=args.chat)
|
||||
score_fraction, score_float = count_score_by_topic(result)
|
||||
|
||||
result_with_score = {
|
||||
"score_fraction": score_fraction,
|
||||
"score_float": score_float,
|
||||
"detail": result,
|
||||
}
|
||||
output_path = (
|
||||
Path(args.output_dir)
|
||||
/ f"{Path(args.dataset_file).stem}_{os.path.basename(model_id)}.json"
|
||||
)
|
||||
logger.info(f"writing result to {output_path}")
|
||||
# Create output directory if it doesn't exist
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / f"{Path(args.dataset_file).stem}_{os.path.basename(model_id)}.json"
|
||||
|
||||
logger.info(f"Writing result to {output_path}")
|
||||
with open(output_path, "w") as f:
|
||||
json.dump(result_with_score, f, indent=4)
|
||||
del llm
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
main()
|
||||
|
|
@ -1,184 +0,0 @@
|
|||
{
|
||||
"questions": [
|
||||
{
|
||||
"question": "Which of the following is a desirable property of a biometric system?",
|
||||
"answers": {
|
||||
"A": "Permanent",
|
||||
"B": "Transferability",
|
||||
"C": "Uniformity",
|
||||
"D": "Forgiveness"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "In TCP/IP networking, which protocol is used to hold network addresses and routing information in a packet?",
|
||||
"answers": {
|
||||
"A": "HTTP",
|
||||
"B": "IP",
|
||||
"C": "Routing Information Protocol (RIP)",
|
||||
"D": "TCP"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "In the context of unexpected negative outcomes in personal privacy policies, what responsibility lies with both the provider and the consumer regarding the retention time of private information?",
|
||||
"answers": {
|
||||
"A": "Both the provider and the consumer are responsible for setting the appropriate retention time",
|
||||
"B": "Neither the provider nor the consumer has a responsibility for setting the appropriate retention time",
|
||||
"C": "Only the consumer is responsible for setting an appropriate retention time",
|
||||
"D": "Only the provider is responsible for ensuring the appropriate retention time"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What type of attack involves an attacker behaving as a store-and-forward or proxy mechanism between two communicating systems?",
|
||||
"answers": {
|
||||
"A": "Buffer overflow attack",
|
||||
"B": "Man-in-the-Middle attack",
|
||||
"C": "DDoS attack",
|
||||
"D": "Sniffer attack"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the fundamental purpose of logging and monitoring in an organization's security measures?",
|
||||
"answers": {
|
||||
"A": "To store backups of critical data",
|
||||
"B": "To track, record, and review activity to detect and respond to security incidents",
|
||||
"C": "To manage and enforce user access controls",
|
||||
"D": "To create a record of all employee activities"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the benefit of structured walk-throughs in disaster recovery testing?",
|
||||
"answers": {
|
||||
"A": "Observe live actions in a controlled environment",
|
||||
"B": "Conduct a full shut-down and restoration at the primary site",
|
||||
"C": "Test operational response to disaster scenarios",
|
||||
"D": "Interrupt real operations at the primary site"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which security process metric would most assist in determining an appropriate backup frequency for a database server?",
|
||||
"answers": {
|
||||
"A": "RTO",
|
||||
"B": "MTBF",
|
||||
"C": "RPO",
|
||||
"D": "MTD"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What constitutional amendment outlines the burden placed on investigators to have a valid search warrant before conducting certain searches?",
|
||||
"answers": {
|
||||
"A": "Third Amendment",
|
||||
"B": "First Amendment",
|
||||
"C": "Second Amendment",
|
||||
"D": "Fourth Amendment"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which tool is frequently used to directly access websites without a browser and is often utilized for testing and identifying potential API vulnerabilities?",
|
||||
"answers": {
|
||||
"A": "SSH",
|
||||
"B": "cURL",
|
||||
"C": "FTP",
|
||||
"D": "Telnet"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the distinguishing characteristic of symmetric-key cryptography?",
|
||||
"answers": {
|
||||
"A": "It tends to be CPU intensive",
|
||||
"B": "It uses the same key for encryption and decryption",
|
||||
"C": "It provides integrity protection to data",
|
||||
"D": "It uses different but related keys for encryption and decryption"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the result of 5521211 modulo 1337?",
|
||||
"answers": {
|
||||
"A": "11111",
|
||||
"B": "33",
|
||||
"C": "602",
|
||||
"D": "738"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is an example of a strong password?",
|
||||
"answers": {
|
||||
"A": "Password123456",
|
||||
"B": "ME_Fgh4_al445!A",
|
||||
"C": "Qwerty!",
|
||||
"D": "123456"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which model consists of the Application Layer, Host-to-Host Layer, Internet Layer, and Network Access Layer?",
|
||||
"answers": {
|
||||
"A": "OSI model",
|
||||
"B": "WAN model",
|
||||
"C": "LAN model",
|
||||
"D": "TCP/IP model"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of 'inference engine' in the context of expert systems?",
|
||||
"answers": {
|
||||
"A": "To infer new facts and solve problems using rules and facts in the knowledge base",
|
||||
"B": "To encrypt messages passed across the Internet",
|
||||
"C": "To enable recognition of an entity by a system",
|
||||
"D": "To control access to resources in a system"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary objective of conducting code reviews for bespoke and custom software applications within the Payment Card Industry Data Security Standard guidelines?",
|
||||
"answers": {
|
||||
"A": "To expedite the deplxoyment of bespoke and custom software into production",
|
||||
"B": "To exploit potential coding vulnerabilities in production software",
|
||||
"C": "To ensure bespoke and custom software meets performance benchmarks under heavy load",
|
||||
"D": "To ensure that bespoke and custom software cannot be exploited via coding vulnerabilities"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which principle of cybersecurity focuses on ensuring that data is accurate and trustworthy?",
|
||||
"answers": {
|
||||
"A": "Resilience",
|
||||
"B": "Availability",
|
||||
"C": "Authenticity",
|
||||
"D": "Integrity"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "An IDS installed on the network perimeter sees a spike in traffic during off-duty hours and begins logging and alerting. Which type of IDS is in place?",
|
||||
"answers": {
|
||||
"A": "Packet filtering",
|
||||
"B": "Anomaly based",
|
||||
"C": "Signature based",
|
||||
"D": "Stateful"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a desirable property of a biometric system?",
|
||||
"answers": {
|
||||
"A": "Permanent",
|
||||
"B": "Transferability",
|
||||
"C": "Uniformity",
|
||||
"D": "Forgiveness"
|
||||
},
|
||||
"solution": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,25 +0,0 @@
|
|||
{
|
||||
"questions": [
|
||||
{
|
||||
"question": "Which of the following is a desirable property of a biometric system?",
|
||||
"answers": {
|
||||
"A": "Permanent",
|
||||
"B": "Transferability",
|
||||
"C": "Uniformity",
|
||||
"D": "Forgiveness"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "In TCP/IP networking, which protocol is used to hold network addresses and routing information in a packet?",
|
||||
"answers": {
|
||||
"A": "HTTP",
|
||||
"B": "IP",
|
||||
"C": "Routing Information Protocol (RIP)",
|
||||
"D": "TCP"
|
||||
},
|
||||
"solution": "B"
|
||||
}
|
||||
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -1,811 +0,0 @@
|
|||
{
|
||||
"questions": [
|
||||
|
||||
{
|
||||
"question": "What is the primary requirement for an Random Bit Generator's (RBG) output to be used for generating cryptographic keys?",
|
||||
"answers": {
|
||||
"A": "The RBG's output should have a length that matches the target data to be protected",
|
||||
"B": "The RBG's output should be computationally indistinguishable from random bits and provide sufficient entropy to support the security strength required for the target data",
|
||||
"C": "The RBG's output should be as long as possible to ensure maximal randomness",
|
||||
"D": "The RBG's output should have precisely the same length as the symmetric key to be generated"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "In cryptography, what is the purpose of using a key-derivation function (KDF)?",
|
||||
"answers": {
|
||||
"A": "Generate public keys",
|
||||
"B": "Authenticate digital signatures",
|
||||
"C": "Encrypt data using a password",
|
||||
"D": "KDF are algorithms used to transform a secret into crucial parameters like keys and Initialization Vectors (IVs)"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of segmentation in the context of PCI DSS?",
|
||||
"answers": {
|
||||
"A": "To reduce the number of PCI DSS requirements applicable to an entity",
|
||||
"B": "To limit the scope of the PCI DSS assessment and minimize the potential for security breaches",
|
||||
"C": "To completely remove the applicability of PCI DSS for a merchant's cardholder data environment",
|
||||
"D": "To eliminate the need for implementing PCI DSS controls"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of using strong cryptography to protect cardholder data during transmission over open, public networks?",
|
||||
"answers": {
|
||||
"A": "To ensure the data is secured from unauthorized access or interception during transmission",
|
||||
"B": "To increase the speed of data transmission over open, public networks",
|
||||
"C": "To provide access to authorized individuals to monitor the transmission process",
|
||||
"D": "To improve the efficiency of network routing protocols"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of periodic reviews of user accounts and access privileges?",
|
||||
"answers": {
|
||||
"A": "To ensure all user accounts have access to the highest privileges",
|
||||
"B": "To demonstrate adherence to regulatory requirements",
|
||||
"C": "To create a record of all user accounts",
|
||||
"D": "To identify and remove any inappropriate access and privileges"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a purpose of having synchronized system clocks?",
|
||||
"answers": {
|
||||
"A": "To compare log files from different systems",
|
||||
"B": "To accelerate system performance",
|
||||
"C": "To standardize file naming conventions",
|
||||
"D": "To reduce power consumption"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is physical security?",
|
||||
"answers": {
|
||||
"A": "The protection of data from hacker attacks",
|
||||
"B": "The prevention of natural disasters caused by environmental factors",
|
||||
"C": "The act of securing a building or location to prevent unauthorized access, theft, or damage to physical assets",
|
||||
"D": "The enforcement of technical security controls to prevent data breaches"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What does the abbreviation 'SDK' stand for in the context of software security?",
|
||||
"answers": {
|
||||
"A": "Software Development Kit",
|
||||
"B": "System Deflection Key",
|
||||
"C": "Software Delegation Kernel",
|
||||
"D": "System Development Key"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the main purpose of multi-factor authentication in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To limit user privileges within the network",
|
||||
"B": "To ensure compliance with government regulations",
|
||||
"C": "To adds an extra layer of security by requiring multiple forms of verification",
|
||||
"D": "To monitor network traffic for security threats"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary objective of a Web Application Firewall (WAF) in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To encrypt network traffic between web servers",
|
||||
"B": "To prevent unauthorized access to network services",
|
||||
"C": "To protect web applications from security threats",
|
||||
"D": "To manage user authentication and authorization for web services"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of periodically changing passwords or passphrases?",
|
||||
"answers": {
|
||||
"A": "To provide more time for a malicious individual to crack the password/passphrase",
|
||||
"B": "To allow for more rapid detection and response to address potentially compromised credentials",
|
||||
"C": "To prevent unauthorized users from gaining access to the user account through use of a shared authentication factor",
|
||||
"D": "To offer less time for a malicious individual to crack a password/passphrase and less time to use a compromised password"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following defines the entity’s security objectives and principles?",
|
||||
"answers": {
|
||||
"A": "Security policies",
|
||||
"B": "Operational procedures",
|
||||
"C": "Risk assessment framework",
|
||||
"D": "Incident response plan"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the focus of a security audit or vulnerability assessment?",
|
||||
"answers": {
|
||||
"A": "Locating vulnerabilities",
|
||||
"B": "Locating threats",
|
||||
"C": "Enacting threats",
|
||||
"D": "Exploiting vulnerabilities"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a mechanism for managing digital certificates through a system of trust?",
|
||||
"answers": {
|
||||
"A": "PKI",
|
||||
"B": "PKCS",
|
||||
"C": "ISA",
|
||||
"D": "SSL"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which protocol is used to create a secure environment in a wireless network?",
|
||||
"answers": {
|
||||
"A": "WAP",
|
||||
"B": "WPA2",
|
||||
"C": "WTLS",
|
||||
"D": "WML3"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which design concept limits access to systems from outside users while protecting users and systems inside the LAN?",
|
||||
"answers": {
|
||||
"A": "DMZ",
|
||||
"B": "VLAN",
|
||||
"C": "I&A",
|
||||
"D": "Router"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the process of making an operating system secure from attack called?",
|
||||
"answers": {
|
||||
"A": "Hardening",
|
||||
"B": "Tuning",
|
||||
"C": "Sealing",
|
||||
"D": "Locking down"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which mechanism is used by PKI to allow immediate verification of a certificate's validity?",
|
||||
"answers": {
|
||||
"A": "CRL",
|
||||
"B": "MD5",
|
||||
"C": "SSHA",
|
||||
"D": "OCSP"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a major security problem with FTP?",
|
||||
"answers": {
|
||||
"A": "Password files are stored in an unsecure area on disk",
|
||||
"B": "Memory traces can corrupt file access",
|
||||
"C": "User IDs and passwords are unencrypted",
|
||||
"D": "FTP sites are unregistered"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following creates a fixed-length output from a variable-length input?",
|
||||
"answers": {
|
||||
"A": "MD5",
|
||||
"B": "SHA1",
|
||||
"C": "SHA3",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of a service-level agreement (SLA) in the context of business continuity?",
|
||||
"answers": {
|
||||
"A": "To define the obligations of the service provider to the client",
|
||||
"B": "To provide high-availability architecture",
|
||||
"C": "To outline a disaster recovery plan",
|
||||
"D": "To ensure the fault tolerance of systems"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What does the Address Resolution Protocol (ARP) do in a network?",
|
||||
"answers": {
|
||||
"A": "Resolves domain names to IP addresses",
|
||||
"B": "Allocates IP addresses to network devices",
|
||||
"C": "Translates IP addresses to media access control (MAC) addresses",
|
||||
"D": "Verifies that a host is reachable"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "In the OSI model, which layer consists of the physical media and infrastructure of networks?",
|
||||
"answers": {
|
||||
"A": "Session",
|
||||
"B": "Presentation",
|
||||
"C": "Data Link",
|
||||
"D": "Physical"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "In binary representation, what is the result of the bitwise XOR operation between 110101 and 101010",
|
||||
"answers": {
|
||||
"A": "011111",
|
||||
"B": "100111",
|
||||
"C": "0",
|
||||
"D": "111111"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the proper sequence of the TCP three-way-handshake?",
|
||||
"answers": {
|
||||
"A": "SYN-ACK, ACK, ACK",
|
||||
"B": "SYN, SYN-ACK, ACK",
|
||||
"C": "SYN-SYN, SYN-ACK, SYN",
|
||||
"D": "ACK, SYN-ACK, SYN"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which topology has built-in redundancy because of its many client connections?",
|
||||
"answers": {
|
||||
"A": "Token ring",
|
||||
"B": "Bus",
|
||||
"C": "Hybrid",
|
||||
"D": "Mesh"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "How can Google hacking be valuable in a cybersecurity investigation?",
|
||||
"answers": {
|
||||
"A": "To create fake Google search results",
|
||||
"B": "To extract data from Google's databases",
|
||||
"C": "To reveal hidden or sensitive information from Google search results",
|
||||
"D": "To manipulate website content in Google's index"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the role of the three-way handshake in establishing a TCP connection?",
|
||||
"answers": {
|
||||
"A": "To request the establishment of a connection",
|
||||
"B": "To acknowledge the request for connection",
|
||||
"C": "To complete the establishment of a connection",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which best describes a vulnerability scan?",
|
||||
"answers": {
|
||||
"A": "A way to find open ports",
|
||||
"B": "A way to diagram a network",
|
||||
"C": "A proxy attack",
|
||||
"D": "A way to automate the discovery of vulnerabilities"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is Tor used for?",
|
||||
"answers": {
|
||||
"A": "To make it more difficult to trace a user's Internet activity",
|
||||
"B": "To hide the process of scanning",
|
||||
"C": "To automate scanning",
|
||||
"D": "To hide the banner on a system"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the maximum number of IP addresses in an IPv6 subnet?",
|
||||
"answers": {
|
||||
"A": "6 bit",
|
||||
"B": "32 bit",
|
||||
"C": "6*16 bit",
|
||||
"D": "8*16 bit"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a characteristic of a worm?",
|
||||
"answers": {
|
||||
"A": "Requires user interaction to spread",
|
||||
"B": "Self-replicates and spreads independently",
|
||||
"C": "Aargets specific individuals or organizations",
|
||||
"D": "A worm is an item that runs silently"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What common tool can be used for launching an ARP poisoning attack?",
|
||||
"answers": {
|
||||
"A": "Cain & Abel",
|
||||
"B": "Nmap",
|
||||
"C": "Scooter",
|
||||
"D": "Tcpdump"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What technology is used to verify information as it is entered into a web application, preventing database manipulation and buffer overflows?",
|
||||
"answers": {
|
||||
"A": "Input Validation",
|
||||
"B": "Cross-Site Scripting (XSS)",
|
||||
"C": "Session Management",
|
||||
"D": "Buffer overflow"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which type of firewall analyzes the status of traffic?",
|
||||
"answers": {
|
||||
"A": "Circuit level",
|
||||
"B": "Packet filtering",
|
||||
"C": "Stateful inspection",
|
||||
"D": "NIDS"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of the Crossover Error Rate (CER) in evaluating biometric devices",
|
||||
"answers": {
|
||||
"A": "To compare the False Rejection Rate (FRR) of different biometric devices",
|
||||
"B": "To compare the accuracy of similar biometric devices from different vendors or different models from the same vendor",
|
||||
"C": "To assess the acceptance level of biometric factors",
|
||||
"D": "To measure the throughput rate of biometric devices"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of a padded cell system in relation to intrusion detection?",
|
||||
"answers": {
|
||||
"A": "Attract unauthorized users",
|
||||
"B": "Simulate a real network for intruders",
|
||||
"C": "Isolate detected intruders",
|
||||
"D": "Send alerts to administrators"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of penetration testing?",
|
||||
"answers": {
|
||||
"A": "To evaluate the robustness of security systems and identify vulnerabilities",
|
||||
"B": "To exploit vulnerabilities and cause damage to the system",
|
||||
"C": "To eliminate all vulnerabilities in the system",
|
||||
"D": "To simulate unauthorized attacks without actually exploiting vulnerabilities"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of RADIUS and TACACS in a network environment?",
|
||||
"answers": {
|
||||
"A": "To centralize the authentication of remote dial-up connections",
|
||||
"B": "To establish secure connections for voice and video conferencing",
|
||||
"C": "To provide security for WAN communication technologies",
|
||||
"D": "To support remote journaling and electronic vaulting"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which symmetric encryption scheme is recommended for use in SRTP?",
|
||||
"answers": {
|
||||
"A": "AES in ECB mode",
|
||||
"B": "DES in CBC mode",
|
||||
"C": "RC4",
|
||||
"D": "AES in GCM mode"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the essential requirement for the security of the Discrete Logarithm Integrated Encryption Scheme (DLIES) to be maintained?",
|
||||
"answers": {
|
||||
"A": "The order q of the base point P should be at least q >= 1024",
|
||||
"B": "Randomly choose two primes p and q such that p*q > 2048 bits",
|
||||
"C": "The length of the prime number p should be at least 3000 bits",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which document provides the recommendation for elliptic curve cryptography?",
|
||||
"answers": {
|
||||
"A": "NIST SP 800-56C",
|
||||
"B": "NIST SP 800-63-3",
|
||||
"C": "NIST SP 800-57 Part 1",
|
||||
"D": "NIST SP 800-186-4"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is an essential requirement for the internal state of deterministic random number generators?",
|
||||
"answers": {
|
||||
"A": "Protection against readout and manipulation",
|
||||
"B": "Constant reseeding requirement",
|
||||
"C": "Dependence on reliable physical resources",
|
||||
"D": "Regular update of entropy sources"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which federal government agency has responsibility for ensuring the security of government computer systems that are not used to process sensitive and/or classified information?",
|
||||
"answers": {
|
||||
"A": "National Security Agency",
|
||||
"B": "Federal Bureau of Investigation",
|
||||
"C": "National Institute of Standards and Technology",
|
||||
"D": "Secret Service"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of implementing monitoring systems?",
|
||||
"answers": {
|
||||
"A": "Monitoring the configuration and configuration change of devices",
|
||||
"B": "Ensuring availability and functionality of systems",
|
||||
"C": "Recording access to information by means of system and security logging",
|
||||
"D": "Protection from compromising radiation"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which principle of cybersecurity focuses on ensuring that data is accurate and trustworthy?",
|
||||
"answers": {
|
||||
"A": "Resilience",
|
||||
"B": "Availability",
|
||||
"C": "Authenticity",
|
||||
"D": "Integrity"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of the Transport Layer Security (TLS) protocol?",
|
||||
"answers": {
|
||||
"A": "Privacy and authentication between two communicating applications",
|
||||
"B": "Privacy and data integrity between two communicating applications",
|
||||
"C": "Authentication and data integrity between two communicating applications",
|
||||
"D": "Privacy, authentication, and data integrity between two communicating applications"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "If you were to see the subnet mask 255.255.248.0, what CIDR notation (prefix) would you use to indicate the same thing?",
|
||||
"answers": {
|
||||
"A": "/23",
|
||||
"B": "/22",
|
||||
"C": "/21",
|
||||
"D": "/20"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is NOT one of the five disaster recovery plan testing types?",
|
||||
"answers": {
|
||||
"A": "Simulation",
|
||||
"B": "Checklist",
|
||||
"C": "Availability",
|
||||
"D": "Full Interruption"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What would be the result of sending the string AAAAAAAAAAAAAAAAA into a variable that has been allocated space for 8 bytes?",
|
||||
"answers": {
|
||||
"A": "Heap spraying",
|
||||
"B": "SQL injection",
|
||||
"C": "Buffer overflow",
|
||||
"D": "Slowloris attack"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is NOT a form of computer/network surveillance?",
|
||||
"answers": {
|
||||
"A": "Keyboard monitoring",
|
||||
"B": "Use of network sniffers",
|
||||
"C": "Use of Closed-Circuit Television (CCTV)",
|
||||
"D": "Review of audit logs"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "How would you calculate risk?",
|
||||
"answers": {
|
||||
"A": "Probability / loss",
|
||||
"B": "Probability * mitigation factor",
|
||||
"C": "(Loss + mitigation factor) * (loss/probability)",
|
||||
"D": "Probability * loss"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the common share name used for interprocess communication?",
|
||||
"answers": {
|
||||
"A": "C$",
|
||||
"B": "IPC$",
|
||||
"C": "ADMIN$",
|
||||
"D": "INTERCOM$"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which port is commonly used by SMB when operating over TCP without NetBIOS?",
|
||||
"answers": {
|
||||
"A": "137",
|
||||
"B": "139",
|
||||
"C": "135",
|
||||
"D": "445"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is an example of a strong password?",
|
||||
"answers": {
|
||||
"A": "Password123456",
|
||||
"B": "ME_Fgh4_al445!A",
|
||||
"C": "Qwerty!",
|
||||
"D": "123456"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which access control model would be found in a firewall?",
|
||||
"answers": {
|
||||
"A": "Mandatory access control",
|
||||
"B": "Discretionary access control",
|
||||
"C": "Role-based access control",
|
||||
"D": "Rule-based access control"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the preferred encryption option for sending large amounts of data?",
|
||||
"answers": {
|
||||
"A": "Symmetric encryption",
|
||||
"B": "Asymmetric encryption",
|
||||
"C": "Public Key Cryptography",
|
||||
"D": "Diffie-Hellman scheme"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is an example of a hashing algorithm?",
|
||||
"answers": {
|
||||
"A": "AES",
|
||||
"B": "RSA",
|
||||
"C": "MD5",
|
||||
"D": "3DES"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which RAID level is not fault tolerant?",
|
||||
"answers": {
|
||||
"A": "RAID 1",
|
||||
"B": "RAID 0",
|
||||
"C": "RAID 5",
|
||||
"D": "RAID 6"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "How many keys exist in a public/private key pair?",
|
||||
"answers": {
|
||||
"A": "0",
|
||||
"B": "2",
|
||||
"C": "1",
|
||||
"D": "4"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of an Information Security Governance strategy?",
|
||||
"answers": {
|
||||
"A": "To minimize the impact of security incidents on the organization",
|
||||
"B": "To obtain senior management commitment and support",
|
||||
"C": "To enforce compliance with information security policies",
|
||||
"D": "To prioritize options to mitigate risks"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the process of hiding text within an image called?",
|
||||
"answers": {
|
||||
"A": "Encryption",
|
||||
"B": "Steganography",
|
||||
"C": "Spyware",
|
||||
"D": "Hiding"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What attack involves inserting an unauthorized person in the communication path between two users to intercept and modify packets?",
|
||||
"answers": {
|
||||
"A": "Known Plaintext Attack",
|
||||
"B": "Chosen Cipher-Text Attack",
|
||||
"C": "MITM Attack",
|
||||
"D": "Brute-Force Attack"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
|
||||
"question": "Who is responsible for implementing the planning, budgeting, and performance of the organization's information security components?",
|
||||
"answers": {
|
||||
"A": "Administrator",
|
||||
"B": "CISO",
|
||||
"C": "MGR",
|
||||
"D": "CEO"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "The primary responsibility of the information security steering committee is:",
|
||||
"answers": {
|
||||
"A": "Information security policy development",
|
||||
"B": "Direction setting and performance monitoring",
|
||||
"C": "Information security control implementation",
|
||||
"D": "Provision of information security training for employees"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "According to NIST guidelines, what is a recommended practice for creating a strong password?",
|
||||
"answers": {
|
||||
"A": "Implementing complex composition rules requiring a mix of character types",
|
||||
"B": "Setting mandatory periodic password changes for all users",
|
||||
"C": "Encouraging the use of longer, more memorable passphrases",
|
||||
"D": "Requiring the use of password hints for easier recall"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of continuously improving cybersecurity guidelines and their application notes?",
|
||||
"answers": {
|
||||
"A": "To categorize security measures into groups",
|
||||
"B": "To react to new technologies, use cases, and risks",
|
||||
"C": "To prevent all identified threats",
|
||||
"D": "To eliminate all cybersecurity risks"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What does DDoS stand for?",
|
||||
"answers": {
|
||||
"A": "Double Denial of Service",
|
||||
"B": "Distributed Denial of Service",
|
||||
"C": "Dual Denial of Services",
|
||||
"D": "Denial of Distributed Services"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which is not a common wireless network authentication mechanism?",
|
||||
"answers": {
|
||||
"A": "WEP",
|
||||
"B": "WPA",
|
||||
"C": "802.1X",
|
||||
"D": "RC4"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which technique aims to identify changes in normal routines by examining patterns of a user's activity?",
|
||||
"answers": {
|
||||
"A": "Vulnerability Assessment",
|
||||
"B": "Behavior-based IDS",
|
||||
"C": "Data Mining",
|
||||
"D": "User Profile Analysis"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of conducting a risk analysis in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To assess the risk level for the organization",
|
||||
"B": "To formulate the most important IT security procedures for the organization",
|
||||
"C": "To create a risk assessment statement",
|
||||
"D": "To determine the level of exposure to identified threats and identify possible safeguards or controls"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary advantage of using UDP over TCP?",
|
||||
"answers": {
|
||||
"A": "Reliability",
|
||||
"B": "Error recovery",
|
||||
"C": "Speed of data transfer",
|
||||
"D": "Data integrity"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What does the CIA triad in cryptography emphasize?",
|
||||
"answers": {
|
||||
"A": "Confidentiality, Integrity, Authenticity",
|
||||
"B": "Confidentiality, Integrity, Availability",
|
||||
"C": "Confidentiality, Authenticity, Nonrepudiation",
|
||||
"D": "Confidentiality, Availability, Nonrepudiation"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which nmap command is used to perform a TCP SYN scan, also known as a half-open scan, to determine which ports are open on a target system?",
|
||||
"answers": {
|
||||
"A": "nmap -sP",
|
||||
"B": "nmap -sT",
|
||||
"C": "nmap -SYN",
|
||||
"D": "nmap -sS"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of business continuity plans?",
|
||||
"answers": {
|
||||
"A": "To recover from natural disasters",
|
||||
"B": "To conduct regular audits of the organization's security systems",
|
||||
"C": "To minimize the effects of a disruptive event on a company",
|
||||
"D": "To increase the cost associated with a disruptive event"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which DoS attack sends large amounts of ICMP Echo traffic to a broadcast IP address with a spoofed source address of a victim?",
|
||||
"answers": {
|
||||
"A": "Smurf attack",
|
||||
"B": "SYN flood attack",
|
||||
"C": "Ping of Death",
|
||||
"D": "Botnet attack"
|
||||
},
|
||||
"solution": "A"
|
||||
}
|
||||
,
|
||||
{
|
||||
"question": "Which of the following tools is commonly used for network packet analysis and troubleshooting?",
|
||||
"answers": {
|
||||
"A": "Wireshark",
|
||||
"B": "Metasploit",
|
||||
"C": "Nmap",
|
||||
"D": "Hydra"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "In the context of encrypting data for secure communication, which encryption algorithm is typically preferred for its speed and efficiency?",
|
||||
"answers": {
|
||||
"A": "RSA-2048",
|
||||
"B": "AES-256",
|
||||
"C": "Both of them",
|
||||
"D": "None of them"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of NOP sleds in buffer overflow exploits?",
|
||||
"answers": {
|
||||
"A": "To encrypt the payload",
|
||||
"B": "To alter the content of the NOP register",
|
||||
"C": "To insert no-operation instructions for padding",
|
||||
"D": "To alter the content of the EIP register"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which security feature can help prevent buffer overflow attacks by marking certain memory regions as non-executable?",
|
||||
"answers": {
|
||||
"A": "Data Execution Prevention (DEP)",
|
||||
"B": "Buffer Overflow Prevention (BOP)",
|
||||
"C": "Intrusion Detection System (IDS)",
|
||||
"D": "Address Space Layout Randomization (ASLR)"
|
||||
},
|
||||
"solution": "A"
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
]
|
||||
}
|
||||
|
|
@ -1,334 +0,0 @@
|
|||
"""
|
||||
CyberMetric Evaluator for LLMs
|
||||
|
||||
This script evaluates the performance of language models on the CyberMetric benchmark.
|
||||
It supports both OpenRouter-hosted models and local Ollama models via LiteLLM proxy.
|
||||
|
||||
Usage:
|
||||
python CyberMetric_evaluator_openrouter_ollama.py --model_name MODEL_NAME [--file_path FILE_PATH] [--api_key API_KEY]
|
||||
|
||||
Arguments:
|
||||
--model_name: Required. Model name with prefix (openrouter/ or ollama/)
|
||||
Examples: openrouter/anthropic/claude-3-opus, ollama/llama3
|
||||
--file_path: Optional. Path to the CyberMetric JSON file (default: CyberMetric-2-v1.json)
|
||||
--api_key: Optional. API key for OpenRouter (can also use OPENROUTER_API_KEY env var)
|
||||
|
||||
Environment Variables:
|
||||
OPENROUTER_API_KEY: API key for OpenRouter (if using OpenRouter models)
|
||||
OPENROUTER_API_BASE: Base URL for OpenRouter API (default: https://openrouter.ai/api/v1)
|
||||
OLLAMA_API_BASE: Base URL for Ollama API via LiteLLM proxy (default: http://localhost:8000/v1)
|
||||
|
||||
Examples:
|
||||
# Run with an OpenRouter model
|
||||
python CyberMetric_evaluator_openrouter_ollama.py --model_name openrouter/qwen/qwen3-32b:free
|
||||
|
||||
# Run with a local Ollama model (requires LiteLLM proxy running)
|
||||
python CyberMetric_evaluator_openrouter_ollama.py --model_name ollama/qwen2.5:14b
|
||||
|
||||
# Specify a different benchmark file
|
||||
python CyberMetric_evaluator_openrouter_ollama.py --model_name openrouter/qwen/qwen3-32b:free --file_path CyberMetric-10000-v1.json
|
||||
|
||||
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
import os
|
||||
import datetime
|
||||
import random
|
||||
import string
|
||||
import argparse
|
||||
from tqdm import tqdm
|
||||
import litellm
|
||||
import requests
|
||||
|
||||
# Default API bases
|
||||
OPENROUTER_API_BASE = "https://openrouter.ai/api/v1"
|
||||
OLLAMA_LITELLM_API_BASE = "http://localhost:8000/v1"
|
||||
|
||||
class CyberMetricEvaluator:
|
||||
def __init__(self, model_name, file_path, api_key=None, openrouter_api_base=None, ollama_litellm_api_base=None):
|
||||
self.model_name = model_name
|
||||
self.file_path = file_path
|
||||
self.failed_questions = []
|
||||
self.failed_count = 0
|
||||
|
||||
# Set API configurations
|
||||
self.openrouter_api_base = openrouter_api_base or os.environ.get("OPENROUTER_API_BASE", OPENROUTER_API_BASE)
|
||||
self.ollama_litellm_api_base = ollama_litellm_api_base or os.environ.get("OLLAMA_LITELLM_API_BASE", OLLAMA_LITELLM_API_BASE)
|
||||
|
||||
# Set API key for OpenRouter if needed
|
||||
self.api_key = None
|
||||
if self.model_name.startswith("openrouter/"):
|
||||
self.api_key = api_key or os.environ.get("OPENROUTER_API_KEY")
|
||||
|
||||
self.start_time = datetime.datetime.now()
|
||||
|
||||
# Create output directory structure
|
||||
self.output_dir = self.create_output_directory()
|
||||
self.info_file = os.path.join(self.output_dir, "information.txt")
|
||||
self.report_file = os.path.join(self.output_dir, "report_failed_questions.json")
|
||||
|
||||
# Initialize info file
|
||||
self.initialize_info_file()
|
||||
|
||||
print("--DEBUG: model_name: ", self.model_name)
|
||||
|
||||
def create_output_directory(self):
|
||||
# Create base directory if it doesn't exist
|
||||
base_dir = "output_cybermetric"
|
||||
if not os.path.exists(base_dir):
|
||||
os.makedirs(base_dir)
|
||||
|
||||
# Format model name for directory (replace / with -)
|
||||
model_dir_name = self.model_name.replace("/", "-")
|
||||
|
||||
# Get current date
|
||||
current_date = datetime.datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
# Create directory name
|
||||
dir_name = f"{model_dir_name}-{current_date}"
|
||||
|
||||
# If directory already exists, add random string
|
||||
full_path = os.path.join(base_dir, dir_name)
|
||||
if os.path.exists(full_path):
|
||||
random_str = ''.join(random.choices(string.ascii_lowercase + string.digits, k=5))
|
||||
dir_name = f"{model_dir_name}-{current_date}-{random_str}"
|
||||
full_path = os.path.join(base_dir, dir_name)
|
||||
|
||||
os.makedirs(full_path)
|
||||
return full_path
|
||||
|
||||
def initialize_info_file(self):
|
||||
with open(self.info_file, 'w') as file:
|
||||
file.write(f"CyberMetric Evaluation\n")
|
||||
file.write(f"=====================\n\n")
|
||||
file.write(f"Model: {self.model_name}\n")
|
||||
file.write(f"Dataset: {self.file_path}\n")
|
||||
file.write(f"Start Time: {self.start_time.strftime('%Y-%m-%d %H:%M:%S')}\n")
|
||||
file.write(f"Status: Running\n")
|
||||
file.write(f"Questions Processed: 0\n")
|
||||
file.write(f"Correct Answers: 0\n")
|
||||
file.write(f"Accuracy: 0.00%\n")
|
||||
|
||||
def update_info_file(self, questions_processed, correct_count, status="Running"):
|
||||
accuracy = correct_count / questions_processed * 100 if questions_processed > 0 else 0
|
||||
|
||||
with open(self.info_file, 'w') as file:
|
||||
file.write(f"CyberMetric Evaluation\n")
|
||||
file.write(f"=====================\n\n")
|
||||
file.write(f"Model: {self.model_name}\n") # "openrouter/qwen/qwen3-32b:free"
|
||||
file.write(f"Dataset: {self.file_path}\n")
|
||||
file.write(f"Start Time: {self.start_time.strftime('%Y-%m-%d %H:%M:%S')}\n")
|
||||
file.write(f"Status: {status}\n")
|
||||
file.write(f"Questions Processed: {questions_processed}\n")
|
||||
file.write(f"Correct Answers: {correct_count}\n")
|
||||
file.write(f"Accuracy: {accuracy:.2f}%\n")
|
||||
|
||||
if status == "Completed":
|
||||
end_time = datetime.datetime.now()
|
||||
duration = end_time - self.start_time
|
||||
file.write(f"End Time: {end_time.strftime('%Y-%m-%d %H:%M:%S')}\n")
|
||||
file.write(f"Duration: {duration}\n")
|
||||
|
||||
def read_json_file(self):
|
||||
with open(self.file_path, 'r') as file:
|
||||
return json.load(file)
|
||||
|
||||
def save_failed_questions(self):
|
||||
report_data = {"questions": self.failed_questions}
|
||||
|
||||
with open(self.report_file, 'w') as file:
|
||||
json.dump(report_data, file, indent=4)
|
||||
|
||||
print(f"Updated failed questions report in {self.report_file}")
|
||||
|
||||
@staticmethod
|
||||
def extract_answer(response):
|
||||
if response.strip():
|
||||
match = re.search(r"ANSWER:?\s*([A-D])", response, re.IGNORECASE)
|
||||
if match:
|
||||
return match.group(1).upper()
|
||||
return None
|
||||
|
||||
def ask_ollama_litellm(self, prompt, max_retries=5):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
response = litellm.completion(
|
||||
model=self.model_name,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a security expert who answers questions."},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
api_base="http://localhost:8000"
|
||||
)
|
||||
if hasattr(response, "choices") and response.choices:
|
||||
content = response.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
print("--DEBUG: result: ", result)
|
||||
return result
|
||||
else:
|
||||
print("Incorrect answer format detected. Attempting the question again.")
|
||||
except Exception as e:
|
||||
print(f"Error: {e}. Attempting the question again in {2 ** attempt} seconds.")
|
||||
time.sleep(2 ** attempt)
|
||||
return None
|
||||
|
||||
def ask_openrouter(self, prompt, max_retries=5):
|
||||
if not self.api_key:
|
||||
raise ValueError("API key is required for OpenRouter models")
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
response = litellm.completion(
|
||||
model=self.model_name,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a security expert who answers questions."},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
api_base=self.openrouter_api_base,
|
||||
api_key=self.api_key,
|
||||
headers={
|
||||
"HTTP-Referer": "https://your-site-url.com",
|
||||
"X-Title": "CyberMetric Evaluator"
|
||||
}
|
||||
)
|
||||
if hasattr(response, "choices") and response.choices:
|
||||
content = response.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
print("--DEBUG: result: ", result)
|
||||
return result
|
||||
else:
|
||||
print("Incorrect answer format detected. Attempting the question again.")
|
||||
except Exception as e:
|
||||
print(f"Error: {e}. Attempting the question again in {2 ** attempt} seconds.")
|
||||
time.sleep(2 ** attempt)
|
||||
return None
|
||||
|
||||
def ask_llm(self, question, answers, max_retries=5):
|
||||
options = ', '.join([f"{key}) {value}" for key, value in answers.items()])
|
||||
prompt = (
|
||||
f"Question: {question}\n"
|
||||
f"Options: {options}\n\n"
|
||||
"Choose the correct answer (A, B, C, or D) only. Always return in this format: 'ANSWER: X'"
|
||||
)
|
||||
print("--DEBUG: prompt: ", prompt)
|
||||
|
||||
if self.model_name.startswith("openrouter/"):
|
||||
return self.ask_openrouter(prompt, max_retries)
|
||||
elif self.model_name.startswith("ollama/"):
|
||||
# Only use the litellm approach with port 8000 for Ollama
|
||||
return self.ask_ollama_litellm(prompt, max_retries)
|
||||
else:
|
||||
print("Error: Model name must start with 'openrouter/' or 'ollama/'")
|
||||
return None
|
||||
|
||||
def run_evaluation(self):
|
||||
if not (self.model_name.startswith("openrouter/") or self.model_name.startswith("ollama/")):
|
||||
print("Error: You must set model name with prefix 'ollama/' or 'openrouter/'")
|
||||
return
|
||||
|
||||
json_data = self.read_json_file()
|
||||
questions_data = json_data['questions']
|
||||
|
||||
correct_count = 0
|
||||
incorrect_answers = []
|
||||
|
||||
with tqdm(total=len(questions_data), desc="Processing Questions") as progress_bar:
|
||||
for i, item in enumerate(questions_data):
|
||||
question = item['question']
|
||||
answers = item['answers']
|
||||
correct_answer = item['solution']
|
||||
|
||||
llm_answer = self.ask_llm(question, answers)
|
||||
if llm_answer == correct_answer:
|
||||
correct_count += 1
|
||||
else:
|
||||
self.failed_questions.append({
|
||||
'question': question,
|
||||
'answers': answers,
|
||||
'solution': correct_answer,
|
||||
'llm_answer': llm_answer
|
||||
})
|
||||
self.failed_count += 1
|
||||
|
||||
if self.failed_count % 2 == 0:
|
||||
self.save_failed_questions()
|
||||
|
||||
incorrect_answers.append({
|
||||
'question': question,
|
||||
'correct_answer': correct_answer,
|
||||
'llm_answer': llm_answer
|
||||
})
|
||||
|
||||
# Update progress and information file
|
||||
questions_processed = i + 1
|
||||
accuracy_rate = correct_count / questions_processed * 100
|
||||
progress_bar.set_postfix_str(f"Accuracy: {accuracy_rate:.2f}%")
|
||||
progress_bar.update(1)
|
||||
|
||||
# Update info file every 5 questions
|
||||
if questions_processed % 5 == 0 or questions_processed == len(questions_data):
|
||||
self.update_info_file(questions_processed, correct_count)
|
||||
|
||||
# Final update with completed status
|
||||
self.update_info_file(len(questions_data), correct_count, "Completed")
|
||||
print(f"\nFinal Accuracy: {correct_count / len(questions_data) * 100:.2f}%")
|
||||
|
||||
if self.failed_questions:
|
||||
self.save_failed_questions() # final failed questions
|
||||
|
||||
if incorrect_answers:
|
||||
print("\nIncorrect Answers:")
|
||||
for item in incorrect_answers:
|
||||
print(f"Question: {item['question']}")
|
||||
print(f"Expected Answer: {item['correct_answer']}, LLM Answer: {item['llm_answer']}\n")
|
||||
|
||||
if __name__ == "__main__":
|
||||
#litellm._turn_on_debug()
|
||||
|
||||
# Create argument parser
|
||||
parser = argparse.ArgumentParser(description='CyberMetric Evaluator for LLMs')
|
||||
parser.add_argument('--model_name', type=str, required=True,
|
||||
help='Model name with prefix (openrouter/ or ollama/)')
|
||||
parser.add_argument('--file_path', type=str, default='CyberMetric-2-v1.json',
|
||||
help='Path to the CyberMetric JSON file')
|
||||
parser.add_argument('--api_key', type=str,
|
||||
help='API key for OpenRouter (can also use OPENROUTER_API_KEY env var)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
model_name = args.model_name
|
||||
file_path = args.file_path
|
||||
api_key = args.api_key or os.environ.get("OPENROUTER_API_KEY")
|
||||
|
||||
if model_name.startswith("ollama/"):
|
||||
# Ollama configuration
|
||||
evaluator = CyberMetricEvaluator(
|
||||
model_name=model_name,
|
||||
file_path=file_path
|
||||
)
|
||||
print(f"Using Ollama configuration with LiteLLM proxy on port 8000")
|
||||
|
||||
elif model_name.startswith("openrouter/"):
|
||||
# OpenRouter configuration
|
||||
if not api_key:
|
||||
raise ValueError("API key must be provided via --api_key or OPENROUTER_API_KEY environment variable for OpenRouter models")
|
||||
|
||||
evaluator = CyberMetricEvaluator(
|
||||
model_name=model_name,
|
||||
file_path=file_path,
|
||||
api_key=api_key,
|
||||
openrouter_api_base=os.environ.get("OPENROUTER_API_BASE", "https://openrouter.ai/api/v1")
|
||||
)
|
||||
print("Using OpenRouter configuration")
|
||||
|
||||
else:
|
||||
raise ValueError("Model name must start with 'ollama/' or 'openrouter/'")
|
||||
|
||||
# Run the evaluation
|
||||
evaluator.run_evaluation()
|
||||
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
# CyberMetric Dataset
|
||||
|
||||
<div align="center">
|
||||
<img width="800" alt="logo" src="https://github.com/cybermetric/CyberMetric/assets/159767263/455e5a31-97da-4179-ad49-fa182fe7d9ad">
|
||||
</div>
|
||||
|
||||
|
||||
# Description
|
||||
|
||||
|
||||
|
||||
The **CyberMetric Dataset** introduces a new benchmarking tool consisting of 10,000 questions designed to evaluate the cybersecurity knowledge of various Large Language Models (LLMs) within the cybersecurity domain. This dataset is created using different LLMs and has been verified by human experts in the cybersecurity field to ensure its relevance and accuracy. The dataset is compiled from various sources including standards, certifications, research papers, books, and other publications within the cybersecurity field. We provide the dataset in four distinct sizes —small, medium, big and large— comprising 80, 500, 2000 and 10,000 questions, respectively.The smallest version is tailored for comparisons between different LLMs and humans. The CyberMetric-80 dataset has been subject to testing with 30 human participants, enabling an effective comparison between human and machine intelligence.
|
||||
|
||||
# Cite
|
||||
|
||||
The CyberMetric paper **"CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge"** has been accepted for publication in the 2024 IEEE International Conference on Cyber Security and Resilience (IEEE CSR 2024).
|
||||
|
||||
IEEE Xplore link: https://ieeexplore.ieee.org/document/10679494
|
||||
|
||||
Cite the paper:
|
||||
```python
|
||||
@INPROCEEDINGS{10679494,
|
||||
author={Tihanyi, Norbert and Ferrag, Mohamed Amine and Jain, Ridhi and Bisztray, Tamas and Debbah, Merouane},
|
||||
booktitle={2024 IEEE International Conference on Cyber Security and Resilience (CSR)},
|
||||
title={CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge},
|
||||
year={2024},
|
||||
volume={},
|
||||
number={},
|
||||
pages={296-302},
|
||||
keywords={Accuracy;Reverse engineering;Benchmark testing;NIST Standards;Risk management;Problem-solving;Computer security},
|
||||
doi={10.1109/CSR61664.2024.10679494}}
|
||||
```
|
||||
|
||||
# Architecture
|
||||
|
||||
The CyberMetric dataset was created by applying different language models using Retrieval-Augmented Generation (RAG), with human validation included in the process. The AI-driven generation framework is illustrated in the following figure.
|
||||
<img width="1102" alt="Framework" src="https://github.com/cybermetric/CyberMetric/assets/159767263/6d3c8d55-289d-4922-8302-4f3e379a9805">
|
||||
|
||||
# Prompt
|
||||
Most of the evaluated models were instruction-fine-tuned, and the following prompts were used to obtain the correct answers in XML format.
|
||||
```python
|
||||
def make_messages(question, answers):
|
||||
"""
|
||||
Formats a single question+answers into a list of message dictionaries for the pipeline.
|
||||
"""
|
||||
options_str = ', '.join([f"{key}) {value}" for key, value in answers.items()])
|
||||
instructions = (
|
||||
"You are a helpful AI assistant.\n"
|
||||
"Instructions:\n"
|
||||
"a. Carefully read the question.\n"
|
||||
"b. Choose the correct answer (A, B, C, or D) only.\n"
|
||||
"c. Do NOT include any explanation or additional text in the response.\n"
|
||||
"d. Always return the answer in this XML format: '<xml>answer</xml>'. "
|
||||
"For example, if the correct answer is D, then return <xml>D</xml>.\n\n"
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": instructions},
|
||||
{"role": "user", "content": f"#Question: {question}\nOptions: {options_str}"}
|
||||
]
|
||||
return messages
|
||||
|
||||
```
|
||||
# LLM Leaderboard on CyberMetric Dataset
|
||||
|
||||
We have assessed and compared state-of-the-art LLM models using the CyberMetric dataset. The most recent evaluation was conducted on December 27th, 2024.
|
||||
|
||||
|
||||
|
||||
|
||||
<div align="center">
|
||||
<img width="1065" alt="Cybermetric_result" src="https://github.com/user-attachments/assets/dc3a0a0f-59d1-464e-bf67-a37d0008bdaf" />
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
# Usage
|
||||
|
||||
We have developed a compact Python script called `CyberMetric_evaluator.py` to showcase how to utilize the Dataset with OpenAI GPT. Simply insert your API key in the script by setting `API_KEY="<YOUR-API-KEY-HERE>"`, and then execute the evaluator program.
|
||||
|
||||
|
||||
Here's an example output generated by the script using the CyberMetric-80 dataset:
|
||||
|
||||
|
||||
|
|
@ -1 +0,0 @@
|
|||
Subproject commit 7aef3176a1a095b40bbfa806a7a2d6c5e203d5b7
|
||||
Loading…
Reference in New Issue