mirror of https://github.com/aliasrobotics/cai.git
cybermetric submodule
This commit is contained in:
Mery-Sanz
parent
3ee4833dd6
commit
54bb4655b2
|
|
@ -1,9 +1,9 @@
|
|||
[submodule "benchmarks/cybermetric"]
|
||||
path = benchmarks/cybermetric
|
||||
url = https://github.com/cybermetric/CyberMetric.git
|
||||
[submodule "benchmarks/cti_bench"]
|
||||
path = benchmarks/cti_bench
|
||||
url = https://github.com/xashru/cti-bench.git
|
||||
[submodule "benchmarks/seceval"]
|
||||
path = benchmarks/seceval
|
||||
url = https://github.com/XuanwuAI/SecEval.git
|
||||
[submodule "benchmarks/cybermetric"]
|
||||
path = benchmarks/cybermetric
|
||||
url = https://github.com/cybermetric/CyberMetric.git
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
Subproject commit 2f5818bd2c19350cd6cfae028b75499ebe4ffd29
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,25 +0,0 @@
|
|||
{
|
||||
"questions": [
|
||||
{
|
||||
"question": "Which of the following is a desirable property of a biometric system?",
|
||||
"answers": {
|
||||
"A": "Permanent",
|
||||
"B": "Transferability",
|
||||
"C": "Uniformity",
|
||||
"D": "Forgiveness"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "In TCP/IP networking, which protocol is used to hold network addresses and routing information in a packet?",
|
||||
"answers": {
|
||||
"A": "HTTP",
|
||||
"B": "IP",
|
||||
"C": "Routing Information Protocol (RIP)",
|
||||
"D": "TCP"
|
||||
},
|
||||
"solution": "B"
|
||||
}
|
||||
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -1,811 +0,0 @@
|
|||
{
|
||||
"questions": [
|
||||
|
||||
{
|
||||
"question": "What is the primary requirement for an Random Bit Generator's (RBG) output to be used for generating cryptographic keys?",
|
||||
"answers": {
|
||||
"A": "The RBG's output should have a length that matches the target data to be protected",
|
||||
"B": "The RBG's output should be computationally indistinguishable from random bits and provide sufficient entropy to support the security strength required for the target data",
|
||||
"C": "The RBG's output should be as long as possible to ensure maximal randomness",
|
||||
"D": "The RBG's output should have precisely the same length as the symmetric key to be generated"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "In cryptography, what is the purpose of using a key-derivation function (KDF)?",
|
||||
"answers": {
|
||||
"A": "Generate public keys",
|
||||
"B": "Authenticate digital signatures",
|
||||
"C": "Encrypt data using a password",
|
||||
"D": "KDF are algorithms used to transform a secret into crucial parameters like keys and Initialization Vectors (IVs)"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of segmentation in the context of PCI DSS?",
|
||||
"answers": {
|
||||
"A": "To reduce the number of PCI DSS requirements applicable to an entity",
|
||||
"B": "To limit the scope of the PCI DSS assessment and minimize the potential for security breaches",
|
||||
"C": "To completely remove the applicability of PCI DSS for a merchant's cardholder data environment",
|
||||
"D": "To eliminate the need for implementing PCI DSS controls"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of using strong cryptography to protect cardholder data during transmission over open, public networks?",
|
||||
"answers": {
|
||||
"A": "To ensure the data is secured from unauthorized access or interception during transmission",
|
||||
"B": "To increase the speed of data transmission over open, public networks",
|
||||
"C": "To provide access to authorized individuals to monitor the transmission process",
|
||||
"D": "To improve the efficiency of network routing protocols"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of periodic reviews of user accounts and access privileges?",
|
||||
"answers": {
|
||||
"A": "To ensure all user accounts have access to the highest privileges",
|
||||
"B": "To demonstrate adherence to regulatory requirements",
|
||||
"C": "To create a record of all user accounts",
|
||||
"D": "To identify and remove any inappropriate access and privileges"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a purpose of having synchronized system clocks?",
|
||||
"answers": {
|
||||
"A": "To compare log files from different systems",
|
||||
"B": "To accelerate system performance",
|
||||
"C": "To standardize file naming conventions",
|
||||
"D": "To reduce power consumption"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is physical security?",
|
||||
"answers": {
|
||||
"A": "The protection of data from hacker attacks",
|
||||
"B": "The prevention of natural disasters caused by environmental factors",
|
||||
"C": "The act of securing a building or location to prevent unauthorized access, theft, or damage to physical assets",
|
||||
"D": "The enforcement of technical security controls to prevent data breaches"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What does the abbreviation 'SDK' stand for in the context of software security?",
|
||||
"answers": {
|
||||
"A": "Software Development Kit",
|
||||
"B": "System Deflection Key",
|
||||
"C": "Software Delegation Kernel",
|
||||
"D": "System Development Key"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the main purpose of multi-factor authentication in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To limit user privileges within the network",
|
||||
"B": "To ensure compliance with government regulations",
|
||||
"C": "To adds an extra layer of security by requiring multiple forms of verification",
|
||||
"D": "To monitor network traffic for security threats"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary objective of a Web Application Firewall (WAF) in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To encrypt network traffic between web servers",
|
||||
"B": "To prevent unauthorized access to network services",
|
||||
"C": "To protect web applications from security threats",
|
||||
"D": "To manage user authentication and authorization for web services"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of periodically changing passwords or passphrases?",
|
||||
"answers": {
|
||||
"A": "To provide more time for a malicious individual to crack the password/passphrase",
|
||||
"B": "To allow for more rapid detection and response to address potentially compromised credentials",
|
||||
"C": "To prevent unauthorized users from gaining access to the user account through use of a shared authentication factor",
|
||||
"D": "To offer less time for a malicious individual to crack a password/passphrase and less time to use a compromised password"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following defines the entity’s security objectives and principles?",
|
||||
"answers": {
|
||||
"A": "Security policies",
|
||||
"B": "Operational procedures",
|
||||
"C": "Risk assessment framework",
|
||||
"D": "Incident response plan"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the focus of a security audit or vulnerability assessment?",
|
||||
"answers": {
|
||||
"A": "Locating vulnerabilities",
|
||||
"B": "Locating threats",
|
||||
"C": "Enacting threats",
|
||||
"D": "Exploiting vulnerabilities"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a mechanism for managing digital certificates through a system of trust?",
|
||||
"answers": {
|
||||
"A": "PKI",
|
||||
"B": "PKCS",
|
||||
"C": "ISA",
|
||||
"D": "SSL"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which protocol is used to create a secure environment in a wireless network?",
|
||||
"answers": {
|
||||
"A": "WAP",
|
||||
"B": "WPA2",
|
||||
"C": "WTLS",
|
||||
"D": "WML3"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which design concept limits access to systems from outside users while protecting users and systems inside the LAN?",
|
||||
"answers": {
|
||||
"A": "DMZ",
|
||||
"B": "VLAN",
|
||||
"C": "I&A",
|
||||
"D": "Router"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the process of making an operating system secure from attack called?",
|
||||
"answers": {
|
||||
"A": "Hardening",
|
||||
"B": "Tuning",
|
||||
"C": "Sealing",
|
||||
"D": "Locking down"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which mechanism is used by PKI to allow immediate verification of a certificate's validity?",
|
||||
"answers": {
|
||||
"A": "CRL",
|
||||
"B": "MD5",
|
||||
"C": "SSHA",
|
||||
"D": "OCSP"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a major security problem with FTP?",
|
||||
"answers": {
|
||||
"A": "Password files are stored in an unsecure area on disk",
|
||||
"B": "Memory traces can corrupt file access",
|
||||
"C": "User IDs and passwords are unencrypted",
|
||||
"D": "FTP sites are unregistered"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following creates a fixed-length output from a variable-length input?",
|
||||
"answers": {
|
||||
"A": "MD5",
|
||||
"B": "SHA1",
|
||||
"C": "SHA3",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of a service-level agreement (SLA) in the context of business continuity?",
|
||||
"answers": {
|
||||
"A": "To define the obligations of the service provider to the client",
|
||||
"B": "To provide high-availability architecture",
|
||||
"C": "To outline a disaster recovery plan",
|
||||
"D": "To ensure the fault tolerance of systems"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What does the Address Resolution Protocol (ARP) do in a network?",
|
||||
"answers": {
|
||||
"A": "Resolves domain names to IP addresses",
|
||||
"B": "Allocates IP addresses to network devices",
|
||||
"C": "Translates IP addresses to media access control (MAC) addresses",
|
||||
"D": "Verifies that a host is reachable"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "In the OSI model, which layer consists of the physical media and infrastructure of networks?",
|
||||
"answers": {
|
||||
"A": "Session",
|
||||
"B": "Presentation",
|
||||
"C": "Data Link",
|
||||
"D": "Physical"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "In binary representation, what is the result of the bitwise XOR operation between 110101 and 101010",
|
||||
"answers": {
|
||||
"A": "011111",
|
||||
"B": "100111",
|
||||
"C": "0",
|
||||
"D": "111111"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the proper sequence of the TCP three-way-handshake?",
|
||||
"answers": {
|
||||
"A": "SYN-ACK, ACK, ACK",
|
||||
"B": "SYN, SYN-ACK, ACK",
|
||||
"C": "SYN-SYN, SYN-ACK, SYN",
|
||||
"D": "ACK, SYN-ACK, SYN"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which topology has built-in redundancy because of its many client connections?",
|
||||
"answers": {
|
||||
"A": "Token ring",
|
||||
"B": "Bus",
|
||||
"C": "Hybrid",
|
||||
"D": "Mesh"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "How can Google hacking be valuable in a cybersecurity investigation?",
|
||||
"answers": {
|
||||
"A": "To create fake Google search results",
|
||||
"B": "To extract data from Google's databases",
|
||||
"C": "To reveal hidden or sensitive information from Google search results",
|
||||
"D": "To manipulate website content in Google's index"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the role of the three-way handshake in establishing a TCP connection?",
|
||||
"answers": {
|
||||
"A": "To request the establishment of a connection",
|
||||
"B": "To acknowledge the request for connection",
|
||||
"C": "To complete the establishment of a connection",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which best describes a vulnerability scan?",
|
||||
"answers": {
|
||||
"A": "A way to find open ports",
|
||||
"B": "A way to diagram a network",
|
||||
"C": "A proxy attack",
|
||||
"D": "A way to automate the discovery of vulnerabilities"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is Tor used for?",
|
||||
"answers": {
|
||||
"A": "To make it more difficult to trace a user's Internet activity",
|
||||
"B": "To hide the process of scanning",
|
||||
"C": "To automate scanning",
|
||||
"D": "To hide the banner on a system"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the maximum number of IP addresses in an IPv6 subnet?",
|
||||
"answers": {
|
||||
"A": "6 bit",
|
||||
"B": "32 bit",
|
||||
"C": "6*16 bit",
|
||||
"D": "8*16 bit"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a characteristic of a worm?",
|
||||
"answers": {
|
||||
"A": "Requires user interaction to spread",
|
||||
"B": "Self-replicates and spreads independently",
|
||||
"C": "Aargets specific individuals or organizations",
|
||||
"D": "A worm is an item that runs silently"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What common tool can be used for launching an ARP poisoning attack?",
|
||||
"answers": {
|
||||
"A": "Cain & Abel",
|
||||
"B": "Nmap",
|
||||
"C": "Scooter",
|
||||
"D": "Tcpdump"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What technology is used to verify information as it is entered into a web application, preventing database manipulation and buffer overflows?",
|
||||
"answers": {
|
||||
"A": "Input Validation",
|
||||
"B": "Cross-Site Scripting (XSS)",
|
||||
"C": "Session Management",
|
||||
"D": "Buffer overflow"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which type of firewall analyzes the status of traffic?",
|
||||
"answers": {
|
||||
"A": "Circuit level",
|
||||
"B": "Packet filtering",
|
||||
"C": "Stateful inspection",
|
||||
"D": "NIDS"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of the Crossover Error Rate (CER) in evaluating biometric devices",
|
||||
"answers": {
|
||||
"A": "To compare the False Rejection Rate (FRR) of different biometric devices",
|
||||
"B": "To compare the accuracy of similar biometric devices from different vendors or different models from the same vendor",
|
||||
"C": "To assess the acceptance level of biometric factors",
|
||||
"D": "To measure the throughput rate of biometric devices"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of a padded cell system in relation to intrusion detection?",
|
||||
"answers": {
|
||||
"A": "Attract unauthorized users",
|
||||
"B": "Simulate a real network for intruders",
|
||||
"C": "Isolate detected intruders",
|
||||
"D": "Send alerts to administrators"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of penetration testing?",
|
||||
"answers": {
|
||||
"A": "To evaluate the robustness of security systems and identify vulnerabilities",
|
||||
"B": "To exploit vulnerabilities and cause damage to the system",
|
||||
"C": "To eliminate all vulnerabilities in the system",
|
||||
"D": "To simulate unauthorized attacks without actually exploiting vulnerabilities"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of RADIUS and TACACS in a network environment?",
|
||||
"answers": {
|
||||
"A": "To centralize the authentication of remote dial-up connections",
|
||||
"B": "To establish secure connections for voice and video conferencing",
|
||||
"C": "To provide security for WAN communication technologies",
|
||||
"D": "To support remote journaling and electronic vaulting"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which symmetric encryption scheme is recommended for use in SRTP?",
|
||||
"answers": {
|
||||
"A": "AES in ECB mode",
|
||||
"B": "DES in CBC mode",
|
||||
"C": "RC4",
|
||||
"D": "AES in GCM mode"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the essential requirement for the security of the Discrete Logarithm Integrated Encryption Scheme (DLIES) to be maintained?",
|
||||
"answers": {
|
||||
"A": "The order q of the base point P should be at least q >= 1024",
|
||||
"B": "Randomly choose two primes p and q such that p*q > 2048 bits",
|
||||
"C": "The length of the prime number p should be at least 3000 bits",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which document provides the recommendation for elliptic curve cryptography?",
|
||||
"answers": {
|
||||
"A": "NIST SP 800-56C",
|
||||
"B": "NIST SP 800-63-3",
|
||||
"C": "NIST SP 800-57 Part 1",
|
||||
"D": "NIST SP 800-186-4"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is an essential requirement for the internal state of deterministic random number generators?",
|
||||
"answers": {
|
||||
"A": "Protection against readout and manipulation",
|
||||
"B": "Constant reseeding requirement",
|
||||
"C": "Dependence on reliable physical resources",
|
||||
"D": "Regular update of entropy sources"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which federal government agency has responsibility for ensuring the security of government computer systems that are not used to process sensitive and/or classified information?",
|
||||
"answers": {
|
||||
"A": "National Security Agency",
|
||||
"B": "Federal Bureau of Investigation",
|
||||
"C": "National Institute of Standards and Technology",
|
||||
"D": "Secret Service"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of implementing monitoring systems?",
|
||||
"answers": {
|
||||
"A": "Monitoring the configuration and configuration change of devices",
|
||||
"B": "Ensuring availability and functionality of systems",
|
||||
"C": "Recording access to information by means of system and security logging",
|
||||
"D": "Protection from compromising radiation"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which principle of cybersecurity focuses on ensuring that data is accurate and trustworthy?",
|
||||
"answers": {
|
||||
"A": "Resilience",
|
||||
"B": "Availability",
|
||||
"C": "Authenticity",
|
||||
"D": "Integrity"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of the Transport Layer Security (TLS) protocol?",
|
||||
"answers": {
|
||||
"A": "Privacy and authentication between two communicating applications",
|
||||
"B": "Privacy and data integrity between two communicating applications",
|
||||
"C": "Authentication and data integrity between two communicating applications",
|
||||
"D": "Privacy, authentication, and data integrity between two communicating applications"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "If you were to see the subnet mask 255.255.248.0, what CIDR notation (prefix) would you use to indicate the same thing?",
|
||||
"answers": {
|
||||
"A": "/23",
|
||||
"B": "/22",
|
||||
"C": "/21",
|
||||
"D": "/20"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is NOT one of the five disaster recovery plan testing types?",
|
||||
"answers": {
|
||||
"A": "Simulation",
|
||||
"B": "Checklist",
|
||||
"C": "Availability",
|
||||
"D": "Full Interruption"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What would be the result of sending the string AAAAAAAAAAAAAAAAA into a variable that has been allocated space for 8 bytes?",
|
||||
"answers": {
|
||||
"A": "Heap spraying",
|
||||
"B": "SQL injection",
|
||||
"C": "Buffer overflow",
|
||||
"D": "Slowloris attack"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is NOT a form of computer/network surveillance?",
|
||||
"answers": {
|
||||
"A": "Keyboard monitoring",
|
||||
"B": "Use of network sniffers",
|
||||
"C": "Use of Closed-Circuit Television (CCTV)",
|
||||
"D": "Review of audit logs"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "How would you calculate risk?",
|
||||
"answers": {
|
||||
"A": "Probability / loss",
|
||||
"B": "Probability * mitigation factor",
|
||||
"C": "(Loss + mitigation factor) * (loss/probability)",
|
||||
"D": "Probability * loss"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the common share name used for interprocess communication?",
|
||||
"answers": {
|
||||
"A": "C$",
|
||||
"B": "IPC$",
|
||||
"C": "ADMIN$",
|
||||
"D": "INTERCOM$"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which port is commonly used by SMB when operating over TCP without NetBIOS?",
|
||||
"answers": {
|
||||
"A": "137",
|
||||
"B": "139",
|
||||
"C": "135",
|
||||
"D": "445"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is an example of a strong password?",
|
||||
"answers": {
|
||||
"A": "Password123456",
|
||||
"B": "ME_Fgh4_al445!A",
|
||||
"C": "Qwerty!",
|
||||
"D": "123456"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which access control model would be found in a firewall?",
|
||||
"answers": {
|
||||
"A": "Mandatory access control",
|
||||
"B": "Discretionary access control",
|
||||
"C": "Role-based access control",
|
||||
"D": "Rule-based access control"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the preferred encryption option for sending large amounts of data?",
|
||||
"answers": {
|
||||
"A": "Symmetric encryption",
|
||||
"B": "Asymmetric encryption",
|
||||
"C": "Public Key Cryptography",
|
||||
"D": "Diffie-Hellman scheme"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is an example of a hashing algorithm?",
|
||||
"answers": {
|
||||
"A": "AES",
|
||||
"B": "RSA",
|
||||
"C": "MD5",
|
||||
"D": "3DES"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which RAID level is not fault tolerant?",
|
||||
"answers": {
|
||||
"A": "RAID 1",
|
||||
"B": "RAID 0",
|
||||
"C": "RAID 5",
|
||||
"D": "RAID 6"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "How many keys exist in a public/private key pair?",
|
||||
"answers": {
|
||||
"A": "0",
|
||||
"B": "2",
|
||||
"C": "1",
|
||||
"D": "4"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of an Information Security Governance strategy?",
|
||||
"answers": {
|
||||
"A": "To minimize the impact of security incidents on the organization",
|
||||
"B": "To obtain senior management commitment and support",
|
||||
"C": "To enforce compliance with information security policies",
|
||||
"D": "To prioritize options to mitigate risks"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the process of hiding text within an image called?",
|
||||
"answers": {
|
||||
"A": "Encryption",
|
||||
"B": "Steganography",
|
||||
"C": "Spyware",
|
||||
"D": "Hiding"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What attack involves inserting an unauthorized person in the communication path between two users to intercept and modify packets?",
|
||||
"answers": {
|
||||
"A": "Known Plaintext Attack",
|
||||
"B": "Chosen Cipher-Text Attack",
|
||||
"C": "MITM Attack",
|
||||
"D": "Brute-Force Attack"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
|
||||
"question": "Who is responsible for implementing the planning, budgeting, and performance of the organization's information security components?",
|
||||
"answers": {
|
||||
"A": "Administrator",
|
||||
"B": "CISO",
|
||||
"C": "MGR",
|
||||
"D": "CEO"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "The primary responsibility of the information security steering committee is:",
|
||||
"answers": {
|
||||
"A": "Information security policy development",
|
||||
"B": "Direction setting and performance monitoring",
|
||||
"C": "Information security control implementation",
|
||||
"D": "Provision of information security training for employees"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "According to NIST guidelines, what is a recommended practice for creating a strong password?",
|
||||
"answers": {
|
||||
"A": "Implementing complex composition rules requiring a mix of character types",
|
||||
"B": "Setting mandatory periodic password changes for all users",
|
||||
"C": "Encouraging the use of longer, more memorable passphrases",
|
||||
"D": "Requiring the use of password hints for easier recall"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of continuously improving cybersecurity guidelines and their application notes?",
|
||||
"answers": {
|
||||
"A": "To categorize security measures into groups",
|
||||
"B": "To react to new technologies, use cases, and risks",
|
||||
"C": "To prevent all identified threats",
|
||||
"D": "To eliminate all cybersecurity risks"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What does DDoS stand for?",
|
||||
"answers": {
|
||||
"A": "Double Denial of Service",
|
||||
"B": "Distributed Denial of Service",
|
||||
"C": "Dual Denial of Services",
|
||||
"D": "Denial of Distributed Services"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which is not a common wireless network authentication mechanism?",
|
||||
"answers": {
|
||||
"A": "WEP",
|
||||
"B": "WPA",
|
||||
"C": "802.1X",
|
||||
"D": "RC4"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which technique aims to identify changes in normal routines by examining patterns of a user's activity?",
|
||||
"answers": {
|
||||
"A": "Vulnerability Assessment",
|
||||
"B": "Behavior-based IDS",
|
||||
"C": "Data Mining",
|
||||
"D": "User Profile Analysis"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of conducting a risk analysis in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To assess the risk level for the organization",
|
||||
"B": "To formulate the most important IT security procedures for the organization",
|
||||
"C": "To create a risk assessment statement",
|
||||
"D": "To determine the level of exposure to identified threats and identify possible safeguards or controls"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary advantage of using UDP over TCP?",
|
||||
"answers": {
|
||||
"A": "Reliability",
|
||||
"B": "Error recovery",
|
||||
"C": "Speed of data transfer",
|
||||
"D": "Data integrity"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What does the CIA triad in cryptography emphasize?",
|
||||
"answers": {
|
||||
"A": "Confidentiality, Integrity, Authenticity",
|
||||
"B": "Confidentiality, Integrity, Availability",
|
||||
"C": "Confidentiality, Authenticity, Nonrepudiation",
|
||||
"D": "Confidentiality, Availability, Nonrepudiation"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which nmap command is used to perform a TCP SYN scan, also known as a half-open scan, to determine which ports are open on a target system?",
|
||||
"answers": {
|
||||
"A": "nmap -sP",
|
||||
"B": "nmap -sT",
|
||||
"C": "nmap -SYN",
|
||||
"D": "nmap -sS"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of business continuity plans?",
|
||||
"answers": {
|
||||
"A": "To recover from natural disasters",
|
||||
"B": "To conduct regular audits of the organization's security systems",
|
||||
"C": "To minimize the effects of a disruptive event on a company",
|
||||
"D": "To increase the cost associated with a disruptive event"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which DoS attack sends large amounts of ICMP Echo traffic to a broadcast IP address with a spoofed source address of a victim?",
|
||||
"answers": {
|
||||
"A": "Smurf attack",
|
||||
"B": "SYN flood attack",
|
||||
"C": "Ping of Death",
|
||||
"D": "Botnet attack"
|
||||
},
|
||||
"solution": "A"
|
||||
}
|
||||
,
|
||||
{
|
||||
"question": "Which of the following tools is commonly used for network packet analysis and troubleshooting?",
|
||||
"answers": {
|
||||
"A": "Wireshark",
|
||||
"B": "Metasploit",
|
||||
"C": "Nmap",
|
||||
"D": "Hydra"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "In the context of encrypting data for secure communication, which encryption algorithm is typically preferred for its speed and efficiency?",
|
||||
"answers": {
|
||||
"A": "RSA-2048",
|
||||
"B": "AES-256",
|
||||
"C": "Both of them",
|
||||
"D": "None of them"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of NOP sleds in buffer overflow exploits?",
|
||||
"answers": {
|
||||
"A": "To encrypt the payload",
|
||||
"B": "To alter the content of the NOP register",
|
||||
"C": "To insert no-operation instructions for padding",
|
||||
"D": "To alter the content of the EIP register"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which security feature can help prevent buffer overflow attacks by marking certain memory regions as non-executable?",
|
||||
"answers": {
|
||||
"A": "Data Execution Prevention (DEP)",
|
||||
"B": "Buffer Overflow Prevention (BOP)",
|
||||
"C": "Intrusion Detection System (IDS)",
|
||||
"D": "Address Space Layout Randomization (ASLR)"
|
||||
},
|
||||
"solution": "A"
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
]
|
||||
}
|
||||
|
|
@ -1,397 +0,0 @@
|
|||
"""
|
||||
CyberMetric Evaluator for LLMs
|
||||
|
||||
This script evaluates the performance of language models on the CyberMetric benchmark.
|
||||
It supports OpenRouter-hosted models, local Ollama models via LiteLLM proxy, and OpenAI models.
|
||||
|
||||
Usage:
|
||||
python CyberMetric_evaluator.py --model_name MODEL_NAME [--file_path FILE_PATH] [--api_key API_KEY]
|
||||
|
||||
Arguments:
|
||||
--model_name: Required. Model name with prefix (openrouter/, ollama/, or openai/)
|
||||
Examples: openrouter/anthropic/claude-3-opus, ollama/llama3, openai/gpt-4o
|
||||
--file_path: Optional. Path to the CyberMetric JSON file (default: CyberMetric-2-v1.json)
|
||||
--api_key: Optional. API key for OpenRouter or OpenAI (can also use env vars)
|
||||
|
||||
Environment Variables:
|
||||
OPENROUTER_API_KEY: API key for OpenRouter (if using OpenRouter models)
|
||||
OPENROUTER_API_BASE: Base URL for OpenRouter API (default: https://openrouter.ai/api/v1)
|
||||
OLLAMA_API_BASE: Base URL for Ollama API via LiteLLM proxy (default: http://localhost:8000/v1)
|
||||
OPENAI_API_KEY: API key for OpenAI (if using OpenAI models)
|
||||
OPENAI_API_BASE: Base URL for OpenAI API (default: https://api.openai.com/v1)
|
||||
|
||||
Examples:
|
||||
# Run with an OpenRouter model
|
||||
python CyberMetric_evaluator.py --model_name openrouter/qwen/qwen3-32b:free
|
||||
|
||||
# Run with a local Ollama model (requires LiteLLM proxy running)
|
||||
python CyberMetric_evaluator.py --model_name ollama/qwen2.5:14b
|
||||
|
||||
# Run with an OpenAI model
|
||||
python CyberMetric_evaluator.py --model_name openai/gpt-4o
|
||||
|
||||
# Specify a different benchmark file
|
||||
python CyberMetric_evaluator.py --model_name openai/gpt-4o --file_path CyberMetric-10000-v1.json
|
||||
|
||||
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
import os
|
||||
import datetime
|
||||
import random
|
||||
import string
|
||||
import argparse
|
||||
from tqdm import tqdm
|
||||
import litellm
|
||||
import requests
|
||||
|
||||
# Enable debug mode for litellm
|
||||
#litellm._turn_on_debug()
|
||||
|
||||
# Default API bases
|
||||
OPENROUTER_API_BASE = "https://openrouter.ai/api/v1"
|
||||
OLLAMA_LITELLM_API_BASE = os.environ.get("OLLAMA_API_BASE", "http://localhost:8000/v1")
|
||||
OPENAI_API_BASE = os.environ.get("OPENAI_API_BASE", "https://api.openai.com/v1")
|
||||
|
||||
class CyberMetricEvaluator:
|
||||
def __init__(self, model_name, file_path, api_key=None, openrouter_api_base=None, ollama_litellm_api_base=None, openai_api_base=None):
|
||||
self.model_name = model_name
|
||||
self.file_path = file_path
|
||||
self.failed_questions = []
|
||||
self.failed_count = 0
|
||||
|
||||
# Set API configurations
|
||||
self.openrouter_api_base = openrouter_api_base or os.environ.get("OPENROUTER_API_BASE", OPENROUTER_API_BASE)
|
||||
self.ollama_litellm_api_base = ollama_litellm_api_base or os.environ.get("OLLAMA_LITELLM_API_BASE", OLLAMA_LITELLM_API_BASE)
|
||||
self.openai_api_base = openai_api_base or os.environ.get("OPENAI_API_BASE", OPENAI_API_BASE)
|
||||
|
||||
# Set API key based on model type
|
||||
self.api_key = None
|
||||
if self.model_name.startswith("openrouter/"):
|
||||
self.api_key = api_key or os.environ.get("OPENROUTER_API_KEY")
|
||||
elif self.model_name.startswith("openai/"):
|
||||
self.api_key = api_key or os.environ.get("OPENAI_API_KEY")
|
||||
|
||||
self.start_time = datetime.datetime.now()
|
||||
|
||||
# Create output directory structure
|
||||
self.output_dir = self.create_output_directory()
|
||||
self.info_file = os.path.join(self.output_dir, "information.txt")
|
||||
self.report_file = os.path.join(self.output_dir, "report_failed_questions.json")
|
||||
|
||||
# Initialize info file
|
||||
self.initialize_info_file()
|
||||
|
||||
print("--DEBUG: model_name: ", self.model_name)
|
||||
|
||||
def create_output_directory(self):
|
||||
# Create base directory if it doesn't exist
|
||||
base_dir = "outputs"
|
||||
if not os.path.exists(base_dir):
|
||||
os.makedirs(base_dir)
|
||||
|
||||
# Format model name for directory (replace / with -)
|
||||
model_dir_name = self.model_name.replace("/", "-")
|
||||
|
||||
# Get current date
|
||||
current_date = datetime.datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
# Create directory name
|
||||
dir_name = f"{model_dir_name}-{current_date}"
|
||||
|
||||
# If directory already exists, add random string
|
||||
full_path = os.path.join(base_dir, dir_name)
|
||||
if os.path.exists(full_path):
|
||||
random_str = ''.join(random.choices(string.ascii_lowercase + string.digits, k=5))
|
||||
dir_name = f"{model_dir_name}-{current_date}-{random_str}"
|
||||
full_path = os.path.join(base_dir, dir_name)
|
||||
|
||||
os.makedirs(full_path)
|
||||
return full_path
|
||||
|
||||
def initialize_info_file(self):
|
||||
with open(self.info_file, 'w') as file:
|
||||
file.write(f"CyberMetric Evaluation\n")
|
||||
file.write(f"=====================\n\n")
|
||||
file.write(f"Model: {self.model_name}\n")
|
||||
file.write(f"Dataset: {self.file_path}\n")
|
||||
file.write(f"Start Time: {self.start_time.strftime('%Y-%m-%d %H:%M:%S')}\n")
|
||||
file.write(f"Status: Running\n")
|
||||
file.write(f"Questions Processed: 0\n")
|
||||
file.write(f"Correct Answers: 0\n")
|
||||
file.write(f"Accuracy: 0.00%\n")
|
||||
|
||||
def update_info_file(self, questions_processed, correct_count, status="Running"):
|
||||
accuracy = correct_count / questions_processed * 100 if questions_processed > 0 else 0
|
||||
|
||||
with open(self.info_file, 'w') as file:
|
||||
file.write(f"CyberMetric Evaluation\n")
|
||||
file.write(f"=====================\n\n")
|
||||
file.write(f"Model: {self.model_name}\n")
|
||||
file.write(f"Dataset: {self.file_path}\n")
|
||||
file.write(f"Start Time: {self.start_time.strftime('%Y-%m-%d %H:%M:%S')}\n")
|
||||
file.write(f"Status: {status}\n")
|
||||
file.write(f"Questions Processed: {questions_processed}\n")
|
||||
file.write(f"Correct Answers: {correct_count}\n")
|
||||
file.write(f"Accuracy: {accuracy:.2f}%\n")
|
||||
|
||||
if status == "Completed":
|
||||
end_time = datetime.datetime.now()
|
||||
duration = end_time - self.start_time
|
||||
file.write(f"End Time: {end_time.strftime('%Y-%m-%d %H:%M:%S')}\n")
|
||||
file.write(f"Duration: {duration}\n")
|
||||
|
||||
def read_json_file(self):
|
||||
with open(self.file_path, 'r') as file:
|
||||
return json.load(file)
|
||||
|
||||
def save_failed_questions(self):
|
||||
report_data = {"questions": self.failed_questions}
|
||||
|
||||
with open(self.report_file, 'w') as file:
|
||||
json.dump(report_data, file, indent=4)
|
||||
|
||||
print(f"Updated failed questions report in {self.report_file}")
|
||||
|
||||
@staticmethod
|
||||
def extract_answer(response):
|
||||
if response.strip():
|
||||
match = re.search(r"ANSWER:?\s*([A-D])", response, re.IGNORECASE)
|
||||
if match:
|
||||
return match.group(1).upper()
|
||||
return None
|
||||
|
||||
def ask_ollama_litellm(self, prompt, max_retries=5):
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
response = litellm.completion(
|
||||
model=self.model_name,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a security expert who answers questions."},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
api_base="http://localhost:8000"
|
||||
)
|
||||
if hasattr(response, "choices") and response.choices:
|
||||
content = response.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
print("--DEBUG: result: ", result)
|
||||
return result
|
||||
else:
|
||||
print("Incorrect answer format detected. Attempting the question again.")
|
||||
except Exception as e:
|
||||
print(f"Error: {e}. Attempting the question again in {2 ** attempt} seconds.")
|
||||
time.sleep(2 ** attempt)
|
||||
return None
|
||||
|
||||
def ask_openrouter(self, prompt, max_retries=5):
|
||||
if not self.api_key:
|
||||
raise ValueError("API key is required for OpenRouter models")
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
response = litellm.completion(
|
||||
model=self.model_name,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a security expert who answers questions."},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
api_base=self.openrouter_api_base,
|
||||
api_key=self.api_key,
|
||||
headers={
|
||||
"HTTP-Referer": "https://your-site-url.com",
|
||||
"X-Title": "CyberMetric Evaluator"
|
||||
}
|
||||
)
|
||||
if hasattr(response, "choices") and response.choices:
|
||||
content = response.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
print("--DEBUG: result: ", result)
|
||||
return result
|
||||
else:
|
||||
print("Incorrect answer format detected. Attempting the question again.")
|
||||
except Exception as e:
|
||||
print(f"Error: {e}. Attempting the question again in {2 ** attempt} seconds.")
|
||||
time.sleep(2 ** attempt)
|
||||
return None
|
||||
|
||||
def ask_openai(self, prompt, max_retries=5):
|
||||
if not self.api_key:
|
||||
raise ValueError("API key is required for OpenAI models")
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
# Extract the actual model name without the 'openai/' prefix
|
||||
actual_model = self.model_name.replace("openai/", "")
|
||||
|
||||
response = litellm.completion(
|
||||
model=actual_model, # Use the actual OpenAI model name
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a security expert who answers questions."},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
api_base=self.openai_api_base,
|
||||
api_key=self.api_key
|
||||
)
|
||||
if hasattr(response, "choices") and response.choices:
|
||||
content = response.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
print("--DEBUG: result: ", result)
|
||||
return result
|
||||
else:
|
||||
print("Incorrect answer format detected. Attempting the question again.")
|
||||
except Exception as e:
|
||||
print(f"Error: {e}. Attempting the question again in {2 ** attempt} seconds.")
|
||||
time.sleep(2 ** attempt)
|
||||
return None
|
||||
|
||||
def ask_llm(self, question, answers, max_retries=5):
|
||||
options = ', '.join([f"{key}) {value}" for key, value in answers.items()])
|
||||
prompt = (
|
||||
f"Question: {question}\n"
|
||||
f"Options: {options}\n\n"
|
||||
"Choose the correct answer (A, B, C, or D) only. Always return in this format: 'ANSWER: X'"
|
||||
)
|
||||
print("--DEBUG: prompt: ", prompt)
|
||||
|
||||
if self.model_name.startswith("openrouter/"):
|
||||
return self.ask_openrouter(prompt, max_retries)
|
||||
elif self.model_name.startswith("ollama/"):
|
||||
return self.ask_ollama_litellm(prompt, max_retries)
|
||||
elif self.model_name.startswith("openai/"):
|
||||
return self.ask_openai(prompt, max_retries)
|
||||
else:
|
||||
print(f"Error: Unsupported model prefix: {self.model_name}")
|
||||
return None
|
||||
|
||||
def run_evaluation(self):
|
||||
if not (self.model_name.startswith("openrouter/") or
|
||||
self.model_name.startswith("ollama/") or
|
||||
self.model_name.startswith("openai/")):
|
||||
print("Error: Model name must start with 'ollama/', 'openrouter/', or 'openai/'")
|
||||
return
|
||||
|
||||
json_data = self.read_json_file()
|
||||
questions_data = json_data['questions']
|
||||
|
||||
correct_count = 0
|
||||
incorrect_answers = []
|
||||
|
||||
with tqdm(total=len(questions_data), desc="Processing Questions") as progress_bar:
|
||||
for i, item in enumerate(questions_data):
|
||||
question = item['question']
|
||||
answers = item['answers']
|
||||
correct_answer = item['solution']
|
||||
|
||||
llm_answer = self.ask_llm(question, answers)
|
||||
if llm_answer == correct_answer:
|
||||
correct_count += 1
|
||||
else:
|
||||
self.failed_questions.append({
|
||||
'question': question,
|
||||
'answers': answers,
|
||||
'solution': correct_answer,
|
||||
'llm_answer': llm_answer
|
||||
})
|
||||
self.failed_count += 1
|
||||
|
||||
if self.failed_count % 2 == 0:
|
||||
self.save_failed_questions()
|
||||
|
||||
incorrect_answers.append({
|
||||
'question': question,
|
||||
'correct_answer': correct_answer,
|
||||
'llm_answer': llm_answer
|
||||
})
|
||||
|
||||
# Update progress and information file
|
||||
questions_processed = i + 1
|
||||
accuracy_rate = correct_count / questions_processed * 100
|
||||
progress_bar.set_postfix_str(f"Accuracy: {accuracy_rate:.2f}%")
|
||||
progress_bar.update(1)
|
||||
|
||||
# Update info file every 5 questions
|
||||
if questions_processed % 5 == 0 or questions_processed == len(questions_data):
|
||||
self.update_info_file(questions_processed, correct_count)
|
||||
|
||||
# Final update with completed status
|
||||
self.update_info_file(len(questions_data), correct_count, "Completed")
|
||||
print(f"\nFinal Accuracy: {correct_count / len(questions_data) * 100:.2f}%")
|
||||
|
||||
if self.failed_questions:
|
||||
self.save_failed_questions() # final failed questions
|
||||
|
||||
if incorrect_answers:
|
||||
print("\nIncorrect Answers:")
|
||||
for item in incorrect_answers:
|
||||
print(f"Question: {item['question']}")
|
||||
print(f"Expected Answer: {item['correct_answer']}, LLM Answer: {item['llm_answer']}\n")
|
||||
|
||||
if __name__ == "__main__":
|
||||
#litellm._turn_on_debug()
|
||||
|
||||
# Create argument parser
|
||||
parser = argparse.ArgumentParser(description='CyberMetric Evaluator for LLMs')
|
||||
parser.add_argument('--model_name', type=str, required=True,
|
||||
help='Model name with prefix (openrouter/, ollama/, or openai/)')
|
||||
parser.add_argument('--file_path', type=str, default='CyberMetric-2-v1.json',
|
||||
help='Path to the CyberMetric JSON file')
|
||||
parser.add_argument('--api_key', type=str,
|
||||
help='API key for OpenRouter or OpenAI (can also use env vars)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
model_name = args.model_name
|
||||
file_path = args.file_path
|
||||
api_key = args.api_key
|
||||
|
||||
if model_name.startswith("ollama/"):
|
||||
# Ollama configuration
|
||||
evaluator = CyberMetricEvaluator(
|
||||
model_name=model_name,
|
||||
file_path=file_path
|
||||
)
|
||||
print(f"Using Ollama configuration with LiteLLM proxy on port 8000")
|
||||
|
||||
elif model_name.startswith("openrouter/"):
|
||||
# OpenRouter configuration
|
||||
api_key = api_key or os.environ.get("OPENROUTER_API_KEY")
|
||||
if not api_key:
|
||||
raise ValueError("API key must be provided via --api_key or OPENROUTER_API_KEY environment variable for OpenRouter models")
|
||||
|
||||
evaluator = CyberMetricEvaluator(
|
||||
model_name=model_name,
|
||||
file_path=file_path,
|
||||
api_key=api_key,
|
||||
openrouter_api_base=os.environ.get("OPENROUTER_API_BASE", "https://openrouter.ai/api/v1")
|
||||
)
|
||||
print("Using OpenRouter configuration")
|
||||
|
||||
elif model_name.startswith("openai/"):
|
||||
# OpenAI configuration
|
||||
api_key = api_key or os.environ.get("OPENAI_API_KEY")
|
||||
if not api_key:
|
||||
raise ValueError("API key must be provided via --api_key or OPENAI_API_KEY environment variable for OpenAI models")
|
||||
|
||||
print(f"API key provided: {api_key[:4]}...{api_key[-4:] if len(api_key) > 8 else ''}")
|
||||
|
||||
evaluator = CyberMetricEvaluator(
|
||||
model_name=model_name,
|
||||
file_path=file_path,
|
||||
api_key=api_key,
|
||||
openai_api_base=os.environ.get("OPENAI_API_BASE", "https://api.openai.com/v1")
|
||||
)
|
||||
print("Using OpenAI configuration")
|
||||
|
||||
else:
|
||||
raise ValueError("Model name must start with 'ollama/', 'openrouter/', or 'openai/'")
|
||||
|
||||
# Run the evaluation
|
||||
evaluator.run_evaluation()
|
||||
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
# CyberMetric Dataset
|
||||
|
||||
<div align="center">
|
||||
<img width="800" alt="logo" src="https://github.com/cybermetric/CyberMetric/assets/159767263/455e5a31-97da-4179-ad49-fa182fe7d9ad">
|
||||
</div>
|
||||
|
||||
|
||||
# Description
|
||||
|
||||
|
||||
|
||||
The **CyberMetric Dataset** introduces a new benchmarking tool consisting of 10,000 questions designed to evaluate the cybersecurity knowledge of various Large Language Models (LLMs) within the cybersecurity domain. This dataset is created using different LLMs and has been verified by human experts in the cybersecurity field to ensure its relevance and accuracy. The dataset is compiled from various sources including standards, certifications, research papers, books, and other publications within the cybersecurity field. We provide the dataset in four distinct sizes —small, medium, big and large— comprising 80, 500, 2000 and 10,000 questions, respectively.The smallest version is tailored for comparisons between different LLMs and humans. The CyberMetric-80 dataset has been subject to testing with 30 human participants, enabling an effective comparison between human and machine intelligence.
|
||||
|
||||
# Cite
|
||||
|
||||
The CyberMetric paper **"CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge"** has been accepted for publication in the 2024 IEEE International Conference on Cyber Security and Resilience (IEEE CSR 2024).
|
||||
|
||||
IEEE Xplore link: https://ieeexplore.ieee.org/document/10679494
|
||||
|
||||
Cite the paper:
|
||||
```python
|
||||
@INPROCEEDINGS{10679494,
|
||||
author={Tihanyi, Norbert and Ferrag, Mohamed Amine and Jain, Ridhi and Bisztray, Tamas and Debbah, Merouane},
|
||||
booktitle={2024 IEEE International Conference on Cyber Security and Resilience (CSR)},
|
||||
title={CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge},
|
||||
year={2024},
|
||||
volume={},
|
||||
number={},
|
||||
pages={296-302},
|
||||
keywords={Accuracy;Reverse engineering;Benchmark testing;NIST Standards;Risk management;Problem-solving;Computer security},
|
||||
doi={10.1109/CSR61664.2024.10679494}}
|
||||
```
|
||||
|
||||
# Architecture
|
||||
|
||||
The CyberMetric dataset was created by applying different language models using Retrieval-Augmented Generation (RAG), with human validation included in the process. The AI-driven generation framework is illustrated in the following figure.
|
||||
<img width="1102" alt="Framework" src="https://github.com/cybermetric/CyberMetric/assets/159767263/6d3c8d55-289d-4922-8302-4f3e379a9805">
|
||||
|
||||
# Prompt
|
||||
Most of the evaluated models were instruction-fine-tuned, and the following prompts were used to obtain the correct answers in XML format.
|
||||
```python
|
||||
def make_messages(question, answers):
|
||||
"""
|
||||
Formats a single question+answers into a list of message dictionaries for the pipeline.
|
||||
"""
|
||||
options_str = ', '.join([f"{key}) {value}" for key, value in answers.items()])
|
||||
instructions = (
|
||||
"You are a helpful AI assistant.\n"
|
||||
"Instructions:\n"
|
||||
"a. Carefully read the question.\n"
|
||||
"b. Choose the correct answer (A, B, C, or D) only.\n"
|
||||
"c. Do NOT include any explanation or additional text in the response.\n"
|
||||
"d. Always return the answer in this XML format: '<xml>answer</xml>'. "
|
||||
"For example, if the correct answer is D, then return <xml>D</xml>.\n\n"
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": instructions},
|
||||
{"role": "user", "content": f"#Question: {question}\nOptions: {options_str}"}
|
||||
]
|
||||
return messages
|
||||
|
||||
```
|
||||
# LLM Leaderboard on CyberMetric Dataset
|
||||
|
||||
We have assessed and compared state-of-the-art LLM models using the CyberMetric dataset. The most recent evaluation was conducted on December 27th, 2024.
|
||||
|
||||
|
||||
|
||||
|
||||
<div align="center">
|
||||
<img width="1065" alt="Cybermetric_result" src="https://github.com/user-attachments/assets/dc3a0a0f-59d1-464e-bf67-a37d0008bdaf" />
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
# Usage
|
||||
|
||||
We have developed a compact Python script called `CyberMetric_evaluator.py` to showcase how to utilize the Dataset with OpenAI GPT. Simply insert your API key in the script by setting `API_KEY="<YOUR-API-KEY-HERE>"`, and then execute the evaluator program.
|
||||
|
||||
|
||||
Here's an example output generated by the script using the CyberMetric-80 dataset:
|
||||
|
||||
|
||||
Loading…
Reference in New Issue