mirror of https://github.com/aliasrobotics/cai.git
add datasets
This commit is contained in:
Mery-Sanz
parent
fa85b79087
commit
433bd9bf65
|
|
@ -1 +0,0 @@
|
|||
.DS_Store
|
||||
|
|
@ -1,150 +0,0 @@
|
|||
# SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models
|
||||
|
||||
[中文版](README_CN.md).
|
||||
|
||||
The advent of large language models has ignited a transformative era for the cybersecurity industry. Pioneering applications are being developed, deployed, and utilized in areas such as cybersecurity knowledge QA, vulnerability hunting, and alert investigation. Various researches have indicated that LLMs primarily acquire their knowledge during the pretraining phase, with fine-tuning serving essentially to align the model with user intentions, providing the ability to follow instructions. This suggests that the knowledge and skills embedded in the foundational model significantly influence the model's potential on specific downstream tas ks
|
||||
|
||||
Yet, a focused evaluation of cybersecurity knowledge is missing in existing datasets. We address this by introducing "SecEval". SecEval is the first benchmark specifically created for evaluating cybersecurity knowledge in Foundation Models. It offers over 2000 multiple-choice questions across 9 domains: Software Security, Application Security, System Security, Web Security, Cryptography, Memory Safety, Network Security, and PenTest.
|
||||
SecEval generates questions by prompting OpenAI GPT4 with authoritative sources such as open-licensed textbooks, official documentation, and industry guidelines and standards. The generation process is meticulously crafted to ensure the dataset meets rigorous quality, diversity, and impartiality criteria. You can explore our dataset the [explore page](https://xuanwuai.github.io/SecEval/explore.html).
|
||||
|
||||
Using SecEval, we conduct an evaluation of 10 state-of-the-art foundational models, providing new insights into their performance in the field of cybersecurity. The results indicate that there is still a long way to go before LLMs can be the master of cybersecurity. We hope that SecEval can serve as a catalyst for future research in this area.
|
||||
|
||||
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Leaderboard](#leaderboard)
|
||||
- [Dataset](#dataset)
|
||||
- [Generation Process](#generation-process)
|
||||
- [Limitations](#limitations)
|
||||
- [Future Work](#future-work)
|
||||
- [Licenses](#licenses)
|
||||
- [Citation](#citation)
|
||||
- [Credits](#credits)
|
||||
|
||||
|
||||
|
||||
## Leaderboard
|
||||
|
||||
| # | Model | Creator | Access | Submission Date | System Security | Application Security | PenTest | Memory Safety | Network Security | Web Security | Vulnerability | Software Security | Cryptography | Overall |
|
||||
|-----|-------------------|-----------|-----------|-----------------|-----------------|----------------------|---------|---------------|------------------|--------------|---------------|-------------------|--------------|---------|
|
||||
| 1 | gpt-4-turbo | OpenAI | API, Web | 2023-12-20 | 73.61 | 75.25 | 80.00 | 70.83 | 75.65 | 82.15 | 76.05 | 73.28 | 64.29 | 79.07 |
|
||||
| 2 | gpt-3.5-turbo | OpenAI | API, Web | 2023-12-20 | 59.15 | 57.18 | 72.00 | 43.75 | 60.87 | 63.00 | 60.18 | 58.19 | 35.71 | 62.09 |
|
||||
| 3 | Yi-6B | 01-AI | Weight | 2023-12-20 | 50.61 | 48.89 | 69.26 | 35.42 | 56.52 | 54.98 | 49.40 | 45.69 | 35.71 | 53.57 |
|
||||
| 4 | Orca-2-7b | Microsoft | Weight | 2023-12-20 | 46.76 | 47.03 | 60.84 | 31.25 | 49.13 | 55.63 | 50.00 | 52.16 | 14.29 | 51.60 |
|
||||
| 5 | Mistral-7B-v0.1 | Mistralai | Weight | 2023-12-20 | 40.19 | 38.37 | 53.47 | 33.33 | 36.52 | 46.57 | 42.22 | 43.10 | 28.57 | 43.65 |
|
||||
| 6 | chatglm3-6b-base | THUDM | Weight | 2023-12-20 | 39.72 | 37.25 | 57.47 | 31.25 | 43.04 | 41.14 | 37.43 | 39.66 | 28.57 | 41.58 |
|
||||
| 7 | Aquila2-7B | BAAI | Weight | 2023-12-20 | 34.84 | 36.01 | 47.16 | 22.92 | 32.17 | 42.04 | 38.02 | 36.21 | 7.14 | 38.29 |
|
||||
| 8 | Qwen-7B | Alibaba | Weight | 2023-12-20 | 28.92 | 28.84 | 41.47 | 18.75 | 29.57 | 33.25 | 31.74 | 30.17 | 14.29 | 31.37 |
|
||||
| 9 | internlm-7b | Sensetime | Weight | 2023-12-20 | 25.92 | 25.87 | 36.21 | 25.00 | 27.83 | 32.86 | 29.34 | 34.05 | 7.14 | 30.29 |
|
||||
| 10 | Llama-2-7b-hf | MetaAI | Weight | 2023-12-20 | 20.94 | 18.69 | 26.11 | 16.67 | 14.35 | 22.77 | 21.56 | 20.26 | 21.43 | 22.15 |
|
||||
|
||||
## Dataset
|
||||
|
||||
### Format
|
||||
|
||||
The dataset is in json format. Each question has the following fields:
|
||||
|
||||
* id: str # unique id for each question
|
||||
* source: str # the source where the question is generated from
|
||||
* question: str # the question description
|
||||
* choices: List[str] # the choices for the question
|
||||
* answer: str # the answer for the question
|
||||
* topics: List[QuestionTopic] # the topics for the question, each question can have multiple topics.
|
||||
* keyword: str # the keyword for the question
|
||||
|
||||
|
||||
### Question Distribution
|
||||
|
||||
| Topic | No. of Questions |
|
||||
|---------------------|-----------------|
|
||||
| SystemSecurity | 1065 |
|
||||
| ApplicationSecurity | 808 |
|
||||
| PenTest | 475 |
|
||||
| MemorySafety | 48 |
|
||||
| NetworkSecurity | 230 |
|
||||
| WebSecurity | 773 |
|
||||
| Vulnerability | 334 |
|
||||
| SoftwareSecurity | 232 |
|
||||
| Cryptography | 14 |
|
||||
| Overall | 2126 |
|
||||
|
||||
|
||||
### Download
|
||||
You can download the json file of the dataset by running.
|
||||
|
||||
```
|
||||
wget https://huggingface.co/datasets/XuanwuAI/SecEval/blob/main/questions.json
|
||||
```
|
||||
|
||||
Or you can load the dataset from [Huggingface](https://huggingface.co/datasets/XuanwuAI/SecEval).
|
||||
|
||||
### Evaluate Your Model on SecEval
|
||||
|
||||
You can use our [evaluation script](https://github.com/XuanwuAI/SecEval/tree/main/eval) to evaluate your model on SecEval dataset.
|
||||
|
||||
|
||||
## Generation Process
|
||||
|
||||
### Data Collection
|
||||
|
||||
- **Textbook**: We selected open-licensed textbooks from the Computer Security courses CS161 at UC Berkeley and 6.858 at MIT. These resources provide extensive information on network security, memory safety, web security, and cryptography.
|
||||
|
||||
- **Official Documentation**: We utilized official documentation, such as Apple Platform Security, Android Security, and Windows Security, to integrate system security and application security knowledge specific to these platforms.
|
||||
|
||||
- **Industrial Guidelines**: To encompass web security, we referred to the Mozilla Web Security Guidelines. In addition, we used the OWASP Web Security Testing Guide (WSTG) and OWASP Mobile Application Security Testing Guide (MASTG) for insights into web and application security testing.
|
||||
|
||||
- **Industrial Standards**: The Common Weakness Enumeration (CWE) was employed to address knowledge of vulnerabilities. For penetration testing, we incorporated the MITRE ATT&CK and MITRE D3fend frameworks.
|
||||
|
||||
### Questions Generation
|
||||
|
||||
To facilitate the evaluation process, we designed the dataset in a multiple-choice question format. Our approach to question generation involved several steps:
|
||||
|
||||
1. **Text Parsing**: We began by parsing the texts according to their hierarchical structure, such as chapters and sections for textbooks, or tactics and techniques for frameworks like ATT&CK.
|
||||
|
||||
2. **Content Sampling**: For texts with extensive content, such as CWE or Windows Security Documentation, we employed a sampling strategy to maintain manageability. For example, we selected the top 25 most common weakness types and 175 random types from CWE.
|
||||
|
||||
3. **Question Generation**: Utilizing GPT-4, we generated multiple-choice questions based on the parsed text, with the level of detail adjusted according to the content's nature. For instance, questions stemming from the CS161 textbook were based on individual sections, while those from ATT&CK were based on techniques.
|
||||
|
||||
4. **Question Refinement**: We then prompted GPT-4 to identify and filter out questions with issues such as too simplistic or not self-contained. Where possible, questions were revised; otherwise, they were discarded.
|
||||
|
||||
5. **Answer Calibration**: We refine the selection of answer options by presenting GPT-4 with both the question and the source text from which the question is derived. Should the response generated by GPT-4 diverge from the previously established answer, this discrepancy suggests that obtaining a consistent answer for the question is inherently challenging. In such cases, we opt to eliminate these problematic questions.
|
||||
|
||||
6. **Classification**: Finally, we organized the questions into 9 topics, and attached a relevant fine-grained keyword to each question.
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
The dataset, while comprehensive, exhibits certain constraints:
|
||||
|
||||
1. **Distribution Imbalance**: The dataset presents an uneven distribution of questions across different domains, resulting in a higher concentration of questions in certain areas while others are less represented.
|
||||
|
||||
2. **Incomplete Scope**: Some topics on Cybersecurity are absent from the dataset, such as content security, reverse engineering, and malware analysis. As such, it does not encapsulate the full breadth of knowledge within the field.
|
||||
|
||||
## Future Work
|
||||
|
||||
1. **Improvement on Distribution**: We aim to broaden the dataset's comprehensiveness by incorporating additional questions, thereby enriching the coverage of existing cybersecurity topics.
|
||||
|
||||
2. **Improvement on Topic Coverage**: Efforts will be made to include a wider array of cybersecurity topics within the dataset, which will help achieve a more equitable distribution of questions across various fields.
|
||||
|
||||
|
||||
## Licenses
|
||||
|
||||
The dataset is released under the [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. The code is released under the [MIT](https://opensource.org/licenses/MIT) license.
|
||||
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{li2023seceval,
|
||||
title={SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models},
|
||||
author={Li, Guancheng and Li, Yifeng and Wang Guannan and Yang, Haoyu and Yu, Yang},
|
||||
publisher = {GitHub},
|
||||
howpublished= "https://github.com/XuanwuAI/SecEval",
|
||||
year={2023}
|
||||
}
|
||||
```
|
||||
|
||||
## Credits
|
||||
|
||||
This work is supported by [Tencent Security Xuanwu Lab](https://xlab.tencent.com/en/). we also apperiate Tencent Spark Talent Program for help.
|
||||
|
|
@ -1,144 +0,0 @@
|
|||
# SecEval：评估基础模型在网络安全知识上的全面基准测试
|
||||
|
||||
[English](README.md)
|
||||
|
||||
随着大型语言模型的出现，为网络安全行业带来了变革性的时代。在网络安全知识问答、漏洞挖掘和告警研判等领域，人们正在开发、部署和使用大模型应用。各种研究表明，LLMs主要在预训练阶段获取知识，而微调基本上是为了使模型与用户意图对齐，提供遵循指令的能力。这表明基础模型中的知识和技能对模型在特定下游任务上的潜力有着重要影响。
|
||||
|
||||
然而，在现有评估数据集中缺乏对网络安全知识的专门性评估。我们通过引入“SecEval”来解决这个问题。SecEval是首个专门为评估基础模型中的网络安全知识而创建的基准测试。它提供了超过2000个多项选择题，覆盖9个领域：软件安全、应用程序安全、系统安全、网络安全、密码学、内存安全、网络安全和渗透测试。
|
||||
|
||||
SecEval通过向OpenAI GPT4提供权威来源的数据来生成相应问题，如开放许可的教科书、官方文档和行业指导方针与标准，来生成问题。生成过程经过精心设计，以确保数据集满足严格的质量、多样性和公正性标准。您可以在[探索页面](https://xuanwuai.github.io/SecEval/explore.html)上探索我们的数据集。
|
||||
|
||||
使用SecEval，我们对10种最先进的基础模型进行了评估，为它们在网络安全领域的表现提供了新的见解。结果表明，在LLMs在网络安全领域还有很长的路要走。我们希望SecEval能够成为未来在这一领域研究的催化剂。
|
||||
## 目录
|
||||
|
||||
- [Leaderboard](#leaderboard)
|
||||
- [数据集](#数据集)
|
||||
- [数据集生成方法](#数据集生成方法)
|
||||
- [局限性](#局限性)
|
||||
- [未来工作](#未来工作)
|
||||
- [许可证](#许可证)
|
||||
- [引用](#引用)
|
||||
|
||||
## Leaderboard
|
||||
|
||||
| # | Model | Creator | Access | Submission Date | System Security | Application Security | PenTest | Memory Safety | Network Security | Web Security | Vulnerability | Software Security | Cryptography | Overall |
|
||||
|-----|-------------------|-----------|-----------|-----------------|-----------------|----------------------|---------|---------------|------------------|--------------|---------------|-------------------|--------------|---------|
|
||||
| 1 | gpt-4-turbo | OpenAI | API, Web | 2023-12-20 | 73.61 | 75.25 | 80.00 | 70.83 | 75.65 | 82.15 | 76.05 | 73.28 | 64.29 | 79.07 |
|
||||
| 2 | gpt-35-turbo | OpenAI | API, Web | 2023-12-20 | 59.15 | 57.18 | 72.00 | 43.75 | 60.87 | 63.00 | 60.18 | 58.19 | 35.71 | 62.09 |
|
||||
| 3 | Yi-6B | 01-AI | Weight | 2023-12-20 | 50.61 | 48.89 | 69.26 | 35.42 | 56.52 | 54.98 | 49.40 | 45.69 | 35.71 | 53.57 |
|
||||
| 4 | Orca-2-7b | Microsoft | Weight | 2023-12-20 | 46.76 | 47.03 | 60.84 | 31.25 | 49.13 | 55.63 | 50.00 | 52.16 | 14.29 | 51.60 |
|
||||
| 5 | Mistral-7B-v0.1 | Mistralai | Weight | 2023-12-20 | 40.19 | 38.37 | 53.47 | 33.33 | 36.52 | 46.57 | 42.22 | 43.10 | 28.57 | 43.65 |
|
||||
| 6 | chatglm3-6b-base | THUDM | Weight | 2023-12-20 | 39.72 | 37.25 | 57.47 | 31.25 | 43.04 | 41.14 | 37.43 | 39.66 | 28.57 | 41.58 |
|
||||
| 7 | Aquila2-7B | BAAI | Weight | 2023-12-20 | 34.84 | 36.01 | 47.16 | 22.92 | 32.17 | 42.04 | 38.02 | 36.21 | 7.14 | 38.29 |
|
||||
| 8 | Qwen-7B | Alibaba | Weight | 2023-12-20 | 28.92 | 28.84 | 41.47 | 18.75 | 29.57 | 33.25 | 31.74 | 30.17 | 14.29 | 31.37 |
|
||||
| 9 | internlm-7b | Sensetime | Weight | 2023-12-20 | 25.92 | 25.87 | 36.21 | 25.00 | 27.83 | 32.86 | 29.34 | 34.05 | 7.14 | 30.29 |
|
||||
| 10 | Llama-2-7b-hf | MetaAI | Weight | 2023-12-20 | 20.94 | 18.69 | 26.11 | 16.67 | 14.35 | 22.77 | 21.56 | 20.26 | 21.43 | 22.15 |
|
||||
|
||||
## 数据集
|
||||
|
||||
### 格式
|
||||
|
||||
数据集采用json格式。每个问题包含以下字段：
|
||||
|
||||
* id: str # 每个问题的唯一id
|
||||
* source: str # 问题来源
|
||||
* question: str # 问题描述
|
||||
* choices: List[str] # 问题的选项
|
||||
* answer: str # 问题的答案
|
||||
* topics: List[QuestionTopic] # 问题的主题，每个问题可以有多个主题。
|
||||
* keyword: str # 问题的关键词
|
||||
|
||||
|
||||
### 问题分布
|
||||
|
||||
| 主题 | 问题数量 |
|
||||
|----------------------|--------------|
|
||||
| 系统安全 | 1065 |
|
||||
| 应用安全 | 808 |
|
||||
| 渗透测试 | 475 |
|
||||
| 内存安全 | 48 |
|
||||
| 网络安全 | 230 |
|
||||
| 网络安全 | 773 |
|
||||
| 漏洞 | 334 |
|
||||
| 软件安全 | 232 |
|
||||
| 密码学 | 14 |
|
||||
| 总体 | 2126 |
|
||||
|
||||
|
||||
### 下载
|
||||
|
||||
您可以通过运行以下命令下载json文件的数据集。
|
||||
```
|
||||
wget https://huggingface.co/datasets/XuanwuAI/SecEval/blob/main/problems.json
|
||||
```
|
||||
|
||||
或者您可以从[Huggingface](https://huggingface.co/datasets/XuanwuAI/SecEval)加载数据集。
|
||||
|
||||
### 在SecEval上评估您的模型
|
||||
|
||||
您可以使用我们的[评估脚本](https://github.com/XuanwuAI/SecEval/tree/main/eval)来在SecEval数据集上评估您的模型。
|
||||
|
||||
## 数据集生成方法
|
||||
|
||||
### 数据收集
|
||||
|
||||
- **教科书**：我们从加州大学伯克利分校的计算机安全课程 CS161 和麻省理工学院的 6.858 选取了开放许可的教科书。这些资源提供了关于网络安全、内存安全、网页安全和密码学的广泛信息。
|
||||
|
||||
- **官方文档**：我们使用了如苹果平台安全、安卓安全和 Windows 安全等官方文档，以整合这些平台特有的系统安全和应用安全知识。
|
||||
|
||||
- **工业指南**：为了包含网络安全，我们参考了 Mozilla 网络安全指南。此外，我们还使用了 OWASP 网络安全测试指南（WSTG）和 OWASP 移动应用安全测试指南（MASTG），以获得网络和应用安全测试的洞见。
|
||||
|
||||
- **工业标准**：我们使用了CWE来解决对漏洞的认知问题。对于渗透测试，我们纳入了 MITRE ATT&CK 和 MITRE D3fend 框架。
|
||||
|
||||
### 生成评估问题
|
||||
|
||||
为了便于评估过程，我们设计了一个多项选择题的数据集格式。我们的问题生成方法包括几个步骤：
|
||||
|
||||
1. **文本解析**：我们首先根据文本的层次结构进行解析，例如教科书的章节，或诸如ATT&CK这样的框架的战略和技术。
|
||||
|
||||
2. **内容抽样**：对于内容丰富的文本，如CWE或Windows安全文档，我们采用抽样策略以保持可管理性。例如，我们从CWE中选择了25种最常见的弱点类型和175种随机类型。
|
||||
|
||||
3. **问题生成**：利用GPT-4基于解析后的文本生成多项选择题并根据文本的特性调整送入GPT参考文本的粒度。例如，CS161教科书的问题是基于各个章节的，而来自ATT&CK的则是基于技术的。
|
||||
|
||||
4. **问题完善**：然后我们指导GPT-4识别并过滤出问题，比如过于简单或不完整的问题。在可能的情况下，我们会尝试将问题修正；否则，它们将被丢弃。
|
||||
|
||||
5. **答案校准**：我们通过向GPT-4提供问题和产生问题的源文本来再次生成答案。如果GPT-4生成的响应与之前问题生成阶段确定的答案不一致，则表明，对于这道题目，获得一致答案本身具有挑战性。在这种情况下，我们选择淘汰这些问题。
|
||||
|
||||
6. **分类**：最后，我们将问题分为9个主题，并为每个问题附上相关的细粒度关键词。
|
||||
|
||||
|
||||
## 局限性
|
||||
|
||||
当前数据集存在一定的局限性：
|
||||
|
||||
1. **分布不平衡**：数据集在不同领域的问题分布不均，导致某些区域的问题集中度较高，而其他区域则较少。
|
||||
|
||||
2. **主题覆盖范围不完整**：数据集中缺少一些网络安全主题，如内容安全、逆向工程和恶意软件分析。因此，它并未涵盖该领域内的全部知识范围。
|
||||
|
||||
## 未来工作
|
||||
|
||||
1. **分布上的改进**：我们旨在通过增加更多问题来提高数据集的全面性，从而丰富现有网络安全主题的覆盖范围。
|
||||
|
||||
2. **主题覆盖上的改进**：我们将努力在数据集中包含更广泛的网络安全主题，这将有助于实现各个领域问题分布的更公平分配。
|
||||
|
||||
|
||||
## 许可证
|
||||
|
||||
数据集在 [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) 许可证下发布。代码在 [MIT](https://opensource.org/licenses/MIT) 许可证下发布。
|
||||
|
||||
|
||||
## 引用
|
||||
|
||||
```bibtex
|
||||
@misc{li2023seceval,
|
||||
title={SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models},
|
||||
author={Li, Guancheng and Li, Yifeng and Wang Guannan and Yang, Haoyu and Yu, Yang},
|
||||
publisher = {GitHub},
|
||||
howpublished= "https://github.com/XuanwuAI/SecEval",
|
||||
year={2023}
|
||||
}
|
||||
```
|
||||
|
||||
## 致谢
|
||||
|
||||
这项工作是在[腾讯安全玄武实验室](https://xlab.tencent.com/en/)的支持下完成的，并获得了腾讯星火计划的帮助。
|
||||
|
|
@ -1,189 +0,0 @@
|
|||
/* next/prev button styling */
|
||||
#multiline{
|
||||
line-height: 1em;
|
||||
}
|
||||
|
||||
input[type=button] {
|
||||
cursor: pointer;
|
||||
background: white;
|
||||
width: 5vw;
|
||||
height: 3vh;
|
||||
border-radius: 5px;
|
||||
font-size: 0.8em;
|
||||
}
|
||||
|
||||
.row {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: flex-start;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
margin-top: var(--nav-height);
|
||||
}
|
||||
|
||||
.optionsbtn {
|
||||
height: 50px;
|
||||
padding: 5px 10px;
|
||||
top: var(--nav-height);
|
||||
left: 0;
|
||||
position: fixed;
|
||||
z-index: 1;
|
||||
|
||||
font-size: 20px;
|
||||
cursor: pointer;
|
||||
background-color: var(--nav-color);
|
||||
color: white;
|
||||
border: none;
|
||||
}
|
||||
|
||||
.optionsbtn:hover {
|
||||
background-color: silver;
|
||||
}
|
||||
|
||||
|
||||
#option-panel {
|
||||
position: sticky;
|
||||
z-index: 2;
|
||||
height: 250px;
|
||||
top: var(--nav-height);
|
||||
left: 0;
|
||||
background-color: silver;
|
||||
overflow-x: hidden;
|
||||
transition: 0.5s;
|
||||
height: calc(100vh - var(--nav-height) - 40px);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
/* justify-content: space-around; */
|
||||
align-items: center;
|
||||
width: 20vw;
|
||||
padding-top: 40px;
|
||||
}
|
||||
|
||||
#option-panel a {
|
||||
padding: 8px 8px 8px 32px;
|
||||
text-decoration: none;
|
||||
font-size: 25px;
|
||||
color: #818181;
|
||||
display: block;
|
||||
transition: 0.3s;
|
||||
}
|
||||
|
||||
#option-panel a:hover {
|
||||
color: #f1f1f1;
|
||||
}
|
||||
|
||||
#option-panel .closebtn {
|
||||
position: absolute;
|
||||
top: 10px;
|
||||
right: 10px;
|
||||
font-size: 36px;
|
||||
}
|
||||
|
||||
#display {
|
||||
float: right;
|
||||
width: 60vw;
|
||||
transition: 0.5s;
|
||||
padding: 0px 10vw;
|
||||
}
|
||||
|
||||
#content-body {
|
||||
/* height: 200vh; */
|
||||
width: 100%;
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: flex-start;
|
||||
}
|
||||
|
||||
.display-col {
|
||||
width: 48%;
|
||||
}
|
||||
|
||||
.box {
|
||||
padding: 10px;
|
||||
margin: 0;
|
||||
background-color: white;
|
||||
border-radius: 5px;
|
||||
border: 1px solid rgba(0, 0, 0, 0.4);
|
||||
/* box-shadow: 5px 5px 5px rgba(0,0,0,0.2); */
|
||||
overflow-y: auto;
|
||||
max-height: 100vh;
|
||||
}
|
||||
|
||||
#title {
|
||||
margin: 50px 0;
|
||||
}
|
||||
|
||||
.question-title {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.question-txt {
|
||||
overflow-y: hidden;
|
||||
color: blue;
|
||||
}
|
||||
|
||||
.hint-txt {
|
||||
margin-bottom: 2px;
|
||||
}
|
||||
|
||||
.question-img {
|
||||
max-width: 100%;
|
||||
max-height: 400px;
|
||||
width: auto;
|
||||
height: auto;
|
||||
display: block;
|
||||
margin: 0 auto;
|
||||
margin-top: 20px;
|
||||
}
|
||||
|
||||
.choices {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.choices-vertical {
|
||||
display: grid;
|
||||
row-gap: 10px;
|
||||
}
|
||||
|
||||
.choice-txt {
|
||||
border: 1px solid silver;
|
||||
/* text-align: center; */
|
||||
padding: 5px 10px;
|
||||
margin-right: 10px;
|
||||
width: fit-content;
|
||||
}
|
||||
|
||||
.answer-txt {
|
||||
color: green;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.optbox {
|
||||
width: calc(90%);
|
||||
margin-bottom: 10px;
|
||||
transition: all 0.5s ease;
|
||||
}
|
||||
|
||||
.opttitle {
|
||||
width: 90%;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.dd-label {
|
||||
margin: 0px 0;
|
||||
display: block;
|
||||
}
|
||||
|
||||
#filter-submit {
|
||||
color: white;
|
||||
border-radius: 4px;
|
||||
text-shadow: 0 1px 1px rgba(0, 0, 0, 0.2);
|
||||
}
|
||||
|
||||
.button-success {
|
||||
background-color: rgb(28, 184, 65);
|
||||
}
|
||||
|
|
@ -1,78 +0,0 @@
|
|||
table {
|
||||
border: 1px solid black;
|
||||
border-collapse: collapse;
|
||||
}
|
||||
|
||||
th {
|
||||
padding: 2px 5px;
|
||||
border: 1px solid black;
|
||||
}
|
||||
|
||||
td {
|
||||
padding: 2px 5px;
|
||||
border: 1px solid black;
|
||||
}
|
||||
|
||||
thead {
|
||||
background: #ddd;
|
||||
}
|
||||
|
||||
table#results.js-sort-0 tbody tr td:nth-child(1),
|
||||
table#results.js-sort-1 tbody tr td:nth-child(2),
|
||||
table#results.js-sort-2 tbody tr td:nth-child(3),
|
||||
table#results.js-sort-3 tbody tr td:nth-child(4),
|
||||
table#results.js-sort-4 tbody tr td:nth-child(5),
|
||||
table#results.js-sort-5 tbody tr td:nth-child(6),
|
||||
table#results.js-sort-6 tbody tr td:nth-child(7),
|
||||
table#results.js-sort-7 tbody tr td:nth-child(8),
|
||||
table#results.js-sort-8 tbody tr td:nth-child(9),
|
||||
table#results.js-sort-9 tbody tr td:nth-child(10),
|
||||
table#results.js-sort-10 tbody tr td:nth-child(11),
|
||||
table#results.js-sort-11 tbody tr td:nth-child(12),
|
||||
table#results.js-sort-12 tbody tr td:nth-child(13),
|
||||
table#results.js-sort-13 tbody tr td:nth-child(14) {
|
||||
background: #dee;
|
||||
}
|
||||
|
||||
/* Table for three sub-tasks */
|
||||
#results {
|
||||
font-family: Arial, Helvetica, sans-serif;
|
||||
font-size: 16px;
|
||||
border-collapse: collapse;
|
||||
width: 100%;
|
||||
text-align: center;
|
||||
margin: auto;
|
||||
margin-top: 1em;
|
||||
margin-bottom: 2em;
|
||||
}
|
||||
|
||||
/*table tile*/
|
||||
#results th {
|
||||
border: 1px solid #ddd;
|
||||
padding: 8px;
|
||||
padding-top: 12px;
|
||||
padding-bottom: 12px;
|
||||
font-size: 16px;
|
||||
text-align: center;
|
||||
font-weight: 600;
|
||||
background-color: var(--UCLA-darkblue);
|
||||
color: white;
|
||||
}
|
||||
|
||||
#results td {
|
||||
border: 1px solid #ddd;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
#results tr:nth-child(even) {
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
|
||||
#results tr:hover {
|
||||
background-color: #ddd;
|
||||
}
|
||||
|
||||
.js-sort-number:hover {
|
||||
text-decoration: underline;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
|
@ -1,317 +0,0 @@
|
|||
/* some constants used throughout the website*/
|
||||
:root {
|
||||
--nav-font: Arial, Helvetica, sans-serif;
|
||||
--nav-height: 6vh;
|
||||
--UCLA-gold: #FFD100;
|
||||
--UCLA-blue: #2774AE;
|
||||
--UCLA-darkblue: #003B5C;
|
||||
--body-font: #212529;
|
||||
--heading: #1859c2;
|
||||
--nav-darkblue: #05004a;
|
||||
--nav-color: #212529;
|
||||
--nav-color-light: #353c42;
|
||||
--nav-darkblue: #05004a;
|
||||
}
|
||||
|
||||
html {
|
||||
overflow-x: hidden;
|
||||
font-family: var(--nav-font);
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
font-size: 16px;
|
||||
line-height: 110%;
|
||||
color: var(--body-font)
|
||||
}
|
||||
|
||||
strong {
|
||||
color: var(--body-font);
|
||||
}
|
||||
|
||||
/* block: heading in the paragraph */
|
||||
h1 {
|
||||
/* color: var(--UCLA-blue); */
|
||||
color: var(--heading);
|
||||
}
|
||||
|
||||
/* element: paragraphs in the body text */
|
||||
p {
|
||||
line-height: 150%;
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
/* links */
|
||||
a {
|
||||
color: black;
|
||||
text-decoration: none;
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
a:focus,
|
||||
a:hover {
|
||||
color: #005587;
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
a.hover {
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
a.hover:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
/* block: paper title in the body text */
|
||||
papername {
|
||||
font-family: Arial;
|
||||
font-size: 18px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
/* element: divider line */
|
||||
hr {
|
||||
width: 75%;
|
||||
color: silver;
|
||||
opacity: 0.5;
|
||||
}
|
||||
|
||||
img {
|
||||
max-width: 100%;
|
||||
}
|
||||
|
||||
ul {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
/* list */
|
||||
li {
|
||||
margin-top: 12px;
|
||||
font-size: 18px;
|
||||
display: list-item;
|
||||
text-align: -webkit-match-parent;
|
||||
}
|
||||
|
||||
/* body of the page */
|
||||
#body {
|
||||
width: 60vw;
|
||||
margin: auto;
|
||||
}
|
||||
|
||||
/* section */
|
||||
.section {
|
||||
width: calc(100% - 20px);
|
||||
padding: 10px;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
/* anchor tag for nav button scrolling */
|
||||
.anchor {
|
||||
position: relative;
|
||||
display: block;
|
||||
/* visibility: hidden; */
|
||||
top: calc(-1 * var(--nav-height));
|
||||
height: 1px;
|
||||
}
|
||||
|
||||
/* the banner */
|
||||
#title {
|
||||
margin-top: var(--nav-height);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
height: calc(50vh - var(--nav-height));
|
||||
width: 100vw;
|
||||
left: 0vw;
|
||||
font-weight: bold;
|
||||
color: white;
|
||||
/* background: linear-gradient(to bottom, var(--nav-darkblue), var(--UCLA-blue)); */
|
||||
background: linear-gradient(180deg, var(--nav-darkblue), var(--UCLA-blue));
|
||||
/* background: linear-gradient(, var(--UCLA-darkblue), var(--UCLA-blue) 60%, white); */
|
||||
}
|
||||
|
||||
#title-icon {
|
||||
width: 4em;
|
||||
margin: 10px;
|
||||
background-color: transparent;
|
||||
}
|
||||
|
||||
/* used to center the title */
|
||||
#title-wrapper {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
#title-text {
|
||||
font-size: 50px;
|
||||
}
|
||||
|
||||
#subtitle-text {
|
||||
font-size: 30px;
|
||||
text-align: center;
|
||||
line-height: 125%;
|
||||
margin: -0.5em;
|
||||
}
|
||||
|
||||
/* change this to adjust the vertical position of the title*/
|
||||
#title-padding-bottom {
|
||||
font-size: 1.5em;
|
||||
}
|
||||
|
||||
/* shows three examples on the home page */
|
||||
#example-box {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
/* data example images */
|
||||
.example-img {
|
||||
/* width: 15vw; */
|
||||
width: 90%;
|
||||
margin: auto;
|
||||
}
|
||||
|
||||
/* makes its children independent columns on the same row */
|
||||
.makecol {
|
||||
width: 100%;
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
#overview-left {
|
||||
width: 70%;
|
||||
}
|
||||
|
||||
#overview-right {
|
||||
width: 35%;
|
||||
}
|
||||
|
||||
/* not very important. Tried to make the website responsive */
|
||||
@media only screen and (max-width: 1000px) {
|
||||
#body {
|
||||
padding: 0;
|
||||
margin: 0;
|
||||
width: 100%;
|
||||
left: 0;
|
||||
}
|
||||
|
||||
.section {
|
||||
width: 98%;
|
||||
padding: 2.5%;
|
||||
}
|
||||
|
||||
#overview-content-wrapper {
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
#overview-left,
|
||||
#overview-right {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
overflow-x: auto;
|
||||
display: block;
|
||||
}
|
||||
|
||||
th, td {
|
||||
white-space: nowrap;
|
||||
}
|
||||
}
|
||||
|
||||
.bibtex {
|
||||
/* font-family: Arial; */
|
||||
padding-left: 1em;
|
||||
padding-right: 1em;
|
||||
padding-top: 1em;
|
||||
padding-bottom: 1em;
|
||||
background-color: #f5f5f5;
|
||||
border: 1px solid #ccc;
|
||||
border-radius: 5px;
|
||||
font-size: 16px;
|
||||
word-wrap: break-word;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
.ext-link {
|
||||
/* color: var(--UCLA-blue); */
|
||||
color: #007bff;
|
||||
/* text-decoration: underline; */
|
||||
}
|
||||
|
||||
#cc-img {
|
||||
--width: 40%;
|
||||
width: var(--width);
|
||||
margin: 0 calc(calc(100% - var(--width)) / 2);
|
||||
}
|
||||
|
||||
/* class: profile */
|
||||
.profile {
|
||||
display: inline-block;
|
||||
width: 180px;
|
||||
}
|
||||
|
||||
/* class: profile image */
|
||||
.profile-img {
|
||||
border-radius: 50%;
|
||||
width: 150px;
|
||||
}
|
||||
|
||||
/* #task-pie-chart */
|
||||
.col-wrapper {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.col {
|
||||
width: 48%;
|
||||
}
|
||||
|
||||
/* Table for three sub-tasks */
|
||||
#tasks {
|
||||
font-family: Arial, Helvetica, sans-serif;
|
||||
border-collapse: collapse;
|
||||
width: 70%;
|
||||
text-align: center;
|
||||
margin: auto;
|
||||
margin-top: 1em;
|
||||
margin-bottom: 2em;
|
||||
}
|
||||
|
||||
/*table tile*/
|
||||
#tasks th {
|
||||
border: 1px solid #ddd;
|
||||
padding: 8px;
|
||||
padding-top: 12px;
|
||||
padding-bottom: 12px;
|
||||
font-size: 16px;
|
||||
text-align: center;
|
||||
font-weight: 600;
|
||||
background-color: var(--UCLA-darkblue);
|
||||
color: white;
|
||||
}
|
||||
|
||||
#tasks td {
|
||||
border: 1px solid #ddd;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
#tasks tr:nth-child(even) {
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
|
||||
#tasks tr:hover {
|
||||
background-color: #ddd;
|
||||
}
|
||||
|
|
@ -1,55 +0,0 @@
|
|||
/* navigation bar styling */
|
||||
#nav {
|
||||
width: 96%;
|
||||
height: var(--nav-height);
|
||||
/* background-color: var(--nav-darkblue); */
|
||||
background-color: #212529;
|
||||
position: fixed;
|
||||
top: 0;
|
||||
z-index: 10;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 0 2vw;
|
||||
opacity: 1;
|
||||
/* border-bottom: 1px solid blue; */
|
||||
}
|
||||
|
||||
#icon {
|
||||
font-size: 1.5em;
|
||||
font-family: var(--nav-font);
|
||||
font-weight: 700;
|
||||
color: white;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.nav-button {
|
||||
margin-left: 20px;
|
||||
font-family: var(--nav-font);
|
||||
color: white;
|
||||
}
|
||||
|
||||
.nav-button:hover {
|
||||
color: var(--UCLA-gold);
|
||||
}
|
||||
|
||||
#nav-icon {
|
||||
width: 1em;
|
||||
margin: 5px;
|
||||
}
|
||||
|
||||
/* footer style */
|
||||
#footer {
|
||||
width: 100%;
|
||||
height: var(--nav-height);
|
||||
background-color: var(--nav-darkblue);
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
opacity: 1;
|
||||
font-size: 0.7em;
|
||||
color: grey;
|
||||
margin-top: 40px;
|
||||
}
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Explore | SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models</title>
|
||||
<link rel="icon" href="img/logo.png" type="image/icon type">
|
||||
|
||||
<link rel="stylesheet" href="https://unpkg.com/purecss@2.1.0/build/pure-min.css"
|
||||
integrity="sha384-yHIFVG6ClnONEA5yB5DJXfW2/KC173DIQrYoZMEtBvGzmf0PKiGyNEqe9N6BNDBH" crossorigin="anonymous">
|
||||
<link rel="stylesheet" href="css/main.css">
|
||||
<link rel="stylesheet" href="css/nav.css">
|
||||
<link rel="stylesheet" href="css/explore.css">
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@latest/dist/katex.min.css">
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/underscore@latest/underscore-umd-min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/katex@latest/dist/katex.min.js"></script>
|
||||
<script src="javascript/questions-data.js"></script>
|
||||
<script src="javascript/explore.js" defer></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="nav">
|
||||
<div id="icon">
|
||||
<img src="img/logo.png" id="nav-icon">
|
||||
<a class="nav-button" href="https://XuanwuAI.github.io/SecEval/"
|
||||
style="margin-left: 2px; font-size: 24px">SecEval
|
||||
</a>
|
||||
</div>
|
||||
<div>
|
||||
<a class="nav-button" href="index.html#home">Home</a>
|
||||
<a class="nav-button" href="index.html#dataset">Dataset</a>
|
||||
<a class="nav-button" href="#paper_placeholder.html " target="_blank">Paper</a>
|
||||
<a class="nav-button" href="https://github.com/XuanwuAI/SecEval" target="_blank">Github</a>
|
||||
<a class="nav-button" href="index.html#citation">Citation</a>
|
||||
<a class="nav-button" href="index.html#contact">Contact</a>
|
||||
<a class="nav-button" href="leaderboard.html">Leaderboard</a>
|
||||
<a class="nav-button" href="explore.html">Explore</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<button class="optionsbtn">☰</button>
|
||||
<div class="row">
|
||||
<div id="option-panel" class="left">
|
||||
<a href="javascript:void(0)" class="closebtn">×</a>
|
||||
<h3 class="opttitle">Sample Filters</h3>
|
||||
<div class="optbox"></div>
|
||||
</div>
|
||||
<div id="display">
|
||||
<br><br>
|
||||
<h1 id="multiline" xt-marked="ok">SecEval (A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models)</h1>
|
||||
<br><br>
|
||||
<div id="content-body">
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 363 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 78 KiB |
|
|
@ -1,136 +0,0 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models</title>
|
||||
<link rel="icon" href="img/logo.png" type="image/icon type">
|
||||
<link rel="stylesheet" href="css/main.css">
|
||||
<link rel="stylesheet" href="css/nav.css">
|
||||
<script src="javascript/index.js" defer></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="nav">
|
||||
<div id="icon">
|
||||
<img src="img/logo.png" id="nav-icon">
|
||||
<a class="nav-button" href="https://XuanwuAI.github.io/SecEval/" style="margin-left: 2px; font-size: 24px">SecEval
|
||||
</a>
|
||||
</div>
|
||||
<div>
|
||||
<a class="nav-button" href="#home">Home</a>
|
||||
<a class="nav-button" href="#dataset">Dataset</a>
|
||||
<a class="nav-button" href="paper_placeholder.html" target="_blank">Paper</a>
|
||||
<a class="nav-button" href="https://github.com/XuanwuAI/SecEval" target="_blank">Github</a>
|
||||
<a class="nav-button" href="#citation">Citation</a>
|
||||
<a class="nav-button" href="#contact">Contact</a>
|
||||
<a class="nav-button" href="leaderboard.html">Leaderboard</a>
|
||||
<a class="nav-button" href="explore.html">Explore</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- anchor for the home button -->
|
||||
<div id="home" style="position: absolute; top: 0;"></div>
|
||||
|
||||
<!-- banner -->
|
||||
<div id="title">
|
||||
<div id="title-wrapper">
|
||||
<img src="img/logo.png" id="title-icon">
|
||||
<p id="title-text">SecEval</p>
|
||||
</div>
|
||||
<p id="subtitle-text">
|
||||
A Comprehensive Benchmark for Evaluating <br>
|
||||
Cybersecurity Knowledge of Foundation Models
|
||||
<br><br>
|
||||
(2023)
|
||||
</p>
|
||||
<p id="title-padding-bottom"> </p>
|
||||
</div>
|
||||
|
||||
<!-- the main body of the page -->
|
||||
<!-- each section uses an empty div as an anchor -->
|
||||
<div id="body">
|
||||
<div class="section">
|
||||
<div id="about" class="anchor"></div>
|
||||
<h1>About SecEval</h1>
|
||||
<p style="line-height: 150%">
|
||||
SecEval is the first benchmark specifically created for evaluating cybersecurity knowledge in Foundation Models. It offers over 2000 multiple-choice questions across 9 domains: Software Security, Application Security, System Security, Web Security, Cryptography, Memory Safety, Network Security, and PenTest.
|
||||
These questions are generated by prompting OpenAI GPT4 with authoritative sources such as open-licensed textbooks, official documentation, and industry guidelines and standards. The generation process is meticulously crafted to ensure the dataset meets rigorous quality, diversity, and impartiality criteria. You can explore our dataset and detailed methodology in our <a class="ext-link" href="paper_placeholder.html" target="_blank"
|
||||
xt-marked="ok">paper</a>, or explore samples by visiting <a class="ext-link" href="explore.html"
|
||||
xt-marked="ok">Explore</a>.
|
||||
</p>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- <hr> -->
|
||||
<div class="section">
|
||||
<div id="dataset" class="anchor"></div>
|
||||
<h1>SecEval Dataset</h1>
|
||||
<p>
|
||||
Our dataset is distributed under the
|
||||
<a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="blank" class="ext-link">
|
||||
CC BY-NC-SA (Attribution-NonCommercial-ShareAlike)
|
||||
</a>
|
||||
license. You can download the dataset from our
|
||||
<a href="https://github.com/XuanwuAI/SecEval" class="ext-link" target="blank">Github
|
||||
Repository</a> or <a href="https://huggingface.co/datasets/XuanwuAI/SecEval" class="ext-link" target="blank">HuggingFace Datasets</a>!
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
<!-- <hr> -->
|
||||
<div class="section">
|
||||
<div id="citation" class="anchor"></div>
|
||||
<h1>Citation</h1>
|
||||
<pre class="bibtex">
|
||||
@misc{li2023seceval,
|
||||
title={SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models},
|
||||
author={Li, Guancheng and Li, Yifeng and Wang Guannan and Yang, Haoyu and Yu, Yang},
|
||||
publisher = {GitHub},
|
||||
howpublished= "https://github.com/XuanwuAI/SecEval",
|
||||
year={2023}
|
||||
}
|
||||
</pre>
|
||||
<br>
|
||||
</div>
|
||||
|
||||
|
||||
<!-- Contact -->
|
||||
<div class="section" id="contact">
|
||||
<h1>Contact</h1>
|
||||
<p>
|
||||
If you have any questions about SecEval, please contact us at, or open
|
||||
up an issue on
|
||||
<a href="https://github.com/XuanwuAI/SecEval/issues" target="blank" class="ext-link">Github</a>.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Affiliation -->
|
||||
<div class="section" style="table-layout: auto;">
|
||||
<center>
|
||||
<table>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>
|
||||
<a href="https://xlab.tencent.com/en/" target="_blank" rel="external">
|
||||
<img class="center-block" src="img/xlab.png"
|
||||
style="height:6em; max-width: 100%;"></a>
|
||||
</td>
|
||||
<!-- <td> </td> -->
|
||||
</tr>
|
||||
</tbody>
|
||||
</table><br>
|
||||
</center>
|
||||
</div>
|
||||
|
||||
|
||||
</div>
|
||||
<!-- footer -->
|
||||
<div id="footer" style="font-size: 12pt">
|
||||
© 2023 Website developed based on the ScienceQA template.
|
||||
</div>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
|
@ -1,204 +0,0 @@
|
|||
let topics = [
|
||||
'All',
|
||||
'SoftwareSecurity',
|
||||
'NetworkSecurity',
|
||||
'Vulnerability',
|
||||
'MemorySafety',
|
||||
'WebSecurity',
|
||||
'AppSecurity',
|
||||
'Cryptography',
|
||||
'SystemSecurity',
|
||||
'PenTest'
|
||||
]
|
||||
|
||||
|
||||
|
||||
let topic_dd = make_dropdown("Choose a topic:", topics, "topic-dd");
|
||||
let optbtn = document.getElementsByClassName("optionsbtn")[0];
|
||||
let closebtn = document.getElementsByClassName("closebtn")[0];
|
||||
let optionpanel = document.getElementById("option-panel");
|
||||
let body = document.getElementById("content-body");
|
||||
let display = document.getElementById("display");
|
||||
let optboxes = document.getElementsByClassName("optbox");
|
||||
let opt_dds = document.getElementsByClassName("opt-dd");
|
||||
|
||||
optboxes[0].innerHTML += topic_dd;
|
||||
|
||||
topic_dd = document.getElementById("topic-dd");
|
||||
topic_dd.addEventListener("change", filter_data);
|
||||
|
||||
|
||||
let filters = {};
|
||||
|
||||
optbtn.addEventListener("click", openNav);
|
||||
closebtn.addEventListener("click", closeNav);
|
||||
|
||||
|
||||
for (each of opt_dds) {
|
||||
each.addEventListener("change", change_filters);
|
||||
}
|
||||
|
||||
filter_data();
|
||||
|
||||
function openNav() {
|
||||
optionpanel.style.width = "20vw";
|
||||
display.style.width = "60vw";
|
||||
for (each of optionpanel.children) {
|
||||
each.style.display = "block";
|
||||
}
|
||||
}
|
||||
|
||||
function closeNav() {
|
||||
optionpanel.style.width = "0";
|
||||
display.style.width = "80vw";
|
||||
for (each of optionpanel.children) {
|
||||
each.style.display = "none";
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
function change_filters(e) {
|
||||
filters.topic = document.getElementById("topic-dd").value;
|
||||
}
|
||||
|
||||
function create_page(d) {
|
||||
if (d.length === 0) {
|
||||
body.innerHTML = "<p>No example satisfies All the filters.</p>";
|
||||
} else {
|
||||
col1 = create_col(d.slice(0, d.length / 2));
|
||||
col2 = create_col(d.slice(d.length / 2));
|
||||
body.innerHTML = col1 + col2;
|
||||
}
|
||||
reflow(body);
|
||||
console.log("reflowed");
|
||||
}
|
||||
|
||||
function escapeHtml(html) {
|
||||
return html
|
||||
.replace(/&/g, "&")
|
||||
.replace(/</g, "<")
|
||||
.replace(/>/g, ">")
|
||||
.replace(/"/g, """)
|
||||
.replace(/'/g, "'");
|
||||
}
|
||||
|
||||
|
||||
function create_col(data) {
|
||||
res = [];
|
||||
for (each of data) {
|
||||
res.push(create_example(each));
|
||||
}
|
||||
return `<div class="display-col"> ${res.join("")} </div>`;
|
||||
}
|
||||
|
||||
|
||||
function create_example(data) {
|
||||
let escapedQuestion = escapeHtml(data.question);
|
||||
let question = make_qt(escapedQuestion);
|
||||
|
||||
let escapedChoices = data.choices.map(choice => escapeHtml(choice));
|
||||
let choices = make_choices(escapedChoices);
|
||||
|
||||
let answer = make_answer(data.answer);
|
||||
html = make_box([question, choices, answer]) + "<hr/>";
|
||||
return html;
|
||||
|
||||
}
|
||||
|
||||
|
||||
function convert_latex(data) {
|
||||
const re = /(\$+)([^\$]*?[^\\])\1|\\begin\{tabular\}.*?\\end\{tabular\}/gs;
|
||||
let html = '';
|
||||
|
||||
let match;
|
||||
let lastIndex = 0;
|
||||
while ((match = re.exec(data)) !== null) {
|
||||
html += `<span>${data.slice(lastIndex, match.index)}</span>`;
|
||||
|
||||
if (match[0].startsWith('\\begin{tabular}')) {
|
||||
const table = match[0];
|
||||
const tableHtml = '...';
|
||||
html += tableHtml;
|
||||
} else {
|
||||
const latex = match[2];
|
||||
const formula = katex.renderToString(latex, {throwOnError: false});
|
||||
html += `<span>${formula}</span>`;
|
||||
}
|
||||
|
||||
lastIndex = match.index + match[0].length;
|
||||
}
|
||||
|
||||
html += `<span>${data.slice(lastIndex)}</span>`;
|
||||
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_qt(text) {
|
||||
let html = `
|
||||
<p><b>Question </b></p>
|
||||
<p class="question-txt">${convert_latex(text).replace(/\\n|\n/g, "<br>")}</p>
|
||||
`;
|
||||
return html;
|
||||
}
|
||||
|
||||
|
||||
function make_box(contents, cls = "") {
|
||||
if (contents.join("").length === 0) return "";
|
||||
let html = `
|
||||
<div class="box ${cls}">
|
||||
${contents.join(" ").replace(/\\n|\n/g, "")}
|
||||
</div>
|
||||
`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_choices(choices) {
|
||||
let temp = "";
|
||||
let len = 0;
|
||||
for (each of choices) {
|
||||
let html = make_choice(each);
|
||||
temp += html;
|
||||
len += each.length;
|
||||
}
|
||||
let html = "";
|
||||
if (len < 60)
|
||||
html = `<p><b>Choices </b></p><div class="choices">${convert_latex(temp).replace(/\\n|\n/g, "<br>")}</div>`;
|
||||
else
|
||||
html = `<p><b>Choices </b></p><div class="choices-vertical">${convert_latex(temp).replace(/\\n|\n/g, "<br>")}</div>`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_choice(choice) {
|
||||
let html = `<div class="choice-txt">${convert_latex(choice).replace(/\\n|\n/g, "<br>")}</div>`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_answer(answer) {
|
||||
let html = `<p><b>Answer </b></p><p class="answer-txt">${convert_latex(answer)}</p>`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_dropdown(label, options, id, default_ind = 0) {
|
||||
let html = "";
|
||||
for (let i = 0; i < options.length; i++) {
|
||||
if (i === default_ind)
|
||||
html += `<option value="${options[i]}" selected> ${options[i]} </option>`;
|
||||
else
|
||||
html += `<option value="${options[i]}"> ${options[i]} </option>`;
|
||||
}
|
||||
html = `<label class="dd-label">${label} <select id="${id}" class="opt-dd"> ${html} </select> </label><br/>`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function filter_data() {
|
||||
change_filters();
|
||||
res = problem_data;
|
||||
if (filters.topic !== "All")
|
||||
res = res.filter(e => e.topics.includes(filters.topic));
|
||||
d = _.sample(res, Math.min(100, res.length));
|
||||
create_page(d);
|
||||
}
|
||||
|
||||
function reflow(elt) {
|
||||
elt.offsetHeight;
|
||||
}
|
||||
|
|
@ -1,172 +0,0 @@
|
|||
leaderboard = [
|
||||
{
|
||||
"Model": "gpt-4-turbo",
|
||||
"Creator": "OpenAI",
|
||||
"Access": "API, Web",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "73.61",
|
||||
"Application Security": "75.25",
|
||||
"PenTest": "80.00",
|
||||
"Memory Safety": "70.83",
|
||||
"Network Security": "75.65",
|
||||
"Web Security": "82.15",
|
||||
"Vulnerability": "76.05",
|
||||
"Software Security": "73.28",
|
||||
"Cryptography": "64.29",
|
||||
"Overall": "79.07",
|
||||
"#": "1"
|
||||
},
|
||||
{
|
||||
"Model": "gpt-3.5-turbo",
|
||||
"Creator": "OpenAI",
|
||||
"Access": "API, Web",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "59.15",
|
||||
"Application Security": "57.18",
|
||||
"PenTest": "72.00",
|
||||
"Memory Safety": "43.75",
|
||||
"Network Security": "60.87",
|
||||
"Web Security": "63.00",
|
||||
"Vulnerability": "60.18",
|
||||
"Software Security": "58.19",
|
||||
"Cryptography": "35.71",
|
||||
"Overall": "62.09",
|
||||
"#": "2"
|
||||
},
|
||||
{
|
||||
"Model": "Yi-6B",
|
||||
"Creator": "01-AI",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "50.61",
|
||||
"Application Security": "48.89",
|
||||
"PenTest": "69.26",
|
||||
"Memory Safety": "35.42",
|
||||
"Network Security": "56.52",
|
||||
"Web Security": "54.98",
|
||||
"Vulnerability": "49.40",
|
||||
"Software Security": "45.69",
|
||||
"Cryptography": "35.71",
|
||||
"Overall": "53.57",
|
||||
"#": "3"
|
||||
},
|
||||
{
|
||||
"Model": "Orca-2-7b",
|
||||
"Creator": "Microsoft",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "46.76",
|
||||
"Application Security": "47.03",
|
||||
"PenTest": "60.84",
|
||||
"Memory Safety": "31.25",
|
||||
"Network Security": "49.13",
|
||||
"Web Security": "55.63",
|
||||
"Vulnerability": "50.00",
|
||||
"Software Security": "52.16",
|
||||
"Cryptography": "14.29",
|
||||
"Overall": "51.60",
|
||||
"#": "4"
|
||||
},
|
||||
{
|
||||
"Model": "Mistral-7B-v0.1",
|
||||
"Creator": "Mistralai",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "40.19",
|
||||
"Application Security": "38.37",
|
||||
"PenTest": "53.47",
|
||||
"Memory Safety": "33.33",
|
||||
"Network Security": "36.52",
|
||||
"Web Security": "46.57",
|
||||
"Vulnerability": "42.22",
|
||||
"Software Security": "43.10",
|
||||
"Cryptography": "28.57",
|
||||
"Overall": "43.65",
|
||||
"#": "5"
|
||||
},
|
||||
{
|
||||
"Model": "chatglm3-6b-base",
|
||||
"Creator": "THUDM",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "39.72",
|
||||
"Application Security": "37.25",
|
||||
"PenTest": "57.47",
|
||||
"Memory Safety": "31.25",
|
||||
"Network Security": "43.04",
|
||||
"Web Security": "41.14",
|
||||
"Vulnerability": "37.43",
|
||||
"Software Security": "39.66",
|
||||
"Cryptography": "28.57",
|
||||
"Overall": "41.58",
|
||||
"#": "6"
|
||||
},
|
||||
{
|
||||
"Model": "Aquila2-7B",
|
||||
"Creator": "BAAI",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "34.84",
|
||||
"Application Security": "36.01",
|
||||
"PenTest": "47.16",
|
||||
"Memory Safety": "22.92",
|
||||
"Network Security": "32.17",
|
||||
"Web Security": "42.04",
|
||||
"Vulnerability": "38.02",
|
||||
"Software Security": "36.21",
|
||||
"Cryptography": "7.14",
|
||||
"Overall": "38.29",
|
||||
"#": "7"
|
||||
},
|
||||
{
|
||||
"Model": "Qwen-7B",
|
||||
"Creator": "Alibaba",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "28.92",
|
||||
"Application Security": "28.84",
|
||||
"PenTest": "41.47",
|
||||
"Memory Safety": "18.75",
|
||||
"Network Security": "29.57",
|
||||
"Web Security": "33.25",
|
||||
"Vulnerability": "31.74",
|
||||
"Software Security": "30.17",
|
||||
"Cryptography": "14.29",
|
||||
"Overall": "31.37",
|
||||
"#": "8"
|
||||
},
|
||||
{
|
||||
"Model": "internlm-7b",
|
||||
"Creator": "Sensetime",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "25.92",
|
||||
"Application Security": "25.87",
|
||||
"PenTest": "36.21",
|
||||
"Memory Safety": "25.00",
|
||||
"Network Security": "27.83",
|
||||
"Web Security": "32.86",
|
||||
"Vulnerability": "29.34",
|
||||
"Software Security": "34.05",
|
||||
"Cryptography": "7.14",
|
||||
"Overall": "30.29",
|
||||
"#": "9"
|
||||
},
|
||||
{
|
||||
"Model": "Llama-2-7b-hf",
|
||||
"Creator": "MetaAI",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "20.94",
|
||||
"Application Security": "18.69",
|
||||
"PenTest": "26.11",
|
||||
"Memory Safety": "16.67",
|
||||
"Network Security": "14.35",
|
||||
"Web Security": "22.77",
|
||||
"Vulnerability": "21.56",
|
||||
"Software Security": "20.26",
|
||||
"Cryptography": "21.43",
|
||||
"Overall": "22.15",
|
||||
"#": "10"
|
||||
}
|
||||
]
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
let obj_to_tr = (obj) => {
|
||||
return `<tr>
|
||||
<td>${obj["#"]}</td>
|
||||
<td>${obj.Model}</td>
|
||||
<td>${obj.Creator}</td>
|
||||
<td>${obj.Access}</td>
|
||||
<td>${obj["Submission Date"]}</td>
|
||||
<td>${obj["Network Security"]}</td>
|
||||
<td>${obj.Vulnerability}</td>
|
||||
<td>${obj["Memory Safety"]}</td>
|
||||
<td>${obj["Web Security"]}</td>
|
||||
<td>${obj["Application Security"]}</td>
|
||||
<td>${obj.Cryptography}</td>
|
||||
<td>${obj["System Security"]}</td>
|
||||
<td>${obj["Software Security"]}</td>
|
||||
<td>${obj["PenTest"]}</td>
|
||||
<td>${obj.Overall}</td>
|
||||
</tr>`
|
||||
}
|
||||
|
||||
let table = document.getElementById("results");
|
||||
html = "";
|
||||
for (let i of leaderboard) {
|
||||
html += obj_to_tr(i);
|
||||
}
|
||||
table.getElementsByTagName("tbody")[0].innerHTML = html;
|
||||
function reflow(elt){
|
||||
console.log(elt.offsetHeight);
|
||||
}
|
||||
reflow(table);
|
||||
|
|
@ -1,309 +0,0 @@
|
|||
/**
|
||||
* sort-table.js
|
||||
* A pure JavaScript (no dependencies) solution to make HTML
|
||||
* Tables sortable
|
||||
*
|
||||
* Copyright (c) 2013 Tyler Uebele
|
||||
* Released under the MIT license. See included LICENSE.txt
|
||||
* or http://opensource.org/licenses/MIT
|
||||
*
|
||||
* latest version available at https://github.com/tyleruebele/sort-table
|
||||
*/
|
||||
|
||||
/**
|
||||
* Sort the rows in a HTML Table
|
||||
*
|
||||
* @param Table The Table DOM object
|
||||
* @param col The zero-based column number by which to sort
|
||||
* @param dir Optional. The sort direction; pass 1 for asc; -1 for desc
|
||||
* @returns void
|
||||
*/
|
||||
function sortTable(Table, col, dir) {
|
||||
var sortClass, i;
|
||||
|
||||
// get previous sort column
|
||||
sortTable.sortCol = -1;
|
||||
sortClass = Table.className.match(/js-sort-\d+/);
|
||||
if (null != sortClass) {
|
||||
sortTable.sortCol = sortClass[0].replace(/js-sort-/, '');
|
||||
Table.className = Table.className.replace(new RegExp(' ?' + sortClass[0] + '\\b'), '');
|
||||
}
|
||||
// If sort column was not passed, use previous
|
||||
if ('undefined' === typeof col) {
|
||||
col = sortTable.sortCol;
|
||||
}
|
||||
|
||||
if ('undefined' !== typeof dir) {
|
||||
// Accept -1 or 'desc' for descending. All else is ascending
|
||||
sortTable.sortDir = dir == -1 || dir == 'desc' ? -1 : 1;
|
||||
} else {
|
||||
// sort direction was not passed, use opposite of previous
|
||||
sortClass = Table.className.match(/js-sort-(a|de)sc/);
|
||||
if (null != sortClass && sortTable.sortCol == col) {
|
||||
sortTable.sortDir = 'js-sort-asc' == sortClass[0] ? -1 : 1;
|
||||
} else {
|
||||
sortTable.sortDir = 1;
|
||||
}
|
||||
}
|
||||
Table.className = Table.className.replace(/ ?js-sort-(a|de)sc/g, '');
|
||||
|
||||
// update sort column
|
||||
Table.className += ' js-sort-' + col;
|
||||
sortTable.sortCol = col;
|
||||
|
||||
// update sort direction
|
||||
Table.className += ' js-sort-' + (sortTable.sortDir == -1 ? 'desc' : 'asc');
|
||||
|
||||
// get sort type
|
||||
if (col < Table.tHead.rows[Table.tHead.rows.length - 1].cells.length) {
|
||||
sortClass = Table.tHead.rows[Table.tHead.rows.length - 1].cells[col].className.match(/js-sort-[-\w]+/);
|
||||
}
|
||||
// Improved support for colspan'd headers
|
||||
for (i = 0; i < Table.tHead.rows[Table.tHead.rows.length - 1].cells.length; i++) {
|
||||
if (col == Table.tHead.rows[Table.tHead.rows.length - 1].cells[i].getAttribute('data-js-sort-colNum')) {
|
||||
sortClass = Table.tHead.rows[Table.tHead.rows.length - 1].cells[i].className.match(/js-sort-[-\w]+/);
|
||||
}
|
||||
}
|
||||
if (null != sortClass) {
|
||||
sortTable.sortFunc = sortClass[0].replace(/js-sort-/, '');
|
||||
} else {
|
||||
sortTable.sortFunc = 'string';
|
||||
}
|
||||
// Set the headers for the active column to have the decorative class
|
||||
Table.querySelectorAll('.js-sort-active').forEach(function(Node) {
|
||||
Node.className = Node.className.replace(/ ?js-sort-active\b/, '');
|
||||
});
|
||||
Table.querySelectorAll('[data-js-sort-colNum="' + col + '"]:not(:empty)').forEach(function(Node) {
|
||||
Node.className += ' js-sort-active';
|
||||
});
|
||||
|
||||
// sort!
|
||||
var rows = [],
|
||||
TBody = Table.tBodies[0];
|
||||
|
||||
for (i = 0; i < TBody.rows.length; i++) {
|
||||
rows[i] = TBody.rows[i];
|
||||
}
|
||||
if ('none' != sortTable.sortFunc) {
|
||||
rows.sort(sortTable.compareRow);
|
||||
}
|
||||
|
||||
while (TBody.firstChild) {
|
||||
TBody.removeChild(TBody.firstChild);
|
||||
}
|
||||
for (i = 0; i < rows.length; i++) {
|
||||
TBody.appendChild(rows[i]);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare two table rows based on current settings
|
||||
*
|
||||
* @param RowA A TR DOM object
|
||||
* @param RowB A TR DOM object
|
||||
* @returns {number} 1 if RowA is greater, -1 if RowB, 0 if equal
|
||||
*/
|
||||
sortTable.compareRow = function(RowA, RowB) {
|
||||
var valA, valB;
|
||||
if ('function' != typeof sortTable[sortTable.sortFunc]) {
|
||||
sortTable.sortFunc = 'string';
|
||||
}
|
||||
valA = sortTable[sortTable.sortFunc](RowA.cells[sortTable.sortCol]);
|
||||
valB = sortTable[sortTable.sortFunc](RowB.cells[sortTable.sortCol]);
|
||||
|
||||
return valA == valB ? 0 : sortTable.sortDir * (valA > valB ? 1 : -1);
|
||||
};
|
||||
|
||||
/**
|
||||
* Strip all HTML, no exceptions
|
||||
* @param html
|
||||
* @returns {string}
|
||||
*/
|
||||
sortTable.stripTags = function(html) {
|
||||
replace_unit = (s) => {
|
||||
let iUnit = (s.indexOf('M') > -1) ? s.indexOf('M') : s.indexOf('B');
|
||||
if (iUnit == -1) return s;
|
||||
let unit = s[iUnit];
|
||||
let val = Number(s.substring(0, iUnit));
|
||||
if (isNaN(val)) return s;
|
||||
val *= (unit == 'M') ? 1000000 : 1000000000;
|
||||
return val.toString();
|
||||
}
|
||||
html = replace_unit(html);
|
||||
return html.replace(/<\/?[a-z][a-z0-9]*\b[^>]*>/gi, '');
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Converts innerHTML to a timestamp, 0 for invalid dates
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {Number}
|
||||
*/
|
||||
sortTable.date = function(Cell) {
|
||||
// If okDate library is available, Use it for advanced Date processing
|
||||
if (typeof okDate !== 'undefined') {
|
||||
var kDate = okDate(sortTable.stripTags(Cell.innerHTML));
|
||||
return kDate ? kDate.getTime() : 0;
|
||||
} else {
|
||||
return (new Date(sortTable.stripTags(Cell.innerHTML))).getTime() || 0;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Converts innerHTML to a JS Number object
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {Number}
|
||||
*/
|
||||
sortTable.number = function(Cell) {
|
||||
return Number(sortTable.stripTags(Cell.innerHTML).replace(/[^-\d.]/g, ''));
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Converts innerHTML to a lower case string for insensitive compare
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {String}
|
||||
*/
|
||||
sortTable.string = function(Cell) {
|
||||
return sortTable.stripTags(Cell.innerHTML).toLowerCase();
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {String}
|
||||
*/
|
||||
sortTable.raw = function(Cell) {
|
||||
return Cell.innerHTML;
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Captures the last space-delimited token from innerHTML
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {String}
|
||||
*/
|
||||
sortTable.last = function(Cell) {
|
||||
return sortTable.stripTags(Cell.innerHTML).split(' ').pop().toLowerCase();
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Captures the value of the first childNode
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {String}
|
||||
*/
|
||||
sortTable.input = function(Cell) {
|
||||
for (var i = 0; i < Cell.children.length; i++) {
|
||||
if ('object' == typeof Cell.children[i]
|
||||
&& 'undefined' != typeof Cell.children[i].value
|
||||
) {
|
||||
return Cell.children[i].value.toLowerCase();
|
||||
}
|
||||
}
|
||||
|
||||
return sortTable.string(Cell);
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that prevents sorting by always returning null
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns null
|
||||
*/
|
||||
sortTable.none = function(Cell) {
|
||||
return null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Return the click handler appropriate to the specified Table and column
|
||||
*
|
||||
* @param Table Table to sort
|
||||
* @param col Column to sort by
|
||||
* @returns {Function} Click Handler
|
||||
*/
|
||||
sortTable.getClickHandler = function(Table, col) {
|
||||
return function() {
|
||||
sortTable(Table, col);
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Attach sortTable() calls to table header cells' onclick events
|
||||
* If the table(s) do not have a THead node, one will be created around the
|
||||
* first row
|
||||
*/
|
||||
sortTable.init = function() {
|
||||
var THead, Tables, Handler;
|
||||
if (document.querySelectorAll) {
|
||||
Tables = document.querySelectorAll('table.js-sort-table');
|
||||
} else {
|
||||
Tables = document.getElementsByTagName('table');
|
||||
}
|
||||
|
||||
for (var i = 0; i < Tables.length; i++) {
|
||||
// Because IE<8 doesn't support querySelectorAll, skip unclassed tables
|
||||
if (!document.querySelectorAll && null === Tables[i].className.match(/\bjs-sort-table\b/)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Prevent repeat processing
|
||||
if (Tables[i].attributes['data-js-sort-table']) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Ensure table has a tHead element
|
||||
if (!Tables[i].tHead) {
|
||||
THead = document.createElement('thead');
|
||||
THead.appendChild(Tables[i].rows[0]);
|
||||
Tables[i].insertBefore(THead, Tables[i].children[0]);
|
||||
} else {
|
||||
THead = Tables[i].tHead;
|
||||
}
|
||||
|
||||
// Attach click events to table header
|
||||
for (var rowNum = 0; rowNum < THead.rows.length; rowNum++) {
|
||||
for (var cellNum = 0, colNum = 0; cellNum < THead.rows[rowNum].cells.length; cellNum++) {
|
||||
// Skip headers marked "js-sort-none"
|
||||
if (THead.rows[rowNum].cells[cellNum].className.match(/\bjs-sort-none\b/)) {
|
||||
continue;
|
||||
}
|
||||
// Define which column the header should invoke sorting for
|
||||
THead.rows[rowNum].cells[cellNum].setAttribute('data-js-sort-colNum', colNum);
|
||||
Handler = sortTable.getClickHandler(Tables[i], colNum);
|
||||
window.addEventListener
|
||||
? THead.rows[rowNum].cells[cellNum].addEventListener('click', Handler)
|
||||
: window.attachEvent && THead.rows[rowNum].cells[cellNum].attachEvent('onclick', Handler);
|
||||
colNum += THead.rows[rowNum].cells[cellNum].colSpan;
|
||||
}
|
||||
}
|
||||
|
||||
// Mark table as processed
|
||||
Tables[i].setAttribute('data-js-sort-table', 'true')
|
||||
}
|
||||
|
||||
// Add default styles as the first style in head so they can be easily overwritten by user styles
|
||||
var element = document.createElement('style');
|
||||
document.head.insertBefore(element, document.head.childNodes[0]);
|
||||
var sheet = element.sheet;
|
||||
sheet.insertRule('table.js-sort-table.js-sort-asc thead tr > .js-sort-active:not(.js-sort-none):after {content: "\\25b2";font-size: 0.7em;padding-left: 3px;line-height: 0.7em;}', 0);
|
||||
sheet.insertRule('table.js-sort-table.js-sort-desc thead tr > .js-sort-active:not(.js-sort-none):after {content: "\\25bc";font-size: 0.7em;padding-left: 3px;line-height: 0.7em;}', 0);
|
||||
};
|
||||
|
||||
// Run sortTable.init() when the page loads
|
||||
window.addEventListener
|
||||
? window.addEventListener('load', sortTable.init, false)
|
||||
: window.attachEvent && window.attachEvent('onload', sortTable.init)
|
||||
;
|
||||
|
||||
// Shim for IE11's lack of NodeList.prototype.forEach
|
||||
if (typeof NodeList.prototype.forEach !== "function") {
|
||||
NodeList.prototype.forEach = Array.prototype.forEach;
|
||||
}
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
import sys, os
|
||||
from math import ceil, floor, sqrt, log2
|
||||
from collections import defaultdict, deque
|
||||
input = lambda: sys.stdin.readline().strip()
|
||||
intlist = lambda: [int(i) for i in input().split()]
|
||||
flolist = lambda: [float(f) for f in input().split()]
|
||||
mat = lambda a, b, v: [[v] * b for _ in range(a)]
|
||||
from rich import print
|
||||
import json
|
||||
import re
|
||||
|
||||
def remove_html_tags(text):
|
||||
"""Remove html tags from a string"""
|
||||
clean = re.compile('<.*?>')
|
||||
return re.sub(clean, '', text)
|
||||
|
||||
def debold(ls):
|
||||
for i in range(len(ls)):
|
||||
ls[i] = ls[i].replace("**", "")
|
||||
return ls
|
||||
|
||||
def process_sources(s):
|
||||
# [source](link) to {Source: source, Link: link}
|
||||
s = s.split("](")
|
||||
source = s[0][1:]
|
||||
link = s[1][:-1]
|
||||
return {"Source": source, "Link": link}
|
||||
|
||||
# read leaderboard.md table and create a list of dictionaries
|
||||
with open("leaderboard.md", "r") as f:
|
||||
header = f.readline().strip().split("|")
|
||||
header = [h.strip() for h in header]
|
||||
header = header[1:-1]
|
||||
debold(header)
|
||||
header = [remove_html_tags(h) for h in header]
|
||||
lines = f.readlines()
|
||||
lines = [l.strip().split("|") for l in lines]
|
||||
lines = [[e.strip() for e in l] for l in lines]
|
||||
lines = [[remove_html_tags(e) for e in l] for l in lines]
|
||||
lines = [l[1:-1] for l in lines]
|
||||
lines = [debold(l) for l in lines]
|
||||
lines = lines[1:]
|
||||
lines = [dict(zip(header, l)) for l in lines]
|
||||
|
||||
for ln in lines:
|
||||
ln["Source"] = process_sources(ln["Source"])
|
||||
|
||||
with open("leaderboard-data.js", "w") as f:
|
||||
f.write("leaderboard = ")
|
||||
json.dump(lines, f, indent=4)
|
||||
|
||||
|
||||
|
|
@ -1,205 +0,0 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def print_result(result):\n",
|
||||
" nums = result.split(\" \")\n",
|
||||
" for num in nums:\n",
|
||||
" print(f\"<td>{num}</td>\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>40.28</td>\n",
|
||||
"<td>46.13</td>\n",
|
||||
"<td>29.25</td>\n",
|
||||
"<td>47.45</td>\n",
|
||||
"<td>40.08</td>\n",
|
||||
"<td>33.66</td>\n",
|
||||
"<td>39.35</td>\n",
|
||||
"<td>40.67</td>\n",
|
||||
"<td>39.83</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Random chance\n",
|
||||
"result = \"40.28 46.13 29.25 47.45 40.08 33.66 39.35 40.67 39.83\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>90.23</td>\n",
|
||||
"<td>84.97</td>\n",
|
||||
"<td>87.48</td>\n",
|
||||
"<td>89.60</td>\n",
|
||||
"<td>87.50</td>\n",
|
||||
"<td>88.10</td>\n",
|
||||
"<td>91.59</td>\n",
|
||||
"<td>82.42</td>\n",
|
||||
"<td>88.40</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Human\n",
|
||||
"result = \"90.23 84.97 87.48 89.60 87.50 88.10 91.59 82.42 88.40\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>75.04</td>\n",
|
||||
"<td>66.59</td>\n",
|
||||
"<td>78.00</td>\n",
|
||||
"<td>74.24</td>\n",
|
||||
"<td>65.74</td>\n",
|
||||
"<td>79.58</td>\n",
|
||||
"<td>76.36</td>\n",
|
||||
"<td>69.87</td>\n",
|
||||
"<td>74.04</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# GPT-3 zero-shot QCM→A\n",
|
||||
"result = \"75.04 66.59 78.00 74.24 65.74 79.58 76.36 69.87 74.04\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>74.64</td>\n",
|
||||
"<td>69.74</td>\n",
|
||||
"<td>76.00</td>\n",
|
||||
"<td>74.44</td>\n",
|
||||
"<td>67.28</td>\n",
|
||||
"<td>77.42</td>\n",
|
||||
"<td>76.80</td>\n",
|
||||
"<td>68.89</td>\n",
|
||||
"<td>73.97</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# GPT-3 2-shot QCM→A\n",
|
||||
"result = \"74.64 69.74 76.00 74.44 67.28 77.42 76.80 68.89 73.97\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>76.60</td>\n",
|
||||
"<td>65.92</td>\n",
|
||||
"<td>77.55</td>\n",
|
||||
"<td>75.51</td>\n",
|
||||
"<td>66.09</td>\n",
|
||||
"<td>79.58</td>\n",
|
||||
"<td>78.49</td>\n",
|
||||
"<td>67.63</td>\n",
|
||||
"<td>74.61</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# GPT-3 (CoT) 2-shot QCM→AE\n",
|
||||
"result = \"76.60 65.92 77.55 75.51 66.09 79.58 78.49 67.63 74.61\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>75.44</td>\n",
|
||||
"<td>70.87</td>\n",
|
||||
"<td>78.09</td>\n",
|
||||
"<td>74.68</td>\n",
|
||||
"<td>67.43</td>\n",
|
||||
"<td>79.93</td>\n",
|
||||
"<td>78.23</td>\n",
|
||||
"<td>69.68</td>\n",
|
||||
"<td>75.17</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# GPT-3 (CoT) 2-shot QCM→ALE\n",
|
||||
"result = \"75.44 70.87 78.09 74.68 67.43 79.93 78.23 69.68 75.17\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.9.1rc1 64-bit",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,83 +0,0 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Leaderboard | SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models</title>
|
||||
<link rel="icon" href="img/logo.png" type="image/icon type">
|
||||
<link rel="stylesheet" href="css/main.css">
|
||||
<link rel="stylesheet" href="css/nav.css">
|
||||
<link rel="stylesheet" href="css/leaderboard.css">
|
||||
<script type="text/javascript" src="javascript/leaderboard/sort-table.js" defer></script>
|
||||
<script type="text/javascript" src="javascript/leaderboard/leaderboard-data.js" defer></script>
|
||||
<script type="text/javascript" src="javascript/leaderboard/leaderboard.js" defer></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="nav">
|
||||
<div id="icon">
|
||||
<img src="img/logo.png" id="nav-icon">
|
||||
<a class="nav-button" href="https://XuanwuAI.github.io/SecEval/"
|
||||
style="margin-left: 2px; font-size: 24px">SecEval
|
||||
</a>
|
||||
</div>
|
||||
<div>
|
||||
<a class="nav-button" href="index.html#home">Home</a>
|
||||
<a class="nav-button" href="index.html#dataset">Dataset</a>
|
||||
<a class="nav-button" href="#paper_placeholder.html" target="_blank">Paper</a>
|
||||
<a class="nav-button" href="https://github.com/XuanwuAI/SecEval" target="_blank">Github</a>
|
||||
<a class="nav-button" href="index.html#citation">Citation</a>
|
||||
<a class="nav-button" href="index.html#contact">Contact</a>
|
||||
<a class="nav-button" href="leaderboard.html">Leaderboard</a>
|
||||
<a class="nav-button" href="explore.html">Explore</a>
|
||||
</div>
|
||||
</div>
|
||||
<div id="body">
|
||||
<br><br><br><br>
|
||||
<div class="section">
|
||||
<h1>Leaderboard - SecEval</h1>
|
||||
<p>
|
||||
Evaluation results of different models.
|
||||
</p>
|
||||
<!--
|
||||
<p>
|
||||
Please fill in this
|
||||
<a href="https://forms.gle/iwEMepqowMkknNvb8" class="ext-link">Google Form
|
||||
</a> to submit your evaluation results.
|
||||
</p>
|
||||
-->
|
||||
|
||||
<!-- https://www.cssscript.com/sort-table-header-column/ -->
|
||||
<table class="js-sort-table" id="results">
|
||||
<thead>
|
||||
<tr>
|
||||
<td class="js-sort-number"><strong>#</strong></td>
|
||||
<td class="js-sort-number"><strong>Model</strong></td>
|
||||
<!-- <td class="js-sort-number"><strong>Learning</strong></td> -->
|
||||
<td class="js-sort-number"><strong>Creator</strong></td>
|
||||
<td class="js-sort-number"><strong>Access</strong></td>
|
||||
<td class="js-sort-number"><strong>Submission Date</strong></td>
|
||||
<td class="js-sort-number"><strong>Network Security</strong></td>
|
||||
<td class="js-sort-number"><strong><u>Vulnerability</u></strong></td>
|
||||
<td class="js-sort-number"><strong>Memory Safety</strong></td>
|
||||
<td class="js-sort-number"><strong>Web Security</strong></td>
|
||||
<td class="js-sort-number"><strong>Application Security</strong></td>
|
||||
<td class="js-sort-number"><strong>Cryptography</strong></td>
|
||||
<td class="js-sort-number"><strong>System Security</strong></td>
|
||||
<td class="js-sort-number"><strong>Software Security</strong></td>
|
||||
<td class="js-sort-number"><strong>PenTest</strong></td>
|
||||
<td class="js-sort-number"><strong>Overall</strong></td>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
</tbody>
|
||||
|
||||
</table>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
|
@ -1 +0,0 @@
|
|||
Coming Soon
|
||||
|
|
@ -1,107 +0,0 @@
|
|||
# Evaluating on SecEval Dataset
|
||||
|
||||
## Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Download Dataset
|
||||
|
||||
You can download the json file of the dataset by running.
|
||||
|
||||
```
|
||||
wget https://huggingface.co/datasets/XuanwuAI/SecEval/blob/main/questions.json
|
||||
```
|
||||
|
||||
## Evaluate on a Model
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
usage: eval.py [-h] [-o OUTPUT_DIR] -d DATASET_FILE [-c] [-b BATCH_SIZE] -B {remote_hf,azure,textgen,local_hf} -m MODELS [MODELS ...]
|
||||
|
||||
SecEval Evaluation CLI
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
-o OUTPUT_DIR, --output_dir OUTPUT_DIR
|
||||
Specify the output directory.
|
||||
-d DATASET_FILE, --dataset_file DATASET_FILE
|
||||
Specify the dataset file to evaluate on.
|
||||
-c, --chat Evaluate on chat model.
|
||||
-b BATCH_SIZE, --batch_size BATCH_SIZE
|
||||
Specify the batch size.
|
||||
-B {remote_hf,azure,textgen,local_hf}, --backend {remote_hf,azure,textgen,local_hf}
|
||||
Specify the llm type. remote_hf: remote huggingface model backed, azure: azure openai model, textgen: textgen backend, local_hf: local huggingface model backed
|
||||
-m MODELS [MODELS ...], --models MODELS [MODELS ...]
|
||||
Specify the models.
|
||||
```
|
||||
|
||||
### Select a Backend
|
||||
|
||||
There are four backends supported for evaluating on SecEval dataset.
|
||||
Some backends require additional environment variables to be set.
|
||||
|
||||
#### Local Huggingface Model Backend
|
||||
|
||||
Local Huggingface Model Backend is used for evaluating on models that are stored locally. You can download the models from [Huggingface](https://huggingface.co/models) and use this backend to evalute the model.
|
||||
|
||||
```bash
|
||||
export HF_LOCAL_MODEL_DIR=<path to local huggingface model>
|
||||
```
|
||||
|
||||
#### Remote Huggingface Model Backend
|
||||
|
||||
Remote Huggingface Model Backend is used for evaluating on models that are stored on Huggingface. You can use this backend to evalute the model.
|
||||
|
||||
|
||||
#### Azure OpenAI Model Backend
|
||||
|
||||
Azure OpenAI Model Backend is used for evaluating on models that are deployed on Azure OpenAI.
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=<your openai api key>
|
||||
export OPENAI_API_ENDPOINT=<your openai api endpoint>
|
||||
```
|
||||
|
||||
|
||||
#### text-generation-webui Backend
|
||||
|
||||
[text-generation-webui](https://github.com/oobabooga/text-generation-webui) is a service used for deploying and testing models. We have supported evaluating on models deployed on text-generation-webui.
|
||||
|
||||
```bash
|
||||
|
||||
export TEXTGEN_MODEL_URL=<your textgenerationwebui api url>
|
||||
```
|
||||
|
||||
### Run Evaluation
|
||||
|
||||
If you want to evaluate on a base model, you can run the following command:
|
||||
|
||||
```bash
|
||||
python eval.py -d <path to dataset file> -B <backend> -m <model name> -o <output directory>
|
||||
```
|
||||
|
||||
If you want to evaluate on a chat model such as gpt-3.5-turbo, you should add the `-c` flag:
|
||||
|
||||
```bash
|
||||
python eval.py -d <path to dataset file> -B <backend> -m <model name> -o <output directory> -c
|
||||
```
|
||||
|
||||
|
||||
## Inspection the Results
|
||||
|
||||
The results will be saved in the output directory. The result is a JSONObject contains following field
|
||||
|
||||
```json
|
||||
{
|
||||
"score_fraction": {
|
||||
"topic_name": "topic_score_fraction",
|
||||
},
|
||||
"score_float": {
|
||||
"topic_name": "topic_score_float",
|
||||
},
|
||||
"detail": ["detail1", "detail2",]
|
||||
}
|
||||
```
|
||||
|
|
@ -1,414 +0,0 @@
|
|||
"""
|
||||
SecEval Evaluation Script
|
||||
|
||||
This script evaluates language models on cybersecurity multiple-choice questions.
|
||||
It supports various LLM backends including HuggingFace, Azure OpenAI, TextGen, and Ollama.
|
||||
The script processes questions in batches and calculates accuracy scores by topic.
|
||||
|
||||
Usage:
|
||||
python3 eval.py -d dataset.json -B backend_type -m model_name [options]
|
||||
Example:
|
||||
python3 eval.py --dataset_file datasets/questions.json --output_dir outputs --backend ollama --model ollama/qwen2.5:14b
|
||||
|
||||
Environment Variables:
|
||||
- OPENAI_API_ENDPOINT: Azure OpenAI endpoint URL
|
||||
- OPENAI_API_KEY: Azure OpenAI API key
|
||||
- TEXTGEN_MODEL_URL: URL for TextGen model
|
||||
- LOCAL_HF_MODEL_DIR: Directory containing local HuggingFace models
|
||||
- OLLAMA_API_BASE: Base URL for Ollama API (default: http://localhost:8000)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
from typing import Any, Dict, List
|
||||
from dotenv import load_dotenv
|
||||
import asyncio
|
||||
load_dotenv()
|
||||
from langchain.llms.huggingface_pipeline import HuggingFacePipeline
|
||||
from langchain.chat_models import AzureChatOpenAI
|
||||
from langchain.llms.textgen import TextGen
|
||||
from langchain.schema.language_model import BaseLanguageModel
|
||||
from langchain.adapters.openai import convert_message_to_dict
|
||||
from langchain.schema.messages import (
|
||||
AIMessage,
|
||||
SystemMessage,
|
||||
HumanMessage,
|
||||
)
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
import os
|
||||
import logging
|
||||
|
||||
# Configure logging
|
||||
logger = logging.getLogger(__name__)
|
||||
import time
|
||||
from langchain.globals import set_llm_cache
|
||||
from langchain.cache import SQLiteCache
|
||||
import litellm
|
||||
|
||||
# Set up caching for LLM responses
|
||||
set_llm_cache(
|
||||
SQLiteCache(
|
||||
database_path=str(Path(__file__).parent.parent / ".langchain.db")
|
||||
)
|
||||
)
|
||||
|
||||
# Configure logging settings
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
logger.addHandler(logging.FileHandler(f"./eval-{int(time.time())}.log", "w"))
|
||||
|
||||
# Instruction template for the LLM
|
||||
instruction = """Below are multiple-choice questions concerning cybersecurity.
|
||||
Please select the correct answers and respond with the correct letters A, B, C, or D.
|
||||
You could select more than one letter.
|
||||
"""
|
||||
|
||||
|
||||
def init_hf_llm(model_id: str) -> HuggingFacePipeline:
|
||||
"""
|
||||
Initialize a HuggingFace language model.
|
||||
|
||||
Args:
|
||||
model_id: The model identifier from HuggingFace
|
||||
|
||||
Returns:
|
||||
HuggingFacePipeline: Initialized model pipeline
|
||||
|
||||
Raises:
|
||||
ImportError: If required dependencies are not installed
|
||||
"""
|
||||
# Check transformers and torch installation
|
||||
try:
|
||||
import transformers
|
||||
except ImportError:
|
||||
raise ImportError("Please install transformers with `pip install transformers`")
|
||||
try:
|
||||
import torch
|
||||
flash_attn_enable = torch.cuda.get_device_capability()[0] >= 8
|
||||
except ImportError:
|
||||
raise ImportError("Please install torch with `pip install torch`")
|
||||
|
||||
# Initialize HuggingFace pipeline with specified parameters
|
||||
llm = HuggingFacePipeline.from_model_id(
|
||||
model_id=model_id,
|
||||
task="text-generation",
|
||||
pipeline_kwargs={"max_new_tokens": 5},
|
||||
device=0,
|
||||
model_kwargs={"trust_remote_code": True, "torch_dtype": torch.bfloat16},
|
||||
)
|
||||
return llm
|
||||
|
||||
|
||||
def init_textgen_llm(model_id: str) -> TextGen:
|
||||
"""
|
||||
Initialize a TextGen language model.
|
||||
|
||||
Args:
|
||||
model_id: The model identifier
|
||||
|
||||
Returns:
|
||||
TextGen: Initialized model
|
||||
|
||||
Raises:
|
||||
RuntimeError: If TEXTGEN_MODEL_URL is not set
|
||||
"""
|
||||
# Check for required environment variable
|
||||
if os.environ.get("TEXTGEN_MODEL_URL") is None:
|
||||
raise RuntimeError("Please set TEXTGEN_MODEL_URL")
|
||||
llm = TextGen(model_url=os.environ["TEXTGEN_MODEL_URL"]) # type: ignore
|
||||
return llm
|
||||
|
||||
|
||||
def init_azure_openai_llm(model_id: str) -> AzureChatOpenAI:
|
||||
"""
|
||||
Initialize an Azure OpenAI language model.
|
||||
|
||||
Args:
|
||||
model_id: The model identifier
|
||||
|
||||
Returns:
|
||||
AzureChatOpenAI: Initialized model
|
||||
|
||||
Raises:
|
||||
RuntimeError: If required environment variables are not set
|
||||
"""
|
||||
if os.environ.get("OPENAI_API_ENDPOINT") is None:
|
||||
raise RuntimeError("Please set OPENAI_API_ENDPOINT")
|
||||
if os.environ.get("OPENAI_API_KEY") is None:
|
||||
raise RuntimeError("Please set OPENAI_API_KEY")
|
||||
|
||||
# Configure Azure OpenAI parameters
|
||||
azure_params = {
|
||||
"model": model_id,
|
||||
"openai_api_base": os.environ["OPENAI_API_ENDPOINT"],
|
||||
"openai_api_key": os.environ["OPENAI_API_KEY"],
|
||||
"openai_api_type": os.environ.get("OPENAI_API_TYPE", "azure"),
|
||||
"openai_api_version": "2023-07-01-preview",
|
||||
}
|
||||
return AzureChatOpenAI(**azure_params) # type: ignore
|
||||
|
||||
|
||||
def init_ollama_llm(model_id: str) -> 'OllamaChat':
|
||||
"""
|
||||
Initialize an Ollama language model.
|
||||
|
||||
Args:
|
||||
model_id: The model identifier
|
||||
|
||||
Returns:
|
||||
OllamaChat: Initialized model wrapper
|
||||
"""
|
||||
class OllamaChat:
|
||||
async def abatch(self, prompts: List[str]) -> List[str]:
|
||||
responses = []
|
||||
for prompt in prompts:
|
||||
try:
|
||||
ollama_api_base = os.getenv("OLLAMA_API_BASE", "http://localhost:8000")
|
||||
api_base = ollama_api_base.rstrip('/v1')
|
||||
completion = litellm.completion(
|
||||
model=model_id,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
api_base=api_base,
|
||||
custom_llm_provider="ollama"
|
||||
)
|
||||
if hasattr(completion, "choices") and completion.choices:
|
||||
content = completion.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
responses.append(result)
|
||||
else:
|
||||
print("Incorrect answer format detected.")
|
||||
responses.append("Error: No result parsed")
|
||||
except Exception as e:
|
||||
logging.error(f"Ollama error: {e}")
|
||||
responses.append(f"Error: {e}")
|
||||
return responses
|
||||
|
||||
def extract_answer(self, text: str) -> str:
|
||||
match = re.findall(r"[A-D]", text.upper())
|
||||
return "".join(sorted(set(match))) if match else ""
|
||||
|
||||
return OllamaChat()
|
||||
|
||||
|
||||
def init_openrouter_llm(model_id: str):
|
||||
class OpenRouterChat:
|
||||
async def abatch(self, prompts: List[str]):
|
||||
responses = []
|
||||
for prompt in prompts:
|
||||
try:
|
||||
api_base = os.getenv("OPENROUTER_API_BASE", "https://openrouter.ai/api/v1/chat/completions")
|
||||
api_key = os.getenv("OPENROUTER_API_KEY")
|
||||
|
||||
if not api_key:
|
||||
raise ValueError("OPENROUTER_API_KEY is not defined in the environment variables.")
|
||||
|
||||
completion = litellm.completion(
|
||||
model=model_id,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
api_base=api_base,
|
||||
api_key=api_key,
|
||||
custom_llm_provider="openrouter"
|
||||
)
|
||||
|
||||
if hasattr(completion, "choices") and completion.choices:
|
||||
content = completion.choices[0].message.content
|
||||
result = self.extract_answer(content)
|
||||
if result:
|
||||
responses.append(result)
|
||||
else:
|
||||
print("Formato de respuesta incorrecto.")
|
||||
responses.append("Error: No se pudo extraer resultado")
|
||||
except Exception as e:
|
||||
logging.error(f"OpenRouter error: {e}")
|
||||
responses.append(f"Error: {e}")
|
||||
return responses
|
||||
|
||||
def extract_answer(self, text: str):
|
||||
match = re.findall(r"[A-D]", text.upper())
|
||||
return "".join(sorted(set(match))) if match else ""
|
||||
|
||||
return OpenRouterChat()
|
||||
|
||||
|
||||
def load_dataset(dataset_path: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Load evaluation dataset from JSON file.
|
||||
|
||||
Args:
|
||||
dataset_path: Path to dataset file
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: Loaded dataset
|
||||
"""
|
||||
with open(dataset_path, "r") as f:
|
||||
dataset = json.load(f)
|
||||
return dataset
|
||||
|
||||
|
||||
async def batch_inference_dataset(llm: BaseLanguageModel, batch: List[Dict[str, Any]], chat: bool = False) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Process a batch of questions through the language model.
|
||||
|
||||
Args:
|
||||
llm: Language model to use
|
||||
batch: List of questions to process
|
||||
chat: Whether to use chat format
|
||||
|
||||
Returns:
|
||||
List[Dict[str, Any]]: Processed results with scores
|
||||
"""
|
||||
results = []
|
||||
llm_inputs = []
|
||||
for dataset_row in batch:
|
||||
question_text = (
|
||||
"Question: " + dataset_row["question"] + " ".join(dataset_row["choices"])
|
||||
)
|
||||
question_text = question_text.replace("\n", " ")
|
||||
if chat:
|
||||
llm_input = [SystemMessage(content=instruction)]
|
||||
else:
|
||||
llm_input = instruction + "\n"
|
||||
|
||||
llm_inputs.append(llm_input)
|
||||
try:
|
||||
llm_outputs = await llm.abatch(llm_inputs)
|
||||
except Exception as e:
|
||||
logging.error(f"error in processing batch {e}")
|
||||
llm_outputs = [f"{e}" * len(llm_inputs)]
|
||||
for idx, llm_output in enumerate(llm_outputs):
|
||||
if type(llm_output) == AIMessage:
|
||||
llm_output: str = llm_output.content # type: ignore
|
||||
if "Answer:" in llm_output:
|
||||
llm_output = llm_output.replace("Answer:", "")
|
||||
if chat:
|
||||
batch[idx]["llm_input"] = convert_message_to_dict(llm_inputs[idx])
|
||||
else:
|
||||
batch[idx]["llm_input"] = llm_inputs[idx]
|
||||
batch[idx]["llm_output"] = llm_output
|
||||
batch[idx]["llm_answer"] = "".join(
|
||||
sorted(list(set(re.findall(r"[A-D]", llm_output))))
|
||||
)
|
||||
batch[idx]["score"] = int(
|
||||
batch[idx]["llm_answer"].lower() == batch[idx]["answer"].lower()
|
||||
)
|
||||
logging.info(
|
||||
f'llm_output: {llm_output}, parsed answer: {batch[idx]["llm_answer"]}, answer: {batch[idx]["answer"]}'
|
||||
)
|
||||
|
||||
print("Question:", batch[idx]["question"])
|
||||
print("Correct Answer:", batch[idx]["answer"])
|
||||
print("LLM Answer:", batch[idx]["llm_answer"])
|
||||
print("LLM Output:", llm_output)
|
||||
print("Score:", batch[idx]["score"])
|
||||
print("--------------------------------")
|
||||
|
||||
results.append(batch[idx])
|
||||
return results
|
||||
|
||||
|
||||
def inference_dataset(
|
||||
llm: BaseLanguageModel,
|
||||
dataset: List[Dict[str, Any]],
|
||||
batch_size: int = 1,
|
||||
chat: bool = False,
|
||||
):
|
||||
# Prepare the batched inference
|
||||
def chunks(lst, n):
|
||||
for i in range(0, len(lst), n):
|
||||
yield lst[i : i + n]
|
||||
|
||||
# Asynchronously process dataset in batches
|
||||
loop = asyncio.get_event_loop()
|
||||
batches = list(chunks(dataset, batch_size))
|
||||
results = []
|
||||
for idx, batch in enumerate(batches):
|
||||
logger.info(f"processing batch {idx+1}/{len(batches)}")
|
||||
results += loop.run_until_complete(batch_inference_dataset(llm, batch, chat))
|
||||
return results
|
||||
|
||||
|
||||
def count_score_by_topic(dataset: List[Dict[str, Any]]):
|
||||
score_by_topic = {}
|
||||
total_score_by_topic = {}
|
||||
score = 0
|
||||
for dataset_row in dataset:
|
||||
for topic in dataset_row["topics"]:
|
||||
if topic not in score_by_topic:
|
||||
score_by_topic[topic] = 0
|
||||
total_score_by_topic[topic] = 0
|
||||
score_by_topic[topic] += dataset_row["score"]
|
||||
total_score_by_topic[topic] += 1
|
||||
score += dataset_row["score"]
|
||||
score_fraction = {
|
||||
k: f"{v}/{total_score_by_topic[k]}" for k, v in score_by_topic.items()
|
||||
}
|
||||
score_float = {
|
||||
k: round(100 * float(v) / float(total_score_by_topic[k]), 4)
|
||||
for k, v in score_by_topic.items()
|
||||
}
|
||||
score_float["Overall"] = round(100 * float(score) / float(len(dataset)), 4)
|
||||
score_fraction["Overall"] = f"{score}/{len(dataset)}"
|
||||
return score_fraction, score_float
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="SecEval Evaluation CLI")
|
||||
|
||||
parser.add_argument("-o", "--output_dir", type=str, default="/tmp", help="Specify the output directory.")
|
||||
parser.add_argument("-d", "--dataset_file", type=str, required=True, help="Specify the dataset file to evaluate on.")
|
||||
parser.add_argument("-c", "--chat", action="store_true", default=False, help="Evaluate on chat model.")
|
||||
parser.add_argument("-b", "--batch_size", type=int, default=1, help="Specify the batch size.")
|
||||
parser.add_argument("-B", "--backend", type=str, choices=["remote_hf", "azure", "textgen", "local_hf", "ollama", "openrouter"], required=True, help="Specify the llm type. remote_hf: remote huggingface model backed, azure: azure openai model, textgen: textgen backend, local_hf: local huggingface model backed, ollama: ollama model, openrouter: openrouter model")
|
||||
parser.add_argument("-m", "--models", type=str, nargs="+", required=True, help="Specify the models.")
|
||||
|
||||
args = parser.parse_args()
|
||||
models = list(args.models)
|
||||
|
||||
logging.info(f"Evaluating models: {models}")
|
||||
for model_id in models:
|
||||
if args.backend == "remote_hf":
|
||||
llm = init_hf_llm(model_id)
|
||||
elif args.backend == "local_hf":
|
||||
model_dir = os.environ.get("LOCAL_HF_MODEL_DIR")
|
||||
if model_dir is None:
|
||||
raise RuntimeError(
|
||||
"Please set LOCAL_HF_MODEL_DIR when using local_hf backend"
|
||||
)
|
||||
model_id = os.path.join(model_dir, model_id)
|
||||
llm = init_hf_llm(model_id)
|
||||
elif args.backend == "textgen":
|
||||
llm = init_textgen_llm(model_id)
|
||||
elif args.backend == "azure":
|
||||
llm = init_azure_openai_llm(model_id)
|
||||
elif args.backend == "ollama":
|
||||
llm = init_ollama_llm(model_id)
|
||||
elif args.backend == "openrouter":
|
||||
llm = init_openrouter_llm(model_id)
|
||||
else:
|
||||
raise RuntimeError("Unknown backend")
|
||||
|
||||
dataset = load_dataset(args.dataset_file)
|
||||
result = inference_dataset(llm, dataset, batch_size=args.batch_size, chat=args.chat)
|
||||
score_fraction, score_float = count_score_by_topic(result)
|
||||
|
||||
result_with_score = {
|
||||
"score_fraction": score_fraction,
|
||||
"score_float": score_float,
|
||||
"detail": result,
|
||||
}
|
||||
# Create output directory if it doesn't exist
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = output_dir / f"{Path(args.dataset_file).stem}_{os.path.basename(model_id)}.json"
|
||||
|
||||
logger.info(f"Writing result to {output_path}")
|
||||
with open(output_path, "w") as f:
|
||||
json.dump(result_with_score, f, indent=4)
|
||||
del llm
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
transformers==4.36.*
|
||||
langchain
|
||||
einops
|
||||
protobuf
|
||||
scipy
|
||||
sentencepiece
|
||||
torch
|
||||
python-dotenv
|
||||
transformers_stream_generator
|
||||
tiktoken
|
||||
litellm
|
||||
langchain-community
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"questions": [
|
||||
{
|
||||
"question": "Which of the following is a desirable property of a biometric system?",
|
||||
"answers": {
|
||||
"A": "Permanent",
|
||||
"B": "Transferability",
|
||||
"C": "Uniformity",
|
||||
"D": "Forgiveness"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "In TCP/IP networking, which protocol is used to hold network addresses and routing information in a packet?",
|
||||
"answers": {
|
||||
"A": "HTTP",
|
||||
"B": "IP",
|
||||
"C": "Routing Information Protocol (RIP)",
|
||||
"D": "TCP"
|
||||
},
|
||||
"solution": "B"
|
||||
}
|
||||
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,811 @@
|
|||
{
|
||||
"questions": [
|
||||
|
||||
{
|
||||
"question": "What is the primary requirement for an Random Bit Generator's (RBG) output to be used for generating cryptographic keys?",
|
||||
"answers": {
|
||||
"A": "The RBG's output should have a length that matches the target data to be protected",
|
||||
"B": "The RBG's output should be computationally indistinguishable from random bits and provide sufficient entropy to support the security strength required for the target data",
|
||||
"C": "The RBG's output should be as long as possible to ensure maximal randomness",
|
||||
"D": "The RBG's output should have precisely the same length as the symmetric key to be generated"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "In cryptography, what is the purpose of using a key-derivation function (KDF)?",
|
||||
"answers": {
|
||||
"A": "Generate public keys",
|
||||
"B": "Authenticate digital signatures",
|
||||
"C": "Encrypt data using a password",
|
||||
"D": "KDF are algorithms used to transform a secret into crucial parameters like keys and Initialization Vectors (IVs)"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of segmentation in the context of PCI DSS?",
|
||||
"answers": {
|
||||
"A": "To reduce the number of PCI DSS requirements applicable to an entity",
|
||||
"B": "To limit the scope of the PCI DSS assessment and minimize the potential for security breaches",
|
||||
"C": "To completely remove the applicability of PCI DSS for a merchant's cardholder data environment",
|
||||
"D": "To eliminate the need for implementing PCI DSS controls"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of using strong cryptography to protect cardholder data during transmission over open, public networks?",
|
||||
"answers": {
|
||||
"A": "To ensure the data is secured from unauthorized access or interception during transmission",
|
||||
"B": "To increase the speed of data transmission over open, public networks",
|
||||
"C": "To provide access to authorized individuals to monitor the transmission process",
|
||||
"D": "To improve the efficiency of network routing protocols"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of periodic reviews of user accounts and access privileges?",
|
||||
"answers": {
|
||||
"A": "To ensure all user accounts have access to the highest privileges",
|
||||
"B": "To demonstrate adherence to regulatory requirements",
|
||||
"C": "To create a record of all user accounts",
|
||||
"D": "To identify and remove any inappropriate access and privileges"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a purpose of having synchronized system clocks?",
|
||||
"answers": {
|
||||
"A": "To compare log files from different systems",
|
||||
"B": "To accelerate system performance",
|
||||
"C": "To standardize file naming conventions",
|
||||
"D": "To reduce power consumption"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is physical security?",
|
||||
"answers": {
|
||||
"A": "The protection of data from hacker attacks",
|
||||
"B": "The prevention of natural disasters caused by environmental factors",
|
||||
"C": "The act of securing a building or location to prevent unauthorized access, theft, or damage to physical assets",
|
||||
"D": "The enforcement of technical security controls to prevent data breaches"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What does the abbreviation 'SDK' stand for in the context of software security?",
|
||||
"answers": {
|
||||
"A": "Software Development Kit",
|
||||
"B": "System Deflection Key",
|
||||
"C": "Software Delegation Kernel",
|
||||
"D": "System Development Key"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the main purpose of multi-factor authentication in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To limit user privileges within the network",
|
||||
"B": "To ensure compliance with government regulations",
|
||||
"C": "To adds an extra layer of security by requiring multiple forms of verification",
|
||||
"D": "To monitor network traffic for security threats"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary objective of a Web Application Firewall (WAF) in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To encrypt network traffic between web servers",
|
||||
"B": "To prevent unauthorized access to network services",
|
||||
"C": "To protect web applications from security threats",
|
||||
"D": "To manage user authentication and authorization for web services"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of periodically changing passwords or passphrases?",
|
||||
"answers": {
|
||||
"A": "To provide more time for a malicious individual to crack the password/passphrase",
|
||||
"B": "To allow for more rapid detection and response to address potentially compromised credentials",
|
||||
"C": "To prevent unauthorized users from gaining access to the user account through use of a shared authentication factor",
|
||||
"D": "To offer less time for a malicious individual to crack a password/passphrase and less time to use a compromised password"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following defines the entity’s security objectives and principles?",
|
||||
"answers": {
|
||||
"A": "Security policies",
|
||||
"B": "Operational procedures",
|
||||
"C": "Risk assessment framework",
|
||||
"D": "Incident response plan"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the focus of a security audit or vulnerability assessment?",
|
||||
"answers": {
|
||||
"A": "Locating vulnerabilities",
|
||||
"B": "Locating threats",
|
||||
"C": "Enacting threats",
|
||||
"D": "Exploiting vulnerabilities"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a mechanism for managing digital certificates through a system of trust?",
|
||||
"answers": {
|
||||
"A": "PKI",
|
||||
"B": "PKCS",
|
||||
"C": "ISA",
|
||||
"D": "SSL"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which protocol is used to create a secure environment in a wireless network?",
|
||||
"answers": {
|
||||
"A": "WAP",
|
||||
"B": "WPA2",
|
||||
"C": "WTLS",
|
||||
"D": "WML3"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which design concept limits access to systems from outside users while protecting users and systems inside the LAN?",
|
||||
"answers": {
|
||||
"A": "DMZ",
|
||||
"B": "VLAN",
|
||||
"C": "I&A",
|
||||
"D": "Router"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the process of making an operating system secure from attack called?",
|
||||
"answers": {
|
||||
"A": "Hardening",
|
||||
"B": "Tuning",
|
||||
"C": "Sealing",
|
||||
"D": "Locking down"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which mechanism is used by PKI to allow immediate verification of a certificate's validity?",
|
||||
"answers": {
|
||||
"A": "CRL",
|
||||
"B": "MD5",
|
||||
"C": "SSHA",
|
||||
"D": "OCSP"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a major security problem with FTP?",
|
||||
"answers": {
|
||||
"A": "Password files are stored in an unsecure area on disk",
|
||||
"B": "Memory traces can corrupt file access",
|
||||
"C": "User IDs and passwords are unencrypted",
|
||||
"D": "FTP sites are unregistered"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following creates a fixed-length output from a variable-length input?",
|
||||
"answers": {
|
||||
"A": "MD5",
|
||||
"B": "SHA1",
|
||||
"C": "SHA3",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of a service-level agreement (SLA) in the context of business continuity?",
|
||||
"answers": {
|
||||
"A": "To define the obligations of the service provider to the client",
|
||||
"B": "To provide high-availability architecture",
|
||||
"C": "To outline a disaster recovery plan",
|
||||
"D": "To ensure the fault tolerance of systems"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What does the Address Resolution Protocol (ARP) do in a network?",
|
||||
"answers": {
|
||||
"A": "Resolves domain names to IP addresses",
|
||||
"B": "Allocates IP addresses to network devices",
|
||||
"C": "Translates IP addresses to media access control (MAC) addresses",
|
||||
"D": "Verifies that a host is reachable"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "In the OSI model, which layer consists of the physical media and infrastructure of networks?",
|
||||
"answers": {
|
||||
"A": "Session",
|
||||
"B": "Presentation",
|
||||
"C": "Data Link",
|
||||
"D": "Physical"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "In binary representation, what is the result of the bitwise XOR operation between 110101 and 101010",
|
||||
"answers": {
|
||||
"A": "011111",
|
||||
"B": "100111",
|
||||
"C": "0",
|
||||
"D": "111111"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the proper sequence of the TCP three-way-handshake?",
|
||||
"answers": {
|
||||
"A": "SYN-ACK, ACK, ACK",
|
||||
"B": "SYN, SYN-ACK, ACK",
|
||||
"C": "SYN-SYN, SYN-ACK, SYN",
|
||||
"D": "ACK, SYN-ACK, SYN"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which topology has built-in redundancy because of its many client connections?",
|
||||
"answers": {
|
||||
"A": "Token ring",
|
||||
"B": "Bus",
|
||||
"C": "Hybrid",
|
||||
"D": "Mesh"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "How can Google hacking be valuable in a cybersecurity investigation?",
|
||||
"answers": {
|
||||
"A": "To create fake Google search results",
|
||||
"B": "To extract data from Google's databases",
|
||||
"C": "To reveal hidden or sensitive information from Google search results",
|
||||
"D": "To manipulate website content in Google's index"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the role of the three-way handshake in establishing a TCP connection?",
|
||||
"answers": {
|
||||
"A": "To request the establishment of a connection",
|
||||
"B": "To acknowledge the request for connection",
|
||||
"C": "To complete the establishment of a connection",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which best describes a vulnerability scan?",
|
||||
"answers": {
|
||||
"A": "A way to find open ports",
|
||||
"B": "A way to diagram a network",
|
||||
"C": "A proxy attack",
|
||||
"D": "A way to automate the discovery of vulnerabilities"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is Tor used for?",
|
||||
"answers": {
|
||||
"A": "To make it more difficult to trace a user's Internet activity",
|
||||
"B": "To hide the process of scanning",
|
||||
"C": "To automate scanning",
|
||||
"D": "To hide the banner on a system"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the maximum number of IP addresses in an IPv6 subnet?",
|
||||
"answers": {
|
||||
"A": "6 bit",
|
||||
"B": "32 bit",
|
||||
"C": "6*16 bit",
|
||||
"D": "8*16 bit"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is a characteristic of a worm?",
|
||||
"answers": {
|
||||
"A": "Requires user interaction to spread",
|
||||
"B": "Self-replicates and spreads independently",
|
||||
"C": "Aargets specific individuals or organizations",
|
||||
"D": "A worm is an item that runs silently"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What common tool can be used for launching an ARP poisoning attack?",
|
||||
"answers": {
|
||||
"A": "Cain & Abel",
|
||||
"B": "Nmap",
|
||||
"C": "Scooter",
|
||||
"D": "Tcpdump"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What technology is used to verify information as it is entered into a web application, preventing database manipulation and buffer overflows?",
|
||||
"answers": {
|
||||
"A": "Input Validation",
|
||||
"B": "Cross-Site Scripting (XSS)",
|
||||
"C": "Session Management",
|
||||
"D": "Buffer overflow"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which type of firewall analyzes the status of traffic?",
|
||||
"answers": {
|
||||
"A": "Circuit level",
|
||||
"B": "Packet filtering",
|
||||
"C": "Stateful inspection",
|
||||
"D": "NIDS"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of the Crossover Error Rate (CER) in evaluating biometric devices",
|
||||
"answers": {
|
||||
"A": "To compare the False Rejection Rate (FRR) of different biometric devices",
|
||||
"B": "To compare the accuracy of similar biometric devices from different vendors or different models from the same vendor",
|
||||
"C": "To assess the acceptance level of biometric factors",
|
||||
"D": "To measure the throughput rate of biometric devices"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of a padded cell system in relation to intrusion detection?",
|
||||
"answers": {
|
||||
"A": "Attract unauthorized users",
|
||||
"B": "Simulate a real network for intruders",
|
||||
"C": "Isolate detected intruders",
|
||||
"D": "Send alerts to administrators"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of penetration testing?",
|
||||
"answers": {
|
||||
"A": "To evaluate the robustness of security systems and identify vulnerabilities",
|
||||
"B": "To exploit vulnerabilities and cause damage to the system",
|
||||
"C": "To eliminate all vulnerabilities in the system",
|
||||
"D": "To simulate unauthorized attacks without actually exploiting vulnerabilities"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of RADIUS and TACACS in a network environment?",
|
||||
"answers": {
|
||||
"A": "To centralize the authentication of remote dial-up connections",
|
||||
"B": "To establish secure connections for voice and video conferencing",
|
||||
"C": "To provide security for WAN communication technologies",
|
||||
"D": "To support remote journaling and electronic vaulting"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which symmetric encryption scheme is recommended for use in SRTP?",
|
||||
"answers": {
|
||||
"A": "AES in ECB mode",
|
||||
"B": "DES in CBC mode",
|
||||
"C": "RC4",
|
||||
"D": "AES in GCM mode"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the essential requirement for the security of the Discrete Logarithm Integrated Encryption Scheme (DLIES) to be maintained?",
|
||||
"answers": {
|
||||
"A": "The order q of the base point P should be at least q >= 1024",
|
||||
"B": "Randomly choose two primes p and q such that p*q > 2048 bits",
|
||||
"C": "The length of the prime number p should be at least 3000 bits",
|
||||
"D": "All of the above"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which document provides the recommendation for elliptic curve cryptography?",
|
||||
"answers": {
|
||||
"A": "NIST SP 800-56C",
|
||||
"B": "NIST SP 800-63-3",
|
||||
"C": "NIST SP 800-57 Part 1",
|
||||
"D": "NIST SP 800-186-4"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is an essential requirement for the internal state of deterministic random number generators?",
|
||||
"answers": {
|
||||
"A": "Protection against readout and manipulation",
|
||||
"B": "Constant reseeding requirement",
|
||||
"C": "Dependence on reliable physical resources",
|
||||
"D": "Regular update of entropy sources"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which federal government agency has responsibility for ensuring the security of government computer systems that are not used to process sensitive and/or classified information?",
|
||||
"answers": {
|
||||
"A": "National Security Agency",
|
||||
"B": "Federal Bureau of Investigation",
|
||||
"C": "National Institute of Standards and Technology",
|
||||
"D": "Secret Service"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of implementing monitoring systems?",
|
||||
"answers": {
|
||||
"A": "Monitoring the configuration and configuration change of devices",
|
||||
"B": "Ensuring availability and functionality of systems",
|
||||
"C": "Recording access to information by means of system and security logging",
|
||||
"D": "Protection from compromising radiation"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which principle of cybersecurity focuses on ensuring that data is accurate and trustworthy?",
|
||||
"answers": {
|
||||
"A": "Resilience",
|
||||
"B": "Availability",
|
||||
"C": "Authenticity",
|
||||
"D": "Integrity"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of the Transport Layer Security (TLS) protocol?",
|
||||
"answers": {
|
||||
"A": "Privacy and authentication between two communicating applications",
|
||||
"B": "Privacy and data integrity between two communicating applications",
|
||||
"C": "Authentication and data integrity between two communicating applications",
|
||||
"D": "Privacy, authentication, and data integrity between two communicating applications"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "If you were to see the subnet mask 255.255.248.0, what CIDR notation (prefix) would you use to indicate the same thing?",
|
||||
"answers": {
|
||||
"A": "/23",
|
||||
"B": "/22",
|
||||
"C": "/21",
|
||||
"D": "/20"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is NOT one of the five disaster recovery plan testing types?",
|
||||
"answers": {
|
||||
"A": "Simulation",
|
||||
"B": "Checklist",
|
||||
"C": "Availability",
|
||||
"D": "Full Interruption"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What would be the result of sending the string AAAAAAAAAAAAAAAAA into a variable that has been allocated space for 8 bytes?",
|
||||
"answers": {
|
||||
"A": "Heap spraying",
|
||||
"B": "SQL injection",
|
||||
"C": "Buffer overflow",
|
||||
"D": "Slowloris attack"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is NOT a form of computer/network surveillance?",
|
||||
"answers": {
|
||||
"A": "Keyboard monitoring",
|
||||
"B": "Use of network sniffers",
|
||||
"C": "Use of Closed-Circuit Television (CCTV)",
|
||||
"D": "Review of audit logs"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "How would you calculate risk?",
|
||||
"answers": {
|
||||
"A": "Probability / loss",
|
||||
"B": "Probability * mitigation factor",
|
||||
"C": "(Loss + mitigation factor) * (loss/probability)",
|
||||
"D": "Probability * loss"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the common share name used for interprocess communication?",
|
||||
"answers": {
|
||||
"A": "C$",
|
||||
"B": "IPC$",
|
||||
"C": "ADMIN$",
|
||||
"D": "INTERCOM$"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which port is commonly used by SMB when operating over TCP without NetBIOS?",
|
||||
"answers": {
|
||||
"A": "137",
|
||||
"B": "139",
|
||||
"C": "135",
|
||||
"D": "445"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is an example of a strong password?",
|
||||
"answers": {
|
||||
"A": "Password123456",
|
||||
"B": "ME_Fgh4_al445!A",
|
||||
"C": "Qwerty!",
|
||||
"D": "123456"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which access control model would be found in a firewall?",
|
||||
"answers": {
|
||||
"A": "Mandatory access control",
|
||||
"B": "Discretionary access control",
|
||||
"C": "Role-based access control",
|
||||
"D": "Rule-based access control"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the preferred encryption option for sending large amounts of data?",
|
||||
"answers": {
|
||||
"A": "Symmetric encryption",
|
||||
"B": "Asymmetric encryption",
|
||||
"C": "Public Key Cryptography",
|
||||
"D": "Diffie-Hellman scheme"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "Which of the following is an example of a hashing algorithm?",
|
||||
"answers": {
|
||||
"A": "AES",
|
||||
"B": "RSA",
|
||||
"C": "MD5",
|
||||
"D": "3DES"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which RAID level is not fault tolerant?",
|
||||
"answers": {
|
||||
"A": "RAID 1",
|
||||
"B": "RAID 0",
|
||||
"C": "RAID 5",
|
||||
"D": "RAID 6"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "How many keys exist in a public/private key pair?",
|
||||
"answers": {
|
||||
"A": "0",
|
||||
"B": "2",
|
||||
"C": "1",
|
||||
"D": "4"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary goal of an Information Security Governance strategy?",
|
||||
"answers": {
|
||||
"A": "To minimize the impact of security incidents on the organization",
|
||||
"B": "To obtain senior management commitment and support",
|
||||
"C": "To enforce compliance with information security policies",
|
||||
"D": "To prioritize options to mitigate risks"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the process of hiding text within an image called?",
|
||||
"answers": {
|
||||
"A": "Encryption",
|
||||
"B": "Steganography",
|
||||
"C": "Spyware",
|
||||
"D": "Hiding"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What attack involves inserting an unauthorized person in the communication path between two users to intercept and modify packets?",
|
||||
"answers": {
|
||||
"A": "Known Plaintext Attack",
|
||||
"B": "Chosen Cipher-Text Attack",
|
||||
"C": "MITM Attack",
|
||||
"D": "Brute-Force Attack"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
|
||||
"question": "Who is responsible for implementing the planning, budgeting, and performance of the organization's information security components?",
|
||||
"answers": {
|
||||
"A": "Administrator",
|
||||
"B": "CISO",
|
||||
"C": "MGR",
|
||||
"D": "CEO"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "The primary responsibility of the information security steering committee is:",
|
||||
"answers": {
|
||||
"A": "Information security policy development",
|
||||
"B": "Direction setting and performance monitoring",
|
||||
"C": "Information security control implementation",
|
||||
"D": "Provision of information security training for employees"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "According to NIST guidelines, what is a recommended practice for creating a strong password?",
|
||||
"answers": {
|
||||
"A": "Implementing complex composition rules requiring a mix of character types",
|
||||
"B": "Setting mandatory periodic password changes for all users",
|
||||
"C": "Encouraging the use of longer, more memorable passphrases",
|
||||
"D": "Requiring the use of password hints for easier recall"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of continuously improving cybersecurity guidelines and their application notes?",
|
||||
"answers": {
|
||||
"A": "To categorize security measures into groups",
|
||||
"B": "To react to new technologies, use cases, and risks",
|
||||
"C": "To prevent all identified threats",
|
||||
"D": "To eliminate all cybersecurity risks"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What does DDoS stand for?",
|
||||
"answers": {
|
||||
"A": "Double Denial of Service",
|
||||
"B": "Distributed Denial of Service",
|
||||
"C": "Dual Denial of Services",
|
||||
"D": "Denial of Distributed Services"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which is not a common wireless network authentication mechanism?",
|
||||
"answers": {
|
||||
"A": "WEP",
|
||||
"B": "WPA",
|
||||
"C": "802.1X",
|
||||
"D": "RC4"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "Which technique aims to identify changes in normal routines by examining patterns of a user's activity?",
|
||||
"answers": {
|
||||
"A": "Vulnerability Assessment",
|
||||
"B": "Behavior-based IDS",
|
||||
"C": "Data Mining",
|
||||
"D": "User Profile Analysis"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of conducting a risk analysis in cybersecurity?",
|
||||
"answers": {
|
||||
"A": "To assess the risk level for the organization",
|
||||
"B": "To formulate the most important IT security procedures for the organization",
|
||||
"C": "To create a risk assessment statement",
|
||||
"D": "To determine the level of exposure to identified threats and identify possible safeguards or controls"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary advantage of using UDP over TCP?",
|
||||
"answers": {
|
||||
"A": "Reliability",
|
||||
"B": "Error recovery",
|
||||
"C": "Speed of data transfer",
|
||||
"D": "Data integrity"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "What does the CIA triad in cryptography emphasize?",
|
||||
"answers": {
|
||||
"A": "Confidentiality, Integrity, Authenticity",
|
||||
"B": "Confidentiality, Integrity, Availability",
|
||||
"C": "Confidentiality, Authenticity, Nonrepudiation",
|
||||
"D": "Confidentiality, Availability, Nonrepudiation"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "Which nmap command is used to perform a TCP SYN scan, also known as a half-open scan, to determine which ports are open on a target system?",
|
||||
"answers": {
|
||||
"A": "nmap -sP",
|
||||
"B": "nmap -sT",
|
||||
"C": "nmap -SYN",
|
||||
"D": "nmap -sS"
|
||||
},
|
||||
"solution": "D"
|
||||
},
|
||||
{
|
||||
"question": "What is the primary purpose of business continuity plans?",
|
||||
"answers": {
|
||||
"A": "To recover from natural disasters",
|
||||
"B": "To conduct regular audits of the organization's security systems",
|
||||
"C": "To minimize the effects of a disruptive event on a company",
|
||||
"D": "To increase the cost associated with a disruptive event"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which DoS attack sends large amounts of ICMP Echo traffic to a broadcast IP address with a spoofed source address of a victim?",
|
||||
"answers": {
|
||||
"A": "Smurf attack",
|
||||
"B": "SYN flood attack",
|
||||
"C": "Ping of Death",
|
||||
"D": "Botnet attack"
|
||||
},
|
||||
"solution": "A"
|
||||
}
|
||||
,
|
||||
{
|
||||
"question": "Which of the following tools is commonly used for network packet analysis and troubleshooting?",
|
||||
"answers": {
|
||||
"A": "Wireshark",
|
||||
"B": "Metasploit",
|
||||
"C": "Nmap",
|
||||
"D": "Hydra"
|
||||
},
|
||||
"solution": "A"
|
||||
},
|
||||
{
|
||||
"question": "In the context of encrypting data for secure communication, which encryption algorithm is typically preferred for its speed and efficiency?",
|
||||
"answers": {
|
||||
"A": "RSA-2048",
|
||||
"B": "AES-256",
|
||||
"C": "Both of them",
|
||||
"D": "None of them"
|
||||
},
|
||||
"solution": "B"
|
||||
},
|
||||
{
|
||||
"question": "What is the purpose of NOP sleds in buffer overflow exploits?",
|
||||
"answers": {
|
||||
"A": "To encrypt the payload",
|
||||
"B": "To alter the content of the NOP register",
|
||||
"C": "To insert no-operation instructions for padding",
|
||||
"D": "To alter the content of the EIP register"
|
||||
},
|
||||
"solution": "C"
|
||||
},
|
||||
{
|
||||
"question": "Which security feature can help prevent buffer overflow attacks by marking certain memory regions as non-executable?",
|
||||
"answers": {
|
||||
"A": "Data Execution Prevention (DEP)",
|
||||
"B": "Buffer Overflow Prevention (BOP)",
|
||||
"C": "Intrusion Detection System (IDS)",
|
||||
"D": "Address Space Layout Randomization (ASLR)"
|
||||
},
|
||||
"solution": "A"
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
]
|
||||
}
|
||||
Loading…
Reference in New Issue