mirror of https://github.com/aliasrobotics/cai.git
add seceval
This commit is contained in:
Mery-Sanz
parent
0242a311cb
commit
0ca20db759
|
|
@ -0,0 +1 @@
|
|||
.DS_Store
|
||||
Binary file not shown.
|
|
@ -0,0 +1,150 @@
|
|||
# SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models
|
||||
|
||||
[中文版](README_CN.md).
|
||||
|
||||
The advent of large language models has ignited a transformative era for the cybersecurity industry. Pioneering applications are being developed, deployed, and utilized in areas such as cybersecurity knowledge QA, vulnerability hunting, and alert investigation. Various researches have indicated that LLMs primarily acquire their knowledge during the pretraining phase, with fine-tuning serving essentially to align the model with user intentions, providing the ability to follow instructions. This suggests that the knowledge and skills embedded in the foundational model significantly influence the model's potential on specific downstream tas ks
|
||||
|
||||
Yet, a focused evaluation of cybersecurity knowledge is missing in existing datasets. We address this by introducing "SecEval". SecEval is the first benchmark specifically created for evaluating cybersecurity knowledge in Foundation Models. It offers over 2000 multiple-choice questions across 9 domains: Software Security, Application Security, System Security, Web Security, Cryptography, Memory Safety, Network Security, and PenTest.
|
||||
SecEval generates questions by prompting OpenAI GPT4 with authoritative sources such as open-licensed textbooks, official documentation, and industry guidelines and standards. The generation process is meticulously crafted to ensure the dataset meets rigorous quality, diversity, and impartiality criteria. You can explore our dataset the [explore page](https://xuanwuai.github.io/SecEval/explore.html).
|
||||
|
||||
Using SecEval, we conduct an evaluation of 10 state-of-the-art foundational models, providing new insights into their performance in the field of cybersecurity. The results indicate that there is still a long way to go before LLMs can be the master of cybersecurity. We hope that SecEval can serve as a catalyst for future research in this area.
|
||||
|
||||
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Leaderboard](#leaderboard)
|
||||
- [Dataset](#dataset)
|
||||
- [Generation Process](#generation-process)
|
||||
- [Limitations](#limitations)
|
||||
- [Future Work](#future-work)
|
||||
- [Licenses](#licenses)
|
||||
- [Citation](#citation)
|
||||
- [Credits](#credits)
|
||||
|
||||
|
||||
|
||||
## Leaderboard
|
||||
|
||||
| # | Model | Creator | Access | Submission Date | System Security | Application Security | PenTest | Memory Safety | Network Security | Web Security | Vulnerability | Software Security | Cryptography | Overall |
|
||||
|-----|-------------------|-----------|-----------|-----------------|-----------------|----------------------|---------|---------------|------------------|--------------|---------------|-------------------|--------------|---------|
|
||||
| 1 | gpt-4-turbo | OpenAI | API, Web | 2023-12-20 | 73.61 | 75.25 | 80.00 | 70.83 | 75.65 | 82.15 | 76.05 | 73.28 | 64.29 | 79.07 |
|
||||
| 2 | gpt-3.5-turbo | OpenAI | API, Web | 2023-12-20 | 59.15 | 57.18 | 72.00 | 43.75 | 60.87 | 63.00 | 60.18 | 58.19 | 35.71 | 62.09 |
|
||||
| 3 | Yi-6B | 01-AI | Weight | 2023-12-20 | 50.61 | 48.89 | 69.26 | 35.42 | 56.52 | 54.98 | 49.40 | 45.69 | 35.71 | 53.57 |
|
||||
| 4 | Orca-2-7b | Microsoft | Weight | 2023-12-20 | 46.76 | 47.03 | 60.84 | 31.25 | 49.13 | 55.63 | 50.00 | 52.16 | 14.29 | 51.60 |
|
||||
| 5 | Mistral-7B-v0.1 | Mistralai | Weight | 2023-12-20 | 40.19 | 38.37 | 53.47 | 33.33 | 36.52 | 46.57 | 42.22 | 43.10 | 28.57 | 43.65 |
|
||||
| 6 | chatglm3-6b-base | THUDM | Weight | 2023-12-20 | 39.72 | 37.25 | 57.47 | 31.25 | 43.04 | 41.14 | 37.43 | 39.66 | 28.57 | 41.58 |
|
||||
| 7 | Aquila2-7B | BAAI | Weight | 2023-12-20 | 34.84 | 36.01 | 47.16 | 22.92 | 32.17 | 42.04 | 38.02 | 36.21 | 7.14 | 38.29 |
|
||||
| 8 | Qwen-7B | Alibaba | Weight | 2023-12-20 | 28.92 | 28.84 | 41.47 | 18.75 | 29.57 | 33.25 | 31.74 | 30.17 | 14.29 | 31.37 |
|
||||
| 9 | internlm-7b | Sensetime | Weight | 2023-12-20 | 25.92 | 25.87 | 36.21 | 25.00 | 27.83 | 32.86 | 29.34 | 34.05 | 7.14 | 30.29 |
|
||||
| 10 | Llama-2-7b-hf | MetaAI | Weight | 2023-12-20 | 20.94 | 18.69 | 26.11 | 16.67 | 14.35 | 22.77 | 21.56 | 20.26 | 21.43 | 22.15 |
|
||||
|
||||
## Dataset
|
||||
|
||||
### Format
|
||||
|
||||
The dataset is in json format. Each question has the following fields:
|
||||
|
||||
* id: str # unique id for each question
|
||||
* source: str # the source where the question is generated from
|
||||
* question: str # the question description
|
||||
* choices: List[str] # the choices for the question
|
||||
* answer: str # the answer for the question
|
||||
* topics: List[QuestionTopic] # the topics for the question, each question can have multiple topics.
|
||||
* keyword: str # the keyword for the question
|
||||
|
||||
|
||||
### Question Distribution
|
||||
|
||||
| Topic | No. of Questions |
|
||||
|---------------------|-----------------|
|
||||
| SystemSecurity | 1065 |
|
||||
| ApplicationSecurity | 808 |
|
||||
| PenTest | 475 |
|
||||
| MemorySafety | 48 |
|
||||
| NetworkSecurity | 230 |
|
||||
| WebSecurity | 773 |
|
||||
| Vulnerability | 334 |
|
||||
| SoftwareSecurity | 232 |
|
||||
| Cryptography | 14 |
|
||||
| Overall | 2126 |
|
||||
|
||||
|
||||
### Download
|
||||
You can download the json file of the dataset by running.
|
||||
|
||||
```
|
||||
wget https://huggingface.co/datasets/XuanwuAI/SecEval/blob/main/questions.json
|
||||
```
|
||||
|
||||
Or you can load the dataset from [Huggingface](https://huggingface.co/datasets/XuanwuAI/SecEval).
|
||||
|
||||
### Evaluate Your Model on SecEval
|
||||
|
||||
You can use our [evaluation script](https://github.com/XuanwuAI/SecEval/tree/main/eval) to evaluate your model on SecEval dataset.
|
||||
|
||||
|
||||
## Generation Process
|
||||
|
||||
### Data Collection
|
||||
|
||||
- **Textbook**: We selected open-licensed textbooks from the Computer Security courses CS161 at UC Berkeley and 6.858 at MIT. These resources provide extensive information on network security, memory safety, web security, and cryptography.
|
||||
|
||||
- **Official Documentation**: We utilized official documentation, such as Apple Platform Security, Android Security, and Windows Security, to integrate system security and application security knowledge specific to these platforms.
|
||||
|
||||
- **Industrial Guidelines**: To encompass web security, we referred to the Mozilla Web Security Guidelines. In addition, we used the OWASP Web Security Testing Guide (WSTG) and OWASP Mobile Application Security Testing Guide (MASTG) for insights into web and application security testing.
|
||||
|
||||
- **Industrial Standards**: The Common Weakness Enumeration (CWE) was employed to address knowledge of vulnerabilities. For penetration testing, we incorporated the MITRE ATT&CK and MITRE D3fend frameworks.
|
||||
|
||||
### Questions Generation
|
||||
|
||||
To facilitate the evaluation process, we designed the dataset in a multiple-choice question format. Our approach to question generation involved several steps:
|
||||
|
||||
1. **Text Parsing**: We began by parsing the texts according to their hierarchical structure, such as chapters and sections for textbooks, or tactics and techniques for frameworks like ATT&CK.
|
||||
|
||||
2. **Content Sampling**: For texts with extensive content, such as CWE or Windows Security Documentation, we employed a sampling strategy to maintain manageability. For example, we selected the top 25 most common weakness types and 175 random types from CWE.
|
||||
|
||||
3. **Question Generation**: Utilizing GPT-4, we generated multiple-choice questions based on the parsed text, with the level of detail adjusted according to the content's nature. For instance, questions stemming from the CS161 textbook were based on individual sections, while those from ATT&CK were based on techniques.
|
||||
|
||||
4. **Question Refinement**: We then prompted GPT-4 to identify and filter out questions with issues such as too simplistic or not self-contained. Where possible, questions were revised; otherwise, they were discarded.
|
||||
|
||||
5. **Answer Calibration**: We refine the selection of answer options by presenting GPT-4 with both the question and the source text from which the question is derived. Should the response generated by GPT-4 diverge from the previously established answer, this discrepancy suggests that obtaining a consistent answer for the question is inherently challenging. In such cases, we opt to eliminate these problematic questions.
|
||||
|
||||
6. **Classification**: Finally, we organized the questions into 9 topics, and attached a relevant fine-grained keyword to each question.
|
||||
|
||||
|
||||
## Limitations
|
||||
|
||||
The dataset, while comprehensive, exhibits certain constraints:
|
||||
|
||||
1. **Distribution Imbalance**: The dataset presents an uneven distribution of questions across different domains, resulting in a higher concentration of questions in certain areas while others are less represented.
|
||||
|
||||
2. **Incomplete Scope**: Some topics on Cybersecurity are absent from the dataset, such as content security, reverse engineering, and malware analysis. As such, it does not encapsulate the full breadth of knowledge within the field.
|
||||
|
||||
## Future Work
|
||||
|
||||
1. **Improvement on Distribution**: We aim to broaden the dataset's comprehensiveness by incorporating additional questions, thereby enriching the coverage of existing cybersecurity topics.
|
||||
|
||||
2. **Improvement on Topic Coverage**: Efforts will be made to include a wider array of cybersecurity topics within the dataset, which will help achieve a more equitable distribution of questions across various fields.
|
||||
|
||||
|
||||
## Licenses
|
||||
|
||||
The dataset is released under the [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license. The code is released under the [MIT](https://opensource.org/licenses/MIT) license.
|
||||
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{li2023seceval,
|
||||
title={SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models},
|
||||
author={Li, Guancheng and Li, Yifeng and Wang Guannan and Yang, Haoyu and Yu, Yang},
|
||||
publisher = {GitHub},
|
||||
howpublished= "https://github.com/XuanwuAI/SecEval",
|
||||
year={2023}
|
||||
}
|
||||
```
|
||||
|
||||
## Credits
|
||||
|
||||
This work is supported by [Tencent Security Xuanwu Lab](https://xlab.tencent.com/en/). we also apperiate Tencent Spark Talent Program for help.
|
||||
|
|
@ -0,0 +1,144 @@
|
|||
# SecEval：评估基础模型在网络安全知识上的全面基准测试
|
||||
|
||||
[English](README.md)
|
||||
|
||||
随着大型语言模型的出现，为网络安全行业带来了变革性的时代。在网络安全知识问答、漏洞挖掘和告警研判等领域，人们正在开发、部署和使用大模型应用。各种研究表明，LLMs主要在预训练阶段获取知识，而微调基本上是为了使模型与用户意图对齐，提供遵循指令的能力。这表明基础模型中的知识和技能对模型在特定下游任务上的潜力有着重要影响。
|
||||
|
||||
然而，在现有评估数据集中缺乏对网络安全知识的专门性评估。我们通过引入“SecEval”来解决这个问题。SecEval是首个专门为评估基础模型中的网络安全知识而创建的基准测试。它提供了超过2000个多项选择题，覆盖9个领域：软件安全、应用程序安全、系统安全、网络安全、密码学、内存安全、网络安全和渗透测试。
|
||||
|
||||
SecEval通过向OpenAI GPT4提供权威来源的数据来生成相应问题，如开放许可的教科书、官方文档和行业指导方针与标准，来生成问题。生成过程经过精心设计，以确保数据集满足严格的质量、多样性和公正性标准。您可以在[探索页面](https://xuanwuai.github.io/SecEval/explore.html)上探索我们的数据集。
|
||||
|
||||
使用SecEval，我们对10种最先进的基础模型进行了评估，为它们在网络安全领域的表现提供了新的见解。结果表明，在LLMs在网络安全领域还有很长的路要走。我们希望SecEval能够成为未来在这一领域研究的催化剂。
|
||||
## 目录
|
||||
|
||||
- [Leaderboard](#leaderboard)
|
||||
- [数据集](#数据集)
|
||||
- [数据集生成方法](#数据集生成方法)
|
||||
- [局限性](#局限性)
|
||||
- [未来工作](#未来工作)
|
||||
- [许可证](#许可证)
|
||||
- [引用](#引用)
|
||||
|
||||
## Leaderboard
|
||||
|
||||
| # | Model | Creator | Access | Submission Date | System Security | Application Security | PenTest | Memory Safety | Network Security | Web Security | Vulnerability | Software Security | Cryptography | Overall |
|
||||
|-----|-------------------|-----------|-----------|-----------------|-----------------|----------------------|---------|---------------|------------------|--------------|---------------|-------------------|--------------|---------|
|
||||
| 1 | gpt-4-turbo | OpenAI | API, Web | 2023-12-20 | 73.61 | 75.25 | 80.00 | 70.83 | 75.65 | 82.15 | 76.05 | 73.28 | 64.29 | 79.07 |
|
||||
| 2 | gpt-35-turbo | OpenAI | API, Web | 2023-12-20 | 59.15 | 57.18 | 72.00 | 43.75 | 60.87 | 63.00 | 60.18 | 58.19 | 35.71 | 62.09 |
|
||||
| 3 | Yi-6B | 01-AI | Weight | 2023-12-20 | 50.61 | 48.89 | 69.26 | 35.42 | 56.52 | 54.98 | 49.40 | 45.69 | 35.71 | 53.57 |
|
||||
| 4 | Orca-2-7b | Microsoft | Weight | 2023-12-20 | 46.76 | 47.03 | 60.84 | 31.25 | 49.13 | 55.63 | 50.00 | 52.16 | 14.29 | 51.60 |
|
||||
| 5 | Mistral-7B-v0.1 | Mistralai | Weight | 2023-12-20 | 40.19 | 38.37 | 53.47 | 33.33 | 36.52 | 46.57 | 42.22 | 43.10 | 28.57 | 43.65 |
|
||||
| 6 | chatglm3-6b-base | THUDM | Weight | 2023-12-20 | 39.72 | 37.25 | 57.47 | 31.25 | 43.04 | 41.14 | 37.43 | 39.66 | 28.57 | 41.58 |
|
||||
| 7 | Aquila2-7B | BAAI | Weight | 2023-12-20 | 34.84 | 36.01 | 47.16 | 22.92 | 32.17 | 42.04 | 38.02 | 36.21 | 7.14 | 38.29 |
|
||||
| 8 | Qwen-7B | Alibaba | Weight | 2023-12-20 | 28.92 | 28.84 | 41.47 | 18.75 | 29.57 | 33.25 | 31.74 | 30.17 | 14.29 | 31.37 |
|
||||
| 9 | internlm-7b | Sensetime | Weight | 2023-12-20 | 25.92 | 25.87 | 36.21 | 25.00 | 27.83 | 32.86 | 29.34 | 34.05 | 7.14 | 30.29 |
|
||||
| 10 | Llama-2-7b-hf | MetaAI | Weight | 2023-12-20 | 20.94 | 18.69 | 26.11 | 16.67 | 14.35 | 22.77 | 21.56 | 20.26 | 21.43 | 22.15 |
|
||||
|
||||
## 数据集
|
||||
|
||||
### 格式
|
||||
|
||||
数据集采用json格式。每个问题包含以下字段：
|
||||
|
||||
* id: str # 每个问题的唯一id
|
||||
* source: str # 问题来源
|
||||
* question: str # 问题描述
|
||||
* choices: List[str] # 问题的选项
|
||||
* answer: str # 问题的答案
|
||||
* topics: List[QuestionTopic] # 问题的主题，每个问题可以有多个主题。
|
||||
* keyword: str # 问题的关键词
|
||||
|
||||
|
||||
### 问题分布
|
||||
|
||||
| 主题 | 问题数量 |
|
||||
|----------------------|--------------|
|
||||
| 系统安全 | 1065 |
|
||||
| 应用安全 | 808 |
|
||||
| 渗透测试 | 475 |
|
||||
| 内存安全 | 48 |
|
||||
| 网络安全 | 230 |
|
||||
| 网络安全 | 773 |
|
||||
| 漏洞 | 334 |
|
||||
| 软件安全 | 232 |
|
||||
| 密码学 | 14 |
|
||||
| 总体 | 2126 |
|
||||
|
||||
|
||||
### 下载
|
||||
|
||||
您可以通过运行以下命令下载json文件的数据集。
|
||||
```
|
||||
wget https://huggingface.co/datasets/XuanwuAI/SecEval/blob/main/problems.json
|
||||
```
|
||||
|
||||
或者您可以从[Huggingface](https://huggingface.co/datasets/XuanwuAI/SecEval)加载数据集。
|
||||
|
||||
### 在SecEval上评估您的模型
|
||||
|
||||
您可以使用我们的[评估脚本](https://github.com/XuanwuAI/SecEval/tree/main/eval)来在SecEval数据集上评估您的模型。
|
||||
|
||||
## 数据集生成方法
|
||||
|
||||
### 数据收集
|
||||
|
||||
- **教科书**：我们从加州大学伯克利分校的计算机安全课程 CS161 和麻省理工学院的 6.858 选取了开放许可的教科书。这些资源提供了关于网络安全、内存安全、网页安全和密码学的广泛信息。
|
||||
|
||||
- **官方文档**：我们使用了如苹果平台安全、安卓安全和 Windows 安全等官方文档，以整合这些平台特有的系统安全和应用安全知识。
|
||||
|
||||
- **工业指南**：为了包含网络安全，我们参考了 Mozilla 网络安全指南。此外，我们还使用了 OWASP 网络安全测试指南（WSTG）和 OWASP 移动应用安全测试指南（MASTG），以获得网络和应用安全测试的洞见。
|
||||
|
||||
- **工业标准**：我们使用了CWE来解决对漏洞的认知问题。对于渗透测试，我们纳入了 MITRE ATT&CK 和 MITRE D3fend 框架。
|
||||
|
||||
### 生成评估问题
|
||||
|
||||
为了便于评估过程，我们设计了一个多项选择题的数据集格式。我们的问题生成方法包括几个步骤：
|
||||
|
||||
1. **文本解析**：我们首先根据文本的层次结构进行解析，例如教科书的章节，或诸如ATT&CK这样的框架的战略和技术。
|
||||
|
||||
2. **内容抽样**：对于内容丰富的文本，如CWE或Windows安全文档，我们采用抽样策略以保持可管理性。例如，我们从CWE中选择了25种最常见的弱点类型和175种随机类型。
|
||||
|
||||
3. **问题生成**：利用GPT-4基于解析后的文本生成多项选择题并根据文本的特性调整送入GPT参考文本的粒度。例如，CS161教科书的问题是基于各个章节的，而来自ATT&CK的则是基于技术的。
|
||||
|
||||
4. **问题完善**：然后我们指导GPT-4识别并过滤出问题，比如过于简单或不完整的问题。在可能的情况下，我们会尝试将问题修正；否则，它们将被丢弃。
|
||||
|
||||
5. **答案校准**：我们通过向GPT-4提供问题和产生问题的源文本来再次生成答案。如果GPT-4生成的响应与之前问题生成阶段确定的答案不一致，则表明，对于这道题目，获得一致答案本身具有挑战性。在这种情况下，我们选择淘汰这些问题。
|
||||
|
||||
6. **分类**：最后，我们将问题分为9个主题，并为每个问题附上相关的细粒度关键词。
|
||||
|
||||
|
||||
## 局限性
|
||||
|
||||
当前数据集存在一定的局限性：
|
||||
|
||||
1. **分布不平衡**：数据集在不同领域的问题分布不均，导致某些区域的问题集中度较高，而其他区域则较少。
|
||||
|
||||
2. **主题覆盖范围不完整**：数据集中缺少一些网络安全主题，如内容安全、逆向工程和恶意软件分析。因此，它并未涵盖该领域内的全部知识范围。
|
||||
|
||||
## 未来工作
|
||||
|
||||
1. **分布上的改进**：我们旨在通过增加更多问题来提高数据集的全面性，从而丰富现有网络安全主题的覆盖范围。
|
||||
|
||||
2. **主题覆盖上的改进**：我们将努力在数据集中包含更广泛的网络安全主题，这将有助于实现各个领域问题分布的更公平分配。
|
||||
|
||||
|
||||
## 许可证
|
||||
|
||||
数据集在 [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) 许可证下发布。代码在 [MIT](https://opensource.org/licenses/MIT) 许可证下发布。
|
||||
|
||||
|
||||
## 引用
|
||||
|
||||
```bibtex
|
||||
@misc{li2023seceval,
|
||||
title={SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models},
|
||||
author={Li, Guancheng and Li, Yifeng and Wang Guannan and Yang, Haoyu and Yu, Yang},
|
||||
publisher = {GitHub},
|
||||
howpublished= "https://github.com/XuanwuAI/SecEval",
|
||||
year={2023}
|
||||
}
|
||||
```
|
||||
|
||||
## 致谢
|
||||
|
||||
这项工作是在[腾讯安全玄武实验室](https://xlab.tencent.com/en/)的支持下完成的，并获得了腾讯星火计划的帮助。
|
||||
|
|
@ -0,0 +1,189 @@
|
|||
/* next/prev button styling */
|
||||
#multiline{
|
||||
line-height: 1em;
|
||||
}
|
||||
|
||||
input[type=button] {
|
||||
cursor: pointer;
|
||||
background: white;
|
||||
width: 5vw;
|
||||
height: 3vh;
|
||||
border-radius: 5px;
|
||||
font-size: 0.8em;
|
||||
}
|
||||
|
||||
.row {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: flex-start;
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
margin-top: var(--nav-height);
|
||||
}
|
||||
|
||||
.optionsbtn {
|
||||
height: 50px;
|
||||
padding: 5px 10px;
|
||||
top: var(--nav-height);
|
||||
left: 0;
|
||||
position: fixed;
|
||||
z-index: 1;
|
||||
|
||||
font-size: 20px;
|
||||
cursor: pointer;
|
||||
background-color: var(--nav-color);
|
||||
color: white;
|
||||
border: none;
|
||||
}
|
||||
|
||||
.optionsbtn:hover {
|
||||
background-color: silver;
|
||||
}
|
||||
|
||||
|
||||
#option-panel {
|
||||
position: sticky;
|
||||
z-index: 2;
|
||||
height: 250px;
|
||||
top: var(--nav-height);
|
||||
left: 0;
|
||||
background-color: silver;
|
||||
overflow-x: hidden;
|
||||
transition: 0.5s;
|
||||
height: calc(100vh - var(--nav-height) - 40px);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
/* justify-content: space-around; */
|
||||
align-items: center;
|
||||
width: 20vw;
|
||||
padding-top: 40px;
|
||||
}
|
||||
|
||||
#option-panel a {
|
||||
padding: 8px 8px 8px 32px;
|
||||
text-decoration: none;
|
||||
font-size: 25px;
|
||||
color: #818181;
|
||||
display: block;
|
||||
transition: 0.3s;
|
||||
}
|
||||
|
||||
#option-panel a:hover {
|
||||
color: #f1f1f1;
|
||||
}
|
||||
|
||||
#option-panel .closebtn {
|
||||
position: absolute;
|
||||
top: 10px;
|
||||
right: 10px;
|
||||
font-size: 36px;
|
||||
}
|
||||
|
||||
#display {
|
||||
float: right;
|
||||
width: 60vw;
|
||||
transition: 0.5s;
|
||||
padding: 0px 10vw;
|
||||
}
|
||||
|
||||
#content-body {
|
||||
/* height: 200vh; */
|
||||
width: 100%;
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: flex-start;
|
||||
}
|
||||
|
||||
.display-col {
|
||||
width: 48%;
|
||||
}
|
||||
|
||||
.box {
|
||||
padding: 10px;
|
||||
margin: 0;
|
||||
background-color: white;
|
||||
border-radius: 5px;
|
||||
border: 1px solid rgba(0, 0, 0, 0.4);
|
||||
/* box-shadow: 5px 5px 5px rgba(0,0,0,0.2); */
|
||||
overflow-y: auto;
|
||||
max-height: 100vh;
|
||||
}
|
||||
|
||||
#title {
|
||||
margin: 50px 0;
|
||||
}
|
||||
|
||||
.question-title {
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.question-txt {
|
||||
overflow-y: hidden;
|
||||
color: blue;
|
||||
}
|
||||
|
||||
.hint-txt {
|
||||
margin-bottom: 2px;
|
||||
}
|
||||
|
||||
.question-img {
|
||||
max-width: 100%;
|
||||
max-height: 400px;
|
||||
width: auto;
|
||||
height: auto;
|
||||
display: block;
|
||||
margin: 0 auto;
|
||||
margin-top: 20px;
|
||||
}
|
||||
|
||||
.choices {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.choices-vertical {
|
||||
display: grid;
|
||||
row-gap: 10px;
|
||||
}
|
||||
|
||||
.choice-txt {
|
||||
border: 1px solid silver;
|
||||
/* text-align: center; */
|
||||
padding: 5px 10px;
|
||||
margin-right: 10px;
|
||||
width: fit-content;
|
||||
}
|
||||
|
||||
.answer-txt {
|
||||
color: green;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.optbox {
|
||||
width: calc(90%);
|
||||
margin-bottom: 10px;
|
||||
transition: all 0.5s ease;
|
||||
}
|
||||
|
||||
.opttitle {
|
||||
width: 90%;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.dd-label {
|
||||
margin: 0px 0;
|
||||
display: block;
|
||||
}
|
||||
|
||||
#filter-submit {
|
||||
color: white;
|
||||
border-radius: 4px;
|
||||
text-shadow: 0 1px 1px rgba(0, 0, 0, 0.2);
|
||||
}
|
||||
|
||||
.button-success {
|
||||
background-color: rgb(28, 184, 65);
|
||||
}
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
table {
|
||||
border: 1px solid black;
|
||||
border-collapse: collapse;
|
||||
}
|
||||
|
||||
th {
|
||||
padding: 2px 5px;
|
||||
border: 1px solid black;
|
||||
}
|
||||
|
||||
td {
|
||||
padding: 2px 5px;
|
||||
border: 1px solid black;
|
||||
}
|
||||
|
||||
thead {
|
||||
background: #ddd;
|
||||
}
|
||||
|
||||
table#results.js-sort-0 tbody tr td:nth-child(1),
|
||||
table#results.js-sort-1 tbody tr td:nth-child(2),
|
||||
table#results.js-sort-2 tbody tr td:nth-child(3),
|
||||
table#results.js-sort-3 tbody tr td:nth-child(4),
|
||||
table#results.js-sort-4 tbody tr td:nth-child(5),
|
||||
table#results.js-sort-5 tbody tr td:nth-child(6),
|
||||
table#results.js-sort-6 tbody tr td:nth-child(7),
|
||||
table#results.js-sort-7 tbody tr td:nth-child(8),
|
||||
table#results.js-sort-8 tbody tr td:nth-child(9),
|
||||
table#results.js-sort-9 tbody tr td:nth-child(10),
|
||||
table#results.js-sort-10 tbody tr td:nth-child(11),
|
||||
table#results.js-sort-11 tbody tr td:nth-child(12),
|
||||
table#results.js-sort-12 tbody tr td:nth-child(13),
|
||||
table#results.js-sort-13 tbody tr td:nth-child(14) {
|
||||
background: #dee;
|
||||
}
|
||||
|
||||
/* Table for three sub-tasks */
|
||||
#results {
|
||||
font-family: Arial, Helvetica, sans-serif;
|
||||
font-size: 16px;
|
||||
border-collapse: collapse;
|
||||
width: 100%;
|
||||
text-align: center;
|
||||
margin: auto;
|
||||
margin-top: 1em;
|
||||
margin-bottom: 2em;
|
||||
}
|
||||
|
||||
/*table tile*/
|
||||
#results th {
|
||||
border: 1px solid #ddd;
|
||||
padding: 8px;
|
||||
padding-top: 12px;
|
||||
padding-bottom: 12px;
|
||||
font-size: 16px;
|
||||
text-align: center;
|
||||
font-weight: 600;
|
||||
background-color: var(--UCLA-darkblue);
|
||||
color: white;
|
||||
}
|
||||
|
||||
#results td {
|
||||
border: 1px solid #ddd;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
#results tr:nth-child(even) {
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
|
||||
#results tr:hover {
|
||||
background-color: #ddd;
|
||||
}
|
||||
|
||||
.js-sort-number:hover {
|
||||
text-decoration: underline;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
|
@ -0,0 +1,317 @@
|
|||
/* some constants used throughout the website*/
|
||||
:root {
|
||||
--nav-font: Arial, Helvetica, sans-serif;
|
||||
--nav-height: 6vh;
|
||||
--UCLA-gold: #FFD100;
|
||||
--UCLA-blue: #2774AE;
|
||||
--UCLA-darkblue: #003B5C;
|
||||
--body-font: #212529;
|
||||
--heading: #1859c2;
|
||||
--nav-darkblue: #05004a;
|
||||
--nav-color: #212529;
|
||||
--nav-color-light: #353c42;
|
||||
--nav-darkblue: #05004a;
|
||||
}
|
||||
|
||||
html {
|
||||
overflow-x: hidden;
|
||||
font-family: var(--nav-font);
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
font-size: 16px;
|
||||
line-height: 110%;
|
||||
color: var(--body-font)
|
||||
}
|
||||
|
||||
strong {
|
||||
color: var(--body-font);
|
||||
}
|
||||
|
||||
/* block: heading in the paragraph */
|
||||
h1 {
|
||||
/* color: var(--UCLA-blue); */
|
||||
color: var(--heading);
|
||||
}
|
||||
|
||||
/* element: paragraphs in the body text */
|
||||
p {
|
||||
line-height: 150%;
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
/* links */
|
||||
a {
|
||||
color: black;
|
||||
text-decoration: none;
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
a:focus,
|
||||
a:hover {
|
||||
color: #005587;
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
a.hover {
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
a.hover:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
/* block: paper title in the body text */
|
||||
papername {
|
||||
font-family: Arial;
|
||||
font-size: 18px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
/* element: divider line */
|
||||
hr {
|
||||
width: 75%;
|
||||
color: silver;
|
||||
opacity: 0.5;
|
||||
}
|
||||
|
||||
img {
|
||||
max-width: 100%;
|
||||
}
|
||||
|
||||
ul {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
/* list */
|
||||
li {
|
||||
margin-top: 12px;
|
||||
font-size: 18px;
|
||||
display: list-item;
|
||||
text-align: -webkit-match-parent;
|
||||
}
|
||||
|
||||
/* body of the page */
|
||||
#body {
|
||||
width: 60vw;
|
||||
margin: auto;
|
||||
}
|
||||
|
||||
/* section */
|
||||
.section {
|
||||
width: calc(100% - 20px);
|
||||
padding: 10px;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
/* anchor tag for nav button scrolling */
|
||||
.anchor {
|
||||
position: relative;
|
||||
display: block;
|
||||
/* visibility: hidden; */
|
||||
top: calc(-1 * var(--nav-height));
|
||||
height: 1px;
|
||||
}
|
||||
|
||||
/* the banner */
|
||||
#title {
|
||||
margin-top: var(--nav-height);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
height: calc(50vh - var(--nav-height));
|
||||
width: 100vw;
|
||||
left: 0vw;
|
||||
font-weight: bold;
|
||||
color: white;
|
||||
/* background: linear-gradient(to bottom, var(--nav-darkblue), var(--UCLA-blue)); */
|
||||
background: linear-gradient(180deg, var(--nav-darkblue), var(--UCLA-blue));
|
||||
/* background: linear-gradient(, var(--UCLA-darkblue), var(--UCLA-blue) 60%, white); */
|
||||
}
|
||||
|
||||
#title-icon {
|
||||
width: 4em;
|
||||
margin: 10px;
|
||||
background-color: transparent;
|
||||
}
|
||||
|
||||
/* used to center the title */
|
||||
#title-wrapper {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
#title-text {
|
||||
font-size: 50px;
|
||||
}
|
||||
|
||||
#subtitle-text {
|
||||
font-size: 30px;
|
||||
text-align: center;
|
||||
line-height: 125%;
|
||||
margin: -0.5em;
|
||||
}
|
||||
|
||||
/* change this to adjust the vertical position of the title*/
|
||||
#title-padding-bottom {
|
||||
font-size: 1.5em;
|
||||
}
|
||||
|
||||
/* shows three examples on the home page */
|
||||
#example-box {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
/* data example images */
|
||||
.example-img {
|
||||
/* width: 15vw; */
|
||||
width: 90%;
|
||||
margin: auto;
|
||||
}
|
||||
|
||||
/* makes its children independent columns on the same row */
|
||||
.makecol {
|
||||
width: 100%;
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
#overview-left {
|
||||
width: 70%;
|
||||
}
|
||||
|
||||
#overview-right {
|
||||
width: 35%;
|
||||
}
|
||||
|
||||
/* not very important. Tried to make the website responsive */
|
||||
@media only screen and (max-width: 1000px) {
|
||||
#body {
|
||||
padding: 0;
|
||||
margin: 0;
|
||||
width: 100%;
|
||||
left: 0;
|
||||
}
|
||||
|
||||
.section {
|
||||
width: 98%;
|
||||
padding: 2.5%;
|
||||
}
|
||||
|
||||
#overview-content-wrapper {
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
#overview-left,
|
||||
#overview-right {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
overflow-x: auto;
|
||||
display: block;
|
||||
}
|
||||
|
||||
th, td {
|
||||
white-space: nowrap;
|
||||
}
|
||||
}
|
||||
|
||||
.bibtex {
|
||||
/* font-family: Arial; */
|
||||
padding-left: 1em;
|
||||
padding-right: 1em;
|
||||
padding-top: 1em;
|
||||
padding-bottom: 1em;
|
||||
background-color: #f5f5f5;
|
||||
border: 1px solid #ccc;
|
||||
border-radius: 5px;
|
||||
font-size: 16px;
|
||||
word-wrap: break-word;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
.ext-link {
|
||||
/* color: var(--UCLA-blue); */
|
||||
color: #007bff;
|
||||
/* text-decoration: underline; */
|
||||
}
|
||||
|
||||
#cc-img {
|
||||
--width: 40%;
|
||||
width: var(--width);
|
||||
margin: 0 calc(calc(100% - var(--width)) / 2);
|
||||
}
|
||||
|
||||
/* class: profile */
|
||||
.profile {
|
||||
display: inline-block;
|
||||
width: 180px;
|
||||
}
|
||||
|
||||
/* class: profile image */
|
||||
.profile-img {
|
||||
border-radius: 50%;
|
||||
width: 150px;
|
||||
}
|
||||
|
||||
/* #task-pie-chart */
|
||||
.col-wrapper {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.col {
|
||||
width: 48%;
|
||||
}
|
||||
|
||||
/* Table for three sub-tasks */
|
||||
#tasks {
|
||||
font-family: Arial, Helvetica, sans-serif;
|
||||
border-collapse: collapse;
|
||||
width: 70%;
|
||||
text-align: center;
|
||||
margin: auto;
|
||||
margin-top: 1em;
|
||||
margin-bottom: 2em;
|
||||
}
|
||||
|
||||
/*table tile*/
|
||||
#tasks th {
|
||||
border: 1px solid #ddd;
|
||||
padding: 8px;
|
||||
padding-top: 12px;
|
||||
padding-bottom: 12px;
|
||||
font-size: 16px;
|
||||
text-align: center;
|
||||
font-weight: 600;
|
||||
background-color: var(--UCLA-darkblue);
|
||||
color: white;
|
||||
}
|
||||
|
||||
#tasks td {
|
||||
border: 1px solid #ddd;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
#tasks tr:nth-child(even) {
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
|
||||
#tasks tr:hover {
|
||||
background-color: #ddd;
|
||||
}
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
/* navigation bar styling */
|
||||
#nav {
|
||||
width: 96%;
|
||||
height: var(--nav-height);
|
||||
/* background-color: var(--nav-darkblue); */
|
||||
background-color: #212529;
|
||||
position: fixed;
|
||||
top: 0;
|
||||
z-index: 10;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 0 2vw;
|
||||
opacity: 1;
|
||||
/* border-bottom: 1px solid blue; */
|
||||
}
|
||||
|
||||
#icon {
|
||||
font-size: 1.5em;
|
||||
font-family: var(--nav-font);
|
||||
font-weight: 700;
|
||||
color: white;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.nav-button {
|
||||
margin-left: 20px;
|
||||
font-family: var(--nav-font);
|
||||
color: white;
|
||||
}
|
||||
|
||||
.nav-button:hover {
|
||||
color: var(--UCLA-gold);
|
||||
}
|
||||
|
||||
#nav-icon {
|
||||
width: 1em;
|
||||
margin: 5px;
|
||||
}
|
||||
|
||||
/* footer style */
|
||||
#footer {
|
||||
width: 100%;
|
||||
height: var(--nav-height);
|
||||
background-color: var(--nav-darkblue);
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
opacity: 1;
|
||||
font-size: 0.7em;
|
||||
color: grey;
|
||||
margin-top: 40px;
|
||||
}
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Explore | SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models</title>
|
||||
<link rel="icon" href="img/logo.png" type="image/icon type">
|
||||
|
||||
<link rel="stylesheet" href="https://unpkg.com/purecss@2.1.0/build/pure-min.css"
|
||||
integrity="sha384-yHIFVG6ClnONEA5yB5DJXfW2/KC173DIQrYoZMEtBvGzmf0PKiGyNEqe9N6BNDBH" crossorigin="anonymous">
|
||||
<link rel="stylesheet" href="css/main.css">
|
||||
<link rel="stylesheet" href="css/nav.css">
|
||||
<link rel="stylesheet" href="css/explore.css">
|
||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@latest/dist/katex.min.css">
|
||||
|
||||
<script src="https://cdn.jsdelivr.net/npm/underscore@latest/underscore-umd-min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/katex@latest/dist/katex.min.js"></script>
|
||||
<script src="javascript/questions-data.js"></script>
|
||||
<script src="javascript/explore.js" defer></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="nav">
|
||||
<div id="icon">
|
||||
<img src="img/logo.png" id="nav-icon">
|
||||
<a class="nav-button" href="https://XuanwuAI.github.io/SecEval/"
|
||||
style="margin-left: 2px; font-size: 24px">SecEval
|
||||
</a>
|
||||
</div>
|
||||
<div>
|
||||
<a class="nav-button" href="index.html#home">Home</a>
|
||||
<a class="nav-button" href="index.html#dataset">Dataset</a>
|
||||
<a class="nav-button" href="#paper_placeholder.html " target="_blank">Paper</a>
|
||||
<a class="nav-button" href="https://github.com/XuanwuAI/SecEval" target="_blank">Github</a>
|
||||
<a class="nav-button" href="index.html#citation">Citation</a>
|
||||
<a class="nav-button" href="index.html#contact">Contact</a>
|
||||
<a class="nav-button" href="leaderboard.html">Leaderboard</a>
|
||||
<a class="nav-button" href="explore.html">Explore</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<button class="optionsbtn">☰</button>
|
||||
<div class="row">
|
||||
<div id="option-panel" class="left">
|
||||
<a href="javascript:void(0)" class="closebtn">×</a>
|
||||
<h3 class="opttitle">Sample Filters</h3>
|
||||
<div class="optbox"></div>
|
||||
</div>
|
||||
<div id="display">
|
||||
<br><br>
|
||||
<h1 id="multiline" xt-marked="ok">SecEval (A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models)</h1>
|
||||
<br><br>
|
||||
<div id="content-body">
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 363 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 78 KiB |
|
|
@ -0,0 +1,136 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models</title>
|
||||
<link rel="icon" href="img/logo.png" type="image/icon type">
|
||||
<link rel="stylesheet" href="css/main.css">
|
||||
<link rel="stylesheet" href="css/nav.css">
|
||||
<script src="javascript/index.js" defer></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="nav">
|
||||
<div id="icon">
|
||||
<img src="img/logo.png" id="nav-icon">
|
||||
<a class="nav-button" href="https://XuanwuAI.github.io/SecEval/" style="margin-left: 2px; font-size: 24px">SecEval
|
||||
</a>
|
||||
</div>
|
||||
<div>
|
||||
<a class="nav-button" href="#home">Home</a>
|
||||
<a class="nav-button" href="#dataset">Dataset</a>
|
||||
<a class="nav-button" href="paper_placeholder.html" target="_blank">Paper</a>
|
||||
<a class="nav-button" href="https://github.com/XuanwuAI/SecEval" target="_blank">Github</a>
|
||||
<a class="nav-button" href="#citation">Citation</a>
|
||||
<a class="nav-button" href="#contact">Contact</a>
|
||||
<a class="nav-button" href="leaderboard.html">Leaderboard</a>
|
||||
<a class="nav-button" href="explore.html">Explore</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- anchor for the home button -->
|
||||
<div id="home" style="position: absolute; top: 0;"></div>
|
||||
|
||||
<!-- banner -->
|
||||
<div id="title">
|
||||
<div id="title-wrapper">
|
||||
<img src="img/logo.png" id="title-icon">
|
||||
<p id="title-text">SecEval</p>
|
||||
</div>
|
||||
<p id="subtitle-text">
|
||||
A Comprehensive Benchmark for Evaluating <br>
|
||||
Cybersecurity Knowledge of Foundation Models
|
||||
<br><br>
|
||||
(2023)
|
||||
</p>
|
||||
<p id="title-padding-bottom"> </p>
|
||||
</div>
|
||||
|
||||
<!-- the main body of the page -->
|
||||
<!-- each section uses an empty div as an anchor -->
|
||||
<div id="body">
|
||||
<div class="section">
|
||||
<div id="about" class="anchor"></div>
|
||||
<h1>About SecEval</h1>
|
||||
<p style="line-height: 150%">
|
||||
SecEval is the first benchmark specifically created for evaluating cybersecurity knowledge in Foundation Models. It offers over 2000 multiple-choice questions across 9 domains: Software Security, Application Security, System Security, Web Security, Cryptography, Memory Safety, Network Security, and PenTest.
|
||||
These questions are generated by prompting OpenAI GPT4 with authoritative sources such as open-licensed textbooks, official documentation, and industry guidelines and standards. The generation process is meticulously crafted to ensure the dataset meets rigorous quality, diversity, and impartiality criteria. You can explore our dataset and detailed methodology in our <a class="ext-link" href="paper_placeholder.html" target="_blank"
|
||||
xt-marked="ok">paper</a>, or explore samples by visiting <a class="ext-link" href="explore.html"
|
||||
xt-marked="ok">Explore</a>.
|
||||
</p>
|
||||
|
||||
</div>
|
||||
|
||||
<!-- <hr> -->
|
||||
<div class="section">
|
||||
<div id="dataset" class="anchor"></div>
|
||||
<h1>SecEval Dataset</h1>
|
||||
<p>
|
||||
Our dataset is distributed under the
|
||||
<a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="blank" class="ext-link">
|
||||
CC BY-NC-SA (Attribution-NonCommercial-ShareAlike)
|
||||
</a>
|
||||
license. You can download the dataset from our
|
||||
<a href="https://github.com/XuanwuAI/SecEval" class="ext-link" target="blank">Github
|
||||
Repository</a> or <a href="https://huggingface.co/datasets/XuanwuAI/SecEval" class="ext-link" target="blank">HuggingFace Datasets</a>!
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
<!-- <hr> -->
|
||||
<div class="section">
|
||||
<div id="citation" class="anchor"></div>
|
||||
<h1>Citation</h1>
|
||||
<pre class="bibtex">
|
||||
@misc{li2023seceval,
|
||||
title={SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models},
|
||||
author={Li, Guancheng and Li, Yifeng and Wang Guannan and Yang, Haoyu and Yu, Yang},
|
||||
publisher = {GitHub},
|
||||
howpublished= "https://github.com/XuanwuAI/SecEval",
|
||||
year={2023}
|
||||
}
|
||||
</pre>
|
||||
<br>
|
||||
</div>
|
||||
|
||||
|
||||
<!-- Contact -->
|
||||
<div class="section" id="contact">
|
||||
<h1>Contact</h1>
|
||||
<p>
|
||||
If you have any questions about SecEval, please contact us at, or open
|
||||
up an issue on
|
||||
<a href="https://github.com/XuanwuAI/SecEval/issues" target="blank" class="ext-link">Github</a>.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Affiliation -->
|
||||
<div class="section" style="table-layout: auto;">
|
||||
<center>
|
||||
<table>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>
|
||||
<a href="https://xlab.tencent.com/en/" target="_blank" rel="external">
|
||||
<img class="center-block" src="img/xlab.png"
|
||||
style="height:6em; max-width: 100%;"></a>
|
||||
</td>
|
||||
<!-- <td> </td> -->
|
||||
</tr>
|
||||
</tbody>
|
||||
</table><br>
|
||||
</center>
|
||||
</div>
|
||||
|
||||
|
||||
</div>
|
||||
<!-- footer -->
|
||||
<div id="footer" style="font-size: 12pt">
|
||||
© 2023 Website developed based on the ScienceQA template.
|
||||
</div>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
|
@ -0,0 +1,204 @@
|
|||
let topics = [
|
||||
'All',
|
||||
'SoftwareSecurity',
|
||||
'NetworkSecurity',
|
||||
'Vulnerability',
|
||||
'MemorySafety',
|
||||
'WebSecurity',
|
||||
'AppSecurity',
|
||||
'Cryptography',
|
||||
'SystemSecurity',
|
||||
'PenTest'
|
||||
]
|
||||
|
||||
|
||||
|
||||
let topic_dd = make_dropdown("Choose a topic:", topics, "topic-dd");
|
||||
let optbtn = document.getElementsByClassName("optionsbtn")[0];
|
||||
let closebtn = document.getElementsByClassName("closebtn")[0];
|
||||
let optionpanel = document.getElementById("option-panel");
|
||||
let body = document.getElementById("content-body");
|
||||
let display = document.getElementById("display");
|
||||
let optboxes = document.getElementsByClassName("optbox");
|
||||
let opt_dds = document.getElementsByClassName("opt-dd");
|
||||
|
||||
optboxes[0].innerHTML += topic_dd;
|
||||
|
||||
topic_dd = document.getElementById("topic-dd");
|
||||
topic_dd.addEventListener("change", filter_data);
|
||||
|
||||
|
||||
let filters = {};
|
||||
|
||||
optbtn.addEventListener("click", openNav);
|
||||
closebtn.addEventListener("click", closeNav);
|
||||
|
||||
|
||||
for (each of opt_dds) {
|
||||
each.addEventListener("change", change_filters);
|
||||
}
|
||||
|
||||
filter_data();
|
||||
|
||||
function openNav() {
|
||||
optionpanel.style.width = "20vw";
|
||||
display.style.width = "60vw";
|
||||
for (each of optionpanel.children) {
|
||||
each.style.display = "block";
|
||||
}
|
||||
}
|
||||
|
||||
function closeNav() {
|
||||
optionpanel.style.width = "0";
|
||||
display.style.width = "80vw";
|
||||
for (each of optionpanel.children) {
|
||||
each.style.display = "none";
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
function change_filters(e) {
|
||||
filters.topic = document.getElementById("topic-dd").value;
|
||||
}
|
||||
|
||||
function create_page(d) {
|
||||
if (d.length === 0) {
|
||||
body.innerHTML = "<p>No example satisfies All the filters.</p>";
|
||||
} else {
|
||||
col1 = create_col(d.slice(0, d.length / 2));
|
||||
col2 = create_col(d.slice(d.length / 2));
|
||||
body.innerHTML = col1 + col2;
|
||||
}
|
||||
reflow(body);
|
||||
console.log("reflowed");
|
||||
}
|
||||
|
||||
function escapeHtml(html) {
|
||||
return html
|
||||
.replace(/&/g, "&")
|
||||
.replace(/</g, "<")
|
||||
.replace(/>/g, ">")
|
||||
.replace(/"/g, """)
|
||||
.replace(/'/g, "'");
|
||||
}
|
||||
|
||||
|
||||
function create_col(data) {
|
||||
res = [];
|
||||
for (each of data) {
|
||||
res.push(create_example(each));
|
||||
}
|
||||
return `<div class="display-col"> ${res.join("")} </div>`;
|
||||
}
|
||||
|
||||
|
||||
function create_example(data) {
|
||||
let escapedQuestion = escapeHtml(data.question);
|
||||
let question = make_qt(escapedQuestion);
|
||||
|
||||
let escapedChoices = data.choices.map(choice => escapeHtml(choice));
|
||||
let choices = make_choices(escapedChoices);
|
||||
|
||||
let answer = make_answer(data.answer);
|
||||
html = make_box([question, choices, answer]) + "<hr/>";
|
||||
return html;
|
||||
|
||||
}
|
||||
|
||||
|
||||
function convert_latex(data) {
|
||||
const re = /(\$+)([^\$]*?[^\\])\1|\\begin\{tabular\}.*?\\end\{tabular\}/gs;
|
||||
let html = '';
|
||||
|
||||
let match;
|
||||
let lastIndex = 0;
|
||||
while ((match = re.exec(data)) !== null) {
|
||||
html += `<span>${data.slice(lastIndex, match.index)}</span>`;
|
||||
|
||||
if (match[0].startsWith('\\begin{tabular}')) {
|
||||
const table = match[0];
|
||||
const tableHtml = '...';
|
||||
html += tableHtml;
|
||||
} else {
|
||||
const latex = match[2];
|
||||
const formula = katex.renderToString(latex, {throwOnError: false});
|
||||
html += `<span>${formula}</span>`;
|
||||
}
|
||||
|
||||
lastIndex = match.index + match[0].length;
|
||||
}
|
||||
|
||||
html += `<span>${data.slice(lastIndex)}</span>`;
|
||||
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_qt(text) {
|
||||
let html = `
|
||||
<p><b>Question </b></p>
|
||||
<p class="question-txt">${convert_latex(text).replace(/\\n|\n/g, "<br>")}</p>
|
||||
`;
|
||||
return html;
|
||||
}
|
||||
|
||||
|
||||
function make_box(contents, cls = "") {
|
||||
if (contents.join("").length === 0) return "";
|
||||
let html = `
|
||||
<div class="box ${cls}">
|
||||
${contents.join(" ").replace(/\\n|\n/g, "")}
|
||||
</div>
|
||||
`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_choices(choices) {
|
||||
let temp = "";
|
||||
let len = 0;
|
||||
for (each of choices) {
|
||||
let html = make_choice(each);
|
||||
temp += html;
|
||||
len += each.length;
|
||||
}
|
||||
let html = "";
|
||||
if (len < 60)
|
||||
html = `<p><b>Choices </b></p><div class="choices">${convert_latex(temp).replace(/\\n|\n/g, "<br>")}</div>`;
|
||||
else
|
||||
html = `<p><b>Choices </b></p><div class="choices-vertical">${convert_latex(temp).replace(/\\n|\n/g, "<br>")}</div>`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_choice(choice) {
|
||||
let html = `<div class="choice-txt">${convert_latex(choice).replace(/\\n|\n/g, "<br>")}</div>`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_answer(answer) {
|
||||
let html = `<p><b>Answer </b></p><p class="answer-txt">${convert_latex(answer)}</p>`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function make_dropdown(label, options, id, default_ind = 0) {
|
||||
let html = "";
|
||||
for (let i = 0; i < options.length; i++) {
|
||||
if (i === default_ind)
|
||||
html += `<option value="${options[i]}" selected> ${options[i]} </option>`;
|
||||
else
|
||||
html += `<option value="${options[i]}"> ${options[i]} </option>`;
|
||||
}
|
||||
html = `<label class="dd-label">${label} <select id="${id}" class="opt-dd"> ${html} </select> </label><br/>`;
|
||||
return html;
|
||||
}
|
||||
|
||||
function filter_data() {
|
||||
change_filters();
|
||||
res = problem_data;
|
||||
if (filters.topic !== "All")
|
||||
res = res.filter(e => e.topics.includes(filters.topic));
|
||||
d = _.sample(res, Math.min(100, res.length));
|
||||
create_page(d);
|
||||
}
|
||||
|
||||
function reflow(elt) {
|
||||
elt.offsetHeight;
|
||||
}
|
||||
|
|
@ -0,0 +1,172 @@
|
|||
leaderboard = [
|
||||
{
|
||||
"Model": "gpt-4-turbo",
|
||||
"Creator": "OpenAI",
|
||||
"Access": "API, Web",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "73.61",
|
||||
"Application Security": "75.25",
|
||||
"PenTest": "80.00",
|
||||
"Memory Safety": "70.83",
|
||||
"Network Security": "75.65",
|
||||
"Web Security": "82.15",
|
||||
"Vulnerability": "76.05",
|
||||
"Software Security": "73.28",
|
||||
"Cryptography": "64.29",
|
||||
"Overall": "79.07",
|
||||
"#": "1"
|
||||
},
|
||||
{
|
||||
"Model": "gpt-3.5-turbo",
|
||||
"Creator": "OpenAI",
|
||||
"Access": "API, Web",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "59.15",
|
||||
"Application Security": "57.18",
|
||||
"PenTest": "72.00",
|
||||
"Memory Safety": "43.75",
|
||||
"Network Security": "60.87",
|
||||
"Web Security": "63.00",
|
||||
"Vulnerability": "60.18",
|
||||
"Software Security": "58.19",
|
||||
"Cryptography": "35.71",
|
||||
"Overall": "62.09",
|
||||
"#": "2"
|
||||
},
|
||||
{
|
||||
"Model": "Yi-6B",
|
||||
"Creator": "01-AI",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "50.61",
|
||||
"Application Security": "48.89",
|
||||
"PenTest": "69.26",
|
||||
"Memory Safety": "35.42",
|
||||
"Network Security": "56.52",
|
||||
"Web Security": "54.98",
|
||||
"Vulnerability": "49.40",
|
||||
"Software Security": "45.69",
|
||||
"Cryptography": "35.71",
|
||||
"Overall": "53.57",
|
||||
"#": "3"
|
||||
},
|
||||
{
|
||||
"Model": "Orca-2-7b",
|
||||
"Creator": "Microsoft",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "46.76",
|
||||
"Application Security": "47.03",
|
||||
"PenTest": "60.84",
|
||||
"Memory Safety": "31.25",
|
||||
"Network Security": "49.13",
|
||||
"Web Security": "55.63",
|
||||
"Vulnerability": "50.00",
|
||||
"Software Security": "52.16",
|
||||
"Cryptography": "14.29",
|
||||
"Overall": "51.60",
|
||||
"#": "4"
|
||||
},
|
||||
{
|
||||
"Model": "Mistral-7B-v0.1",
|
||||
"Creator": "Mistralai",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "40.19",
|
||||
"Application Security": "38.37",
|
||||
"PenTest": "53.47",
|
||||
"Memory Safety": "33.33",
|
||||
"Network Security": "36.52",
|
||||
"Web Security": "46.57",
|
||||
"Vulnerability": "42.22",
|
||||
"Software Security": "43.10",
|
||||
"Cryptography": "28.57",
|
||||
"Overall": "43.65",
|
||||
"#": "5"
|
||||
},
|
||||
{
|
||||
"Model": "chatglm3-6b-base",
|
||||
"Creator": "THUDM",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "39.72",
|
||||
"Application Security": "37.25",
|
||||
"PenTest": "57.47",
|
||||
"Memory Safety": "31.25",
|
||||
"Network Security": "43.04",
|
||||
"Web Security": "41.14",
|
||||
"Vulnerability": "37.43",
|
||||
"Software Security": "39.66",
|
||||
"Cryptography": "28.57",
|
||||
"Overall": "41.58",
|
||||
"#": "6"
|
||||
},
|
||||
{
|
||||
"Model": "Aquila2-7B",
|
||||
"Creator": "BAAI",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "34.84",
|
||||
"Application Security": "36.01",
|
||||
"PenTest": "47.16",
|
||||
"Memory Safety": "22.92",
|
||||
"Network Security": "32.17",
|
||||
"Web Security": "42.04",
|
||||
"Vulnerability": "38.02",
|
||||
"Software Security": "36.21",
|
||||
"Cryptography": "7.14",
|
||||
"Overall": "38.29",
|
||||
"#": "7"
|
||||
},
|
||||
{
|
||||
"Model": "Qwen-7B",
|
||||
"Creator": "Alibaba",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "28.92",
|
||||
"Application Security": "28.84",
|
||||
"PenTest": "41.47",
|
||||
"Memory Safety": "18.75",
|
||||
"Network Security": "29.57",
|
||||
"Web Security": "33.25",
|
||||
"Vulnerability": "31.74",
|
||||
"Software Security": "30.17",
|
||||
"Cryptography": "14.29",
|
||||
"Overall": "31.37",
|
||||
"#": "8"
|
||||
},
|
||||
{
|
||||
"Model": "internlm-7b",
|
||||
"Creator": "Sensetime",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "25.92",
|
||||
"Application Security": "25.87",
|
||||
"PenTest": "36.21",
|
||||
"Memory Safety": "25.00",
|
||||
"Network Security": "27.83",
|
||||
"Web Security": "32.86",
|
||||
"Vulnerability": "29.34",
|
||||
"Software Security": "34.05",
|
||||
"Cryptography": "7.14",
|
||||
"Overall": "30.29",
|
||||
"#": "9"
|
||||
},
|
||||
{
|
||||
"Model": "Llama-2-7b-hf",
|
||||
"Creator": "MetaAI",
|
||||
"Access": "Weight",
|
||||
"Submission Date": "2023-12-20",
|
||||
"System Security": "20.94",
|
||||
"Application Security": "18.69",
|
||||
"PenTest": "26.11",
|
||||
"Memory Safety": "16.67",
|
||||
"Network Security": "14.35",
|
||||
"Web Security": "22.77",
|
||||
"Vulnerability": "21.56",
|
||||
"Software Security": "20.26",
|
||||
"Cryptography": "21.43",
|
||||
"Overall": "22.15",
|
||||
"#": "10"
|
||||
}
|
||||
]
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
let obj_to_tr = (obj) => {
|
||||
return `<tr>
|
||||
<td>${obj["#"]}</td>
|
||||
<td>${obj.Model}</td>
|
||||
<td>${obj.Creator}</td>
|
||||
<td>${obj.Access}</td>
|
||||
<td>${obj["Submission Date"]}</td>
|
||||
<td>${obj["Network Security"]}</td>
|
||||
<td>${obj.Vulnerability}</td>
|
||||
<td>${obj["Memory Safety"]}</td>
|
||||
<td>${obj["Web Security"]}</td>
|
||||
<td>${obj["Application Security"]}</td>
|
||||
<td>${obj.Cryptography}</td>
|
||||
<td>${obj["System Security"]}</td>
|
||||
<td>${obj["Software Security"]}</td>
|
||||
<td>${obj["PenTest"]}</td>
|
||||
<td>${obj.Overall}</td>
|
||||
</tr>`
|
||||
}
|
||||
|
||||
let table = document.getElementById("results");
|
||||
html = "";
|
||||
for (let i of leaderboard) {
|
||||
html += obj_to_tr(i);
|
||||
}
|
||||
table.getElementsByTagName("tbody")[0].innerHTML = html;
|
||||
function reflow(elt){
|
||||
console.log(elt.offsetHeight);
|
||||
}
|
||||
reflow(table);
|
||||
|
|
@ -0,0 +1,309 @@
|
|||
/**
|
||||
* sort-table.js
|
||||
* A pure JavaScript (no dependencies) solution to make HTML
|
||||
* Tables sortable
|
||||
*
|
||||
* Copyright (c) 2013 Tyler Uebele
|
||||
* Released under the MIT license. See included LICENSE.txt
|
||||
* or http://opensource.org/licenses/MIT
|
||||
*
|
||||
* latest version available at https://github.com/tyleruebele/sort-table
|
||||
*/
|
||||
|
||||
/**
|
||||
* Sort the rows in a HTML Table
|
||||
*
|
||||
* @param Table The Table DOM object
|
||||
* @param col The zero-based column number by which to sort
|
||||
* @param dir Optional. The sort direction; pass 1 for asc; -1 for desc
|
||||
* @returns void
|
||||
*/
|
||||
function sortTable(Table, col, dir) {
|
||||
var sortClass, i;
|
||||
|
||||
// get previous sort column
|
||||
sortTable.sortCol = -1;
|
||||
sortClass = Table.className.match(/js-sort-\d+/);
|
||||
if (null != sortClass) {
|
||||
sortTable.sortCol = sortClass[0].replace(/js-sort-/, '');
|
||||
Table.className = Table.className.replace(new RegExp(' ?' + sortClass[0] + '\\b'), '');
|
||||
}
|
||||
// If sort column was not passed, use previous
|
||||
if ('undefined' === typeof col) {
|
||||
col = sortTable.sortCol;
|
||||
}
|
||||
|
||||
if ('undefined' !== typeof dir) {
|
||||
// Accept -1 or 'desc' for descending. All else is ascending
|
||||
sortTable.sortDir = dir == -1 || dir == 'desc' ? -1 : 1;
|
||||
} else {
|
||||
// sort direction was not passed, use opposite of previous
|
||||
sortClass = Table.className.match(/js-sort-(a|de)sc/);
|
||||
if (null != sortClass && sortTable.sortCol == col) {
|
||||
sortTable.sortDir = 'js-sort-asc' == sortClass[0] ? -1 : 1;
|
||||
} else {
|
||||
sortTable.sortDir = 1;
|
||||
}
|
||||
}
|
||||
Table.className = Table.className.replace(/ ?js-sort-(a|de)sc/g, '');
|
||||
|
||||
// update sort column
|
||||
Table.className += ' js-sort-' + col;
|
||||
sortTable.sortCol = col;
|
||||
|
||||
// update sort direction
|
||||
Table.className += ' js-sort-' + (sortTable.sortDir == -1 ? 'desc' : 'asc');
|
||||
|
||||
// get sort type
|
||||
if (col < Table.tHead.rows[Table.tHead.rows.length - 1].cells.length) {
|
||||
sortClass = Table.tHead.rows[Table.tHead.rows.length - 1].cells[col].className.match(/js-sort-[-\w]+/);
|
||||
}
|
||||
// Improved support for colspan'd headers
|
||||
for (i = 0; i < Table.tHead.rows[Table.tHead.rows.length - 1].cells.length; i++) {
|
||||
if (col == Table.tHead.rows[Table.tHead.rows.length - 1].cells[i].getAttribute('data-js-sort-colNum')) {
|
||||
sortClass = Table.tHead.rows[Table.tHead.rows.length - 1].cells[i].className.match(/js-sort-[-\w]+/);
|
||||
}
|
||||
}
|
||||
if (null != sortClass) {
|
||||
sortTable.sortFunc = sortClass[0].replace(/js-sort-/, '');
|
||||
} else {
|
||||
sortTable.sortFunc = 'string';
|
||||
}
|
||||
// Set the headers for the active column to have the decorative class
|
||||
Table.querySelectorAll('.js-sort-active').forEach(function(Node) {
|
||||
Node.className = Node.className.replace(/ ?js-sort-active\b/, '');
|
||||
});
|
||||
Table.querySelectorAll('[data-js-sort-colNum="' + col + '"]:not(:empty)').forEach(function(Node) {
|
||||
Node.className += ' js-sort-active';
|
||||
});
|
||||
|
||||
// sort!
|
||||
var rows = [],
|
||||
TBody = Table.tBodies[0];
|
||||
|
||||
for (i = 0; i < TBody.rows.length; i++) {
|
||||
rows[i] = TBody.rows[i];
|
||||
}
|
||||
if ('none' != sortTable.sortFunc) {
|
||||
rows.sort(sortTable.compareRow);
|
||||
}
|
||||
|
||||
while (TBody.firstChild) {
|
||||
TBody.removeChild(TBody.firstChild);
|
||||
}
|
||||
for (i = 0; i < rows.length; i++) {
|
||||
TBody.appendChild(rows[i]);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare two table rows based on current settings
|
||||
*
|
||||
* @param RowA A TR DOM object
|
||||
* @param RowB A TR DOM object
|
||||
* @returns {number} 1 if RowA is greater, -1 if RowB, 0 if equal
|
||||
*/
|
||||
sortTable.compareRow = function(RowA, RowB) {
|
||||
var valA, valB;
|
||||
if ('function' != typeof sortTable[sortTable.sortFunc]) {
|
||||
sortTable.sortFunc = 'string';
|
||||
}
|
||||
valA = sortTable[sortTable.sortFunc](RowA.cells[sortTable.sortCol]);
|
||||
valB = sortTable[sortTable.sortFunc](RowB.cells[sortTable.sortCol]);
|
||||
|
||||
return valA == valB ? 0 : sortTable.sortDir * (valA > valB ? 1 : -1);
|
||||
};
|
||||
|
||||
/**
|
||||
* Strip all HTML, no exceptions
|
||||
* @param html
|
||||
* @returns {string}
|
||||
*/
|
||||
sortTable.stripTags = function(html) {
|
||||
replace_unit = (s) => {
|
||||
let iUnit = (s.indexOf('M') > -1) ? s.indexOf('M') : s.indexOf('B');
|
||||
if (iUnit == -1) return s;
|
||||
let unit = s[iUnit];
|
||||
let val = Number(s.substring(0, iUnit));
|
||||
if (isNaN(val)) return s;
|
||||
val *= (unit == 'M') ? 1000000 : 1000000000;
|
||||
return val.toString();
|
||||
}
|
||||
html = replace_unit(html);
|
||||
return html.replace(/<\/?[a-z][a-z0-9]*\b[^>]*>/gi, '');
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Converts innerHTML to a timestamp, 0 for invalid dates
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {Number}
|
||||
*/
|
||||
sortTable.date = function(Cell) {
|
||||
// If okDate library is available, Use it for advanced Date processing
|
||||
if (typeof okDate !== 'undefined') {
|
||||
var kDate = okDate(sortTable.stripTags(Cell.innerHTML));
|
||||
return kDate ? kDate.getTime() : 0;
|
||||
} else {
|
||||
return (new Date(sortTable.stripTags(Cell.innerHTML))).getTime() || 0;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Converts innerHTML to a JS Number object
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {Number}
|
||||
*/
|
||||
sortTable.number = function(Cell) {
|
||||
return Number(sortTable.stripTags(Cell.innerHTML).replace(/[^-\d.]/g, ''));
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Converts innerHTML to a lower case string for insensitive compare
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {String}
|
||||
*/
|
||||
sortTable.string = function(Cell) {
|
||||
return sortTable.stripTags(Cell.innerHTML).toLowerCase();
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {String}
|
||||
*/
|
||||
sortTable.raw = function(Cell) {
|
||||
return Cell.innerHTML;
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Captures the last space-delimited token from innerHTML
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {String}
|
||||
*/
|
||||
sortTable.last = function(Cell) {
|
||||
return sortTable.stripTags(Cell.innerHTML).split(' ').pop().toLowerCase();
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that converts a table cell (TD) to a comparable value
|
||||
* Captures the value of the first childNode
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns {String}
|
||||
*/
|
||||
sortTable.input = function(Cell) {
|
||||
for (var i = 0; i < Cell.children.length; i++) {
|
||||
if ('object' == typeof Cell.children[i]
|
||||
&& 'undefined' != typeof Cell.children[i].value
|
||||
) {
|
||||
return Cell.children[i].value.toLowerCase();
|
||||
}
|
||||
}
|
||||
|
||||
return sortTable.string(Cell);
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper function that prevents sorting by always returning null
|
||||
*
|
||||
* @param Cell A TD DOM object
|
||||
* @returns null
|
||||
*/
|
||||
sortTable.none = function(Cell) {
|
||||
return null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Return the click handler appropriate to the specified Table and column
|
||||
*
|
||||
* @param Table Table to sort
|
||||
* @param col Column to sort by
|
||||
* @returns {Function} Click Handler
|
||||
*/
|
||||
sortTable.getClickHandler = function(Table, col) {
|
||||
return function() {
|
||||
sortTable(Table, col);
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Attach sortTable() calls to table header cells' onclick events
|
||||
* If the table(s) do not have a THead node, one will be created around the
|
||||
* first row
|
||||
*/
|
||||
sortTable.init = function() {
|
||||
var THead, Tables, Handler;
|
||||
if (document.querySelectorAll) {
|
||||
Tables = document.querySelectorAll('table.js-sort-table');
|
||||
} else {
|
||||
Tables = document.getElementsByTagName('table');
|
||||
}
|
||||
|
||||
for (var i = 0; i < Tables.length; i++) {
|
||||
// Because IE<8 doesn't support querySelectorAll, skip unclassed tables
|
||||
if (!document.querySelectorAll && null === Tables[i].className.match(/\bjs-sort-table\b/)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Prevent repeat processing
|
||||
if (Tables[i].attributes['data-js-sort-table']) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Ensure table has a tHead element
|
||||
if (!Tables[i].tHead) {
|
||||
THead = document.createElement('thead');
|
||||
THead.appendChild(Tables[i].rows[0]);
|
||||
Tables[i].insertBefore(THead, Tables[i].children[0]);
|
||||
} else {
|
||||
THead = Tables[i].tHead;
|
||||
}
|
||||
|
||||
// Attach click events to table header
|
||||
for (var rowNum = 0; rowNum < THead.rows.length; rowNum++) {
|
||||
for (var cellNum = 0, colNum = 0; cellNum < THead.rows[rowNum].cells.length; cellNum++) {
|
||||
// Skip headers marked "js-sort-none"
|
||||
if (THead.rows[rowNum].cells[cellNum].className.match(/\bjs-sort-none\b/)) {
|
||||
continue;
|
||||
}
|
||||
// Define which column the header should invoke sorting for
|
||||
THead.rows[rowNum].cells[cellNum].setAttribute('data-js-sort-colNum', colNum);
|
||||
Handler = sortTable.getClickHandler(Tables[i], colNum);
|
||||
window.addEventListener
|
||||
? THead.rows[rowNum].cells[cellNum].addEventListener('click', Handler)
|
||||
: window.attachEvent && THead.rows[rowNum].cells[cellNum].attachEvent('onclick', Handler);
|
||||
colNum += THead.rows[rowNum].cells[cellNum].colSpan;
|
||||
}
|
||||
}
|
||||
|
||||
// Mark table as processed
|
||||
Tables[i].setAttribute('data-js-sort-table', 'true')
|
||||
}
|
||||
|
||||
// Add default styles as the first style in head so they can be easily overwritten by user styles
|
||||
var element = document.createElement('style');
|
||||
document.head.insertBefore(element, document.head.childNodes[0]);
|
||||
var sheet = element.sheet;
|
||||
sheet.insertRule('table.js-sort-table.js-sort-asc thead tr > .js-sort-active:not(.js-sort-none):after {content: "\\25b2";font-size: 0.7em;padding-left: 3px;line-height: 0.7em;}', 0);
|
||||
sheet.insertRule('table.js-sort-table.js-sort-desc thead tr > .js-sort-active:not(.js-sort-none):after {content: "\\25bc";font-size: 0.7em;padding-left: 3px;line-height: 0.7em;}', 0);
|
||||
};
|
||||
|
||||
// Run sortTable.init() when the page loads
|
||||
window.addEventListener
|
||||
? window.addEventListener('load', sortTable.init, false)
|
||||
: window.attachEvent && window.attachEvent('onload', sortTable.init)
|
||||
;
|
||||
|
||||
// Shim for IE11's lack of NodeList.prototype.forEach
|
||||
if (typeof NodeList.prototype.forEach !== "function") {
|
||||
NodeList.prototype.forEach = Array.prototype.forEach;
|
||||
}
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
import sys, os
|
||||
from math import ceil, floor, sqrt, log2
|
||||
from collections import defaultdict, deque
|
||||
input = lambda: sys.stdin.readline().strip()
|
||||
intlist = lambda: [int(i) for i in input().split()]
|
||||
flolist = lambda: [float(f) for f in input().split()]
|
||||
mat = lambda a, b, v: [[v] * b for _ in range(a)]
|
||||
from rich import print
|
||||
import json
|
||||
import re
|
||||
|
||||
def remove_html_tags(text):
|
||||
"""Remove html tags from a string"""
|
||||
clean = re.compile('<.*?>')
|
||||
return re.sub(clean, '', text)
|
||||
|
||||
def debold(ls):
|
||||
for i in range(len(ls)):
|
||||
ls[i] = ls[i].replace("**", "")
|
||||
return ls
|
||||
|
||||
def process_sources(s):
|
||||
# [source](link) to {Source: source, Link: link}
|
||||
s = s.split("](")
|
||||
source = s[0][1:]
|
||||
link = s[1][:-1]
|
||||
return {"Source": source, "Link": link}
|
||||
|
||||
# read leaderboard.md table and create a list of dictionaries
|
||||
with open("leaderboard.md", "r") as f:
|
||||
header = f.readline().strip().split("|")
|
||||
header = [h.strip() for h in header]
|
||||
header = header[1:-1]
|
||||
debold(header)
|
||||
header = [remove_html_tags(h) for h in header]
|
||||
lines = f.readlines()
|
||||
lines = [l.strip().split("|") for l in lines]
|
||||
lines = [[e.strip() for e in l] for l in lines]
|
||||
lines = [[remove_html_tags(e) for e in l] for l in lines]
|
||||
lines = [l[1:-1] for l in lines]
|
||||
lines = [debold(l) for l in lines]
|
||||
lines = lines[1:]
|
||||
lines = [dict(zip(header, l)) for l in lines]
|
||||
|
||||
for ln in lines:
|
||||
ln["Source"] = process_sources(ln["Source"])
|
||||
|
||||
with open("leaderboard-data.js", "w") as f:
|
||||
f.write("leaderboard = ")
|
||||
json.dump(lines, f, indent=4)
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,205 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def print_result(result):\n",
|
||||
" nums = result.split(\" \")\n",
|
||||
" for num in nums:\n",
|
||||
" print(f\"<td>{num}</td>\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>40.28</td>\n",
|
||||
"<td>46.13</td>\n",
|
||||
"<td>29.25</td>\n",
|
||||
"<td>47.45</td>\n",
|
||||
"<td>40.08</td>\n",
|
||||
"<td>33.66</td>\n",
|
||||
"<td>39.35</td>\n",
|
||||
"<td>40.67</td>\n",
|
||||
"<td>39.83</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Random chance\n",
|
||||
"result = \"40.28 46.13 29.25 47.45 40.08 33.66 39.35 40.67 39.83\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>90.23</td>\n",
|
||||
"<td>84.97</td>\n",
|
||||
"<td>87.48</td>\n",
|
||||
"<td>89.60</td>\n",
|
||||
"<td>87.50</td>\n",
|
||||
"<td>88.10</td>\n",
|
||||
"<td>91.59</td>\n",
|
||||
"<td>82.42</td>\n",
|
||||
"<td>88.40</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Human\n",
|
||||
"result = \"90.23 84.97 87.48 89.60 87.50 88.10 91.59 82.42 88.40\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>75.04</td>\n",
|
||||
"<td>66.59</td>\n",
|
||||
"<td>78.00</td>\n",
|
||||
"<td>74.24</td>\n",
|
||||
"<td>65.74</td>\n",
|
||||
"<td>79.58</td>\n",
|
||||
"<td>76.36</td>\n",
|
||||
"<td>69.87</td>\n",
|
||||
"<td>74.04</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# GPT-3 zero-shot QCM→A\n",
|
||||
"result = \"75.04 66.59 78.00 74.24 65.74 79.58 76.36 69.87 74.04\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>74.64</td>\n",
|
||||
"<td>69.74</td>\n",
|
||||
"<td>76.00</td>\n",
|
||||
"<td>74.44</td>\n",
|
||||
"<td>67.28</td>\n",
|
||||
"<td>77.42</td>\n",
|
||||
"<td>76.80</td>\n",
|
||||
"<td>68.89</td>\n",
|
||||
"<td>73.97</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# GPT-3 2-shot QCM→A\n",
|
||||
"result = \"74.64 69.74 76.00 74.44 67.28 77.42 76.80 68.89 73.97\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>76.60</td>\n",
|
||||
"<td>65.92</td>\n",
|
||||
"<td>77.55</td>\n",
|
||||
"<td>75.51</td>\n",
|
||||
"<td>66.09</td>\n",
|
||||
"<td>79.58</td>\n",
|
||||
"<td>78.49</td>\n",
|
||||
"<td>67.63</td>\n",
|
||||
"<td>74.61</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# GPT-3 (CoT) 2-shot QCM→AE\n",
|
||||
"result = \"76.60 65.92 77.55 75.51 66.09 79.58 78.49 67.63 74.61\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<td>75.44</td>\n",
|
||||
"<td>70.87</td>\n",
|
||||
"<td>78.09</td>\n",
|
||||
"<td>74.68</td>\n",
|
||||
"<td>67.43</td>\n",
|
||||
"<td>79.93</td>\n",
|
||||
"<td>78.23</td>\n",
|
||||
"<td>69.68</td>\n",
|
||||
"<td>75.17</td>\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# GPT-3 (CoT) 2-shot QCM→ALE\n",
|
||||
"result = \"75.44 70.87 78.09 74.68 67.43 79.93 78.23 69.68 75.17\"\n",
|
||||
"print_result(result)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.9.1rc1 64-bit",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.1"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"vscode": {
|
||||
"interpreter": {
|
||||
"hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
|
||||
}
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,83 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Leaderboard | SecEval: A Comprehensive Benchmark for Evaluating Cybersecurity Knowledge of Foundation Models</title>
|
||||
<link rel="icon" href="img/logo.png" type="image/icon type">
|
||||
<link rel="stylesheet" href="css/main.css">
|
||||
<link rel="stylesheet" href="css/nav.css">
|
||||
<link rel="stylesheet" href="css/leaderboard.css">
|
||||
<script type="text/javascript" src="javascript/leaderboard/sort-table.js" defer></script>
|
||||
<script type="text/javascript" src="javascript/leaderboard/leaderboard-data.js" defer></script>
|
||||
<script type="text/javascript" src="javascript/leaderboard/leaderboard.js" defer></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="nav">
|
||||
<div id="icon">
|
||||
<img src="img/logo.png" id="nav-icon">
|
||||
<a class="nav-button" href="https://XuanwuAI.github.io/SecEval/"
|
||||
style="margin-left: 2px; font-size: 24px">SecEval
|
||||
</a>
|
||||
</div>
|
||||
<div>
|
||||
<a class="nav-button" href="index.html#home">Home</a>
|
||||
<a class="nav-button" href="index.html#dataset">Dataset</a>
|
||||
<a class="nav-button" href="#paper_placeholder.html" target="_blank">Paper</a>
|
||||
<a class="nav-button" href="https://github.com/XuanwuAI/SecEval" target="_blank">Github</a>
|
||||
<a class="nav-button" href="index.html#citation">Citation</a>
|
||||
<a class="nav-button" href="index.html#contact">Contact</a>
|
||||
<a class="nav-button" href="leaderboard.html">Leaderboard</a>
|
||||
<a class="nav-button" href="explore.html">Explore</a>
|
||||
</div>
|
||||
</div>
|
||||
<div id="body">
|
||||
<br><br><br><br>
|
||||
<div class="section">
|
||||
<h1>Leaderboard - SecEval</h1>
|
||||
<p>
|
||||
Evaluation results of different models.
|
||||
</p>
|
||||
<!--
|
||||
<p>
|
||||
Please fill in this
|
||||
<a href="https://forms.gle/iwEMepqowMkknNvb8" class="ext-link">Google Form
|
||||
</a> to submit your evaluation results.
|
||||
</p>
|
||||
-->
|
||||
|
||||
<!-- https://www.cssscript.com/sort-table-header-column/ -->
|
||||
<table class="js-sort-table" id="results">
|
||||
<thead>
|
||||
<tr>
|
||||
<td class="js-sort-number"><strong>#</strong></td>
|
||||
<td class="js-sort-number"><strong>Model</strong></td>
|
||||
<!-- <td class="js-sort-number"><strong>Learning</strong></td> -->
|
||||
<td class="js-sort-number"><strong>Creator</strong></td>
|
||||
<td class="js-sort-number"><strong>Access</strong></td>
|
||||
<td class="js-sort-number"><strong>Submission Date</strong></td>
|
||||
<td class="js-sort-number"><strong>Network Security</strong></td>
|
||||
<td class="js-sort-number"><strong><u>Vulnerability</u></strong></td>
|
||||
<td class="js-sort-number"><strong>Memory Safety</strong></td>
|
||||
<td class="js-sort-number"><strong>Web Security</strong></td>
|
||||
<td class="js-sort-number"><strong>Application Security</strong></td>
|
||||
<td class="js-sort-number"><strong>Cryptography</strong></td>
|
||||
<td class="js-sort-number"><strong>System Security</strong></td>
|
||||
<td class="js-sort-number"><strong>Software Security</strong></td>
|
||||
<td class="js-sort-number"><strong>PenTest</strong></td>
|
||||
<td class="js-sort-number"><strong>Overall</strong></td>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
</tbody>
|
||||
|
||||
</table>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
|
@ -0,0 +1 @@
|
|||
Coming Soon
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
# Evaluating on SecEval Dataset
|
||||
|
||||
## Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Download Dataset
|
||||
|
||||
You can download the json file of the dataset by running.
|
||||
|
||||
```
|
||||
wget https://huggingface.co/datasets/XuanwuAI/SecEval/blob/main/questions.json
|
||||
```
|
||||
|
||||
## Evaluate on a Model
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
usage: eval.py [-h] [-o OUTPUT_DIR] -d DATASET_FILE [-c] [-b BATCH_SIZE] -B {remote_hf,azure,textgen,local_hf} -m MODELS [MODELS ...]
|
||||
|
||||
SecEval Evaluation CLI
|
||||
|
||||
options:
|
||||
-h, --help show this help message and exit
|
||||
-o OUTPUT_DIR, --output_dir OUTPUT_DIR
|
||||
Specify the output directory.
|
||||
-d DATASET_FILE, --dataset_file DATASET_FILE
|
||||
Specify the dataset file to evaluate on.
|
||||
-c, --chat Evaluate on chat model.
|
||||
-b BATCH_SIZE, --batch_size BATCH_SIZE
|
||||
Specify the batch size.
|
||||
-B {remote_hf,azure,textgen,local_hf}, --backend {remote_hf,azure,textgen,local_hf}
|
||||
Specify the llm type. remote_hf: remote huggingface model backed, azure: azure openai model, textgen: textgen backend, local_hf: local huggingface model backed
|
||||
-m MODELS [MODELS ...], --models MODELS [MODELS ...]
|
||||
Specify the models.
|
||||
```
|
||||
|
||||
### Select a Backend
|
||||
|
||||
There are four backends supported for evaluating on SecEval dataset.
|
||||
Some backends require additional environment variables to be set.
|
||||
|
||||
#### Local Huggingface Model Backend
|
||||
|
||||
Local Huggingface Model Backend is used for evaluating on models that are stored locally. You can download the models from [Huggingface](https://huggingface.co/models) and use this backend to evalute the model.
|
||||
|
||||
```bash
|
||||
export HF_LOCAL_MODEL_DIR=<path to local huggingface model>
|
||||
```
|
||||
|
||||
#### Remote Huggingface Model Backend
|
||||
|
||||
Remote Huggingface Model Backend is used for evaluating on models that are stored on Huggingface. You can use this backend to evalute the model.
|
||||
|
||||
|
||||
#### Azure OpenAI Model Backend
|
||||
|
||||
Azure OpenAI Model Backend is used for evaluating on models that are deployed on Azure OpenAI.
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=<your openai api key>
|
||||
export OPENAI_API_ENDPOINT=<your openai api endpoint>
|
||||
```
|
||||
|
||||
|
||||
#### text-generation-webui Backend
|
||||
|
||||
[text-generation-webui](https://github.com/oobabooga/text-generation-webui) is a service used for deploying and testing models. We have supported evaluating on models deployed on text-generation-webui.
|
||||
|
||||
```bash
|
||||
|
||||
export TEXTGEN_MODEL_URL=<your textgenerationwebui api url>
|
||||
```
|
||||
|
||||
### Run Evaluation
|
||||
|
||||
If you want to evaluate on a base model, you can run the following command:
|
||||
|
||||
```bash
|
||||
python eval.py -d <path to dataset file> -B <backend> -m <model name> -o <output directory>
|
||||
```
|
||||
|
||||
If you want to evaluate on a chat model such as gpt-3.5-turbo, you should add the `-c` flag:
|
||||
|
||||
```bash
|
||||
python eval.py -d <path to dataset file> -B <backend> -m <model name> -o <output directory> -c
|
||||
```
|
||||
|
||||
|
||||
## Inspection the Results
|
||||
|
||||
The results will be saved in the output directory. The result is a JSONObject contains following field
|
||||
|
||||
```json
|
||||
{
|
||||
"score_fraction": {
|
||||
"topic_name": "topic_score_fraction",
|
||||
},
|
||||
"score_float": {
|
||||
"topic_name": "topic_score_float",
|
||||
},
|
||||
"detail": ["detail1", "detail2",]
|
||||
}
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,327 @@
|
|||
import argparse
|
||||
from typing import Any, Dict, List
|
||||
from dotenv import load_dotenv
|
||||
import asyncio
|
||||
load_dotenv()
|
||||
from langchain_community.llms import HuggingFacePipeline, TextGen
|
||||
from langchain_community.chat_models import AzureChatOpenAI
|
||||
from langchain_community.adapters.openai import convert_message_to_dict
|
||||
from langchain_community.cache import SQLiteCache
|
||||
from langchain.schema.language_model import BaseLanguageModel
|
||||
from langchain.adapters.openai import convert_message_to_dict
|
||||
from langchain.schema.messages import (
|
||||
AIMessage,
|
||||
SystemMessage,
|
||||
HumanMessage,
|
||||
)
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
import os
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
import time
|
||||
from langchain.globals import set_llm_cache
|
||||
from langchain.cache import SQLiteCache
|
||||
|
||||
set_llm_cache(
|
||||
SQLiteCache(
|
||||
database_path=str(Path(__file__).parent.parent / ".langchain.db")
|
||||
)
|
||||
)
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(message)s",
|
||||
)
|
||||
logger.addHandler(logging.FileHandler(f"./eval-{int(time.time())}.log", "w"))
|
||||
instruction = "Below are multiple-choice questions concerning cybersecurity. Please select the correct answers and respond with the letters ABCD only."
|
||||
chat_few_shot = [
|
||||
HumanMessage(
|
||||
content="Question: Which mitigation prevent stack overflow bug? A: Stack Canary. B: ALSR. C: CFI. D: Code Signing.",
|
||||
),
|
||||
AIMessage(
|
||||
content="Answer: ABC",
|
||||
),
|
||||
]
|
||||
few_shot = """
|
||||
Question: You are reviewing a piece of code to identify potential vulnerabilities related to NULL pointer dereferences. Which of the following strategies should you consider implementing to mitigate the risk? Choose all that apply. A: Use a programming language that inherently manages pointers and prevents direct memory access. B: Apply input validation to all variables and data stores that receive data from external sources to ensure they are initialized to expected values. C: Explicitly initialize all variables and other data stores, either during declaration or just before the first usage. D: Implement a custom memory management system that automatically assigns a unique identifier to each pointer.
|
||||
Answer: ABC
|
||||
Question: Analyze the code snippet provided. Which of the following practices should be implemented in a multithreaded application to prevent NULL pointer dereference vulnerabilities? Select all that apply.A: Ensure that proper locking APIs are used to lock the pointer before the if statement and unlock when it has finished to prevent race conditions. B: Check the return value of all functions that return a value, including network-related functions, and verify that it is non-null before using it.C: Use automated static analysis tools that target this type of weakness, understanding that while not perfect, they can still be effective.D: Verify that a non-nil 'response' is present before deferring 'response.Body.Close()' to handle cases where the 'Do' method returns an error.
|
||||
Answer: ABCD
|
||||
Question: In a bank management application, a method called `createBankAccount` allows the creation of bank accounts without user identity verification or authority checks. Based on common security practices, which of the following measures would improve security?A: Implement a centralized authentication mechanism for operations that require a proven user identity.B: Duplicate client-side authentication checks on the server side to prevent bypassing of security checks.C: Use a vetted library or framework that offers robust authentication capabilities to prevent security weaknesses.D: Apply strong authentication requirements for all users accessing data stored in the cloud.
|
||||
Answer: ABCD
|
||||
Question: What are the potential consequences if an application handling sensitive data does not require authentication for data access, and what are the appropriate mitigation strategies?A: Attackers could easily access sensitive data from any location; require strong authentication for data access using cloud provider controls.B: Security checks might be bypassed if only performed on the client side; implement checks on both client and server sides.C: Unauthenticated users could alter product functionality; do not use authentication for critical functionality in products.D: Sensitive data may be accessed without proper credentials; utilize authentication capabilities provided by the framework or operating system.
|
||||
Answer: ABD
|
||||
Question: To prevent security vulnerabilities related to deserialization of untrusted data in a Java application, which of the following practices should a developer implement?A: Use the signing/sealing features of the programming language to assure that deserialized data has not been tainted.B: Explicitly define a final readObject() method to throw an exception and prevent deserialization.C: Populate a new object by deserializing data to ensure data flows through safe input validation functions.D: Make fields transient to protect them from deserialization and prevent carrying over sensitive variables.
|
||||
Answer: ABCD
|
||||
"""
|
||||
|
||||
|
||||
def init_hf_llm(model_id: str):
|
||||
# check transformers and torch installation
|
||||
try:
|
||||
import transformers
|
||||
except ImportError:
|
||||
raise ImportError("Please install transformers with `pip install transformers`")
|
||||
try:
|
||||
import torch
|
||||
|
||||
flash_attn_enable = torch.cuda.get_device_capability()[0] >= 8
|
||||
except ImportError:
|
||||
raise ImportError("Please install torch with `pip install torch`")
|
||||
|
||||
# todo: add flash_attn_enable to the model_kwargs
|
||||
|
||||
llm = HuggingFacePipeline.from_model_id(
|
||||
model_id=model_id,
|
||||
task="text-generation",
|
||||
pipeline_kwargs={"max_new_tokens": 5},
|
||||
device=0,
|
||||
model_kwargs={"trust_remote_code": True, "torch_dtype": torch.bfloat16},
|
||||
)
|
||||
return llm
|
||||
|
||||
|
||||
def init_textgen_llm(model_id: str):
|
||||
if os.environ.get("TEXTGEN_MODEL_URL") is None:
|
||||
raise RuntimeError("Please set TEXTGEN_MODEL_URL")
|
||||
llm = TextGen(model_url=os.environ["TEXTGEN_MODEL_URL"]) # type: ignore
|
||||
return llm
|
||||
|
||||
|
||||
def init_azure_openai_llm(model_id: str):
|
||||
if os.environ.get("OPENAI_API_ENDPOINT") is None:
|
||||
raise RuntimeError("Please set OPENAI_API_ENDPOINT")
|
||||
if os.environ.get("OPENAI_API_KEY") is None:
|
||||
raise RuntimeError("Please set OPENAI_API_KEY")
|
||||
azure_params = {
|
||||
"model": model_id,
|
||||
"openai_api_base": os.environ["OPENAI_API_ENDPOINT"],
|
||||
"openai_api_key": os.environ["OPENAI_API_KEY"],
|
||||
"openai_api_type": os.environ.get("OPENAI_API_TYPE", "azure"),
|
||||
"openai_api_version": "2023-07-01-preview",
|
||||
}
|
||||
return AzureChatOpenAI(**azure_params) # type: ignore
|
||||
|
||||
|
||||
def init_ollama_llm(model_id: str):
|
||||
import openai
|
||||
|
||||
# Establecer el endpoint de Ollama
|
||||
openai.api_base = os.getenv("OLLAMA_API_BASE", "http://localhost:8000/v1")
|
||||
openai.api_key = "ollama" # No se usa pero es requerido por el cliente
|
||||
|
||||
class OllamaChat:
|
||||
async def abatch(self, prompts: List[str]):
|
||||
responses = []
|
||||
for prompt in prompts:
|
||||
try:
|
||||
completion = openai.ChatCompletion.create(
|
||||
model=model_id,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
)
|
||||
responses.append(AIMessage(content=completion.choices[0].message["content"]))
|
||||
except Exception as e:
|
||||
logging.error(f"Ollama error: {e}")
|
||||
responses.append(f"Error: {e}")
|
||||
return responses
|
||||
|
||||
return OllamaChat()
|
||||
|
||||
def load_dataset(dataset_path: str):
|
||||
with open(dataset_path, "r") as f:
|
||||
dataset = json.load(f)
|
||||
return dataset
|
||||
|
||||
|
||||
async def batch_inference_dataset(
|
||||
llm: BaseLanguageModel, batch: List[Dict[str, Any]], chat=False
|
||||
):
|
||||
results = []
|
||||
llm_inputs = []
|
||||
for dataset_row in batch:
|
||||
question_text = (
|
||||
"Question: " + dataset_row["question"] + " ".join(dataset_row["choices"])
|
||||
)
|
||||
question_text = question_text.replace("\n", " ")
|
||||
if chat:
|
||||
llm_input = (
|
||||
[SystemMessage(content=instruction)]
|
||||
+ chat_few_shot
|
||||
+ [HumanMessage(content=question_text)]
|
||||
)
|
||||
else:
|
||||
llm_input = instruction + few_shot + question_text + "\n"
|
||||
|
||||
llm_inputs.append(llm_input)
|
||||
try:
|
||||
llm_outputs = await llm.abatch(llm_inputs)
|
||||
except Exception as e:
|
||||
logging.error(f"error in processing batch {e}")
|
||||
llm_outputs = [f"{e}" * len(llm_inputs)]
|
||||
for idx, llm_output in enumerate(llm_outputs):
|
||||
if type(llm_output) == AIMessage:
|
||||
llm_output: str = llm_output.content # type: ignore
|
||||
if "Answer:" in llm_output:
|
||||
llm_output = llm_output.replace("Answer:", "")
|
||||
if chat:
|
||||
batch[idx]["llm_input"] = convert_message_to_dict(llm_inputs[idx])
|
||||
else:
|
||||
batch[idx]["llm_input"] = llm_inputs[idx]
|
||||
batch[idx]["llm_output"] = llm_output
|
||||
batch[idx]["llm_answer"] = "".join(
|
||||
sorted(list(set(re.findall(r"[A-D]", llm_output))))
|
||||
)
|
||||
batch[idx]["score"] = int(
|
||||
batch[idx]["llm_answer"].lower() == batch[idx]["answer"].lower()
|
||||
)
|
||||
logging.info(
|
||||
f'llm_output: {llm_output}, parsed answer: {batch[idx]["llm_answer"]}, answer: {batch[idx]["answer"]}'
|
||||
)
|
||||
results.append(batch[idx])
|
||||
return results
|
||||
|
||||
|
||||
def inference_dataset(
|
||||
llm: BaseLanguageModel,
|
||||
dataset: List[Dict[str, Any]],
|
||||
batch_size: int = 1,
|
||||
chat: bool = False,
|
||||
):
|
||||
# Prepare the batched inference
|
||||
def chunks(lst, n):
|
||||
for i in range(0, len(lst), n):
|
||||
yield lst[i : i + n]
|
||||
|
||||
# Asynchronously process dataset in batches
|
||||
loop = asyncio.get_event_loop()
|
||||
batches = list(chunks(dataset, batch_size))
|
||||
results = []
|
||||
for idx, batch in enumerate(batches):
|
||||
logger.info(f"processing batch {idx+1}/{len(batches)}")
|
||||
results += loop.run_until_complete(batch_inference_dataset(llm, batch, chat))
|
||||
return results
|
||||
|
||||
|
||||
def count_score_by_topic(dataset: List[Dict[str, Any]]):
|
||||
score_by_topic = {}
|
||||
total_score_by_topic = {}
|
||||
score = 0
|
||||
for dataset_row in dataset:
|
||||
for topic in dataset_row["topics"]:
|
||||
if topic not in score_by_topic:
|
||||
score_by_topic[topic] = 0
|
||||
total_score_by_topic[topic] = 0
|
||||
score_by_topic[topic] += dataset_row["score"]
|
||||
total_score_by_topic[topic] += 1
|
||||
score += dataset_row["score"]
|
||||
score_fraction = {
|
||||
k: f"{v}/{total_score_by_topic[k]}" for k, v in score_by_topic.items()
|
||||
}
|
||||
score_float = {
|
||||
k: round(100 * float(v) / float(total_score_by_topic[k]), 4)
|
||||
for k, v in score_by_topic.items()
|
||||
}
|
||||
score_float["Overall"] = round(100 * float(score) / float(len(dataset)), 4)
|
||||
score_fraction["Overall"] = f"{score}/{len(dataset)}"
|
||||
return score_fraction, score_float
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="SecEval Evaluation CLI")
|
||||
|
||||
parser.add_argument(
|
||||
"-o",
|
||||
"--output_dir",
|
||||
type=str,
|
||||
default="/tmp",
|
||||
help="Specify the output directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-d",
|
||||
"--dataset_file",
|
||||
type=str,
|
||||
required=True,
|
||||
help="Specify the dataset file to evaluate on.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-c",
|
||||
"--chat",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Evaluate on chat model.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-b",
|
||||
"--batch_size",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Specify the batch size.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-B",
|
||||
"--backend",
|
||||
type=str,
|
||||
choices=["remote_hf", "azure", "textgen", "local_hf", "ollama"],
|
||||
required=True,
|
||||
help="Specify the llm type. remote_hf: remote huggingface model backed, azure: azure openai model, textgen: textgen backend, local_hf: local huggingface model backed",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-m",
|
||||
"--models",
|
||||
type=str,
|
||||
nargs="+",
|
||||
required=True,
|
||||
help="Specify the models.",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
models = list(args.models)
|
||||
|
||||
logging.info(f"evaluating models: {models}")
|
||||
for model_id in models:
|
||||
if args.backend == "remote_hf":
|
||||
llm = init_hf_llm(model_id)
|
||||
elif args.backend == "local_hf":
|
||||
model_dir = os.environ.get("LOCAL_HF_MODEL_DIR")
|
||||
if model_dir is None:
|
||||
raise RuntimeError(
|
||||
"Please set LOCAL_HF_MODEL_DIR when using local_hf backend"
|
||||
)
|
||||
model_id = os.path.join(model_dir, model_id)
|
||||
llm = init_hf_llm(model_id)
|
||||
elif args.backend == "textgen":
|
||||
llm = init_textgen_llm(model_id)
|
||||
elif args.backend == "azure":
|
||||
llm = init_azure_openai_llm(model_id)
|
||||
elif args.backend == "ollama":
|
||||
if not args.model:
|
||||
raise RuntimeError("Please specify --model for Ollama backend")
|
||||
llm = init_ollama_llm(args.model)
|
||||
else:
|
||||
raise RuntimeError("Unknown backend")
|
||||
|
||||
dataset = load_dataset(args.dataset_file)
|
||||
result = inference_dataset(
|
||||
llm, dataset, batch_size=args.batch_size, chat=args.chat
|
||||
)
|
||||
score_fraction, score_float = count_score_by_topic(result)
|
||||
result_with_score = {
|
||||
"score_fraction": score_fraction,
|
||||
"score_float": score_float,
|
||||
"detail": result,
|
||||
}
|
||||
output_path = (
|
||||
Path(args.output_dir)
|
||||
/ f"{Path(args.dataset_file).stem}_{os.path.basename(model_id)}.json"
|
||||
)
|
||||
logger.info(f"writing result to {output_path}")
|
||||
with open(output_path, "w") as f:
|
||||
json.dump(result_with_score, f, indent=4)
|
||||
del llm
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
transformers==4.36.*
|
||||
langchain
|
||||
einops
|
||||
protobuf
|
||||
scipy
|
||||
sentencepiece
|
||||
torch
|
||||
python-dotenv
|
||||
transformers_stream_generator
|
||||
tiktoken
|
||||
langchain-community
|
||||
Loading…
Reference in New Issue