53 lines
5.3 KiB
Markdown
53 lines
5.3 KiB
Markdown
# Prompt Scan — LLM Vulnerability Testing Platform
|
|
|
|
## Overview
|
|
|
|
Build a deliberately vulnerable AI/LLM chatbot application that exposes all ten OWASP Top 10 for Large Language Model Applications vulnerabilities as live, exploitable HTTP endpoints. The platform provides a controlled target environment for security professionals, penetration testers, and students to practice LLM-specific attack techniques — prompt injection, insecure output handling, model DoS, plugin exploitation, and more — without risk to production systems.
|
|
|
|
## Step-by-Step Instructions
|
|
|
|
1. **Understand the OWASP Top 10 for LLM Applications** by studying each risk category: LLM01 (Prompt Injection) attacks manipulate model behaviour through crafted input; LLM02 (Insecure Output Handling) occurs when raw LLM output is rendered or executed without sanitisation; LLM03 (Training Data Poisoning) injects malicious content into the model's context; LLM04 (Model Denial of Service) exhausts API quotas; LLM05 (Supply Chain) arises from untrusted plugins; LLM06 (Sensitive Information Disclosure) leaks credentials and API keys; LLM07 (Insecure Plugin Design) enables arbitrary code execution; LLM08 (Excessive Agency) grants unrestricted system permissions; LLM09 (Overreliance) accepts unvalidated output as ground truth; LLM10 (Model Theft) exposes model architecture and prompts.
|
|
|
|
2. **Design the architecture** with a Flask backend that intentionally disables every security control. Create a `Config` class that stores and exposes the OpenAI API key, admin credentials, and system prompt through a public endpoint, implementing LLM06 at the infrastructure level.
|
|
|
|
3. **Implement the vulnerable OpenAI client** wrapping the SDK with deliberate flaws: pass user input directly without sanitisation (LLM01), return full metadata including system prompt (LLM06/LLM10), expose `generate_long_response` with `max_tokens=4000` and no rate limiting (LLM04), and track `request_count` for usage pattern disclosure (LLM10).
|
|
|
|
4. **Build the injectable SQLite database helper** with three SQL injection classes: string-interpolated `search_users` enabling `' OR 1=1--` attacks, `update_user` accepting arbitrary field/value parameters, and `execute_raw_sql` running any SQL statement. Seed with fake users and a secrets table containing plaintext credentials.
|
|
|
|
5. **Create the dangerous plugin system** with `command_executor.py` using `subprocess.run(shell=True)` with no whitelist (LLM07/LLM08), and `file_reader.py` reading any path without validation. Add a `load_external_plugin` endpoint accepting a `plugin_url` to demonstrate LLM05.
|
|
|
|
6. **Implement Flask route handlers** mapping each OWASP category to exploitable endpoints: `/api/chat` (LLM01/02/04/06), `/api/config` (LLM06 full dump), `/api/prompt-injection` (LLM01), `/api/dos/long-response` (LLM04), `/api/plugin/execute` (LLM07/08), `/api/admin/delete-user/<id>` unauthenticated (LLM08), `/api/system/info` dumping `os.environ` (LLM06).
|
|
|
|
7. **Build the web interface** as a single HTML page using Bootstrap. Render LLM responses with `innerHTML` so XSS payloads execute in the browser (LLM02). Add a vulnerability panel with all exploitable endpoints and a config viewer calling `/api/config`.
|
|
|
|
8. **Document every vulnerability** in the `learn/` directory: `owasp-llm-overview.md` explaining each risk, `attack-walkthrough.md` with curl commands, and `mitigations.md` with concrete fixes.
|
|
|
|
9. **Containerise** with Docker using `uv` for dependency management, binding only to `127.0.0.1:5000`, with `HEALTHCHECK` on `/health` and `data/` mounted as a volume.
|
|
|
|
10. **Validate** by running every payload in `tests/attack_payloads.json`, confirming all ten OWASP LLM categories are reproducible, and verifying Docker isolation.
|
|
|
|
## Key Concepts to Learn
|
|
- OWASP Top 10 for Large Language Model Applications
|
|
- Prompt injection — direct, indirect, and role manipulation
|
|
- LLM output handling and XSS/SQLi/RCE attack vectors
|
|
- Rate limiting, token budgets, and denial-of-service mitigation
|
|
- Plugin architecture security and principle of least privilege
|
|
- Secrets management and environment variable hygiene
|
|
- Supply chain risks in AI/ML applications
|
|
- Model metadata disclosure and model theft vectors
|
|
|
|
## Deliverables
|
|
- Flask backend with all ten OWASP LLM vulnerabilities as live endpoints
|
|
- Vulnerable OpenAI client, SQLite helper, and plugin system
|
|
- Docker Compose environment with `uv` and volume isolation
|
|
- `learn/` directory with OWASP overview, attack walkthrough, and mitigations
|
|
- `tests/attack_payloads.json` with documented payloads for all ten categories
|
|
- Complete README with architecture diagram, API reference, and safety warnings
|
|
|
|
---
|
|
[](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/advanced/prompt-scan)
|
|
[](https://www.python.org)
|
|
[](https://flask.palletsprojects.com)
|
|
[](https://opensource.org/licenses/MIT)
|
|
[](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
|