Cybersecurity-Projects/SYNOPSES/advanced/Prompt.Scan.md

Prompt Scan — LLM Vulnerability Testing Platform

Overview

Build a deliberately vulnerable AI/LLM chatbot application that exposes all ten OWASP Top 10 for Large Language Model Applications vulnerabilities as live, exploitable HTTP endpoints. The platform provides a controlled target environment for security professionals, penetration testers, and students to practice LLM-specific attack techniques — prompt injection, insecure output handling, model DoS, plugin exploitation, and more — without risk to production systems.

Step-by-Step Instructions

1. Understand the OWASP Top 10 for LLM Applications by studying each risk category: LLM01 (Prompt Injection) attacks manipulate model behaviour through crafted input; LLM02 (Insecure Output Handling) occurs when raw LLM output is rendered or executed without sanitisation; LLM03 (Training Data Poisoning) injects malicious content into the model's context; LLM04 (Model Denial of Service) exhausts API quotas; LLM05 (Supply Chain) arises from untrusted plugins; LLM06 (Sensitive Information Disclosure) leaks credentials and API keys; LLM07 (Insecure Plugin Design) enables arbitrary code execution; LLM08 (Excessive Agency) grants unrestricted system permissions; LLM09 (Overreliance) accepts unvalidated output as ground truth; LLM10 (Model Theft) exposes model architecture and prompts.

2. Design the architecture with a Flask backend that intentionally disables every security control. Create a Config class that stores and exposes the OpenAI API key, admin credentials, and system prompt through a public endpoint, implementing LLM06 at the infrastructure level.

3. Implement the vulnerable OpenAI client wrapping the SDK with deliberate flaws: pass user input directly without sanitisation (LLM01), return full metadata including system prompt (LLM06/LLM10), expose generate_long_response with max_tokens=4000 and no rate limiting (LLM04), and track request_count for usage pattern disclosure (LLM10).

4. Build the injectable SQLite database helper with three SQL injection classes: string-interpolated search_users enabling ' OR 1=1-- attacks, update_user accepting arbitrary field/value parameters, and execute_raw_sql running any SQL statement. Seed with fake users and a secrets table containing plaintext credentials.

5. Create the dangerous plugin system with command_executor.py using subprocess.run(shell=True) with no whitelist (LLM07/LLM08), and file_reader.py reading any path without validation. Add a load_external_plugin endpoint accepting a plugin_url to demonstrate LLM05.

6. Implement Flask route handlers mapping each OWASP category to exploitable endpoints: /api/chat (LLM01/02/04/06), /api/config (LLM06 full dump), /api/prompt-injection (LLM01), /api/dos/long-response (LLM04), /api/plugin/execute (LLM07/08), /api/admin/delete-user/<id> unauthenticated (LLM08), /api/system/info dumping os.environ (LLM06).

7. Build the web interface as a single HTML page using Bootstrap. Render LLM responses with innerHTML so XSS payloads execute in the browser (LLM02). Add a vulnerability panel with all exploitable endpoints and a config viewer calling /api/config.

8. Document every vulnerability in the learn/ directory: owasp-llm-overview.md explaining each risk, attack-walkthrough.md with curl commands, and mitigations.md with concrete fixes.

9. Containerise with Docker using uv for dependency management, binding only to 127.0.0.1:5000, with HEALTHCHECK on /health and data/ mounted as a volume.

10. Validate by running every payload in tests/attack_payloads.json, confirming all ten OWASP LLM categories are reproducible, and verifying Docker isolation.

Key Concepts to Learn

• OWASP Top 10 for Large Language Model Applications
• Prompt injection — direct, indirect, and role manipulation
• LLM output handling and XSS/SQLi/RCE attack vectors
• Rate limiting, token budgets, and denial-of-service mitigation
• Plugin architecture security and principle of least privilege
• Secrets management and environment variable hygiene
• Supply chain risks in AI/ML applications
• Model metadata disclosure and model theft vectors

Deliverables

• Flask backend with all ten OWASP LLM vulnerabilities as live endpoints
• Vulnerable OpenAI client, SQLite helper, and plugin system
• Docker Compose environment with uv and volume isolation
• learn/ directory with OWASP overview, attack walkthrough, and mitigations
• tests/attack_payloads.json with documented payloads for all ten categories
• Complete README with architecture diagram, API reference, and safety warnings

---