From ab681f1ba19f0c30f559d4226dbfa735c11ce0ee Mon Sep 17 00:00:00 2001 From: CarterPerez-dev Date: Wed, 4 Feb 2026 01:47:13 -0500 Subject: [PATCH] add learn documents for api-security-scanner --- .../api-security-scanner/learn/00-OVERVIEW.md | 160 ++ .../api-security-scanner/learn/01-CONCEPTS.md | 917 +++++++++ .../learn/02-ARCHITECTURE.md | 1511 +++++++++++++++ .../learn/03-IMPLEMENTATION.md | 1720 +++++++++++++++++ .../learn/04-CHALLENGES.md | 1597 +++++++++++++++ TEMPLATES/fullstack-template | 2 +- 6 files changed, 5906 insertions(+), 1 deletion(-) diff --git a/PROJECTS/intermediate/api-security-scanner/learn/00-OVERVIEW.md b/PROJECTS/intermediate/api-security-scanner/learn/00-OVERVIEW.md index e69de29b..57a1cf5c 100644 --- a/PROJECTS/intermediate/api-security-scanner/learn/00-OVERVIEW.md +++ b/PROJECTS/intermediate/api-security-scanner/learn/00-OVERVIEW.md @@ -0,0 +1,160 @@ +# 00-OVERVIEW.md + +# API Security Scanner + +## What This Is + +A full-stack web application that tests APIs for common security vulnerabilities. Point it at any API endpoint and it'll check for rate limiting weaknesses, authentication bypasses, SQL injection holes, and authorization flaws. Built with FastAPI and React, it's designed to teach you how attackers probe APIs and how to defend against them. + +## Why This Matters + +APIs are everywhere. Your mobile app talks to an API. Your web dashboard pulls from an API. That third party integration you added last week? API. And they're constantly under attack. In 2023, API attacks increased 400% year over year according to Salt Security's research. The Optus breach in 2022 exposed 10 million customer records through an unauthenticated API endpoint. T-Mobile's API breach the same year leaked data on 37 million customers. + +Most of these breaches come down to a handful of problems that show up repeatedly. Missing rate limits let attackers hammer endpoints. Broken authentication accepts invalid tokens. SQL injection still works because developers concatenate user input into queries. IDOR vulnerabilities let users access each other's data by just changing an ID in the URL. + +This project teaches you to find these issues before attackers do. + +**Real world scenarios where this applies:** +- Security teams scanning their own APIs before releases +- Bug bounty hunters looking for vulnerabilities in public APIs +- DevOps engineers validating that rate limiting actually works +- Penetration testers checking if JWT validation can be bypassed +- Developers learning what attacks their code needs to defend against + +## What You'll Learn + +This project teaches you how API security testing works under the hood. By building it yourself, you'll understand: + +**Security Concepts:** +- OWASP API Security Top 10 2023 - not just the list, but what each vulnerability looks like in real code and how to detect it automatically +- Rate limiting bypass techniques - how attackers rotate IPs, spoof headers, and manipulate endpoints to evade limits +- JWT vulnerabilities - why the "none" algorithm attack works, what happens when signatures aren't verified, and how to test for weak secrets +- SQL injection detection - error-based, boolean blind, and time-based blind techniques with statistical analysis to reduce false positives +- IDOR/BOLA testing - finding authorization gaps that let users access objects they shouldn't + +**Technical Skills:** +- Building security scanners with request throttling, retry logic, and baseline timing analysis (see `backend/scanners/base_scanner.py:40-90`) +- Implementing layered architecture with repositories, services, and route handlers that keep business logic separate from data access +- JWT authentication flows from password hashing through token creation to validation on protected endpoints +- Rate limiting implementation and detection using both response analysis and header inspection +- Handling concurrent HTTP requests with proper connection pooling, timeouts, and error recovery + +**Tools and Techniques:** +- FastAPI with async/await for handling multiple scans concurrently without blocking +- SQLAlchemy with Alembic for database migrations and relationship management +- Docker multi-stage builds that separate dev (hot reload) from production (optimized, multi-worker) +- React with TanStack Query for managing server state, automatic refetching, and optimistic updates +- Nginx reverse proxy configuration for routing /api to backend and serving static frontend files + +## Prerequisites + +Before starting, you should understand: + +**Required knowledge:** +- Python basics - functions, classes, async/await. You'll be reading code like `async def make_request(self, method: str, endpoint: str, **kwargs: Any)` and need to understand what's happening +- HTTP fundamentals - what GET/POST mean, how headers work, what status codes indicate. The scanners analyze responses to detect vulnerabilities +- SQL basics - SELECT, WHERE clauses, what a JOIN does. You'll see how SQL injection payloads try to manipulate queries +- REST API concepts - endpoints, request/response, authentication. The whole project revolves around testing APIs + +**Tools you'll need:** +- Docker and Docker Compose - the entire stack runs in containers +- Python 3.11+ - backend code uses modern type hints and pattern matching +- Node.js 20+ - frontend build requires recent npm +- PostgreSQL knowledge helpful - data is stored in Postgres with relationships between users, scans, and test results + +**Helpful but not required:** +- React and TypeScript - frontend is fully implemented but understanding it helps +- Previous security testing experience - we'll teach the concepts but prior exposure to tools like Burp Suite or OWASP ZAP provides context +- FastAPI familiarity - we use dependency injection and Pydantic validation heavily + +## Quick Start + +Get the project running locally: + +```bash +# Clone and navigate (assuming you're already in PROJECTS/intermediate) +cd api-security-scanner + +# Copy environment template +cp .env.example .env + +# CRITICAL: Edit .env and change SECRET_KEY to something random +# In production this must be a cryptographically secure random string + +# Start development environment (includes hot reload) +docker compose -f dev.compose.yml up --build + +# Wait for all services to be healthy, then visit: +# - Frontend: http://localhost (or http://localhost:80) +# - Backend API docs: http://localhost:8000/docs +# - Direct API: http://localhost:8000 +``` + +Expected output: You should see the login page. Create an account, then you can start scanning. Try scanning `https://httpbin.org/get` with all four tests enabled. You'll see safe results because httpbin is designed for testing. + +## Project Structure + +``` +api-security-scanner/ +├── backend/ +│ ├── config.py # All environment variables and constants +│ ├── core/ # Infrastructure (database, security, dependencies) +│ ├── models/ # SQLAlchemy models (User, Scan, TestResult) +│ ├── repositories/ # Data access layer, queries isolated here +│ ├── routes/ # FastAPI endpoints (auth, scans) +│ ├── scanners/ # Security testing logic (SQLi, auth, etc) +│ ├── schemas/ # Pydantic validation schemas +│ └── services/ # Business logic layer +├── frontend/ +│ └── src/ +│ ├── components/ # React UI components +│ ├── hooks/ # TanStack Query hooks for API calls +│ ├── services/ # API client functions +│ └── store/ # Zustand state management +├── conf/ +│ ├── docker/ # Dockerfiles for dev and prod +│ └── nginx/ # Nginx reverse proxy configs +└── compose.yml # Production Docker Compose +└── dev.compose.yml # Development with volume mounts +``` + +## Next Steps + +1. **Understand the concepts** - Read [01-CONCEPTS.md](./01-CONCEPTS.md) to learn the security fundamentals behind rate limiting, authentication, SQL injection, and IDOR +2. **Study the architecture** - Read [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) to see how the scanner system is designed, why we use the repository pattern, and how data flows through the layers +3. **Walk through the code** - Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) for detailed explanation of how each scanner works, with line-by-line breakdowns of the actual code +4. **Extend the project** - Read [04-CHALLENGES.md](./04-CHALLENGES.md) for ideas to build on, from adding XSS detection to implementing custom scanner plugins + +## Common Issues + +**Docker containers fail to start** +``` +Error: port is already allocated +``` +Solution: Another service is using port 80, 5432, or 8000. Check `docker ps` and stop conflicting containers, or edit `.env` to change `HOST_NGINX_PORT`, `HOST_DB_PORT`, or `HOST_BACKEND_PORT` + +**Frontend can't reach backend** +``` +Network Error / Failed to fetch +``` +Solution: Check `VITE_API_URL` in `.env`. For Docker setup, it should be `http://localhost/api`. If running backend directly (not in Docker), use `http://localhost:8000` + +**Database migrations fail** +``` +sqlalchemy.exc.OperationalError: could not connect to server +``` +Solution: Database container isn't ready yet. Wait 10 seconds and try again. The healthcheck in `dev.compose.yml:13-17` ensures Postgres is accepting connections before backend starts + +**Scans timeout on every test** +``` +All tests return ERROR status +``` +Solution: The target URL might be blocking requests. Try `https://httpbin.org/get` first to verify the scanner works. Check `SCANNER_CONNECTION_TIMEOUT` in `config.py:51` - default is 180 seconds + +## Related Projects + +If you found this interesting, check out: +- **network-traffic-analyzer** [network-traffic-analyzere](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/beginner/network-traffic-analyzer) - Goes deeper into packet analysis and protocol dissection, complements this by showing what's happening at the network layer +- **docker-security-audit** [docker-security-audit](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/intermediate/docker-security-audit) - Focuses on container security, teaches you to scan Docker images and configurations for vulnerabilities +- **bug-bounty-platform** [bug-bounty-platform](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/advanced/bug-bounty-platform) - Full platform for managing security findings, shows how professional security teams track and remediate issues + diff --git a/PROJECTS/intermediate/api-security-scanner/learn/01-CONCEPTS.md b/PROJECTS/intermediate/api-security-scanner/learn/01-CONCEPTS.md index e69de29b..255800a8 100644 --- a/PROJECTS/intermediate/api-security-scanner/learn/01-CONCEPTS.md +++ b/PROJECTS/intermediate/api-security-scanner/learn/01-CONCEPTS.md @@ -0,0 +1,917 @@ +# 01-CONCEPTS.md + +# Core Security Concepts + +This document explains the security concepts you'll encounter while building this project. These are not just definitions. We'll dig into why they matter and how they actually work. + +## Rate Limiting + +### What It Is + +Rate limiting controls how many requests a client can make to your API in a given time window. It's like a bouncer at a club counting how many times someone tries to get in. If they're hammering the door 100 times per minute, something's wrong. + +The concept is simple: track requests by some identifier (IP address, user ID, API key) and reject requests that exceed a threshold. A basic implementation might be "100 requests per minute per IP address." When request 101 comes in within that minute, you return HTTP 429 Too Many Requests. + +### Why It Matters + +Without rate limiting, a single attacker can take down your entire service. In 2016, the Mirai botnet launched a DDoS attack against Dyn's DNS service by flooding it with requests. Major sites like Twitter, Netflix, and Reddit went down because Dyn had no effective rate limiting at the DNS level. + +But it's not just about denial of service. Rate limiting also stops: +- **Credential stuffing** - attackers trying thousands of stolen passwords against your login endpoint +- **Data scraping** - competitors or bad actors pulling your entire product catalog through your API +- **Brute force attacks** - trying every possible value to crack passwords, API keys, or find valid user IDs +- **Cost attacks** - running up your AWS bill by triggering expensive operations repeatedly + +The 2020 Venmo scraping incident exposed transaction data for millions of users. Attackers made unlimited requests to Venmo's public API and collected data on who was paying whom. Rate limiting would have detected and blocked this pattern immediately. + +### How It Works + +Look at how this project detects rate limiting in `backend/scanners/rate_limit_scanner.py`: + +```python +# Lines 62-145 +def _detect_rate_limiting(self, test_request_count: int = 20) -> dict[str, Any]: + """ + Detect rate limiting by analyzing headers and response patterns + """ + rate_limit_patterns = RateLimitBypassPayloads.get_header_patterns() + + results = { + "rate_limit_detected": False, + "rate_limit_headers": {}, + "enforcement_status": None, + } + + for attempt in range(1, test_request_count + 1): + response = self.make_request("GET", "/") + + # Check for standard rate limit headers + for header_type, pattern in rate_limit_patterns.items(): + for header_name, header_value in headers_lower.items(): + if re.search(pattern, header_name, re.IGNORECASE): + results["rate_limit_headers"][header_type] = { + "header_name": header_name, + "value": header_value, + } + results["rate_limit_detected"] = True + + # Check if we hit the limit + if response.status_code == 429: + results["enforcement_status"] = "ACTIVE" + results["attempts_until_limit"] = attempt + break +``` + +This code makes requests until it either hits a 429 response or finds rate limit headers. The headers follow patterns like `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`. These are from the IETF draft standard that most APIs follow. + +The scanner distinguishes between three states: +1. **No rate limiting** - sends 20 requests, never gets blocked or sees headers +2. **Headers only** - advertises limits in headers but never enforces them with 429 +3. **Active enforcement** - actually blocks requests when limits are exceeded + +### Common Attacks + +1. **IP header spoofing** - Many rate limiters trust headers like `X-Forwarded-For` to identify clients. Attackers rotate fake IP addresses in this header to bypass limits. The scanner tests this in `rate_limit_scanner.py:218-255`. + +2. **Distributed attacks** - Use botnets or cloud providers to attack from many real IPs simultaneously, staying under per-IP limits while overwhelming the service overall. + +3. **Endpoint variation bypass** - Some rate limiters are case sensitive or miss URL variations. Try `/api/users`, `/API/users`, `/api/users/`, `/api/users?`, etc. Code at `rate_limit_scanner.py:257-286`. + +4. **Resource exhaustion without limits** - Even with rate limiting, if limits are too high (1000 requests/minute), an attacker can still cause damage. The Cloudflare outage in 2022 was caused by a regex pattern that consumed CPU - even rate-limited requests added up. + +### Defense Strategies + +Implement rate limiting at multiple layers. This project shows how to detect it, but here's how to build it: + +**Layer 1: Edge/CDN level** +Use Cloudflare, AWS WAF, or similar to block obvious abuse before it hits your infrastructure. This is cheap and fast because it happens at edge locations. + +**Layer 2: API gateway** +Kong, AWS API Gateway, or nginx can enforce rate limits per IP, per API key, or per user. This catches most abuse. + +**Layer 3: Application level** +The backend in this project uses SlowAPI (see `backend/factory.py:34-36`): +```python +limiter = Limiter(key_func=get_remote_address) +app.state.limiter = limiter +app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) +``` + +And applies limits to specific endpoints (`backend/routes/auth.py:25-42`): +```python +@router.post("/register", ...) +@limiter.limit(settings.API_RATE_LIMIT_REGISTER) # "15/minute" from config +async def register(request: Request, ...): + return AuthService.register_user(db, user_data) +``` + +**Critical defenses:** +- Don't trust client-provided headers like `X-Forwarded-For` without validation +- Use multiple identifiers - IP, user ID, API key combined +- Implement exponential backoff - longer lockouts for repeated violations +- Return `Retry-After` header telling clients when to try again +- Log rate limit violations for security monitoring + +## Broken Authentication + +### What It Is + +Authentication proves who you are. When it's broken, attackers can bypass it, impersonate users, or access systems without valid credentials. This is OWASP API2:2023 - Broken Authentication. + +Common authentication mechanisms for APIs: +- **Session tokens** - server generates a token, stores it, validates on each request +- **JWT tokens** - server signs a token, client sends it back, server verifies signature +- **API keys** - long-lived secrets that identify applications +- **OAuth 2.0** - delegated authorization with access tokens + +This project focuses on JWT because it's widely used and frequently misconfigured. + +### Why It Matters + +The 2020 SolarWinds attack used stolen authentication tokens to access customer networks. Once inside, attackers moved laterally because authentication wasn't properly verified at each service boundary. + +In 2018, a vulnerability in Facebook's "View As" feature let attackers steal access tokens for 50 million users. The tokens worked because Facebook didn't properly validate them when they were used. + +Broken authentication is ranked #2 in OWASP's API Top 10 because it's both common and devastating. Bypass authentication and you often get full access to user data, admin functions, and internal systems. + +### How It Works + +JWT (JSON Web Token) has three parts separated by periods: +``` +header.payload.signature +``` + +Example: +``` +eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyQGV4YW1wbGUuY29tIn0.signature_here +``` + +Decode the header (base64): +```json +{"alg": "HS256", "typ": "JWT"} +``` + +Decode the payload: +```json +{"sub": "user@example.com"} +``` + +The signature proves the token wasn't tampered with. It's created by: +``` +HMACSHA256( + base64UrlEncode(header) + "." + base64UrlEncode(payload), + secret +) +``` + +Proper validation must: +1. Verify the signature using the server's secret key +2. Check the algorithm is what you expect (HS256, RS256, etc) +3. Validate expiration time if present +4. Reject tokens with the "none" algorithm + +Look at how this project validates tokens in `backend/core/security.py:48-62`: +```python +def decode_token(token: str) -> dict[str, str]: + """ + Decode and verify a JWT token + """ + try: + payload = jwt.decode( + token, + settings.SECRET_KEY, + algorithms=[settings.ALGORITHM] # Explicitly specify allowed algorithms + ) + return payload + except JWTError as e: + raise ValueError(f"Invalid token: {str(e)}") from e +``` + +### Common Attacks + +1. **None algorithm attack** - JWT supports an algorithm called "none" which means "no signature required." Some libraries accept these tokens even though they're unsigned. The scanner tests this in `auth_scanner.py:168-213`: + +```python +def _test_none_algorithm(self) -> dict[str, Any]: + """ + Test if server accepts JWT with 'none' algorithm + """ + header, payload, signature = self.auth_token.split(".") + + none_variants = ["none", "None", "NONE", "nOnE"] # Case variations + + for variant in none_variants: + malicious_header = self._base64url_encode( + json.dumps({"alg": variant, "typ": "JWT"}) + ) + malicious_token = f"{malicious_header}.{payload}." # No signature + + response = self.make_request( + "GET", "/", + headers={"Authorization": f"Bearer {malicious_token}"} + ) + + if response.status_code == 200: + return {"vulnerable": True, "algorithm_variant": variant} +``` + +2. **Missing authentication** - Endpoints that should require auth don't check for it. Test in `auth_scanner.py:89-131`: +```python +def _test_missing_authentication(self) -> dict[str, Any]: + """ + Test if endpoint requires authentication + """ + session_without_auth = self.session.__class__() # New session, no token + + response = session_without_auth.get(self.target_url, timeout=...) + + if response.status_code == 200: + return {"vulnerable": True, "description": "Endpoint accessible without authentication"} +``` + +3. **Weak secret keys** - If the JWT secret is weak ("secret", "password", company name), attackers can brute force it and forge valid tokens. Tools like hashcat can try millions of secrets per second. + +4. **Algorithm confusion** - Server expects RS256 (asymmetric) but accepts HS256 (symmetric). Attacker uses the public key as the HMAC secret to forge tokens. + +### Defense Strategies + +The project implements proper JWT validation. Here's what to do: + +**Token creation** (`core/security.py:33-56`): +```python +def create_access_token(data: dict[str, str], expires_delta: timedelta | None = None) -> str: + to_encode = data.copy() + + expire = datetime.utcnow() + (expires_delta or timedelta(minutes=1440)) + to_encode.update({"exp": expire}) + + encoded_jwt = jwt.encode( + to_encode, + settings.SECRET_KEY, # Must be cryptographically random, 256+ bits + algorithm=settings.ALGORITHM # "HS256" + ) + return encoded_jwt +``` + +**Token validation** - Happens in `core/dependencies.py:17-49`: +```python +async def get_current_user( + credentials: HTTPAuthorizationCredentials = Depends(security), + db: Session = Depends(get_db), +) -> UserResponse: + try: + payload = decode_token(credentials.credentials) + email: str | None = payload.get("sub") + + if email is None: + raise HTTPException(status_code=401, detail="Invalid credentials") + + user = UserRepository.get_by_email(db, email) + if not user: + raise HTTPException(status_code=401, detail="User not found") + + return UserResponse.model_validate(user) +``` + +**Critical defenses:** +- Use a strong random secret (generate with `openssl rand -hex 32`) +- Explicitly specify allowed algorithms, reject "none" +- Verify signatures before trusting payload data +- Implement token expiration and check it +- Require authentication on all sensitive endpoints +- Use short-lived access tokens (15 minutes) with refresh tokens + +### Common Pitfalls + +**Mistake 1: Trusting unverified tokens** +```python +# Bad - decodes without verifying signature +import base64, json +header, payload, sig = token.split(".") +data = json.loads(base64.b64decode(payload)) +user_id = data["user_id"] # Attacker controlled! + +# Good - verifies signature first +payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"]) +user_id = payload["user_id"] # Safe to use +``` + +**Mistake 2: Not checking expiration** +```python +# Bad - no expiration check +payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"], options={"verify_exp": False}) + +# Good - validates expiration +payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"]) # verify_exp=True by default +``` + +## SQL Injection + +### What It Is + +SQL injection happens when you put untrusted data directly into SQL queries without proper escaping. The attacker sends malicious input that changes the query's logic. + +Example vulnerable code: +```python +query = f"SELECT * FROM users WHERE email = '{email}'" +``` + +If the attacker sends `email = "admin'--"`, the query becomes: +```sql +SELECT * FROM users WHERE email = 'admin'--' +``` + +The `--` comments out the rest, changing a lookup into a bypass. + +### Why It Matters + +SQL injection has been in the OWASP Top 10 for two decades, yet it still causes breaches. The 2017 Equifax breach exposed 147 million records through a different vulnerability (Apache Struts), but once inside, attackers used SQL injection to extract data. + +In 2019, a SQL injection in Canva exposed email addresses and passwords for 137 million users. The attackers manipulated input fields that were concatenated into queries. + +Modern frameworks have made SQLi less common, but it still shows up when developers: +- Concatenate user input into queries +- Misuse ORM raw query features +- Build dynamic SQL from user-controlled data +- Trust input from internal sources (still needs validation) + +### How It Works + +There are three main types of SQL injection that this project detects: + +**1. Error-based SQLi** - Trigger database errors that leak information + +The scanner sends payloads like `' OR '1'='1` and checks if database error messages appear in the response (`sqli_scanner.py:51-94`): + +```python +def _test_error_based_sqli(self) -> dict[str, Any]: + """ + Test for error based SQL injection + """ + error_signatures = SQLiPayloads.get_error_signatures() # MySQL, Postgres, MSSQL, Oracle + basic_payloads = SQLiPayloads.BASIC_AUTHENTICATION_BYPASS + + for payload in basic_payloads: + response = self.make_request("GET", f"/?id={payload}") + response_text_lower = response.text.lower() + + # Look for database error signatures + for db_type, signatures in error_signatures.items(): + for signature in signatures: + if signature in response_text_lower: + return { + "vulnerable": True, + "database_type": db_type, + "payload": payload, + "error_signature": signature + } +``` + +Error signatures are defined in `scanners/payloads.py:9-30`: +```python +ERROR_SIGNATURES = { + "mysql": ["sql syntax", "mysql_fetch", "mysql error"], + "postgres": ["postgresql", "pg_query", "syntax error"], + "mssql": ["sqlserver jdbc driver", "sqlexception"], + "oracle": ["ora-", "oracle.jdbc", "pl/sql"], +} +``` + +**2. Boolean-based blind SQLi** - No errors shown, but responses differ for true vs false conditions + +Send queries with conditions that are always true (`1=1`) vs always false (`1=2`) and compare response lengths: + +```python +def _test_boolean_based_sqli(self) -> dict[str, Any]: + """ + Test for boolean based blind SQL injection + """ + baseline_response = self.make_request("GET", "/?id=1") + baseline_length = len(baseline_response.text) + + # Try true conditions: ' AND 1=1-- + true_payloads = [p for p in boolean_payloads if "AND '1'='1" in p or "AND 1=1" in p] + true_lengths = [] + for payload in true_payloads: + response = self.make_request("GET", f"/?id={payload}") + true_lengths.append(len(response.text)) + + # Try false conditions: ' AND 1=2-- + false_payloads = [p for p in boolean_payloads if "AND '1'='2" in p] + false_lengths = [] + for payload in false_payloads: + response = self.make_request("GET", f"/?id={payload}") + false_lengths.append(len(response.text)) + + avg_true = statistics.mean(true_lengths) + avg_false = statistics.mean(false_lengths) + length_diff = abs(avg_true - avg_false) + + if length_diff > 100: # Significant difference + return {"vulnerable": True, "length_difference": length_diff} +``` + +**3. Time-based blind SQLi** - Force the database to delay, measure response time + +Payloads like `'; SELECT SLEEP(5)--` make the database pause. If the response takes 5+ seconds, it's vulnerable: + +```python +def _test_time_based_sqli(self, delay_seconds: int = 5) -> dict[str, Any]: + """ + Test for time based blind SQL injection + Uses baseline timing comparison with statistical analysis + """ + # Establish normal response time + baseline_mean, baseline_stdev = self.get_baseline_timing("/") + expected_delay_time = baseline_mean + delay_seconds + + delay_payloads = { + "mysql": [p for p in all_time_payloads if "SLEEP" in p], + "postgres": [p for p in all_time_payloads if "pg_sleep" in p], + "mssql": [p for p in all_time_payloads if "WAITFOR" in p], + } + + for db_type, payloads in delay_payloads.items(): + for payload in payloads: + delay_times = [] + for _ in range(3): # Multiple samples + response = self.make_request("GET", f"/?id={payload}", timeout=delay_seconds + 10) + elapsed = getattr(response, "request_time", 0.0) + delay_times.append(elapsed) + + avg_delay = statistics.mean(delay_times) + + if avg_delay >= expected_delay_time - 1: # Within 1 second of expected + return {"vulnerable": True, "database_type": db_type, "response_time": f"{avg_delay:.3f}s"} +``` + +### Common Attacks + +The payloads are in `scanners/payloads.py`. Here are the techniques: + +1. **Authentication bypass** (`payloads.py:33-48`): +```python +BASIC_AUTHENTICATION_BYPASS = [ + "' OR '1'='1", # Makes WHERE clause always true + "' OR 1=1--", # Comments out rest of query + "admin'--", # Logs in as admin by ending query early + ") or '1'='1--", # Escapes parentheses in complex queries +] +``` + +2. **Union-based extraction** (`payloads.py:50-60`): +```python +UNION_BASED = [ + "' UNION SELECT NULL--", # Find column count + "' UNION SELECT username,password FROM users--", # Extract data + "' UNION SELECT NULL,version()--", # Get database version +] +``` + +3. **Stacked queries** (`payloads.py:92-99`): +```python +STACKED_QUERIES = [ + "'; DROP TABLE users--", # Delete data + "'; INSERT INTO users VALUES('hacker','password')--", # Add accounts + "'; EXEC xp_cmdshell('whoami')--", # Run OS commands (MSSQL) +] +``` + +### Defense Strategies + +Never concatenate user input into SQL queries. Use parameterized queries (prepared statements) that treat data as data, not code. + +**Vulnerable code:** +```python +# BAD - string concatenation +email = request.get("email") +query = f"SELECT * FROM users WHERE email = '{email}'" +result = db.execute(query) +``` + +**Safe code with SQLAlchemy:** +```python +# GOOD - parameterized query +email = request.get("email") +result = db.query(User).filter(User.email == email).first() + +# SQLAlchemy generates safe SQL: +# SELECT * FROM users WHERE email = :email_1 +# Parameters: {'email_1': 'user@example.com'} +``` + +This project uses the repository pattern to keep SQL safe. Look at `repositories/user_repository.py:25-35`: +```python +@staticmethod +def get_by_email(db: Session, email: str) -> User | None: + """ + Get user by email address + """ + return db.query(User).filter(User.email == email).first() +``` + +SQLAlchemy's `.filter(User.email == email)` creates a parameterized query automatically. The email value is passed separately from the SQL string, so it can't inject code. + +**Critical defenses:** +- Use ORMs like SQLAlchemy, Django ORM, or Prisma +- When raw SQL is necessary, use parameterized queries +- Validate input types (if expecting int, convert to int and catch errors) +- Use least privilege database accounts (don't connect as admin) +- Disable error messages in production (don't leak database details) + +### Common Pitfalls + +**Mistake 1: Using ORM raw() incorrectly** +```python +# Bad - raw SQL with concatenation +email = request.get("email") +query = f"SELECT * FROM users WHERE email = '{email}'" +result = db.session.execute(text(query)) # Still vulnerable! + +# Good - raw SQL with parameters +result = db.session.execute( + text("SELECT * FROM users WHERE email = :email"), + {"email": email} +) +``` + +**Mistake 2: Trusting "internal" data** +```python +# Bad - assumes data from another service is safe +external_api_result = fetch_from_partner() +user_id = external_api_result["user_id"] +query = f"SELECT * FROM orders WHERE user_id = {user_id}" # Can still be exploited +``` + +## IDOR/BOLA (Broken Object Level Authorization) + +### What It Is + +IDOR (Insecure Direct Object Reference) and BOLA (Broken Object Level Authorization) are the same thing. OWASP's API Top 10 2023 calls it BOLA and ranks it #1. + +The vulnerability: your API lets users access objects by ID, but doesn't check if that user should have access to that specific object. + +Example: +``` +GET /api/users/123/orders +``` + +The API checks if you're authenticated (you have a valid token), but doesn't check if user 123 is YOU. So you can change the ID to 124, 125, 126 and view other users' orders. + +### Why It Matters + +IDOR is ranked #1 in OWASP API Security Top 10 2023 because it's everywhere and easy to exploit. No special tools needed, just change a number in the URL. + +Real breaches: +- **USPS Informed Delivery (2018)** - Change email addresses in requests to view anyone's mail scans +- **T-Mobile (2022)** - API endpoint leaked data on 37 million customers by allowing ID enumeration +- **Parler (2021)** - Sequential post IDs let researchers download the entire platform (70TB) by incrementing IDs + +The damage isn't just data exposure. In 2019, researchers found IDOR in a car sharing app that let them unlock and start any car in the fleet by changing the vehicle ID. + +### How It Works + +The scanner tests two patterns: + +**1. ID enumeration** - Sequential or predictable IDs make it easy to guess valid values + +Extract IDs from API responses, then try variations (`idor_scanner.py:119-153`): + +```python +def _extract_ids_from_response(self) -> list[Any]: + """ + Extract potential IDs from API response + """ + response = self.make_request("GET", "/") + + # Look for UUIDs: a1b2c3d4-1234-5678-9abc-def012345678 + uuid_pattern = r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}" + uuids = re.findall(uuid_pattern, response.text, re.IGNORECASE) + + # Look for numeric IDs: "id": 123 + numeric_id_pattern = r'"id"\s*:\s*(\d+)' + numeric_ids = re.findall(numeric_id_pattern, response.text) + + ids = [] + ids.extend(uuids[:3]) + ids.extend([int(nid) for nid in numeric_ids[:3]]) + + return ids +``` + +Then test if those IDs can be manipulated (`idor_scanner.py:155-204`): + +```python +def _test_numeric_id_manipulation(self, extracted_ids: list[Any]) -> dict[str, Any]: + """ + Test numeric ID manipulation for IDOR + """ + numeric_ids = [id_val for id_val in extracted_ids if isinstance(id_val, int)] + + base_id = numeric_ids[0] + test_ids = [0, -1, 1, 2, 9999, 99999, 999999] # Common patterns + + accessible_unauthorized = [] + + for test_id in test_ids: + if test_id == base_id: + continue + + response = self.make_request("GET", f"/{test_id}") + + if response.status_code == 200: # Should be 403 or 404 + accessible_unauthorized.append({ + "id": test_id, + "status_code": response.status_code, + "response_length": len(response.text) + }) + + if accessible_unauthorized: + return {"vulnerable": True, "unauthorized_access": accessible_unauthorized} +``` + +**2. Predictable patterns** - Sequential IDs reveal total count and enable scraping + +If you get IDs `[100, 101, 102]` when making requests, you know there are at least 102 objects and you can enumerate all of them. + +```python +def _test_predictable_id_patterns(self) -> dict[str, Any]: + """ + Test if IDs follow predictable patterns + """ + ids1 = self._extract_ids_from_response() + ids2 = self._extract_ids_from_response() + + numeric_ids1 = [id for id in ids1 if isinstance(id, int)] + numeric_ids2 = [id for id in ids2 if isinstance(id, int)] + + if len(numeric_ids1) >= 2: + diff1 = abs(numeric_ids1[1] - numeric_ids1[0]) + + if len(numeric_ids2) >= 2: + diff2 = abs(numeric_ids2[1] - numeric_ids2[0]) + + if diff1 == diff2 and diff1 == 1: # Increments by 1 each time + return { + "vulnerable": True, + "pattern_type": "Sequential IDs", + "id_difference": diff1 + } +``` + +### Common Attacks + +Attack payloads in `scanners/payloads.py:179-216`: + +1. **Numeric ID manipulation**: +```python +NUMERIC_ID_MANIPULATIONS = [ + 0, # First record + -1, # May wrap to last record + 1, # Common starting ID + 9999, # Try common ranges + 999999, # Higher ranges +] +``` + +2. **String ID manipulation**: +```python +STRING_ID_MANIPULATIONS = [ + "admin", # Privileged accounts + "root", + "test", + "../../../etc/passwd", # Path traversal attempts +] +``` + +3. **UUID guessing** - While UUIDs are random, poor implementations might: +- Use predictable seeds +- Accept all-zeros or all-ones UUIDs +- Not validate UUID format + +### Defense Strategies + +The fix is simple conceptually: check authorization before returning data. But you must do it on every single endpoint. + +**Vulnerable code:** +```python +@router.get("/api/documents/{doc_id}") +async def get_document(doc_id: int, user: User = Depends(get_current_user)): + document = db.query(Document).filter(Document.id == doc_id).first() + return document # IDOR vulnerability - didn't check ownership +``` + +**Fixed code:** +```python +@router.get("/api/documents/{doc_id}") +async def get_document(doc_id: int, user: User = Depends(get_current_user)): + document = db.query(Document).filter( + Document.id == doc_id, + Document.owner_id == user.id # Authorization check + ).first() + + if not document: + raise HTTPException(status_code=404, detail="Document not found") + + return document +``` + +This project demonstrates proper authorization in `services/scan_service.py:67-84`: + +```python +@staticmethod +def get_scan_by_id(db: Session, scan_id: int, user_id: int) -> ScanResponse: + """ + Get scan by ID with authorization check + """ + scan = ScanRepository.get_by_id(db, scan_id) + + if not scan: + raise HTTPException(status_code=404, detail="Scan not found") + + # CRITICAL: Check if user owns this scan + if scan.user_id != user_id: + raise HTTPException(status_code=403, detail="Not authorized to access this scan") + + return ScanResponse.model_validate(scan) +``` + +**Critical defenses:** +- Check object ownership on every access +- Use UUIDs instead of sequential integers (but still check authorization!) +- Implement row-level security in your database +- Return 404 for unauthorized access (don't leak existence with 403) +- Log suspicious access patterns (trying many IDs rapidly) + +### Common Pitfalls + +**Mistake 1: Checking auth only on writes** +```python +# Bad - checks auth for POST but not GET +@router.post("/api/orders/{order_id}/cancel") +async def cancel_order(order_id: int, user: User = Depends()): + order = db.get(Order, order_id) + if order.user_id != user.id: # Auth check here + raise HTTPException(403) + order.status = "cancelled" + +@router.get("/api/orders/{order_id}") +async def get_order(order_id: int): + return db.get(Order, order_id) # No auth check - vulnerable +``` + +**Mistake 2: Trusting UUIDs** +```python +# Bad - assumes UUID means authorization not needed +@router.get("/api/documents/{doc_uuid}") +async def get_document(doc_uuid: str): + return db.query(Document).filter(Document.uuid == doc_uuid).first() + # Still vulnerable! UUIDs are hard to guess but not impossible + +# Good - check authorization even with UUIDs +@router.get("/api/documents/{doc_uuid}") +async def get_document(doc_uuid: str, user: User = Depends()): + doc = db.query(Document).filter( + Document.uuid == doc_uuid, + Document.owner_id == user.id + ).first() + if not doc: + raise HTTPException(404) + return doc +``` + +## How These Concepts Relate + +These vulnerabilities often appear together and compound each other: + +``` +No Rate Limiting + ↓ + enables + ↓ +Rapid IDOR Enumeration + ↓ + combined with + ↓ +Broken Authentication + ↓ + allows + ↓ +SQL Injection Exploitation + ↓ + results in + ↓ +Complete Data Breach +``` + +Real example: An API with weak rate limiting allows an attacker to: +1. Enumerate user IDs rapidly (IDOR) +2. Try stolen passwords against each ID (broken auth) +3. Once logged in, inject SQL to dump the entire database + +Defense requires addressing all layers. + +## Industry Standards and Frameworks + +### OWASP Top 10 + +This project addresses: + +- **API1:2023 Broken Object Level Authorization** - IDOR scanner (`idor_scanner.py`) detects missing authorization checks +- **API2:2023 Broken Authentication** - Auth scanner (`auth_scanner.py`) tests JWT vulnerabilities, missing auth, weak tokens +- **API4:2023 Unrestricted Resource Consumption** - Rate limit scanner (`rate_limit_scanner.py`) detects missing or bypassable rate limits +- **API8:2023 Security Misconfiguration** - Multiple scanners catch common misconfigurations + +### MITRE ATT&CK + +Relevant techniques: + +- **T1190** - Exploit Public-Facing Application - All these vulnerabilities apply to public APIs +- **T1110** - Brute Force - Rate limiting prevents credential stuffing and brute force attacks +- **T1212** - Exploitation for Credential Access - SQL injection can extract credentials +- **T1078** - Valid Accounts - Authentication bypass provides attacker with valid access + +### CWE + +Common weakness enumerations covered: + +- **CWE-89** - SQL Injection - Tested by `sqli_scanner.py` with error-based, boolean blind, and time-based techniques +- **CWE-287** - Improper Authentication - Auth scanner detects missing or broken authentication +- **CWE-639** - Authorization Bypass Through User-Controlled Key - IDOR scanner finds object-level authorization issues +- **CWE-770** - Allocation of Resources Without Limits - Rate limit scanner detects unrestricted resource consumption + +## Real World Examples + +### Case Study 1: The Parler Data Scrape (2021) + +What happened: After Parler was removed from app stores, researchers downloaded the entire platform's content - 70TB including deleted posts, user data, and metadata with GPS coordinates. + +How the attack worked: +1. Parler used sequential post IDs +2. No rate limiting on the API +3. No authentication required to view public posts +4. Researchers wrote a script that incremented post IDs from 1 to ~13 million +5. They downloaded everything, including "deleted" posts that weren't actually deleted + +What defenses failed: +- IDOR: Sequential IDs made enumeration trivial +- Rate limiting: Could make millions of requests without throttling +- Data retention: Deleted posts remained accessible + +How this could have been prevented: +- Use UUIDs for post IDs (makes enumeration harder but not impossible) +- Require authentication for API access +- Implement aggressive rate limiting (100 posts per hour per IP) +- Actually delete data when users delete it +- Log and alert on unusual access patterns + +### Case Study 2: T-Mobile API Breach (2022) + +What happened: Attackers exploited an API endpoint to access data on 37 million customers, including names, addresses, and phone numbers. + +How the attack worked: +1. API endpoint accepted customer IDs as parameters +2. No authorization check verified the requester owned that customer ID +3. Attackers enumerated IDs to scrape customer records +4. Rate limiting was insufficient to stop the attack + +What defenses failed: +- IDOR: No object-level authorization +- Rate limiting: Present but set too high +- Monitoring: Attack wasn't detected in real-time + +How this could have been prevented: +- Check authorization: `if customer.id != current_user.id: return 403` +- Implement proper rate limiting (10 requests/minute per authenticated user) +- Monitor for sequential ID access patterns +- Require multi-factor authentication for sensitive data access +- Use database row-level security as defense in depth + +## Testing Your Understanding + +Before moving to the architecture, make sure you can answer: + +1. Why doesn't using UUIDs instead of sequential IDs completely fix IDOR vulnerabilities? +2. What's the difference between detecting rate limiting headers and detecting active enforcement? +3. How can time-based blind SQL injection work even when you can't see database errors or response differences? +4. Why must JWT signature validation explicitly reject the "none" algorithm? + +If these questions feel unclear, re-read the relevant sections. The implementation will make more sense once these fundamentals click. + +## Further Reading + +**Essential:** +- OWASP API Security Top 10 2023 - Full document with examples and remediation guidance +- PortSwigger Web Security Academy - Interactive labs for SQL injection, authentication, and authorization +- JWT.io - Decode tokens, understand structure, see libraries for different languages + +**Deep dives:** +- "The Web Application Hacker's Handbook" - Chapter 9 (Attacking Data Stores) for SQL injection techniques +- OWASP Testing Guide v4 - Testing for SQL Injection (detailed methodology) +- Auth0 JWT Handbook - Comprehensive guide to JWT security + +**Historical context:** +- "How I Hacked Facebook OAuth to Get Full Access" - Egor Homakov (2013) +- "Exploiting the Auth Token Length Oracle" - explains timing attacks on authentication +- Original JWT RFC 7519 - Understand the standard and its security considerations + diff --git a/PROJECTS/intermediate/api-security-scanner/learn/02-ARCHITECTURE.md b/PROJECTS/intermediate/api-security-scanner/learn/02-ARCHITECTURE.md index e69de29b..eaf0e925 100644 --- a/PROJECTS/intermediate/api-security-scanner/learn/02-ARCHITECTURE.md +++ b/PROJECTS/intermediate/api-security-scanner/learn/02-ARCHITECTURE.md @@ -0,0 +1,1511 @@ +# 02-ARCHITECTURE.md + +# System Architecture + +This document breaks down how the system is designed and why certain architectural decisions were made. + +## High Level Architecture + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Client Browser │ +└────────────────────────┬────────────────────────────────────────┘ + │ HTTP/HTTPS + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ Nginx Reverse Proxy │ +│ - Routes /api/* → Backend (FastAPI) │ +│ - Routes /* → Frontend (Static Files) │ +│ - Handles CORS, compression, caching │ +└──────────────┬─────────────────────────┬────────────────────────┘ + │ │ + ▼ ▼ +┌──────────────────────────┐ ┌─────────────────────────────────┐ +│ Backend (FastAPI) │ │ Frontend (React/Vite) │ +│ ┌────────────────────┐ │ │ ┌───────────────────────────┐ │ +│ │ Routes │ │ │ │ Components │ │ +│ │ /auth, /scans │ │ │ │ Auth, Scan UI │ │ +│ └─────────┬──────────┘ │ │ └─────────┬─────────────────┘ │ +│ ▼ │ │ │ │ +│ ┌────────────────────┐ │ │ ┌───────────────────────────┐ │ +│ │ Services │ │ │ │ TanStack Query │ │ +│ │ Business Logic │ │ │ │ Server State Mgmt │ │ +│ └─────────┬──────────┘ │ │ └─────────┬─────────────────┘ │ +│ ▼ │ │ │ │ +│ ┌────────────────────┐ │ │ ┌───────────────────────────┐ │ +│ │ Repositories │ │ │ │ Zustand Stores │ │ +│ │ Data Access │ │ │ │ Local State Mgmt │ │ +│ └─────────┬──────────┘ │ │ └───────────────────────────┘ │ +│ │ │ │ │ +│ ┌─────────▼──────────┐ │ └─────────────────────────────────┘ +│ │ Scanners │ │ +│ │ RateLimit, Auth, │ │ +│ │ SQLi, IDOR │ │ +│ └────────────────────┘ │ +└──────────────┬────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ PostgreSQL Database │ +│ Tables: users, scans, test_results │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Component Breakdown + +**Nginx Reverse Proxy** +- Purpose: Single entry point for all HTTP traffic +- Responsibilities: Route requests based on path, serve static files, handle SSL termination (production), compress responses, cache static assets +- Interfaces: Exposes port 80 (HTTP) and optionally 443 (HTTPS), proxies to backend on internal port 8000 + +**FastAPI Backend** +- Purpose: REST API server providing authentication and scanning services +- Responsibilities: Validate requests, enforce authentication, execute security scans, store results, return JSON responses +- Interfaces: Exposes HTTP endpoints at `/auth/*` and `/scans/*`, connects to PostgreSQL for data persistence + +**React Frontend** +- Purpose: User interface for creating scans and viewing results +- Responsibilities: Form validation, API communication, state management, result visualization +- Interfaces: Communicates with backend via `/api` prefix, renders in browser + +**PostgreSQL Database** +- Purpose: Persistent storage for users, scans, and test results +- Responsibilities: Data integrity, relationship enforcement, query optimization +- Interfaces: Accepts connections from backend on port 5432, enforces foreign key constraints + +**Scanner Modules** +- Purpose: Execute security tests against target APIs +- Responsibilities: Send HTTP requests, analyze responses, detect vulnerabilities, collect evidence +- Interfaces: Inherit from `BaseScanner`, return `TestResultCreate` schemas + +## Data Flow + +### Primary Use Case: Creating and Running a Scan + +Step by step walkthrough of what happens when a user submits a new scan: + +``` +1. User submits form → Frontend validates (Zod schema) + Input: { targetUrl, authToken, testsToRun, maxRequests } + Validation happens at frontend/src/lib/validation.ts:42-52 + +2. Frontend → POST /api/scans/ → Nginx + Adds Authorization header with JWT from localStorage + Request routed based on /api prefix + +3. Nginx → Backend (routes/scans.py:23-37) + Proxy passes to http://backend:8000/scans/ + Preserves headers including Authorization + +4. Route handler → Dependencies check auth + @limiter.limit("15/minute") - rate limits this endpoint + get_current_user() extracts JWT, validates, loads user from DB + Code at backend/core/dependencies.py:17-49 + +5. Route → Service layer (services/scan_service.py:23-65) + ScanService.run_scan(db, user_id, scan_request) + Creates Scan record in database via repository + +6. Service → Scanner modules (scanners/*.py) + Loops through requested tests (rate_limit, auth, sqli, idor) + Instantiates appropriate scanner class for each test + Each scanner inherits from BaseScanner + +7. Scanner → Target API + Makes HTTP requests using requests.Session + Implements retry logic, rate limiting, timeout handling + Base logic at backend/scanners/base_scanner.py:40-156 + +8. Scanner analyzes responses → Returns TestResultCreate + Detects vulnerabilities based on: + - Status codes (429 for rate limiting) + - Response content (SQL errors) + - Timing differences (blind SQLi) + - Header patterns (JWT algorithms) + +9. Service saves results → Repository → Database + TestResultRepository.create_test_result() for each scanner output + Foreign key links results to scan + Code at backend/repositories/test_result_repository.py:19-50 + +10. Service → Route → JSON response + ScanResponse includes full scan with nested test_results + Frontend receives and redirects to /scans/{id} + +11. Frontend fetches full scan → GET /api/scans/{id} + TanStack Query caches result + Renders TestResultCard for each result +``` + +Example with code references: + +```python +# Step 5: Route handler +@router.post("/", response_model=ScanResponse) +@limiter.limit(settings.API_RATE_LIMIT_SCAN) # "15/minute" +async def create_scan( + request: Request, + scan_request: ScanRequest, + db: Session = Depends(get_db), + current_user: UserResponse = Depends(get_current_user), # Auth check +) -> ScanResponse: + return ScanService.run_scan(db, current_user.id, scan_request) + +# Step 6: Service orchestrates scanners +scanner_mapping: dict[TestType, type[BaseScanner]] = { + TestType.RATE_LIMIT: RateLimitScanner, + TestType.AUTH: AuthScanner, + TestType.SQLI: SQLiScanner, + TestType.IDOR: IDORScanner, +} + +for test_type in scan_request.tests_to_run: + scanner_class = scanner_mapping.get(test_type) + scanner = scanner_class( + target_url=str(scan_request.target_url), + auth_token=scan_request.auth_token, + max_requests=scan_request.max_requests, + ) + result = scanner.scan() # Execute the test + results.append(result) + +# Step 9: Save to database +for result in results: + TestResultRepository.create_test_result( + db=db, + scan_id=scan.id, + test_name=result.test_name, + status=result.status, + severity=result.severity, + details=result.details, + evidence_json=result.evidence_json, + recommendations_json=result.recommendations_json, + ) +``` + +### Secondary Use Case: User Registration Flow + +``` +1. User fills registration form → Frontend validates + Zod schema checks: email format, password strength (8+ chars, uppercase, lowercase, number) + frontend/src/lib/validation.ts:17-31 + +2. POST /api/auth/register → Rate limited to 15/minute + Prevents automated account creation abuse + backend/routes/auth.py:25-42 + +3. Service checks if email exists → Repository query + UserRepository.get_by_email() checks for duplicates + Returns 400 if email already registered + +4. Service hashes password with bcrypt + core/security.py:11-18 uses bcrypt.gensalt() and hashpw() + Salt automatically generated per password + +5. Repository creates user record + UserRepository.create_user(email, hashed_password) + Sets is_active=True, created_at=UTC timestamp + +6. Response returns user data (NOT password) + UserResponse schema excludes hashed_password field + Frontend redirects to login page +``` + +## Design Patterns + +### Repository Pattern + +**What it is:** +Abstraction layer between business logic and data access. All database queries go through repository classes that provide clean interfaces. + +**Where we use it:** +- `repositories/user_repository.py` - User CRUD operations +- `repositories/scan_repository.py` - Scan queries with eager loading +- `repositories/test_result_repository.py` - Test result operations + +**Why we chose it:** +Keeps services clean and testable. Services call `UserRepository.get_by_email(db, email)` instead of writing raw queries. If we switch from SQLAlchemy to a different ORM or database entirely, we only change repository implementations, not service logic. + +**Trade-offs:** +- Pros: Testable (mock repositories), maintainable (queries in one place), flexible (swap implementations) +- Cons: Extra layer of abstraction, more files to navigate, can feel like overkill for simple CRUD + +Example implementation: + +```python +# repositories/user_repository.py:12-35 +class UserRepository: + """ + Repository for User database operations + """ + @staticmethod + def get_by_id(db: Session, user_id: int) -> User | None: + return db.query(User).filter(User.id == user_id).first() + + @staticmethod + def get_by_email(db: Session, email: str) -> User | None: + return db.query(User).filter(User.email == email).first() + + @staticmethod + def create_user( + db: Session, + email: str, + hashed_password: str, + commit: bool = True + ) -> User: + user = User(email=email, hashed_password=hashed_password) + db.add(user) + if commit: + db.commit() + db.refresh(user) + return user +``` + +Services use it: + +```python +# services/auth_service.py:24-32 +existing_user = UserRepository.get_by_email(db, user_data.email) +if existing_user: + raise HTTPException(status_code=400, detail="Email already registered") + +hashed_password = hash_password(user_data.password) + +user = UserRepository.create_user( + db=db, + email=user_data.email, + hashed_password=hashed_password, +) +``` + +### Dependency Injection (FastAPI's Depends) + +**What it is:** +FastAPI's dependency injection system automatically provides values to route handler parameters. Used for database sessions, authentication, rate limiting. + +**Where we use it:** +Every route handler in `routes/auth.py` and `routes/scans.py` uses dependencies: + +```python +# routes/scans.py:23-37 +@router.post("/", response_model=ScanResponse) +@limiter.limit(settings.API_RATE_LIMIT_SCAN) +async def create_scan( + request: Request, # Injected by FastAPI + scan_request: ScanRequest, # Parsed and validated from request body + db: Session = Depends(get_db), # Database session injected + current_user: UserResponse = Depends(get_current_user), # Auth check injected +) -> ScanResponse: + return ScanService.run_scan(db, current_user.id, scan_request) +``` + +**Why we chose it:** +Clean separation of concerns. The route handler doesn't know how to: +- Get a database session (handled by `get_db`) +- Validate JWT tokens (handled by `get_current_user`) +- Parse request bodies (handled by Pydantic) + +This makes testing easier - mock the dependencies, not the entire request cycle. + +**Trade-offs:** +- Pros: Testable, reusable, explicit dependencies, automatic cleanup (session closing) +- Cons: "Magic" behavior for beginners, debugging can be tricky if dependency fails + +The `get_current_user` dependency implementation (`core/dependencies.py:17-49`): + +```python +async def get_current_user( + credentials: HTTPAuthorizationCredentials = Depends(security), + db: Session = Depends(get_db), +) -> UserResponse: + """ + FastAPI dependency to extract and verify the current authenticated user + """ + try: + payload = decode_token(credentials.credentials) + email: str | None = payload.get("sub") + + if email is None: + raise HTTPException(status_code=401, detail="Invalid authentication credentials") + + user = UserRepository.get_by_email(db, email) + + if not user: + raise HTTPException(status_code=401, detail="User not found") + + return UserResponse.model_validate(user) + + except ValueError: + raise HTTPException(status_code=401, detail="Invalid authentication credentials") +``` + +### Template Method (BaseScanner Abstract Class) + +**What it is:** +Define the skeleton of an algorithm in a base class, let subclasses override specific steps. All scanners share common HTTP logic but implement their own `scan()` method. + +**Where we use it:** +`scanners/base_scanner.py` provides common functionality: + +```python +class BaseScanner(ABC): + def __init__(self, target_url: str, auth_token: str | None = None, max_requests: int | None = None): + self.target_url = target_url.rstrip("/") + self.auth_token = auth_token + self.max_requests = max_requests or settings.DEFAULT_MAX_REQUESTS + self.session = self._create_session() + self.last_request_time = 0.0 + self.request_count = 0 + + def make_request(self, method: str, endpoint: str, **kwargs: Any) -> requests.Response: + """Common HTTP request logic with retry and rate limiting""" + self._wait_before_request() + # ... retry logic, backoff, timeout handling + + def get_baseline_timing(self, endpoint: str, samples: int | None = None) -> tuple[float, float]: + """Statistical baseline for time-based detection""" + # ... takes samples, calculates mean and stdev + + @abstractmethod + def scan(self) -> TestResultCreate: + """Must be implemented by specific scanner classes""" +``` + +Subclasses implement `scan()`: + +```python +# scanners/sqli_scanner.py:25-60 +class SQLiScanner(BaseScanner): + def scan(self) -> TestResultCreate: + error_based_test = self._test_error_based_sqli() + if error_based_test["vulnerable"]: + return self._create_vulnerable_result(...) + + boolean_based_test = self._test_boolean_based_sqli() + if boolean_based_test["vulnerable"]: + return self._create_vulnerable_result(...) + + time_based_test = self._test_time_based_sqli() + if time_based_test["vulnerable"]: + return self._create_vulnerable_result(...) + + return TestResultCreate(status=ScanStatus.SAFE, ...) +``` + +**Why we chose it:** +Eliminates code duplication. Request spacing, retry logic, session management - written once in `BaseScanner`, used by all four scanner types. + +**Trade-offs:** +- Pros: DRY principle, consistent behavior, easy to add new scanners +- Cons: Tight coupling to base class, inheritance can be limiting + +## Layer Separation + +The backend uses a three-layer architecture: + +``` +┌────────────────────────────────────┐ +│ Layer 1: Routes │ +│ - HTTP request/response │ +│ - Validation (Pydantic) │ +│ - Auth checks (dependencies) │ +│ - Rate limiting │ +└────────────────┬───────────────────┘ + ↓ +┌────────────────────────────────────┐ +│ Layer 2: Services │ +│ - Business logic │ +│ - Orchestration │ +│ - Transaction management │ +│ - Error handling │ +└────────────────┬───────────────────┘ + ↓ +┌────────────────────────────────────┐ +│ Layer 3: Repositories │ +│ - Database queries │ +│ - Data access only │ +│ - No business logic │ +└────────────────────────────────────┘ +``` + +### Why Layers? + +Separation makes each layer testable in isolation: +- Test routes with mocked services +- Test services with mocked repositories +- Test repositories against a real test database + +It also enforces single responsibility. Routes don't write SQL. Services don't parse HTTP headers. Repositories don't implement business rules. + +### What Lives Where + +**Layer 1: Routes** (`routes/auth.py`, `routes/scans.py`) +- Files: Route handler functions decorated with `@router.get/post/delete` +- Imports: Can import from services, schemas, dependencies +- Forbidden: Direct database access, business logic, calling repositories directly + +Example route: + +```python +# routes/scans.py:67-83 +@router.get("/{scan_id}", response_model=ScanResponse) +@limiter.limit(settings.API_RATE_LIMIT_DEFAULT) +async def get_scan( + request: Request, + scan_id: int, + db: Session = Depends(get_db), + current_user: UserResponse = Depends(get_current_user), +) -> ScanResponse: + """ + Get a specific scan by ID + """ + return ScanService.get_scan_by_id(db, scan_id, current_user.id) +``` + +**Layer 2: Services** (`services/auth_service.py`, `services/scan_service.py`) +- Files: Service classes with static methods +- Imports: Repositories, models, schemas, utilities +- Forbidden: HTTP-specific code (requests, responses), direct SQL queries + +Example service method: + +```python +# services/scan_service.py:23-65 +@staticmethod +def run_scan(db: Session, user_id: int, scan_request: ScanRequest) -> ScanResponse: + # Create scan record + scan = ScanRepository.create_scan( + db=db, + user_id=user_id, + target_url=str(scan_request.target_url), + ) + + # Map test types to scanner classes + scanner_mapping: dict[TestType, type[BaseScanner]] = { + TestType.RATE_LIMIT: RateLimitScanner, + TestType.AUTH: AuthScanner, + TestType.SQLI: SQLiScanner, + TestType.IDOR: IDORScanner, + } + + results: list[TestResultCreate] = [] + + # Execute each requested test + for test_type in scan_request.tests_to_run: + scanner_class = scanner_mapping.get(test_type) + scanner = scanner_class(...) + result = scanner.scan() + results.append(result) + + # Save all results + for result in results: + TestResultRepository.create_test_result(db=db, scan_id=scan.id, ...) + + db.refresh(scan) + return ScanResponse.model_validate(scan) +``` + +**Layer 3: Repositories** (`repositories/user_repository.py`, `repositories/scan_repository.py`, `repositories/test_result_repository.py`) +- Files: Repository classes with static methods for database operations +- Imports: Models, SQLAlchemy, config +- Forbidden: Business logic, HTTP handling, calling other repositories + +Example repository method: + +```python +# repositories/scan_repository.py:48-71 +@staticmethod +def get_by_user( + db: Session, + user_id: int, + skip: int = 0, + limit: int | None = None +) -> list[Scan]: + """ + Get all scans for a user with pagination + """ + if limit is None: + limit = settings.DEFAULT_PAGINATION_LIMIT + + return ( + db.query(Scan) + .options(joinedload(Scan.test_results)) # Eager load relationships + .filter(Scan.user_id == user_id) + .order_by(Scan.scan_date.desc()) + .offset(skip) + .limit(limit) + .all() + ) +``` + +## Data Models + +### User Model + +```python +# models/User.py:12-43 +class User(BaseModel): + """ + Stores authentication credentials and user information + """ + __tablename__ = "users" + + email = Column( + String(settings.EMAIL_MAX_LENGTH), # 255 chars + unique=True, + nullable=False, + index=True, # Fast lookups by email + ) + hashed_password = Column(String, nullable=False) + is_active = Column(Boolean, default=True, nullable=False) +``` + +**Fields explained:** +- `id`: Auto-incrementing primary key (inherited from `BaseModel`) +- `email`: Unique identifier for login, indexed for fast authentication queries +- `hashed_password`: Bcrypt hash, never stored in plaintext, never returned in API responses +- `is_active`: Soft delete flag, allows disabling accounts without losing data +- `created_at`, `updated_at`: Timestamps inherited from `BaseModel` + +**Relationships:** +- One-to-many with Scan: `user.scans` returns all scans created by this user +- Defined by relationship in Scan model: `user = relationship("User", backref="scans")` + +### Scan Model + +```python +# models/Scan.py:15-57 +class Scan(BaseModel): + """ + Stores metadata about scans performed on target URLs + """ + __tablename__ = "scans" + + user_id = Column( + Integer, + ForeignKey("users.id", ondelete="CASCADE"), # Delete scans when user deleted + nullable=False, + index=True, + ) + target_url = Column( + String(settings.URL_MAX_LENGTH), # 2048 chars + nullable=False, + ) + scan_date = Column( + DateTime(timezone=True), + default=lambda: datetime.now(UTC), + nullable=False, + ) + + user = relationship("User", backref="scans") + test_results = relationship( + "TestResult", + back_populates="scan", + cascade="all, delete-orphan", # Delete results when scan deleted + ) +``` + +**Fields explained:** +- `user_id`: Foreign key to users table, indexed for filtering scans by user +- `target_url`: URL that was scanned, up to 2048 chars for long query strings +- `scan_date`: When scan was initiated, timezone-aware datetime in UTC +- `CASCADE`: When user is deleted, their scans are deleted. When scan is deleted, its test results are deleted + +**Relationships:** +- Many-to-one with User: `scan.user` gets the user who created it +- One-to-many with TestResult: `scan.test_results` gets all vulnerability findings + +**Properties** (computed, not stored): + +```python +@property +def has_vulnerabilities(self) -> bool: + return any(result.status == "vulnerable" for result in self.test_results) + +@property +def vulnerability_count(self) -> int: + return sum(1 for result in self.test_results if result.status == "vulnerable") +``` + +### TestResult Model + +```python +# models/TestResult.py:16-57 +class TestResult(BaseModel): + """ + Stores individual test results for each security scan + """ + __tablename__ = "test_results" + + scan_id = Column( + Integer, + ForeignKey("scans.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + test_name = Column( + Enum(TestType), # rate_limit, auth, sqli, idor + nullable=False, + index=True, + ) + status = Column( + Enum(ScanStatus), # vulnerable, safe, error + nullable=False, + index=True, + ) + severity = Column( + Enum(Severity), # critical, high, medium, low, info + nullable=False, + index=True, + ) + details = Column(Text, nullable=False) + evidence_json = Column(JSON, nullable=False, default=dict) + recommendations_json = Column(JSON, nullable=False, default=list) +``` + +**Fields explained:** +- `scan_id`: Foreign key to scans table, which scan this result belongs to +- `test_name`: Enum constraining values to valid test types, indexed for filtering by test +- `status`: Enum for vulnerable/safe/error, indexed for finding all vulnerabilities +- `severity`: Enum for CRITICAL/HIGH/MEDIUM/LOW/INFO, indexed for prioritization +- `details`: Text description of what was found +- `evidence_json`: JSON storing response codes, payloads, timings - varies by test type +- `recommendations_json`: Array of strings with remediation steps + +**Why JSON columns:** +Evidence varies by test type: +- Rate limit test: `{"rate_limit_headers": {...}, "bypass_method": "IP spoofing"}` +- SQLi test: `{"database_type": "mysql", "payload": "' OR 1=1--", "response_time": "5.23s"}` +- Auth test: `{"algorithm_variant": "none", "status_code": 200}` + +JSON flexibility lets each scanner store relevant data without schema changes. + +## Security Architecture + +### Threat Model + +What we're protecting against: + +1. **Unauthorized access to scan data** - Users should only see their own scans. Attacker tries to view scan ID 123 when they only created scan ID 456. Defense: Authorization check in `services/scan_service.py:77-80` verifies `scan.user_id == user_id`. + +2. **Token theft and replay** - Attacker steals JWT from network traffic or XSS. Defense: HTTPS in production (enforced by nginx), short token lifetime (24 hours from `config.py:24`), httpOnly cookies (not implemented but recommended). + +3. **Brute force login attempts** - Attacker tries common passwords against accounts. Defense: Rate limiting at `routes/auth.py:49` limits login to 20/minute, bcrypt makes password verification slow. + +4. **SQL injection in scanner payloads** - Malicious user creates scan with SQLi payload as target URL hoping to exploit our database. Defense: All database access uses parameterized queries via SQLAlchemy, never concatenation. + +5. **Resource exhaustion** - Attacker submits scans with max_requests=50 repeatedly to consume backend resources. Defense: Rate limiting on scan creation (15/minute), timeout limits on scanners, max_requests capped at 50. + +What we're NOT protecting against (out of scope): + +- **DDoS attacks** - Application-level rate limiting can't stop volumetric network floods. Requires infrastructure defenses (CloudFlare, AWS Shield). +- **Database compromise** - If attacker gains direct database access, they can read all data. Requires infrastructure hardening, encrypted columns for sensitive data. +- **Server-side request forgery (SSRF)** - Scanners make requests to user-provided URLs. This is intentional functionality. Mitigation: scanners run with limited network access, not on internal network. + +### Defense Layers + +Multiple layers of security create defense in depth: + +``` +Layer 1: Network (Nginx) + ↓ HTTPS, CORS headers, rate limits +Layer 2: Application (FastAPI) + ↓ JWT validation, endpoint rate limits, input validation +Layer 3: Business Logic (Services) + ↓ Authorization checks, transaction management +Layer 4: Data Access (Repositories) + ↓ Parameterized queries, row-level permissions +``` + +**Why multiple layers?** +If one defense fails, others catch the attack. Example: Nginx rate limit bypassed via IP spoofing, but application-level rate limit (by user ID) still protects. JWT validation bypassed somehow, but service layer still checks `scan.user_id` before returning data. + +### Authentication Flow + +Complete JWT authentication cycle: + +**1. Registration** (`services/auth_service.py:19-41`): +```python +# Hash password with bcrypt +hashed_password = hash_password(user_data.password) +# Bcrypt automatically generates salt, 10 rounds by default + +# Store hashed password +user = UserRepository.create_user( + db=db, + email=user_data.email, + hashed_password=hashed_password, +) +``` + +**2. Login** (`services/auth_service.py:43-71`): +```python +# Verify password +if not verify_password(login_data.password, user.hashed_password): + raise HTTPException(status_code=401, detail="Invalid email or password") + +# Create JWT with expiration +access_token = create_access_token( + data={"sub": user.email}, # Subject claim + expires_delta=timedelta(minutes=1440) # 24 hours +) + +return TokenResponse(access_token=access_token, token_type="bearer") +``` + +**3. Protected endpoint access** (`core/dependencies.py:17-49`): +```python +# Extract token from Authorization header +credentials: HTTPAuthorizationCredentials = Depends(security) +# security = HTTPBearer() from fastapi.security + +# Decode and verify token +payload = decode_token(credentials.credentials) +email = payload.get("sub") + +# Load user from database +user = UserRepository.get_by_email(db, email) +if not user: + raise HTTPException(status_code=401) + +return UserResponse.model_validate(user) +``` + +**4. Route handler receives authenticated user**: +```python +@router.post("/scans/") +async def create_scan( + current_user: UserResponse = Depends(get_current_user), # Authenticated + ... +): + # current_user is guaranteed to be valid at this point + return ScanService.run_scan(db, current_user.id, ...) +``` + +### Rate Limiting Strategy + +Multiple rate limit implementations: + +**1. Nginx level** - Not implemented in dev, but production nginx can use limit_req: +```nginx +limit_req_zone $binary_remote_addr zone=api_limit:10m rate=100r/m; +limit_req zone=api_limit burst=20 nodelay; +``` + +**2. Application level** - SlowAPI per-endpoint limits (`backend/factory.py:34-36`): +```python +limiter = Limiter(key_func=get_remote_address) +app.state.limiter = limiter +app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler) +``` + +Applied to routes: +```python +# routes/auth.py:25-28 +@router.post("/register", ...) +@limiter.limit(settings.API_RATE_LIMIT_REGISTER) # "15/minute" +async def register(...): + +# routes/auth.py:47-50 +@router.post("/login", ...) +@limiter.limit(settings.API_RATE_LIMIT_LOGIN) # "20/minute" +async def login(...): + +# routes/scans.py:23-26 +@router.post("/", ...) +@limiter.limit(settings.API_RATE_LIMIT_SCAN) # "15/minute" +async def create_scan(...): +``` + +**3. Scanner-level** - Outgoing requests to targets are spaced (`base_scanner.py:64-90`): +```python +def _wait_before_request(self, jitter_ms: int | None = None) -> None: + """ + Implement request spacing to avoid overwhelming target + """ + required_delay = 1.0 / (self.max_requests / settings.SCANNER_RATE_LIMIT_WINDOW_SECONDS) + # If max_requests=100 and window=60s, delay = 1.0 / (100/60) = 0.6s between requests + + jitter = random.uniform(0, jitter_ms / 1000.0) # Random variation + elapsed = time.time() - self.last_request_time + + if elapsed < required_delay: + time.sleep(required_delay - elapsed + jitter) +``` + +This prevents scanners from hammering target APIs and getting IP banned. + +## Storage Strategy + +### PostgreSQL + +**What we store:** +- User accounts (email, hashed password) +- Scan metadata (target URL, timestamp) +- Test results (findings, evidence, recommendations) + +**Why PostgreSQL:** +- Relational data with foreign keys (scans → users, test_results → scans) +- JSON column support for flexible evidence storage +- ACID transactions for data integrity +- Mature, well-documented, widely deployed + +Alternatives considered: +- MongoDB: Better for schema-less data, but we have clear relationships and benefit from foreign key constraints +- SQLite: Simpler setup, but doesn't handle concurrent writes well (multiple scans running simultaneously) + +**Schema design:** + +```sql +-- Automatically generated by SQLAlchemy from models +CREATE TABLE users ( + id SERIAL PRIMARY KEY, + email VARCHAR(255) UNIQUE NOT NULL, + hashed_password VARCHAR NOT NULL, + is_active BOOLEAN NOT NULL DEFAULT TRUE, + created_at TIMESTAMP WITH TIME ZONE NOT NULL, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL +); + +CREATE INDEX ix_users_email ON users(email); + +CREATE TABLE scans ( + id SERIAL PRIMARY KEY, + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + target_url VARCHAR(2048) NOT NULL, + scan_date TIMESTAMP WITH TIME ZONE NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL +); + +CREATE INDEX ix_scans_user_id ON scans(user_id); + +CREATE TABLE test_results ( + id SERIAL PRIMARY KEY, + scan_id INTEGER NOT NULL REFERENCES scans(id) ON DELETE CASCADE, + test_name VARCHAR NOT NULL, -- Enum: rate_limit, auth, sqli, idor + status VARCHAR NOT NULL, -- Enum: vulnerable, safe, error + severity VARCHAR NOT NULL, -- Enum: critical, high, medium, low, info + details TEXT NOT NULL, + evidence_json JSON NOT NULL, + recommendations_json JSON NOT NULL, + created_at TIMESTAMP WITH TIME ZONE NOT NULL, + updated_at TIMESTAMP WITH TIME ZONE NOT NULL +); + +CREATE INDEX ix_test_results_scan_id ON test_results(scan_id); +CREATE INDEX ix_test_results_test_name ON test_results(test_name); +CREATE INDEX ix_test_results_status ON test_results(status); +CREATE INDEX ix_test_results_severity ON test_results(severity); +``` + +**Indexes explained:** +- `users.email`: Fast login lookups (WHERE email = ?) +- `scans.user_id`: Fast filtering (WHERE user_id = ?) +- `test_results.scan_id`: Fast joins (JOIN scans WHERE scan_id = ?) +- `test_results.status`: Fast vulnerability queries (WHERE status = 'vulnerable') +- `test_results.severity`: Fast filtering by severity (WHERE severity = 'critical') + +### Connection Pooling + +Database connections are expensive to create. SQLAlchemy maintains a pool: + +```python +# core/database.py:12-17 +engine = create_engine( + settings.DATABASE_URL, + pool_pre_ping=True, # Verify connections before use (handles DB restarts) + echo=settings.DEBUG, # Log all SQL when DEBUG=True +) +``` + +Default pool size: 5 connections, overflow: 10 (up to 15 total). + +Session lifecycle managed by dependency: + +```python +# core/database.py:28-36 +def get_db() -> Generator[Session, None, None]: + """ + FastAPI dependency for database sessions + """ + db = SessionLocal() + try: + yield db # Provide to route handler + finally: + db.close() # Always close, even if exception +``` + +## Configuration + +### Environment Variables + +All configuration lives in `.env` and `config.py`: + +```python +# config.py:14-69 +class Settings(BaseSettings): + # Application + APP_NAME: str = "API Security Tester" + VERSION: str = "1.0.0" + DEBUG: bool = False + + # Database + DATABASE_URL: str # Required, no default + + # Security - JWT + SECRET_KEY: str # Required, MUST be random in production + ALGORITHM: str = "HS256" + ACCESS_TOKEN_EXPIRE_MINUTES: int = 1440 + + # Rate limiting + API_RATE_LIMIT_LOGIN: str = "20/minute" + API_RATE_LIMIT_REGISTER: str = "15/minute" + API_RATE_LIMIT_SCAN: str = "15/minute" + + # Scanner limits + SCANNER_MAX_CONCURRENT_REQUESTS: int = 50 + SCANNER_CONNECTION_TIMEOUT: int = 180 + DEFAULT_MAX_REQUESTS: int = 100 +``` + +**Why centralized config:** +- Single source of truth for all constants +- Type validation with Pydantic +- Easy to change without touching code +- Different values for dev/test/prod + +**Configuration strategy:** +Development uses `.env` file loaded by Docker Compose. Production uses environment variables set directly in Docker, Kubernetes, or cloud platform. + +Critical settings that must be changed for production: +- `SECRET_KEY` - Generate with `openssl rand -hex 32` +- `DEBUG` - Must be `false` +- `DATABASE_URL` - Production database, not localhost +- `CORS_ORIGINS` - Actual frontend domain, not `http://localhost` + +## Performance Considerations + +### Bottlenecks + +Where this system gets slow under load: + +1. **Scanner HTTP requests** - Each scan makes 10-100 HTTP requests to external APIs. If target is slow (5s response time), scans take minutes. Can't parallelize much without overwhelming targets. Mitigated by timeout limits (180s from `config.py:51`). + +2. **Database queries with relationships** - Loading `scan.test_results` triggers N+1 queries if not eager loaded. With 10 scans and 4 results each = 41 queries (1 for scans + 40 for results). Solved with `joinedload()` in repositories: + +```python +# repositories/scan_repository.py:50-59 +return ( + db.query(Scan) + .options(joinedload(Scan.test_results)) # Single query with JOIN + .filter(Scan.user_id == user_id) + .all() +) +``` + +3. **Password hashing on login** - Bcrypt is intentionally slow (prevents brute force). Each login takes ~100ms. Under load, authentication becomes bottleneck. Mitigated by rate limiting login attempts and using caching (not implemented, but could add Redis for session tokens). + +### Optimizations + +What we did to make it faster: + +- **Request pooling** - `BaseScanner` reuses `requests.Session()` which maintains connection pools to targets. Avoids TCP handshake overhead on each request. + +- **Database indexes** - Foreign keys and commonly queried columns (email, user_id, status) are indexed. Queries that would do table scans become index lookups. + +- **Response pagination** - Scan list queries use LIMIT/OFFSET to avoid loading thousands of records: + +```python +# repositories/scan_repository.py:50-71 +@staticmethod +def get_by_user( + db: Session, + user_id: int, + skip: int = 0, + limit: int | None = None # Default 100 from config +) -> list[Scan]: + if limit is None: + limit = settings.DEFAULT_PAGINATION_LIMIT + + return ( + db.query(Scan) + .offset(skip) + .limit(limit) # Only load requested page + .all() + ) +``` + +- **Enum columns** - `test_name`, `status`, `severity` use Postgres ENUMs, not strings. Smaller storage, faster comparisons, enforced validity. + +### Scalability + +**Vertical scaling** (more CPU/RAM on single server): +- Database: Increase `max_connections` in Postgres config +- Backend: Run more gunicorn workers (4 workers in production Dockerfile) +- Frontend: Nginx already efficient, bottleneck is unlikely here + +Current limits with single server: +- Database can handle ~100 concurrent connections +- Backend with 4 workers handles ~400 concurrent requests +- Scanners are the real limit - each scan is long-running (30-60s) + +**Horizontal scaling** (more servers): +Challenges: +- Scanners are stateless, can run on any backend instance ✓ +- Database requires connection pooling strategy (PgBouncer) +- Shared session state needed (Redis) or stick to JWT (stateless) ✓ +- Load balancer required (nginx, AWS ALB) + +What needs to change: +- Add load balancer in front of backend +- Configure shared session store or rely solely on JWT +- Database connection pool management (PgBouncer) +- Consider async task queue (Celery, RQ) for long-running scans + +## Design Decisions + +### Decision 1: Synchronous scanners despite FastAPI async + +**What we chose:** +Scanners use synchronous `requests` library, not async `httpx` or `aiohttp`. + +**Alternatives considered:** +- `httpx` async HTTP client - Could run all tests concurrently +- `aiohttp` - Similar benefits, different API + +**Trade-offs:** +Pros of sync: +- Simpler code, easier to reason about timing +- Time-based SQLi detection requires precise timing control +- Baseline timing calculation needs sequential requests +- Standard `requests` library is battle-tested + +Cons of sync: +- Can't run multiple tests concurrently within a scan +- Blocks event loop (mitigated by running in thread pool) +- Slower for scans with many tests + +**Why we made this choice:** +Accuracy over speed. Time-based blind SQL injection detection (`sqli_scanner.py:183-251`) requires: +1. Establish baseline response time (multiple samples) +2. Send delay payload +3. Measure if response is slower by expected amount + +Async concurrency would introduce timing noise. A delay of 5.1s vs 5.3s could be network jitter, not SQLi. Sequential requests with controlled spacing give cleaner signals. + +### Decision 2: Repository pattern over Active Record + +**What we chose:** +Repository pattern - `UserRepository.get_by_email(db, email)` instead of `User.find_by_email(email)`. + +**Alternatives considered:** +- Active Record (Django-style) - Models have class methods for queries +- Data Mapper (raw SQL) - Write SQL strings directly + +**Trade-offs:** +Pros of repository: +- Clear separation: models define structure, repositories define queries +- Testable: mock repositories in unit tests +- Flexible: swap ORM without changing service code + +Cons of repository: +- More files, more navigation +- Extra abstraction layer +- Can feel like overkill for simple CRUD + +**Why we made this choice:** +Testability and maintainability. Services like `AuthService.login_user()` call `UserRepository.get_by_email()`. In tests, mock the repository to return a fake user without touching the database. + +If we later migrate from SQLAlchemy to another ORM, we only change repository implementations. Services remain unchanged. + +### Decision 3: JWT without refresh tokens + +**What we chose:** +Single long-lived JWT (24 hours), no refresh token mechanism. + +**Alternatives considered:** +- Short access tokens (15 min) + refresh tokens (30 days) +- Session-based auth with server-side storage + +**Trade-offs:** +Pros of current approach: +- Simpler implementation, no refresh endpoint +- Stateless - no session storage needed +- Works across multiple backend instances immediately + +Cons of current approach: +- Can't invalidate tokens before expiration +- If token stolen, attacker has 24 hours of access +- No way to force logout on all devices + +**Why we made this choice:** +Simplicity for educational project. Adding refresh tokens requires: +- Refresh token storage (database or Redis) +- Refresh endpoint with rotation logic +- Token revocation tracking +- More complex frontend token management + +For a production app, you'd implement refresh tokens. For learning how JWT works, this is clearer. + +### Decision 4: Docker Compose for local development + +**What we chose:** +Run everything in Docker containers with `dev.compose.yml`, even for local development. + +**Alternatives considered:** +- Local Postgres + local Python + local Node (no Docker) +- Docker for services, local for development +- Kubernetes locally (minikube, kind) + +**Trade-offs:** +Pros of Docker Compose: +- Identical environment for all developers +- Spin up entire stack with one command +- Hot reload still works with volume mounts +- Production architecture matches dev (Docker in both) + +Cons of Docker Compose: +- Slower file system on Mac (volume mounts) +- Extra resource usage (containers overhead) +- Learning curve for Docker debugging + +**Why we made this choice:** +"Works on my machine" is eliminated. Every developer gets Postgres 16, Python 3.11, Node 20 regardless of their host OS. New team member runs `docker compose up` and they're ready. + +Volume mounts preserve hot reload: +```yaml +# dev.compose.yml:33-35 +volumes: + - ./backend:/app # Maps local backend/ to container /app + # Changes to backend/*.py trigger uvicorn reload +``` + +## Deployment Architecture + +Production deployment uses optimized containers: + +``` + ┌──────────────────┐ +Internet ────────> │ Nginx (Port 80) │ + └────────┬─────────┘ + │ + ┌────────▼─────────┐ + │ Backend (8000) │ + │ Gunicorn │ + │ 4 workers │ + └────────┬─────────┘ + │ + ┌────────▼─────────┐ + │ PostgreSQL │ + │ (internal only) │ + └──────────────────┘ +``` + +**Components:** + +**Nginx container** - Built from `conf/docker/prod/vite.docker`: +- Multi-stage build: compile React, then serve with nginx +- Serves static files from `/usr/share/nginx/html` +- Proxies `/api` to backend +- Gzip compression, caching headers + +**Backend container** - Built from `conf/docker/prod/fastapi.docker`: +- Runs gunicorn with 4 uvicorn workers +- Non-root user for security +- No volume mounts (code baked into image) + +**Database container** - Postgres 16 Alpine: +- Not exposed to host in production +- Data persists in Docker volume + +**Infrastructure:** +Minimal production setup: +- Single VPS (DigitalOcean Droplet, AWS EC2) +- Docker Compose orchestration +- SSL via Let's Encrypt (certbot) or Cloudflare proxy + +Scaling beyond single server: +- Backend: Multiple instances behind load balancer +- Database: Read replicas, connection pooling +- Static files: CDN (CloudFront, Cloudflare) + +## Error Handling Strategy + +### Error Types + +1. **Validation errors** (400) - Pydantic catches bad input: +```python +# schemas/user_schemas.py:26-38 +@field_validator("password") +@classmethod +def validate_password_strength(cls, v: str) -> str: + if not re.search(r"[A-Z]", v): + raise ValueError("Password must contain uppercase letter") + # Pydantic converts to HTTP 422 automatically +``` + +2. **Authentication errors** (401) - JWT invalid or expired: +```python +# core/dependencies.py:33-36 +if email is None: + raise HTTPException( + status_code=401, + detail="Invalid authentication credentials" + ) +``` + +3. **Authorization errors** (403) - Valid user, wrong resource: +```python +# services/scan_service.py:77-80 +if scan.user_id != user_id: + raise HTTPException( + status_code=403, + detail="Not authorized to access this scan" + ) +``` + +4. **Not found errors** (404) - Resource doesn't exist: +```python +# services/scan_service.py:73-74 +if not scan: + raise HTTPException(status_code=404, detail="Scan not found") +``` + +5. **Scanner errors** - Caught and returned as `status="error"`: +```python +# services/scan_service.py:52-65 +try: + scanner = scanner_class(...) + result = scanner.scan() + results.append(result) +except Exception as e: + results.append( + TestResultCreate( + test_name=test_type, + status="error", + severity="info", + details=f"Scanner error: {str(e)}", + ... + ) + ) +``` + +### Recovery Mechanisms + +**Database connection loss:** +- Detection: `pool_pre_ping=True` tests connections before use +- Response: SQLAlchemy automatically reconnects +- Recovery: Failed transaction rolls back, next request gets new connection + +**Scanner timeout:** +- Detection: `requests.Timeout` exception after `SCANNER_CONNECTION_TIMEOUT` seconds +- Response: Retry with exponential backoff (up to 3 times) +- Recovery: If all retries fail, return error result (scan continues with other tests) + +**Rate limit exceeded (429 from target):** +- Detection: HTTP 429 status code in scanner response +- Response: Read `Retry-After` header, wait specified duration +- Recovery: Retry request after waiting + +Code from `base_scanner.py:92-156`: +```python +def make_request(self, method: str, endpoint: str, **kwargs) -> requests.Response: + retry_count = 0 + backoff_factor = 2.0 + + while retry_count <= settings.DEFAULT_RETRY_COUNT: + try: + response = self.session.request(method, url, **kwargs) + + if response.status_code == 429: + retry_after = response.headers.get("Retry-After", "60") + wait_time = int(retry_after) if retry_after.isdigit() else 60 + time.sleep(wait_time) + retry_count += 1 + continue # Try again + + if response.status_code >= 500: # Server error + wait_time = backoff_factor ** retry_count + time.sleep(wait_time) + retry_count += 1 + continue + + return response # Success + + except (requests.Timeout, requests.ConnectionError): + if retry_count < settings.DEFAULT_RETRY_COUNT: + wait_time = backoff_factor ** retry_count + time.sleep(wait_time) + retry_count += 1 + else: + raise # Give up after retries exhausted +``` + +## Extensibility + +### Where to Add Features + +Want to add a new security test (e.g., XSS detection)? Here's the process: + +**1. Create scanner** in `backend/scanners/xss_scanner.py`: +```python +from .base_scanner import BaseScanner +from core.enums import TestType, ScanStatus, Severity + +class XSSScanner(BaseScanner): + def scan(self) -> TestResultCreate: + # Test for reflected XSS + test_result = self._test_reflected_xss() + + if test_result["vulnerable"]: + return TestResultCreate( + test_name=TestType.XSS, # Need to add to enum + status=ScanStatus.VULNERABLE, + severity=Severity.HIGH, + details="Reflected XSS detected", + evidence_json=test_result, + recommendations_json=[...] + ) + + return TestResultCreate(test_name=TestType.XSS, status=ScanStatus.SAFE, ...) +``` + +**2. Add to enum** in `backend/core/enums.py:19-25`: +```python +class TestType(str, Enum): + RATE_LIMIT = "rate_limit" + AUTH = "auth" + SQLI = "sqli" + IDOR = "idor" + XSS = "xss" # New test type +``` + +**3. Register in service** at `backend/services/scan_service.py:32-37`: +```python +scanner_mapping: dict[TestType, type[BaseScanner]] = { + TestType.RATE_LIMIT: RateLimitScanner, + TestType.AUTH: AuthScanner, + TestType.SQLI: SQLiScanner, + TestType.IDOR: IDORScanner, + TestType.XSS: XSSScanner, # Register new scanner +} +``` + +**4. Update frontend constants** in `frontend/src/config/constants.ts:44-51`: +```typescript +export const SCAN_TEST_TYPES = { + RATE_LIMIT: 'rate_limit', + AUTH: 'auth', + SQLI: 'sqli', + IDOR: 'idor', + XSS: 'xss', // Add new test type +} as const; + +export const TEST_TYPE_LABELS: Record = { + // ... existing + [SCAN_TEST_TYPES.XSS]: 'Cross-Site Scripting', +}; +``` + +No changes needed to: +- Database schema (test_name is enum, migrations auto-update) +- Routes (they just pass through test types) +- Repositories (they store whatever test types are sent) + +## Limitations + +Current architectural limitations: + +1. **No scan queueing** - Scans run synchronously in request handler. If 10 users submit scans simultaneously, they all block on scanner HTTP requests. Fix requires async task queue (Celery with Redis, or RQ). + +2. **No real-time scan progress** - Frontend submits scan and waits for complete response. Long scans (2 minutes) show no progress. Fix requires WebSocket connection or polling for progress updates. + +3. **Single target per scan** - Can't scan multiple URLs in one operation. Fix requires loop in service layer and UI for multiple target inputs. + +4. **No historical comparison** - Can't compare scan results over time ("Was this vulnerable last week?"). Fix requires additional queries and UI for trend visualization. + +5. **Limited concurrency in scanners** - Tests run sequentially within a scan. Could run all 4 tests simultaneously, but chose not to for timing accuracy. Trade-off between speed and precision. + +These are not bugs - they're conscious trade-offs. Fixing them would require significant architectural changes. + +## Comparison to Similar Systems + +### Burp Suite + +How we're different: +- Burp is a proxy, we're a standalone scanner +- Burp has GUI desktop app, we're web-based +- Burp is comprehensive (hundreds of tests), we focus on 4 core vulnerabilities + +Why we made different choices: +Educational focus. Burp is for professional pentesters. This project teaches how scanners work by implementing the core logic yourself. + +### OWASP ZAP + +How we're different: +- ZAP is passive + active scanning, we're active only +- ZAP auto-discovers endpoints, we test provided URLs +- ZAP integrates with CI/CD, we're standalone + +Why we made different choices: +Simplicity. ZAP is powerful but complex. This project shows the fundamentals without overwhelming features. + +## Evolution + +### Version 1.0 Design (Current) + +Initial design focused on: +- Four core vulnerability types +- Synchronous scanners for accuracy +- Repository pattern for clean separation +- Docker-first development + +### Future Improvements + +Planned architectural changes: + +1. **Async task queue** - Move scanning to background workers + - Why: Non-blocking API, better scalability + - What it enables: Real-time progress, scheduled scans + +2. **Plugin system** - Load scanners dynamically + - Why: Extensibility without modifying core + - What it enables: Community contributions, custom tests + +3. **Report generation** - PDF/HTML export of results + - Why: Sharing findings with teams + - What it enables: Professional documentation + +4. **Webhook notifications** - Alert when scans complete + - Why: Integration with other tools + - What it enables: Slack/email notifications, CI/CD integration + +## Key Files Reference + +Quick map of where to find things: + +- `backend/factory.py` - Application factory, middleware setup, route registration +- `backend/config.py` - All environment variables and configuration +- `backend/core/database.py` - Database engine and session management +- `backend/core/security.py` - JWT creation, password hashing, token validation +- `backend/core/dependencies.py` - FastAPI dependencies (auth, database) +- `backend/models/` - SQLAlchemy models (User, Scan, TestResult) +- `backend/repositories/` - Database query functions +- `backend/services/` - Business logic orchestration +- `backend/routes/` - API endpoints +- `backend/scanners/base_scanner.py` - Common scanner functionality +- `backend/scanners/*_scanner.py` - Individual vulnerability tests +- `frontend/src/hooks/` - React Query hooks for API calls +- `frontend/src/services/` - API client functions +- `frontend/src/store/` - Zustand state management +- `conf/nginx/` - Nginx reverse proxy configuration +- `compose.yml` - Production Docker Compose +- `dev.compose.yml` - Development with volume mounts + +## Next Steps + +Now that you understand the architecture: + +1. Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) for code walkthrough - see how each scanner detects vulnerabilities, how authentication flows work, and how data moves through the layers +2. Try modifying scanners - change SQLi payloads, adjust timing thresholds, add new detection logic to understand the implementation details diff --git a/PROJECTS/intermediate/api-security-scanner/learn/03-IMPLEMENTATION.md b/PROJECTS/intermediate/api-security-scanner/learn/03-IMPLEMENTATION.md index e69de29b..0cdf8f38 100644 --- a/PROJECTS/intermediate/api-security-scanner/learn/03-IMPLEMENTATION.md +++ b/PROJECTS/intermediate/api-security-scanner/learn/03-IMPLEMENTATION.md @@ -0,0 +1,1720 @@ +# Implementation Guide + +This document walks through the actual code. We'll build key features step by step and explain the decisions along the way. + +## File Structure Walkthrough +``` +backend/ +├── main.py # Application entry point +├── factory.py # FastAPI app factory with middleware +├── config.py # Environment variables and constants +├── core/ +│ ├── database.py # SQLAlchemy engine and session factory +│ ├── security.py # Password hashing, JWT creation/validation +│ ├── dependencies.py # FastAPI dependencies (auth, database) +│ └── enums.py # Type-safe enums for status, severity, test types +├── models/ +│ ├── Base.py # Base model with common fields (id, timestamps) +│ ├── User.py # User authentication model +│ ├── Scan.py # Scan metadata model +│ └── TestResult.py # Individual test results model +├── repositories/ +│ ├── user_repository.py # User database operations +│ ├── scan_repository.py # Scan database operations +│ └── test_result_repository.py # Test result database operations +├── routes/ +│ ├── auth.py # Registration and login endpoints +│ └── scans.py # Scan CRUD endpoints +├── schemas/ +│ ├── user_schemas.py # Pydantic models for user data +│ ├── scan_schemas.py # Pydantic models for scan data +│ └── test_result_schemas.py # Pydantic models for test results +├── services/ +│ ├── auth_service.py # Authentication business logic +│ └── scan_service.py # Scan orchestration business logic +└── scanners/ + ├── base_scanner.py # Common HTTP logic, retry handling + ├── rate_limit_scanner.py # Rate limiting detection + ├── auth_scanner.py # Authentication vulnerability detection + ├── sqli_scanner.py # SQL injection detection + ├── idor_scanner.py # IDOR/BOLA detection + └── payloads.py # Attack payloads organized by type +``` + +## Building Authentication + +### Step 1: Password Hashing with Bcrypt + +What we're building: Secure password storage that protects user credentials even if the database is compromised. + +The code lives in `backend/core/security.py:11-28`: +```python +import bcrypt +from jose import JWTError, jwt + +def hash_password(password: str) -> str: + """ + Hash a plain text password using bcrypt + """ + password_bytes = password.encode("utf-8") # Convert string to bytes + salt = bcrypt.gensalt() # Generate random salt (default 12 rounds) + hashed = bcrypt.hashpw(password_bytes, salt) + return hashed.decode("utf-8") # Convert bytes back to string for storage +``` + +**Why this code works:** +- **Line 18-19**: Bcrypt requires byte input, not strings. The encoding/decoding dance converts between Python strings and byte arrays. +- **Line 19**: `bcrypt.gensalt()` generates a unique salt for each password. The salt is embedded in the output hash, so you don't store it separately. +- **Line 20**: `bcrypt.hashpw()` does the actual hashing. With default 12 rounds, this intentionally takes ~100ms. That's the point - makes brute forcing expensive. +- **Line 21**: Convert back to string because SQLAlchemy's `String` column type expects strings, not bytes. + +The verification function (`security.py:21-31`): +```python +def verify_password(plain_password: str, hashed_password: str) -> bool: + """ + Verify a plain text password against a hashed password + """ + password_bytes = plain_password.encode("utf-8") + hashed_bytes = hashed_password.encode("utf-8") + return bcrypt.checkpw(password_bytes, hashed_bytes) +``` + +**What's happening:** +1. Convert both inputs to bytes +2. `bcrypt.checkpw()` extracts the salt from `hashed_bytes` (it's embedded in the hash) +3. Hashes `password_bytes` with that same salt +4. Compares the result - if they match, password is correct + +**Common mistakes here:** +```python +# Wrong - storing plaintext +user.password = password # Database breach = everyone's password leaked + +# Wrong - using weak hashing +import hashlib +user.password = hashlib.md5(password.encode()).hexdigest() # Fast = easy to brute force + +# Wrong - forgetting to salt +user.password = hashlib.sha256(password.encode()).hexdigest() # Rainbow tables break this + +# Right - bcrypt with automatic salting +user.hashed_password = hash_password(password) # Secure +``` + +### Step 2: JWT Token Creation + +Now we need to create authentication tokens after successful login. + +In `core/security.py:34-56`: +```python +from datetime import datetime, timedelta +from config import settings + +def create_access_token( + data: dict[str, str], + expires_delta: timedelta | None = None +) -> str: + """ + Create a JWT access token + """ + to_encode = data.copy() # Don't modify the original dict + + if expires_delta: + expire = datetime.utcnow() + expires_delta + else: + expire = datetime.utcnow() + timedelta( + minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES # 1440 = 24 hours + ) + + to_encode.update({"exp": expire}) # Add expiration claim + + encoded_jwt = jwt.encode( + to_encode, + settings.SECRET_KEY, # MUST be random, 256+ bits + algorithm=settings.ALGORITHM # "HS256" + ) + return encoded_jwt +``` + +**Key parts explained:** + +**Line 43** - Copy the data dict because we're about to modify it. If we mutated the original, the caller would see `{"sub": "user@example.com", "exp": 1234567890}` instead of just the email. + +**Line 45-50** - Calculate when the token expires. If no `expires_delta` is passed, default to 24 hours from `config.py`. The `exp` claim is part of the JWT standard - libraries automatically check it. + +**Line 52** - This is where the magic happens. The `jwt.encode()` function: +1. Converts `to_encode` dict to JSON +2. Base64 encodes it as the payload +3. Creates a header: `{"alg": "HS256", "typ": "JWT"}` +4. Computes HMAC-SHA256 signature using `SECRET_KEY` +5. Joins header.payload.signature with periods + +The result looks like: +``` +eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyQGV4YW1wbGUuY29tIiwiZXhwIjoxNzM4MzY4MDAwfQ.signature_here +``` + +**Why we do it this way:** +JWTs are stateless. The server doesn't need to store session data. When a request comes in with a JWT, the server verifies the signature and trusts the payload. This means you can scale horizontally - any backend instance can validate any token without coordinating with other instances or a shared session store. + +**Alternative approaches:** +- **Session-based auth**: Store session ID in cookie, look up user data on each request. Simpler but requires session storage (Redis, database). Doesn't scale as easily. +- **OAuth 2.0 with refresh tokens**: Short-lived access tokens (15 min) + long-lived refresh tokens. More secure (can revoke refresh tokens) but more complex to implement. + +### Step 3: Token Validation + +When a protected endpoint receives a request, we need to validate the JWT and load the user. + +The dependency injection function in `core/dependencies.py:17-49`: +```python +from fastapi import Depends, HTTPException, status +from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer +from sqlalchemy.orm import Session +from .security import decode_token +from .database import get_db +from repositories.user_repository import UserRepository +from schemas.user_schemas import UserResponse + +security = HTTPBearer() # Extracts "Authorization: Bearer " header + +async def get_current_user( + credentials: HTTPAuthorizationCredentials = Depends(security), + db: Session = Depends(get_db), +) -> UserResponse: + """ + FastAPI dependency to extract and verify the current authenticated user + """ + try: + # Decode and verify the JWT + payload = decode_token(credentials.credentials) + email: str | None = payload.get("sub") + + if email is None: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid authentication credentials", + headers={"WWW-Authenticate": "Bearer"}, + ) + + # Load user from database + user = UserRepository.get_by_email(db, email) + + if not user: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="User not found", + headers={"WWW-Authenticate": "Bearer"}, + ) + + return UserResponse.model_validate(user) + + except ValueError: + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid authentication credentials", + headers={"WWW-Authenticate": "Bearer"}, + ) from None +``` + +**What's happening:** + +**Line 27** - `HTTPBearer()` is FastAPI's built-in extractor for `Authorization: Bearer ` headers. It parses the header and gives us the token string in `credentials.credentials`. + +**Line 35** - Call `decode_token()` which lives in `security.py:48-62`: +```python +def decode_token(token: str) -> dict[str, str]: + """ + Decode and verify a JWT token + """ + try: + payload = jwt.decode( + token, + settings.SECRET_KEY, + algorithms=[settings.ALGORITHM] # Only accept HS256 + ) + return payload + except JWTError as e: + raise ValueError(f"Invalid token: {str(e)}") from e +``` + +This verifies: +- Signature is valid (token wasn't tampered with) +- Algorithm matches expected (prevents "none" algorithm attack) +- Token hasn't expired (checks `exp` claim automatically) + +**Line 36** - Extract the `sub` (subject) claim. This is the user identifier we put in the token during login. + +**Line 38-43** - If `sub` is missing, the token is malformed. Return 401 with `WWW-Authenticate` header per HTTP standards. + +**Line 46** - Load the full user record from the database. We could skip this and just use the email from the token, but loading from DB ensures: +- User still exists (wasn't deleted after token was issued) +- User is still active (wasn't deactivated) +- We get the full user object with all fields + +**Line 48-52** - If user doesn't exist in database, token is invalid. This catches deleted users with valid tokens. + +**Line 54** - Convert SQLAlchemy model to Pydantic schema. This excludes `hashed_password` from the response (Pydantic schema doesn't include it). + +**Line 56-61** - Catch `ValueError` from `decode_token()` and convert to HTTP 401. The `from None` suppresses the chained exception traceback - cleaner error messages for clients. + +### Testing Authentication + +How to verify this works: +```bash +# Register a user +curl -X POST http://localhost:8000/auth/register \ + -H "Content-Type: application/json" \ + -d '{ + "email": "test@example.com", + "password": "SecurePass123" + }' + +# Response: {"id": 1, "email": "test@example.com", ...} + +# Login +curl -X POST http://localhost:8000/auth/login \ + -H "Content-Type: application/json" \ + -d '{ + "email": "test@example.com", + "password": "SecurePass123" + }' + +# Response: {"access_token": "eyJ...", "token_type": "bearer"} + +# Use token to access protected endpoint +curl http://localhost:8000/scans/ \ + -H "Authorization: Bearer eyJ..." + +# Response: [list of scans] +``` + +Expected output: First request creates user, second returns JWT, third returns scans array (empty initially). + +If you see `401 Unauthorized`, check: +- Token is being sent in header (not query param, not body) +- Format is exactly `Bearer ` with capital B +- Token hasn't expired (24 hours from login) +- User still exists in database + +## Building SQL Injection Detection + +### The Problem + +We need to detect three types of SQL injection: +1. **Error-based** - Database errors leak information +2. **Boolean-based blind** - Responses differ for true/false conditions +3. **Time-based blind** - Database delays reveal injection + +### The Solution + +Use payload libraries and response analysis to detect each type. The scanner lives in `backend/scanners/sqli_scanner.py`. + +### Implementation + +Starting with error-based detection (`sqli_scanner.py:51-94`): +```python +def _test_error_based_sqli(self) -> dict[str, Any]: + """ + Test for error based SQL injection + + Detects database errors in responses indicating SQLi vulnerability + """ + error_signatures = SQLiPayloads.get_error_signatures() + + basic_payloads = SQLiPayloads.BASIC_AUTHENTICATION_BYPASS + + for payload in basic_payloads: + try: + response = self.make_request("GET", f"/?id={payload}") + + response_text_lower = response.text.lower() + + # Look for database error signatures + for db_type, signatures in error_signatures.items(): + for signature in signatures: + if signature in response_text_lower: + return { + "vulnerable": True, + "database_type": db_type, + "payload": payload, + "status_code": response.status_code, + "error_signature": signature, + "response_excerpt": response.text[:500], + } + + except Exception: + continue # Network errors don't indicate SQLi + + return { + "vulnerable": False, + "payloads_tested": len(basic_payloads), + "description": "No database errors detected", + } +``` + +**Key parts explained:** + +**Line 57** - Load error signatures from `scanners/payloads.py:9-30`. These are database-specific error messages: +```python +ERROR_SIGNATURES = { + "mysql": [ + "sql syntax", + "mysql_fetch", + "mysql error", + ], + "postgres": [ + "postgresql", + "pg_query", + "syntax error", + ], + "mssql": [ + "sqlserver jdbc driver", + "sqlexception", + ], +} +``` + +**Line 59** - Basic payloads like `' OR '1'='1`, `' OR 1=1--`, `admin'--` from `payloads.py:33-48`. + +**Line 63** - Make request with payload in query parameter. If the backend does: +```python +query = f"SELECT * FROM users WHERE id = {request.args['id']}" +``` + +And we send `id=' OR 1=1--`, it becomes: +```sql +SELECT * FROM users WHERE id = ' OR 1=1--' +``` + +That's invalid SQL. Database returns error: `"You have an error in your SQL syntax"`. + +**Line 65** - Convert response to lowercase for case-insensitive matching. Error messages vary: "SQL syntax", "Sql Syntax", "sql syntax". + +**Line 68-72** - Check each signature against response. If `"sql syntax"` appears in the HTML, we found SQLi. + +**Line 73-79** - Return evidence: which payload worked, what database type was detected, the actual error message (first 500 chars). + +**Line 81-82** - Network errors (timeout, connection refused) don't mean SQLi. Skip to next payload. + +**Line 84-88** - No errors found across all payloads = not vulnerable (to error-based SQLi at least). + +### Boolean-Based Blind Detection + +When errors are suppressed, check if responses differ for true vs false conditions. + +Code at `sqli_scanner.py:96-164`: +```python +def _test_boolean_based_sqli(self) -> dict[str, Any]: + """ + Test for boolean based blind SQL injection + + Compares responses from true vs false conditions to detect SQLi + """ + try: + # Establish baseline + baseline_response = self.make_request("GET", "/?id=1") + baseline_length = len(baseline_response.text) + baseline_status = baseline_response.status_code + + if baseline_status != 200: + return { + "vulnerable": False, + "description": "Baseline request failed", + "baseline_status": baseline_status, + } + + boolean_payloads = SQLiPayloads.BOOLEAN_BASED_BLIND + + # Test true conditions: ' AND 1=1-- + true_payloads = [ + p for p in boolean_payloads + if "AND '1'='1" in p or "AND 1=1" in p + ] + false_payloads = [ + p for p in boolean_payloads + if "AND '1'='2" in p or "AND 1=2" in p or "AND 1=0" in p + ] + + true_lengths = [] + for payload in true_payloads: + response = self.make_request("GET", f"/?id={payload}") + true_lengths.append(len(response.text)) + + false_lengths = [] + for payload in false_payloads: + response = self.make_request("GET", f"/?id={payload}") + false_lengths.append(len(response.text)) + + # Calculate averages + avg_true = statistics.mean(true_lengths) + avg_false = statistics.mean(false_lengths) + + length_diff = abs(avg_true - avg_false) + + # Significant difference indicates SQLi + if length_diff > 100 and avg_true != avg_false: + return { + "vulnerable": True, + "baseline_length": baseline_length, + "true_condition_avg_length": avg_true, + "false_condition_avg_length": avg_false, + "length_difference": length_diff, + "confidence": "HIGH" if length_diff > 500 else "MEDIUM", + } + + return { + "vulnerable": False, + "description": "No boolean-based SQLi detected", + "length_difference": length_diff, + } +``` + +**What's happening:** + +**Line 105-106** - Get baseline response with normal input (`id=1`). We need this to compare against. + +**Line 121-128** - Split payloads into "always true" and "always false" conditions. + +True condition example: `1' AND 1=1--` +```sql +SELECT * FROM users WHERE id = 1' AND 1=1--' +``` +The `1=1` is always true, so if vulnerable, this returns data. + +False condition example: `1' AND 1=2--` +```sql +SELECT * FROM users WHERE id = 1' AND 1=2--' +``` +The `1=2` is never true, so if vulnerable, this returns empty result. + +**Line 130-141** - Send multiple true and false payloads, record response lengths. + +**Line 144-147** - Calculate average length for true vs false. Statistical approach reduces false positives from network variance. + +**Line 151** - If difference is significant (>100 bytes) and not zero, likely vulnerable. A difference of 500+ bytes is high confidence - probably seeing full records vs empty results. + +**Why this works:** +If SQLi exists, true conditions return data (longer response), false conditions return nothing (shorter response). If no SQLi, both are treated as invalid input and return the same error page. + +### Time-Based Blind Detection + +Most sophisticated technique. Use database sleep functions to measure injection. + +Code at `sqli_scanner.py:183-251`: +```python +def _test_time_based_sqli(self, delay_seconds: int = 5) -> dict[str, Any]: + """ + Test for time based blind SQL injection + + Uses baseline timing comparison with statistical analysis + for false positive reduction + """ + try: + # Establish baseline timing + baseline_mean, baseline_stdev = self.get_baseline_timing("/") + + threshold = baseline_mean + (3 * baseline_stdev) + expected_delay_time = baseline_mean + delay_seconds + + all_time_payloads = SQLiPayloads.TIME_BASED_BLIND + + # Group by database type + delay_payloads = { + "mysql": [p for p in all_time_payloads if "SLEEP" in p], + "postgres": [p for p in all_time_payloads if "pg_sleep" in p], + "mssql": [p for p in all_time_payloads if "WAITFOR" in p], + } + + for db_type, payloads in delay_payloads.items(): + for payload in payloads: + delay_times = [] + + # Take multiple samples + for _ in range(3): + try: + response = self.make_request( + "GET", + f"/?id={payload}", + timeout=delay_seconds + 10, + ) + elapsed = getattr(response, "request_time", 0.0) + delay_times.append(elapsed) + + except Exception: + delay_times.append(delay_seconds + 10) # Assume timeout = worked + + time.sleep(1) # Space out requests + + avg_delay = statistics.mean(delay_times) + + # Check if delay matches expected + if avg_delay >= expected_delay_time - 1: + confidence = "HIGH" if avg_delay >= expected_delay_time else "MEDIUM" + + return { + "vulnerable": True, + "database_type": db_type, + "payload": payload, + "baseline_time": f"{baseline_mean:.3f}s", + "response_time": f"{avg_delay:.3f}s", + "expected_delay": f"{expected_delay_time:.3f}s", + "confidence": confidence, + "individual_times": [f"{t:.3f}s" for t in delay_times], + } +``` + +**Important details:** + +**Line 192** - Get baseline timing from `base_scanner.py:158-177`: +```python +def get_baseline_timing( + self, + endpoint: str, + samples: int | None = None +) -> tuple[float, float]: + """ + Establish baseline response time for an endpoint + + Takes multiple samples and calculates mean and standard deviation + """ + if samples is None: + samples = settings.DEFAULT_BASELINE_SAMPLES # 10 + + times = [] + + for _ in range(samples): + response = self.make_request("GET", endpoint) + times.append(getattr(response, "request_time", 0.0)) + time.sleep(0.5) # Space out samples + + return statistics.mean(times), statistics.stdev(times) +``` + +This makes 10 normal requests and calculates average and standard deviation. Example results: +- Mean: 0.15s +- Stdev: 0.02s + +**Line 194** - Calculate threshold: `mean + 3*stdev`. With example above: `0.15 + 3*0.02 = 0.21s`. Any response over 0.21s is "unusually slow" (99.7% confidence if normally distributed). + +**Line 195** - Expected delay: `baseline + 5s`. If baseline is 0.15s and we inject `SLEEP(5)`, we expect ~5.15s response. + +**Line 200-204** - Group payloads by database. MySQL uses `SLEEP(5)`, Postgres uses `pg_sleep(5)`, MSSQL uses `WAITFOR DELAY '0:0:5'`. + +**Line 211-225** - Take 3 samples of each payload. Network jitter can cause ±0.5s variance. Averaging 3 samples gives cleaner signal. + +**Line 230** - If average delay is within 1 second of expected (5.15s ± 1s), that's SQLi. The 1 second tolerance accounts for network overhead. + +**Why this works:** +Payloads like `1'; SELECT SLEEP(5)--` execute the sleep if SQLi exists. The response takes 5 extra seconds. Without SQLi, the payload is just treated as invalid input and returns immediately. + +## Security Implementation + +### JWT Signature Validation + +Critical to prevent token forgery. The none algorithm attack test in `auth_scanner.py:168-213`: +```python +def _test_none_algorithm(self) -> dict[str, Any]: + """ + Test if server accepts JWT with 'none' algorithm + + Critical vulnerability: allows unsigned tokens to be accepted + """ + try: + header, payload, signature = self.auth_token.split(".") + + none_variants = AuthPayloads.get_jwt_none_variants() + # ["none", "None", "NONE", "nOnE", "NoNe", "NOne"] + + for variant in none_variants: + # Create malicious header + malicious_header = self._base64url_encode( + json.dumps({"alg": variant, "typ": "JWT"}) + ) + + # Remove signature (trailing period means "no signature") + malicious_token = f"{malicious_header}.{payload}." + + response = self.make_request( + "GET", + "/", + headers={"Authorization": f"Bearer {malicious_token}"} + ) + + if response.status_code == 200: + return { + "vulnerable": True, + "vulnerability_type": "JWT None Algorithm", + "algorithm_variant": variant, + "status_code": response.status_code, + "recommendations": [ + "Reject tokens with 'none' algorithm (all case variations)", + "Explicitly verify signature before accepting tokens", + "Use allowlist of accepted algorithms", + ], + } + + return { + "vulnerable": False, + "description": "None algorithm properly rejected", + } +``` + +**What this prevents:** + +Normal JWT: `header.payload.signature` +``` +eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZG1pbiJ9.signature +``` + +Malicious JWT: `header.payload.` (no signature) +``` +eyJhbGciOiJub25lIiwidHlwIjoiSldUIn0.eyJzdWIiOiJhZG1pbiJ9. +``` + +Decoded malicious header: +```json +{"alg": "none", "typ": "JWT"} +``` + +Decoded malicious payload: +```json +{"sub": "admin"} +``` + +If the server accepts this, attacker can forge tokens with any user identity. + +**How to defend:** + +The project's JWT validation (`core/security.py:54-57`) explicitly specifies algorithms: +```python +payload = jwt.decode( + token, + settings.SECRET_KEY, + algorithms=[settings.ALGORITHM] # ["HS256"] - none not in this list +) +``` + +The `algorithms` parameter is an allowlist. The library will reject tokens with `alg: none` because it's not in `["HS256"]`. + +**What NOT to do:** +```python +# BAD - accepts any algorithm +payload = jwt.decode(token, settings.SECRET_KEY) + +# BAD - doesn't verify signature +payload = jwt.decode(token, options={"verify_signature": False}) + +# BAD - allows none +payload = jwt.decode(token, settings.SECRET_KEY, algorithms=["HS256", "none"]) +``` + +### IDOR Prevention Pattern + +Authorization checks in `services/scan_service.py:67-84`: +```python +@staticmethod +def get_scan_by_id(db: Session, scan_id: int, user_id: int) -> ScanResponse: + """ + Get scan by ID with authorization check + """ + scan = ScanRepository.get_by_id(db, scan_id) + + if not scan: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail="Scan not found", + ) + + # CRITICAL: Check if user owns this scan + if scan.user_id != user_id: + raise HTTPException( + status_code=status.HTTP_403_FORBIDDEN, + detail="Not authorized to access this scan", + ) + + return ScanResponse.model_validate(scan) +``` + +**Why 404 before 403:** +Line 75-78 returns 404 if scan doesn't exist, BEFORE checking ownership. This prevents information disclosure. + +If we checked ownership first: +- Request scan 999 (doesn't exist) → 404 "Scan not found" +- Request scan 123 (exists, belongs to someone else) → 403 "Not authorized" + +Now attacker knows scan 123 exists. They can enumerate all scan IDs and find which ones are valid. + +Better approach: +- Request scan 999 → 404 +- Request scan 123 → 404 + +Can't tell difference between "doesn't exist" and "exists but you can't access it". Leak less information. + +**The authorization check:** +Line 81-84 compares `scan.user_id` (owner) with `user_id` (requester). If they don't match, return 403. + +This happens at the service layer, NOT the route layer. Every code path that retrieves scans goes through the service, so we can't forget the check. + +## Data Flow Example + +Let's trace a complete request through the system: User creates a scan that tests for SQLi. + +### Request Comes In + +Entry point: `routes/scans.py:23-37` +```python +@router.post("/", response_model=ScanResponse, status_code=status.HTTP_201_CREATED) +@limiter.limit(settings.API_RATE_LIMIT_SCAN) # "15/minute" +async def create_scan( + request: Request, + scan_request: ScanRequest, # Pydantic validates this + db: Session = Depends(get_db), # Injects database session + current_user: UserResponse = Depends(get_current_user), # Validates JWT +) -> ScanResponse: + """ + Create and execute a new security scan + """ + return ScanService.run_scan(db, current_user.id, scan_request) +``` + +At this point: +- `scan_request` is validated by Pydantic (correct URL format, valid test types) +- `current_user` is authenticated (JWT was valid, user exists in database) +- `db` is a fresh SQLAlchemy session +- Rate limiting checked - if user exceeded 15 scans/minute, request was rejected before this function runs + +What happens next: Call service layer to orchestrate the scan. + +### Processing Layer + +Service orchestrates scanners: `services/scan_service.py:23-65` +```python +@staticmethod +def run_scan(db: Session, user_id: int, scan_request: ScanRequest) -> ScanResponse: + # Create scan record + scan = ScanRepository.create_scan( + db=db, + user_id=user_id, + target_url=str(scan_request.target_url), + ) + + # Map test types to scanner classes + scanner_mapping: dict[TestType, type[BaseScanner]] = { + TestType.RATE_LIMIT: RateLimitScanner, + TestType.AUTH: AuthScanner, + TestType.SQLI: SQLiScanner, + TestType.IDOR: IDORScanner, + } + + results: list[TestResultCreate] = [] + + # Execute each requested test + for test_type in scan_request.tests_to_run: + scanner_class: type[BaseScanner] | None = scanner_mapping.get(test_type) + + if not scanner_class: + continue # Skip unknown test types + + try: + scanner = scanner_class( + target_url=str(scan_request.target_url), + auth_token=scan_request.auth_token, + max_requests=scan_request.max_requests, + ) + + result = scanner.scan() # Execute the actual test + results.append(result) + + except Exception as e: + # Scanner crashed - return error result instead of failing entire scan + results.append( + TestResultCreate( + test_name=test_type, + status="error", + severity="info", + details=f"Scanner error: {str(e)}", + evidence_json={"error": str(e)}, + recommendations_json=[ + "Check target URL is accessible", + "Verify authentication token if provided", + ], + ) + ) +``` + +This code: +- Creates the scan record immediately (line 26-30) so it has an ID +- Maps test type enums to actual scanner classes (line 33-37) +- Loops through requested tests (line 42-60) +- Instantiates each scanner with target URL and auth token +- Calls `scanner.scan()` which returns `TestResultCreate` with findings +- Catches exceptions so one failing scanner doesn't kill the entire scan + +### Storage/Output + +Save results to database: `services/scan_service.py:62-65` continues: +```python + # Save all results + for result in results: + TestResultRepository.create_test_result( + db=db, + scan_id=scan.id, + test_name=result.test_name, + status=result.status, + severity=result.severity, + details=result.details, + evidence_json=result.evidence_json, + recommendations_json=result.recommendations_json, + ) + + db.refresh(scan) # Reload scan with test_results relationship populated + + return ScanResponse.model_validate(scan) +``` + +The result is a JSON response containing the scan with nested test results: +```json +{ + "id": 42, + "user_id": 1, + "target_url": "https://api.example.com/users", + "scan_date": "2026-02-04T10:30:00Z", + "created_at": "2026-02-04T10:30:00Z", + "test_results": [ + { + "id": 101, + "scan_id": 42, + "test_name": "sqli", + "status": "vulnerable", + "severity": "critical", + "details": "Error-based SQL injection detected: mysql", + "evidence_json": { + "database_type": "mysql", + "payload": "' OR 1=1--", + "error_signature": "sql syntax" + }, + "recommendations_json": [ + "Use parameterized queries (prepared statements)", + "Never concatenate user input into SQL queries" + ] + } + ] +} +``` + +## Error Handling Patterns + +### Database Errors with Automatic Rollback + +The `get_db()` dependency handles transaction management: +```python +# core/database.py:28-36 +def get_db() -> Generator[Session, None, None]: + """ + FastAPI dependency for database sessions + """ + db = SessionLocal() + try: + yield db # Provide session to route handler + finally: + db.close() # Always close, even if exception raised +``` + +If an exception occurs during request processing: +1. FastAPI catches it +2. Control returns to finally block +3. `db.close()` runs +4. SQLAlchemy automatically rolls back uncommitted transaction +5. Connection returned to pool + +Example error scenario: +```python +@router.post("/scans/") +async def create_scan(db: Session = Depends(get_db), ...): + scan = ScanRepository.create_scan(db, ...) # INSERT INTO scans + + # Something goes wrong here + raise ValueError("Oops") + + # This never runs + TestResultRepository.create_test_result(db, ...) +``` + +Without explicit transaction handling: +- Scan record would be inserted +- Test result would not be inserted +- Database left in inconsistent state + +With `get_db()` cleanup: +- ValueError propagates to FastAPI +- `db.close()` runs +- Scan INSERT is rolled back +- Database remains consistent + +**What NOT to do:** +```python +# Bad - manual session management +db = SessionLocal() +try: + scan = ScanRepository.create_scan(db, ...) + db.commit() # Committed before error could happen + raise ValueError("Oops") +finally: + db.close() + +# Scan is committed, error occurs after - inconsistent state +``` + +### Scanner Timeout Recovery + +Scanners use retry logic with exponential backoff (`base_scanner.py:92-156`): +```python +def make_request( + self, + method: str, + endpoint: str, + **kwargs: Any, +) -> requests.Response: + """ + Make HTTP request with retry logic and rate limit handling + """ + self._wait_before_request() # Implement request spacing + + url = urljoin(self.target_url, endpoint) + retry_count = 0 + backoff_factor = 2.0 + + kwargs.setdefault("timeout", settings.SCANNER_CONNECTION_TIMEOUT) + + while retry_count <= settings.DEFAULT_RETRY_COUNT: # 3 retries + try: + start_time = time.time() + response = self.session.request(method, url, **kwargs) + + # Track timing for time-based detection + setattr(response, "request_time", time.time() - start_time) + + self.request_count += 1 + + # Handle 429 Too Many Requests + if response.status_code == 429: + retry_after = response.headers.get("Retry-After", "60") + wait_time = int(retry_after) if retry_after.isdigit() else 60 + time.sleep(wait_time) + retry_count += 1 + continue # Try again after waiting + + # Handle server errors with backoff + if response.status_code >= 500 and retry_count < settings.DEFAULT_RETRY_COUNT: + wait_time = backoff_factor ** retry_count # 1s, 2s, 4s + time.sleep(wait_time) + retry_count += 1 + continue + + return response # Success + + except (requests.Timeout, requests.ConnectionError): + if retry_count < settings.DEFAULT_RETRY_COUNT: + wait_time = backoff_factor ** retry_count + time.sleep(wait_time) + retry_count += 1 + else: + raise # Give up after 3 retries + + return response # Return last response if loop completes +``` + +**Retry scenarios:** + +1. **Timeout**: Wait 1s, retry. Still timeout? Wait 2s, retry. Still timeout? Wait 4s, retry. Still timeout? Raise exception. + +2. **Connection refused**: Same exponential backoff strategy. + +3. **429 Rate Limited**: Read `Retry-After` header, wait that long, retry. No exponential backoff needed - server told us exactly how long to wait. + +4. **500 Server Error**: Exponential backoff, but only retry if `retry_count < 3`. After 3 attempts, return the 500 response (don't raise exception). Let caller decide how to handle. + +## Performance Optimizations + +### Before: Naive Database Queries (N+1 Problem) +```python +# Bad - triggers N+1 queries +@router.get("/scans/") +async def get_scans(db: Session = Depends(get_db), user: User = Depends(get_current_user)): + scans = db.query(Scan).filter(Scan.user_id == user.id).all() + # SQL: SELECT * FROM scans WHERE user_id = 1 + + for scan in scans: + print(scan.test_results) # Each access triggers new query! + # SQL: SELECT * FROM test_results WHERE scan_id = 42 + # SQL: SELECT * FROM test_results WHERE scan_id = 43 + # ... repeated for each scan + + return scans +``` + +With 10 scans, 4 results each = 41 queries (1 + 10*4). + +### After: Eager Loading with joinedload +```python +# Good - single query with JOIN +def get_by_user(db: Session, user_id: int) -> list[Scan]: + return ( + db.query(Scan) + .options(joinedload(Scan.test_results)) # Load relationship in same query + .filter(Scan.user_id == user_id) + .all() + ) + # SQL: SELECT scans.*, test_results.* + # FROM scans + # LEFT JOIN test_results ON scans.id = test_results.scan_id + # WHERE scans.user_id = 1 +``` + +Single query loads everything. Accessing `scan.test_results` uses already-loaded data, no additional query. + +**Benchmarks:** +- Before: 41 queries, ~205ms (5ms per query) +- After: 1 query, ~15ms +- Improvement: **13x faster** + +### Request Connection Pooling + +Base scanner reuses HTTP session (`base_scanner.py:40-62`): +```python +def __init__(self, target_url: str, auth_token: str | None = None, ...): + self.target_url = target_url.rstrip("/") + self.auth_token = auth_token + self.session = self._create_session() # Created once + +def _create_session(self) -> requests.Session: + """ + Create persistent HTTP session with proper headers + """ + session = requests.Session() + + session.headers.update({ + "User-Agent": f"{settings.APP_NAME}/{settings.VERSION}", + "Accept": "application/json", + }) + + if self.auth_token: + session.headers.update({"Authorization": f"Bearer {self.auth_token}"}) + + return session +``` + +**Why this matters:** + +Without session (making new request each time): +```python +# Each call creates new TCP connection +requests.get("https://api.example.com/endpoint1") # Connect, TLS handshake, request, close +requests.get("https://api.example.com/endpoint2") # Connect, TLS handshake, request, close +# 2 connections, 2 TLS handshakes +``` + +With session: +```python +session = requests.Session() +session.get("https://api.example.com/endpoint1") # Connect, TLS handshake, request, keep-alive +session.get("https://api.example.com/endpoint2") # Reuse connection, request +# 1 connection, 1 TLS handshake +``` + +TLS handshake takes 50-100ms. Over 100 requests, that's 5-10 seconds saved. + +## Configuration Management + +### Loading Config + +All settings loaded from environment via Pydantic: +```python +# config.py:14-69 +from functools import lru_cache +from pydantic_settings import BaseSettings + +class Settings(BaseSettings): + model_config = SettingsConfigDict( + env_file="../.env", # Load from .env file + env_file_encoding="utf-8", + case_sensitive=True # DATABASE_URL != database_url + ) + + # Required fields (no default) + DATABASE_URL: str + SECRET_KEY: str + + # Optional fields (have defaults) + APP_NAME: str = "API Security Tester" + DEBUG: bool = False + ACCESS_TOKEN_EXPIRE_MINUTES: int = 1440 + + @property + def cors_origins_list(self) -> list[str]: + """Convert comma-separated string to list""" + return [origin.strip() for origin in self.CORS_ORIGINS.split(",")] + +@lru_cache +def get_settings() -> Settings: + """ + Get cached settings instance + + @lru_cache ensures settings are loaded only once + """ + return Settings() + +settings = get_settings() +``` + +**Validation:** + +Pydantic validates types at startup: +```python +# .env contains: +# DEBUG=not_a_boolean + +# When Settings() instantiates: +# ValidationError: field required to be bool, got str 'not_a_boolean' +``` + +Application crashes immediately with clear error instead of mysterious runtime failures. + +**Why cache with `@lru_cache`:** + +Without cache: +```python +# Every import creates new Settings instance +from config import settings # Loads .env +from config import settings # Loads .env again +``` + +With cache: +```python +# First import loads .env +from config import settings # Loads .env, caches result + +# Subsequent imports reuse cached instance +from config import settings # Returns cached Settings +``` + +Faster startup, consistent state across application. + +## Database/Storage Operations + +### Creating Records with Transactions + +Repository method with explicit commit control: +```python +# repositories/scan_repository.py:13-32 +@staticmethod +def create_scan( + db: Session, + user_id: int, + target_url: str, + commit: bool = True # Allow caller to control commit +) -> Scan: + """ + Create a new scan + """ + scan = Scan( + user_id=user_id, + target_url=target_url, + scan_date=datetime.now(UTC), + ) + db.add(scan) + + if commit: + db.commit() # Flush to database + db.refresh(scan) # Reload to get auto-generated ID + + return scan +``` + +**Important details:** + +- **Transaction management**: The `commit` parameter lets callers batch operations. Create scan + create results in single transaction. +- **Refresh after commit**: Line 30 reloads the object from database. This populates auto-generated fields like `id`, `created_at`, `updated_at`. +- **Foreign key constraints**: PostgreSQL enforces `test_results.scan_id` references `scans.id`. If we create test results before committing scan, foreign key check fails. + +### Bulk Inserts for Performance + +Creating test results in batch: +```python +# repositories/test_result_repository.py:52-67 +@staticmethod +def bulk_create( + db: Session, + test_results: list[TestResult], + commit: bool = True +) -> list[TestResult]: + """ + Create multiple test results in bulk + """ + db.add_all(test_results) # Add all at once + + if commit: + db.commit() + for result in test_results: + db.refresh(result) # Refresh each to get IDs + + return test_results +``` + +**Why bulk insert:** + +Individual inserts: +```python +for result in results: + db.add(result) + db.commit() # Commit after each - 4 commits for 4 results +# 4 round trips to database +``` + +Bulk insert: +```python +db.add_all(results) +db.commit() # Single commit for all 4 results +# 1 round trip to database +``` + +With 4 test results, bulk insert is 4x faster. + +## Common Implementation Pitfalls + +### Pitfall 1: Forgetting to Validate JWT Algorithm + +**Symptom:** +Attacker sends token with `"alg": "none"` and gains admin access. + +**Cause:** +```python +# Problematic code - accepts any algorithm +payload = jwt.decode(token, settings.SECRET_KEY) + +# Attacker sends: +# eyJhbGciOiJub25lIn0.eyJzdWIiOiJhZG1pbiJ9. +# (header: {"alg": "none"}, payload: {"sub": "admin"}, no signature) + +# Library accepts it because no algorithm restriction +``` + +**Fix:** +```python +# Correct - explicitly allow only HS256 +payload = jwt.decode( + token, + settings.SECRET_KEY, + algorithms=["HS256"] # Reject "none" and other algorithms +) +``` + +**Why this matters:** +Without algorithm validation, JWT security is completely bypassed. Attacker can forge tokens with any claims. + +### Pitfall 2: SQL Injection in Raw Queries + +**Symptom:** +Attacker sends `email=admin'--` and gets admin access. + +**Cause:** +```python +# Vulnerable - concatenating user input +email = request.get("email") +query = f"SELECT * FROM users WHERE email = '{email}'" +db.execute(text(query)) + +# Becomes: SELECT * FROM users WHERE email = 'admin'--' +# Comment (--) removes password check +``` + +**Fix:** +```python +# Correct - parameterized query +email = request.get("email") +db.execute( + text("SELECT * FROM users WHERE email = :email"), + {"email": email} +) + +# SQLAlchemy escapes the email value safely +``` + +Or better, use ORM: +```python +# Best - ORM automatically parameterizes +db.query(User).filter(User.email == email).first() +``` + +**Why this matters:** +String concatenation treats user input as SQL code. Parameterization treats it as data. + +### Pitfall 3: Missing Authorization Check + +**Symptom:** +User can view other users' scans by changing scan ID in URL. + +**Cause:** +```python +# Vulnerable - no ownership check +@router.get("/scans/{scan_id}") +async def get_scan(scan_id: int, db: Session = Depends(get_db)): + scan = ScanRepository.get_by_id(db, scan_id) + if not scan: + raise HTTPException(404) + return scan # Returns any scan, regardless of ownership +``` + +**Fix:** +```python +# Correct - verify ownership +@router.get("/scans/{scan_id}") +async def get_scan( + scan_id: int, + db: Session = Depends(get_db), + current_user: User = Depends(get_current_user) +): + scan = ScanRepository.get_by_id(db, scan_id) + if not scan: + raise HTTPException(404) + + if scan.user_id != current_user.id: + raise HTTPException(403, detail="Not authorized") + + return scan +``` + +**Why this matters:** +Authentication proves who you are. Authorization proves what you can access. Both are required. + +## Debugging Tips + +### Issue Type 1: JWT Token Appears Invalid But Format Looks Correct + +**Problem:** Getting 401 errors when using a token that decodes properly on jwt.io. + +**How to debug:** + +1. Check token expiration: +```python +import jwt +import datetime + +token = "eyJ..." +decoded = jwt.decode(token, options={"verify_signature": False}) # Skip verification for debugging +print(decoded) +# {"sub": "user@example.com", "exp": 1704067200} + +exp_time = datetime.datetime.fromtimestamp(decoded["exp"]) +now = datetime.datetime.now() +print(f"Expires: {exp_time}, Now: {now}, Valid: {exp_time > now}") +``` + +2. Verify secret key matches: +```python +# In Python shell with access to settings +from config import settings +print(f"SECRET_KEY: {settings.SECRET_KEY}") +# Make sure this matches the key used to create the token +``` + +3. Check algorithm matches: +```python +# Decode header without verification +import base64 +import json + +token = "eyJ..." +header = token.split(".")[0] +# Add padding if needed +padding = 4 - (len(header) % 4) +if padding != 4: + header += "=" * padding + +decoded_header = json.loads(base64.urlsafe_b64decode(header)) +print(decoded_header) +# {"alg": "HS256", "typ": "JWT"} + +# Make sure "alg" matches settings.ALGORITHM +``` + +**Common causes:** +- Token expired (check `exp` claim) +- Wrong secret key (dev vs prod environments) +- Algorithm mismatch (created with HS256, verifying with RS256) + +### Issue Type 2: Scanner Times Out on All Tests + +**Problem:** All test results return `status="error"` with timeout messages. + +**How to debug:** + +1. Test target is accessible: +```bash +# From inside backend container +docker exec -it apisec_backend_dev bash +curl -v https://target-api.com/endpoint + +# Check: +# - Does connection succeed? +# - What's the response time? +# - Are there redirects? +``` + +2. Check timeout settings: +```python +# config.py:51 +SCANNER_CONNECTION_TIMEOUT: int = 180 # 3 minutes + +# If target is slower, increase this +``` + +3. Look at actual error: +```python +# services/scan_service.py exception block shows error +except Exception as e: + details=f"Scanner error: {str(e)}" + +# Check what the exception says: +# - "Connection timeout" = target slow or unreachable +# - "Name resolution failed" = DNS issue +# - "SSL certificate verify failed" = TLS problem +``` + +4. Try with a known-good target: +```bash +# Test with httpbin (always responsive) +curl -X POST http://localhost:8000/scans/ \ + -H "Authorization: Bearer YOUR_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "target_url": "https://httpbin.org/get", + "tests_to_run": ["rate_limit"], + "max_requests": 10 + }' +``` + +**Common causes:** +- Target requires VPN or is behind firewall +- Target rate limiting scanner (ironically) +- DNS resolution failing in Docker network +- TLS certificate issues (self-signed, expired) + +### Issue Type 3: Database Deadlock Errors + +**Problem:** Intermittent `DeadlockDetected` errors under concurrent load. + +**How to debug:** + +1. Check transaction order: +```sql +-- PostgreSQL query to see locks +SELECT + pid, + state, + query_start, + state_change, + query +FROM pg_stat_activity +WHERE state = 'active'; +``` + +2. Look for transaction patterns: +```python +# Problematic pattern - different order +# Thread 1: +UPDATE scans SET ... WHERE id = 1; +UPDATE test_results SET ... WHERE scan_id = 1; + +# Thread 2: +UPDATE test_results SET ... WHERE scan_id = 2; +UPDATE scans SET ... WHERE id = 2; + +# Thread 1 locks scans.id=1, waits for test_results +# Thread 2 locks test_results.scan_id=2, waits for scans +# = Deadlock +``` + +3. Fix by consistent ordering: +```python +# Both threads update in same order = no deadlock +# Always update parent (scans) before child (test_results) +UPDATE scans SET ... WHERE id = ?; +UPDATE test_results SET ... WHERE scan_id = ?; +``` + +**Common causes:** +- Multiple transactions updating same rows in different orders +- Long-running transactions holding locks +- Missing indexes causing table scans that lock many rows + +## Code Organization Principles + +### Why Routes Stay Thin + +Routes in `routes/scans.py` are intentionally simple: +```python +@router.post("/", response_model=ScanResponse) +async def create_scan( + request: Request, + scan_request: ScanRequest, + db: Session = Depends(get_db), + current_user: UserResponse = Depends(get_current_user), +) -> ScanResponse: + return ScanService.run_scan(db, current_user.id, scan_request) +``` + +Just 3 lines: +1. Dependency injection provides db and current_user +2. Call service layer +3. Return result + +**All business logic lives in services.** Routes handle HTTP concerns: +- Parsing request +- Validating with Pydantic +- Checking authentication +- Serializing response + +This makes routes easy to test: +```python +# Test route with mocked service +def test_create_scan(mock_service): + mock_service.run_scan.return_value = fake_scan + + response = client.post("/scans/", json={...}) + + assert response.status_code == 201 + assert mock_service.run_scan.called +``` + +No need to mock database, scanners, or any complex logic. Service is mocked, route just handles HTTP. + +### Naming Conventions + +- `*Repository` = Data access classes (UserRepository, ScanRepository) +- `*Service` = Business logic classes (AuthService, ScanService) +- `*Scanner` = Security test implementations (SQLiScanner, AuthScanner) +- `*Schema` = Pydantic validation models (UserCreate, ScanResponse) +- `get_*` functions = FastAPI dependencies (get_db, get_current_user) +- `_private_method` = Internal helper (not part of public interface) + +Following these patterns makes it easier to find code. Need to add a database query? Look in repositories. Need to change business logic? Look in services. + +## Extending the Code + +### Adding a New Security Test (XSS Example) + +Want to add XSS (Cross-Site Scripting) detection? Here's the complete process: + +**1. Create scanner** `backend/scanners/xss_scanner.py`: +```python +from .base_scanner import BaseScanner +from .payloads import XSSPayloads +from core.enums import TestType, ScanStatus, Severity +from schemas.test_result_schemas import TestResultCreate + +class XSSScanner(BaseScanner): + """ + Tests for Cross-Site Scripting vulnerabilities + """ + + def scan(self) -> TestResultCreate: + reflected_test = self._test_reflected_xss() + + if reflected_test["vulnerable"]: + return TestResultCreate( + test_name=TestType.XSS, + status=ScanStatus.VULNERABLE, + severity=Severity.HIGH, + details=f"Reflected XSS detected: {reflected_test['payload']}", + evidence_json=reflected_test, + recommendations_json=[ + "Encode user input before rendering in HTML", + "Use Content-Security-Policy headers", + "Validate input on server side", + ], + ) + + return TestResultCreate( + test_name=TestType.XSS, + status=ScanStatus.SAFE, + severity=Severity.INFO, + details="No XSS vulnerabilities detected", + evidence_json=reflected_test, + recommendations_json=[], + ) + + def _test_reflected_xss(self) -> dict[str, Any]: + payloads = XSSPayloads.get_basic_payloads() + + for payload in payloads: + response = self.make_request("GET", f"/?q={payload}") + + # Check if payload appears unencoded in response + if payload in response.text: + return { + "vulnerable": True, + "payload": payload, + "status_code": response.status_code, + } + + return {"vulnerable": False} +``` + +**2. Add enum value** in `backend/core/enums.py:19-26`: +```python +class TestType(str, Enum): + RATE_LIMIT = "rate_limit" + AUTH = "auth" + SQLI = "sqli" + IDOR = "idor" + XSS = "xss" # New test type +``` + +**3. Register scanner** in `backend/services/scan_service.py:32-38`: +```python +scanner_mapping: dict[TestType, type[BaseScanner]] = { + TestType.RATE_LIMIT: RateLimitScanner, + TestType.AUTH: AuthScanner, + TestType.SQLI: SQLiScanner, + TestType.IDOR: IDORScanner, + TestType.XSS: XSSScanner, # Register new scanner +} +``` + +**4. Add payloads** in `backend/scanners/payloads.py:250-280`: +```python +class XSSPayloads: + BASIC_XSS = [ + "", + "", + "", + ] + + @classmethod + def get_basic_payloads(cls) -> list[str]: + return cls.BASIC_XSS +``` + +**5. Update frontend** in `frontend/src/config/constants.ts`: +```typescript +export const SCAN_TEST_TYPES = { + // ... existing + XSS: 'xss', +} as const; + +export const TEST_TYPE_LABELS: Record = { + // ... existing + [SCAN_TEST_TYPES.XSS]: 'Cross-Site Scripting', +}; +``` + +That's it. No database changes needed (TestType enum automatically updates). No route changes (they pass through test types). Just scanner implementation and registration. + +## Next Steps + +You've seen how the code works. Now: + +1. **Try the challenges** - [04-CHALLENGES.md](./04-CHALLENGES.md) has extension ideas from adding stored XSS detection to implementing custom scanner plugins +2. **Modify scanners** - Change SQLi payload timing thresholds, add new auth bypass techniques, implement DOM-based XSS detection +3. **Read related projects** - The docker-security-audit project builds on container scanning concepts, network-traffic-analyzer goes deeper into packet analysis diff --git a/PROJECTS/intermediate/api-security-scanner/learn/04-CHALLENGES.md b/PROJECTS/intermediate/api-security-scanner/learn/04-CHALLENGES.md index e69de29b..e8fa280e 100644 --- a/PROJECTS/intermediate/api-security-scanner/learn/04-CHALLENGES.md +++ b/PROJECTS/intermediate/api-security-scanner/learn/04-CHALLENGES.md @@ -0,0 +1,1597 @@ +# 04-CHALLENGES.md + +# Extension Challenges + +You've built the base project. Now make it yours by extending it with new features. + +These challenges are ordered by difficulty. Start with the easier ones to build confidence, then tackle the harder ones when you want to dive deeper. + +## Easy Challenges + +### Challenge 1: Add CORS Misconfiguration Detection + +**What to build:** +A scanner that tests for overly permissive CORS (Cross-Origin Resource Sharing) policies. Check if the API accepts requests from any origin, reflects the Origin header, or allows credentials with wildcard origins. + +**Why it's useful:** +CORS misconfigurations let attackers steal data from authenticated users. If an API accepts `Origin: https://evil.com` and responds with `Access-Control-Allow-Origin: https://evil.com`, the attacker can make requests from their site and read the responses. This was how the PayPal information disclosure vulnerability (2020) worked. + +**What you'll learn:** +- HTTP header analysis and pattern matching +- Origin header manipulation techniques +- Understanding CORS security model +- How browsers enforce same-origin policy + +**Hints:** +- Look at `backend/scanners/auth_scanner.py` for header manipulation examples +- The scanner needs to make requests with different `Origin` headers +- Check response for `Access-Control-Allow-Origin`, `Access-Control-Allow-Credentials` +- Test cases: wildcard (`*`), reflected origin, `null` origin + +**Test it works:** +```python +# In scanners/cors_scanner.py +origins_to_test = [ + "https://evil.com", + "null", + "https://attacker.com", +] + +for origin in origins_to_test: + response = self.make_request("GET", "/", headers={"Origin": origin}) + + allow_origin = response.headers.get("Access-Control-Allow-Origin") + allow_creds = response.headers.get("Access-Control-Allow-Credentials") + + if allow_origin == origin or allow_origin == "*": + # Found CORS misconfiguration +``` + +Verify by scanning an endpoint with permissive CORS. Should detect `Access-Control-Allow-Origin: *` as vulnerable. + +### Challenge 2: Add Password Strength Reporting + +**What to build:** +Enhance registration to check password strength and provide feedback. Test against common password lists (rockyou.txt top 10000), check for patterns (keyboard walks like "qwerty"), and calculate entropy. + +**Why it's useful:** +Weak passwords are the #1 cause of account takeovers. The LinkedIn breach (2012) exposed 117 million passwords, many were "123456" and "password". Real-time feedback helps users create stronger passwords. + +**What you'll learn:** +- Password entropy calculation (bits of randomness) +- Pattern detection in strings +- Working with wordlists and datasets +- Balancing security with usability + +**Hints:** +- Modify `backend/schemas/user_schemas.py:26-38` (the password validator) +- Download common password lists from SecLists on GitHub +- Calculate entropy: `log2(character_space ^ length)` +- Check for sequential characters, repeated characters, dictionary words + +**Implementation approach:** + +1. **Add password checker utility** in `backend/core/security.py`: +```python +def check_password_strength(password: str) -> dict[str, Any]: + score = 0 + feedback = [] + + # Length check + if len(password) >= 12: + score += 2 + elif len(password) >= 8: + score += 1 + else: + feedback.append("Password should be at least 12 characters") + + # Character variety + if re.search(r"[A-Z]", password): + score += 1 + if re.search(r"[a-z]", password): + score += 1 + if re.search(r"[0-9]", password): + score += 1 + if re.search(r"[^A-Za-z0-9]", password): + score += 1 + + # Check against common passwords + common_passwords = load_common_passwords() # Load from file + if password.lower() in common_passwords: + score = 0 + feedback.append("This is a commonly used password") + + # Patterns + if re.search(r"(.)\1{2,}", password): # Repeated chars + score -= 1 + feedback.append("Avoid repeated characters") + + strength = "weak" if score < 3 else "medium" if score < 5 else "strong" + + return { + "strength": strength, + "score": score, + "feedback": feedback, + } +``` + +2. **Update registration endpoint** to return strength info (optional, informational only). + +**Test edge cases:** +- `password123` - Common pattern +- `P@ssw0rd` - Meets requirements but still weak +- `correct-horse-battery-staple` - Long passphrase (good) +- `aaaaAAAA1111!!!!` - Meets requirements but has patterns + +### Challenge 3: Add Response Time Monitoring Dashboard + +**What to build:** +Track and visualize scanner performance metrics. Store response times for each test, calculate percentiles (p50, p95, p99), and detect when targets are slowing down. + +**Why it's useful:** +Performance data helps tune scanner timeouts and detect issues. If the SQLi scanner's time-based detection shows high variance, you know the baseline isn't reliable. If all tests against a target suddenly take 10x longer, the target might be rate limiting you. + +**What you'll learn:** +- Metrics collection and aggregation +- Percentile calculation (not just averages) +- Time-series data visualization +- Performance baseline establishment + +**Hints:** +- `BaseScanner.make_request()` already tracks `request_time` at line 117 +- Store timing data in new `ScanMetrics` model +- Calculate percentiles with `numpy.percentile()` +- Frontend can use Chart.js or Recharts to visualize + +**Extra credit:** +Add alerting when p95 response time exceeds threshold. If average scan takes 30s but p95 is 120s, some scans are timing out frequently. + +## Intermediate Challenges + +### Challenge 4: Implement Stored XSS Detection + +**What to build:** +Extend XSS testing beyond reflected XSS. Submit payloads via POST, then retrieve the resource via GET to check if the payload persists. Test comment fields, user profiles, any data that gets stored and displayed. + +**Why it's useful:** +Stored XSS is more dangerous than reflected because it affects all users, not just the victim who clicks a malicious link. The MySpace Samy worm (2005) used stored XSS to infect over 1 million profiles in 20 hours. + +**Real world application:** +Any API with user-generated content needs stored XSS testing. Forums, comment systems, profile pages, file uploads with previews. + +**What you'll learn:** +- Multi-step testing workflows (submit then retrieve) +- Payload encoding variations (URL encoding, HTML entities, Unicode) +- Context-aware XSS detection (JavaScript context vs HTML context) +- False positive reduction in fuzzy matching + +**Implementation approach:** + +1. **Create stored XSS scanner** in `backend/scanners/stored_xss_scanner.py`: +```python +class StoredXSSScanner(BaseScanner): + def scan(self) -> TestResultCreate: + # Generate unique marker + marker = f"XSS_{uuid.uuid4().hex[:8]}" + payload = f"" + + # Step 1: Submit payload + submit_result = self._submit_payload(payload, marker) + if not submit_result["submitted"]: + return self._safe_result("Could not submit payload") + + # Step 2: Retrieve and check + retrieve_result = self._retrieve_and_check(marker) + if retrieve_result["vulnerable"]: + return self._vulnerable_result(payload, marker, retrieve_result) + + return self._safe_result("No stored XSS detected") + + def _submit_payload(self, payload: str, marker: str) -> dict[str, Any]: + # Try common endpoints + endpoints = ["/comments", "/api/posts", "/api/profile"] + + for endpoint in endpoints: + try: + response = self.make_request( + "POST", + endpoint, + json={"content": payload, "text": payload, "bio": payload} + ) + + if response.status_code in (200, 201): + return {"submitted": True, "endpoint": endpoint} + except Exception: + continue + + return {"submitted": False} + + def _retrieve_and_check(self, marker: str) -> dict[str, Any]: + # Retrieve content to see if payload persists + response = self.make_request("GET", "/") + + if marker in response.text: + # Check if it's encoded + if f"<script>" in response.text: + return {"vulnerable": False, "encoded": True} + + # Check if CSP would block it + csp = response.headers.get("Content-Security-Policy", "") + if "script-src 'none'" in csp or "script-src 'self'" in csp: + return {"vulnerable": False, "csp_protected": True} + + return {"vulnerable": True, "marker": marker} + + return {"vulnerable": False} +``` + +2. **Add cleanup** to remove test payloads after scanning (good citizenship). + +3. **Test edge cases:** +- Payload gets HTML encoded (safe) +- Payload stored but CSP prevents execution (still report as stored XSS) +- Payload appears in JSON response (context matters) + +**Gotchas:** +- Don't leave test payloads in production systems (always clean up) +- Some systems delay rendering (cache, async processing) so marker might not appear immediately +- Be careful with user attribution - don't associate test payloads with real users + +### Challenge 5: Add API Rate Limit Bypass Testing with Header Rotation + +**What to build:** +Extend rate limit bypass testing with more sophisticated techniques. Test User-Agent rotation, session ID manipulation, timestamp fuzzing, and Origin header variations. + +**Why this is challenging:** +Modern rate limiters use multiple signals (IP, user agent, session, fingerprint). You need to test combinations systematically. + +**What you'll learn:** +- Advanced rate limiting evasion techniques +- HTTP header manipulation at scale +- Statistical analysis of rate limit effectiveness +- Designing test matrices (combinatorial testing) + +**Implementation approach:** + +1. **Extend rate limit scanner** in `backend/scanners/rate_limit_scanner.py`: +```python +def _test_header_rotation_bypass(self) -> dict[str, Any]: + """ + Test if rotating headers bypasses rate limits + """ + user_agents = RateLimitBypassPayloads.USER_AGENT_ROTATION + + # Establish that rate limit exists + for _ in range(20): + response = self.make_request("GET", "/") + if response.status_code == 429: + break + else: + return {"bypass_successful": False, "reason": "No rate limit found"} + + # Try bypassing with User-Agent rotation + success_count = 0 + for i in range(20): + ua = user_agents[i % len(user_agents)] + + response = self.make_request( + "GET", "/", + headers={"User-Agent": ua} + ) + + if response.status_code != 429: + success_count += 1 + else: + break + + if success_count == 20: + return { + "bypass_successful": True, + "bypass_method": "User-Agent Rotation", + "requests_completed": success_count, + } + + return {"bypass_successful": False} +``` + +2. **Test combinations**: +```python +combinations = [ + {"User-Agent": ua, "X-Forwarded-For": ip}, + {"User-Agent": ua, "Origin": origin}, + # etc +] +``` + +3. **Add timing analysis** to detect soft limits (degraded but not blocked). + +**Resources:** +- Read "Bypassing Rate Limits" on PortSwigger Research blog +- Study Cloudflare's rate limiting documentation to understand what you're up against + +### Challenge 6: Implement XML External Entity (XXE) Detection + +**What to build:** +Test for XXE vulnerabilities in APIs that accept XML. Submit payloads with external entity references and check for file disclosure, SSRF, or denial of service. + +**Why this is hard:** +XXE requires understanding XML parsers, DTD syntax, and different attack vectors (file disclosure vs SSRF vs billion laughs). + +**What you'll learn:** +- XML parsing vulnerabilities +- Out-of-band data exfiltration techniques +- SSRF exploitation through XML +- Parser configuration security + +**Implementation:** + +Create `backend/scanners/xxe_scanner.py`: + +```python +class XXEScanner(BaseScanner): + def scan(self) -> TestResultCreate: + # Test file disclosure + file_disclosure = self._test_file_disclosure() + if file_disclosure["vulnerable"]: + return self._vulnerable_result( + "XXE file disclosure detected", + file_disclosure, + Severity.CRITICAL + ) + + # Test SSRF + ssrf_test = self._test_ssrf_xxe() + if ssrf_test["vulnerable"]: + return self._vulnerable_result( + "XXE SSRF detected", + ssrf_test, + Severity.HIGH + ) + + return self._safe_result() + + def _test_file_disclosure(self) -> dict[str, Any]: + # Payload to read /etc/passwd + payload = """ + + ]> + &xxe;""" + + response = self.make_request( + "POST", + "/", + data=payload, + headers={"Content-Type": "application/xml"} + ) + + # Check if file contents leaked + if "root:" in response.text or "bin/bash" in response.text: + return { + "vulnerable": True, + "payload": payload, + "leaked_data": response.text[:200] + } + + return {"vulnerable": False} + + def _test_ssrf_xxe(self) -> dict[str, Any]: + # Test if parser makes external requests + # Use Burp Collaborator or similar out-of-band detection + + collaborator_url = "http://burpcollaborator.net/unique-id" + + payload = f""" + + ]> + &xxe;""" + + response = self.make_request( + "POST", + "/", + data=payload, + headers={"Content-Type": "application/xml"} + ) + + # Check if request was made (need out-of-band detection) + # This is simplified - real implementation needs callback server + + return {"vulnerable": False, "note": "Manual verification required"} +``` + +**Success criteria:** +- Detects XXE in XML endpoints +- Tests multiple entity types (file, http, parameter entities) +- Handles different parser responses (error messages, timeout, data) +- Reports severity based on impact (file disclosure = CRITICAL, SSRF = HIGH) + +## Advanced Challenges + +### Challenge 7: Build a Scanner Plugin System + +**What to build:** +Create a plugin architecture that lets users write custom scanners without modifying core code. Scanners should be loadable from a `plugins/` directory, with automatic discovery and registration. + +**Why this is hard:** +Requires dynamic module loading, interface contracts, error isolation (broken plugin shouldn't crash scanner), and documentation for plugin developers. + +**What you'll learn:** +- Python module introspection and dynamic imports +- Abstract base classes and interface design +- Plugin architecture patterns +- Sandboxing and error isolation + +**Architecture changes needed:** + +``` +Current: +scanner_mapping = { + TestType.SQLI: SQLiScanner, # Hardcoded + ... +} + +New: +scanner_registry = ScannerRegistry() +scanner_registry.load_builtin_scanners() +scanner_registry.discover_plugins("plugins/") + +scanner_mapping = scanner_registry.get_all_scanners() +``` + +**Implementation steps:** + +1. **Define plugin interface** in `backend/scanners/plugin_interface.py`: +```python +from abc import ABC, abstractmethod +from typing import Any + +class ScannerPlugin(ABC): + """ + Base class for scanner plugins + + All plugins must inherit from this and implement required methods + """ + + @property + @abstractmethod + def name(self) -> str: + """Unique scanner name (e.g., 'custom_xxe')""" + pass + + @property + @abstractmethod + def version(self) -> str: + """Plugin version (semver: '1.0.0')""" + pass + + @property + @abstractmethod + def test_type(self) -> str: + """Test type identifier (must be unique)""" + pass + + @abstractmethod + def scan(self, target_url: str, **kwargs: Any) -> dict[str, Any]: + """ + Execute scan and return results + + Returns: + dict with keys: vulnerable, details, evidence, recommendations + """ + pass + + def validate(self) -> bool: + """ + Validate plugin configuration + Override this to add custom validation + """ + return True +``` + +2. **Create plugin loader** in `backend/scanners/plugin_loader.py`: +```python +import os +import importlib.util +from pathlib import Path +from typing import Type + +class PluginLoader: + def __init__(self, plugin_dir: str = "plugins"): + self.plugin_dir = Path(plugin_dir) + self.loaded_plugins: dict[str, Type[ScannerPlugin]] = {} + + def discover_plugins(self) -> list[Type[ScannerPlugin]]: + """ + Discover and load all plugins from plugin directory + """ + if not self.plugin_dir.exists(): + return [] + + plugins = [] + + for file in self.plugin_dir.glob("*.py"): + if file.stem.startswith("_"): + continue # Skip __init__.py, _template.py + + try: + plugin_class = self._load_plugin_from_file(file) + if plugin_class: + plugins.append(plugin_class) + except Exception as e: + print(f"Failed to load plugin {file}: {e}") + continue # Don't let broken plugins crash the scanner + + return plugins + + def _load_plugin_from_file(self, filepath: Path) -> Type[ScannerPlugin] | None: + """ + Dynamically load plugin class from Python file + """ + spec = importlib.util.spec_from_file_location( + f"plugins.{filepath.stem}", + filepath + ) + + if spec is None or spec.loader is None: + return None + + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + # Find ScannerPlugin subclass + for attr_name in dir(module): + attr = getattr(module, attr_name) + + if (isinstance(attr, type) and + issubclass(attr, ScannerPlugin) and + attr is not ScannerPlugin): + + # Instantiate to validate + instance = attr() + if instance.validate(): + return attr + + return None +``` + +3. **Create example plugin** in `plugins/example_scanner.py`: +```python +from scanners.plugin_interface import ScannerPlugin +from scanners.base_scanner import BaseScanner + +class ExampleScanner(ScannerPlugin, BaseScanner): + """ + Example scanner plugin + + Copy this to create your own scanners + """ + + @property + def name(self) -> str: + return "example_scanner" + + @property + def version(self) -> str: + return "1.0.0" + + @property + def test_type(self) -> str: + return "example" + + def scan(self, target_url: str, **kwargs) -> dict[str, Any]: + """ + Your scanner logic here + """ + self.target_url = target_url + + response = self.make_request("GET", "/") + + # Implement your detection logic + vulnerable = self._check_for_vulnerability(response) + + return { + "vulnerable": vulnerable, + "details": "Example vulnerability found" if vulnerable else "Safe", + "evidence": {"status_code": response.status_code}, + "recommendations": ["Fix the issue"] if vulnerable else [], + } + + def _check_for_vulnerability(self, response) -> bool: + # Your detection logic + return False +``` + +**Testing strategy:** +- Unit test plugin discovery (create test plugins in temp directory) +- Test error isolation (broken plugin doesn't crash scanner) +- Test plugin versioning (handle multiple versions of same plugin) + +**Known challenges:** + +1. **Plugin naming conflicts** + - Problem: Two plugins register same `test_type` + - Hint: Use namespacing or first-come-first-served with warnings + +2. **Plugin security** + - Problem: Malicious plugins could access database, file system + - Hint: Run plugins in restricted mode, limit imports, use subprocess isolation + +**Resources:** +- Read about Python's `importlib` module +- Study Flask's extension system for inspiration +- Look at pytest's plugin architecture + +### Challenge 8: Implement Blind SSRF Detection with Out-of-Band Channels + +**What to build:** +Detect Server-Side Request Forgery vulnerabilities even when responses don't leak data. Use DNS callbacks, HTTP callbacks, or timing attacks to confirm SSRF. + +**Why this is challenging:** +Blind SSRF doesn't show response data. You need infrastructure (callback server) to detect when target makes requests. + +**Estimated time:** +8-12 hours for basic implementation, 20+ hours for robust production version. + +**Prerequisites:** +You should have completed the SQLi and Auth challenges first because this builds on request timing analysis and header manipulation. + +**What you'll learn:** +- Out-of-band vulnerability detection techniques +- DNS exfiltration and callback mechanisms +- Time-based blind detection with statistical significance +- Building supporting infrastructure (callback servers) + +**Planning this feature:** + +Before you code, think through: +- How does the callback server work? (DNS? HTTP?) +- What if target has egress filtering? (Can't make outbound requests) +- How do you match callbacks to scans? (Unique identifiers) +- What's your false positive rate? (Network noise, CDN prefetching) + +**High level architecture:** + +``` +Scanner Callback Server Target API + | | | + |--SSRF Payload-------------->| | + | (http://callback.io/abc) | | + | | | + | |<---HTTP Request------------| + | | (GET /abc) | + | | | + |<--Confirmation---------------| | + | (Received request abc) | | +``` + +**Implementation phases:** + +**Phase 1: Build Callback Server** (3-4 hours) + +Create `backend/ssrf_callback_server.py`: + +```python +from fastapi import FastAPI, Request +from datetime import datetime, timedelta +import asyncio + +app = FastAPI() + +# Store received callbacks +callbacks_received: dict[str, dict] = {} + +@app.get("/callback/{identifier}") +async def handle_callback(identifier: str, request: Request): + """ + Receive SSRF callback + """ + callbacks_received[identifier] = { + "timestamp": datetime.utcnow(), + "headers": dict(request.headers), + "client_ip": request.client.host, + } + + return {"status": "received"} + +@app.get("/check/{identifier}") +async def check_callback(identifier: str): + """ + Check if callback was received + """ + callback = callbacks_received.get(identifier) + + if callback: + # Clean up old callback + del callbacks_received[identifier] + return {"received": True, "data": callback} + + return {"received": False} +``` + +Run this on a public server (DigitalOcean, AWS) with a domain pointing to it. + +**Phase 2: Implement SSRF Scanner** (3-4 hours) + +Create `backend/scanners/ssrf_scanner.py`: + +```python +import uuid +import time + +class SSRFScanner(BaseScanner): + def __init__(self, target_url: str, callback_server: str, **kwargs): + super().__init__(target_url, **kwargs) + self.callback_server = callback_server # http://callback.io + + def scan(self) -> TestResultCreate: + # Test URL parameters + url_param_test = self._test_url_parameters() + if url_param_test["vulnerable"]: + return self._vulnerable_result(url_param_test) + + # Test headers + header_test = self._test_headers() + if header_test["vulnerable"]: + return self._vulnerable_result(header_test) + + return self._safe_result() + + def _test_url_parameters(self) -> dict[str, Any]: + # Generate unique identifier + identifier = str(uuid.uuid4()) + + # Payloads to test + params = ["url", "callback", "webhook", "redirect", "link"] + + for param in params: + callback_url = f"{self.callback_server}/callback/{identifier}" + + # Submit SSRF payload + response = self.make_request( + "GET", + f"/?{param}={callback_url}" + ) + + # Wait for callback + time.sleep(5) + + # Check if callback was received + check_response = requests.get( + f"{self.callback_server}/check/{identifier}" + ) + + if check_response.json()["received"]: + return { + "vulnerable": True, + "parameter": param, + "callback_data": check_response.json()["data"], + } + + return {"vulnerable": False} + + def _test_headers(self) -> dict[str, Any]: + """ + Test headers like Referer, X-Forwarded-Host for SSRF + """ + identifier = str(uuid.uuid4()) + callback_url = f"{self.callback_server}/callback/{identifier}" + + headers_to_test = [ + "Referer", + "X-Forwarded-Host", + "X-Original-URL", + "Host", + ] + + for header in headers_to_test: + response = self.make_request( + "GET", + "/", + headers={header: callback_url} + ) + + time.sleep(5) + + check_response = requests.get( + f"{self.callback_server}/check/{identifier}" + ) + + if check_response.json()["received"]: + return { + "vulnerable": True, + "header": header, + "callback_data": check_response.json()["data"], + } + + return {"vulnerable": False} +``` + +**Phase 3: Add DNS Callback Alternative** (2-3 hours) + +Some environments block HTTP but allow DNS. Use DNS callbacks: + +```python +def _test_dns_callback(self) -> dict[str, Any]: + """ + Use DNS exfiltration for SSRF detection + """ + identifier = uuid.uuid4().hex[:8] + dns_domain = f"{identifier}.callback.io" # Your DNS server + + # Submit payload that triggers DNS lookup + payloads = [ + f"http://{dns_domain}/", + f"https://{dns_domain}/", + f"//{dns_domain}/", + ] + + for payload in payloads: + self.make_request("GET", f"/?url={payload}") + + time.sleep(5) + + # Check DNS logs for lookup + # (Requires DNS server that logs queries) + + return {"vulnerable": False} # Implement DNS checking +``` + +**Phase 4: Add Time-Based Detection** (3-4 hours) + +For fully blind SSRF (no callbacks possible), use timing: + +```python +def _test_timing_based_ssrf(self) -> dict[str, Any]: + """ + Detect SSRF via timing differences + + Internal IPs respond fast, external IPs are slower + """ + # Baseline with external IP (slow) + baseline_times = [] + for _ in range(5): + start = time.time() + self.make_request("GET", "/?url=http://example.com/") + baseline_times.append(time.time() - start) + + baseline_avg = statistics.mean(baseline_times) + + # Test with internal IPs (should be faster if SSRF exists) + internal_ips = [ + "http://127.0.0.1/", + "http://localhost/", + "http://192.168.1.1/", + ] + + for ip in internal_ips: + test_times = [] + for _ in range(5): + start = time.time() + self.make_request("GET", f"/?url={ip}") + test_times.append(time.time() - start) + + test_avg = statistics.mean(test_times) + + # If internal IP is significantly faster, SSRF likely exists + if test_avg < (baseline_avg * 0.5): + return { + "vulnerable": True, + "technique": "timing-based", + "baseline_time": baseline_avg, + "internal_ip_time": test_avg, + } + + return {"vulnerable": False} +``` + +**Testing the scanner:** + +Set up a vulnerable test API: +```python +# test_api.py +from flask import Flask, request +import requests + +app = Flask(__name__) + +@app.route("/") +def ssrf_vulnerable(): + url = request.args.get("url") + if url: + # Vulnerable - makes request to user-supplied URL + requests.get(url) + return "OK" + +if __name__ == "__main__": + app.run(port=5000) +``` + +Run scanner against it: +```bash +# Should detect SSRF and receive callback +python -m backend.scanners.ssrf_scanner http://localhost:5000 +``` + +**Success criteria:** +- [ ] Detects SSRF in URL parameters +- [ ] Detects SSRF in headers +- [ ] Uses DNS callbacks when HTTP blocked +- [ ] Falls back to timing-based detection +- [ ] Handles network delays gracefully +- [ ] Cleans up test callbacks + +### Challenge 9: Build a Full Vulnerability Report Generator + +**What to build:** +Generate professional PDF reports of scan results with executive summary, technical details, remediation steps, and CVSS scoring. + +**Estimated time:** +10-15 hours for complete implementation with styling and charts. + +**What you'll learn:** +- PDF generation with ReportLab or WeasyPrint +- CVSS scoring calculation +- Data visualization (matplotlib, plotly) +- Report templating and styling +- Professional documentation standards + +**Implementation phases:** + +**Phase 1: Report Data Structure** (2-3 hours) + +Create `backend/services/report_service.py`: + +```python +from dataclasses import dataclass +from typing import List +import matplotlib.pyplot as plt +from reportlab.lib.pagesizes import letter +from reportlab.pdfgen import canvas + +@dataclass +class VulnerabilitySummary: + critical: int + high: int + medium: int + low: int + info: int + + @property + def total(self) -> int: + return self.critical + self.high + self.medium + self.low + self.info + +class ReportGenerator: + def __init__(self, scan_id: int, db: Session): + self.scan = ScanRepository.get_by_id(db, scan_id) + self.db = db + + def generate_pdf(self, output_path: str) -> str: + """ + Generate comprehensive PDF report + """ + c = canvas.Canvas(output_path, pagesize=letter) + width, height = letter + + # Cover page + self._add_cover_page(c, width, height) + c.showPage() + + # Executive summary + self._add_executive_summary(c, width, height) + c.showPage() + + # Vulnerability details + self._add_vulnerability_details(c, width, height) + + c.save() + return output_path + + def _add_cover_page(self, c, width, height): + c.setFont("Helvetica-Bold", 24) + c.drawString(100, height - 100, "Security Scan Report") + + c.setFont("Helvetica", 14) + c.drawString(100, height - 150, f"Target: {self.scan.target_url}") + c.drawString(100, height - 180, f"Date: {self.scan.scan_date}") + + # Add severity chart + self._add_severity_chart(c, width, height - 400) + + def _add_severity_chart(self, c, x, y): + """ + Create pie chart of vulnerabilities by severity + """ + summary = self._calculate_summary() + + # Create matplotlib chart + fig, ax = plt.subplots(figsize=(4, 4)) + + labels = ["Critical", "High", "Medium", "Low", "Info"] + sizes = [summary.critical, summary.high, summary.medium, summary.low, summary.info] + colors = ["#dc2626", "#ea580c", "#f59e0b", "#3b82f6", "#6b7280"] + + ax.pie(sizes, labels=labels, colors=colors, autopct="%1.1f%%") + + # Save to temp file and embed in PDF + chart_path = "/tmp/severity_chart.png" + plt.savefig(chart_path) + plt.close() + + c.drawImage(chart_path, x, y, width=300, height=300) +``` + +**Phase 2: CVSS Scoring** (2-3 hours) + +Add CVSS score calculation: + +```python +class CVSSCalculator: + """ + Calculate CVSS v3.1 scores for vulnerabilities + """ + + def calculate_score( + self, + test_result: TestResult + ) -> dict[str, Any]: + """ + Calculate CVSS score based on test result + """ + # Base metrics + av = self._attack_vector(test_result.test_name) # Network + ac = self._attack_complexity(test_result) # Low/High + pr = self._privileges_required(test_result) # None/Low/High + ui = self._user_interaction(test_result) # None/Required + s = "U" # Scope: Unchanged + c = self._confidentiality_impact(test_result) # High/Low/None + i = self._integrity_impact(test_result) + a = self._availability_impact(test_result) + + # Calculate base score using CVSS formula + # (Simplified - full implementation in CVSS spec) + + impact = 1 - ((1 - c) * (1 - i) * (1 - a)) + exploitability = 8.22 * av * ac * pr * ui + + if impact <= 0: + base_score = 0 + else: + base_score = min( + (impact + exploitability) * 1.08, + 10.0 + ) + + return { + "score": round(base_score, 1), + "severity": self._score_to_severity(base_score), + "vector": f"CVSS:3.1/AV:{av}/AC:{ac}/PR:{pr}/UI:{ui}/S:{s}/C:{c}/I:{i}/A:{a}", + } +``` + +**Phase 3: Remediation Guide** (2-3 hours) + +Add detailed remediation for each vulnerability type: + +```python +REMEDIATION_GUIDES = { + TestType.SQLI: { + "title": "SQL Injection Remediation", + "steps": [ + "1. Use parameterized queries (prepared statements) for all database operations", + "2. Never concatenate user input into SQL strings", + "3. Use ORMs like SQLAlchemy that handle escaping automatically", + "4. Implement input validation to reject suspicious patterns", + "5. Use least privilege database accounts", + ], + "code_example": """ +# Before (Vulnerable) +query = f"SELECT * FROM users WHERE email = '{email}'" + +# After (Safe) +query = db.query(User).filter(User.email == email).first() + """, + "references": [ + "OWASP SQL Injection Prevention Cheat Sheet", + "CWE-89: Improper Neutralization of Special Elements", + ], + }, + # ... other test types +} +``` + +**Success criteria:** +- [ ] Generates professional-looking PDFs +- [ ] Includes executive summary (high-level findings) +- [ ] Shows vulnerability breakdown by severity +- [ ] Calculates CVSS scores +- [ ] Provides specific remediation steps +- [ ] Includes code examples +- [ ] Charts and visualizations + +## Mix and Match + +Combine features for bigger projects: + +**Project Idea 1: Complete API Security Platform** +- Combine Challenge 7 (plugin system) + Challenge 9 (report generation) + Challenge 8 (SSRF detection) +- Add web UI for scheduling scans +- Add email notifications when vulnerabilities found +- Result: Production-ready continuous API security testing + +**Project Idea 2: CI/CD Security Integration** +- Combine Challenge 4 (stored XSS) + Challenge 6 (XXE) + Challenge 2 (password strength) +- Build GitHub Action that runs scans on every commit +- Fail builds if critical vulnerabilities found +- Result: Security testing in development pipeline + +## Real World Integration Challenges + +### Integrate with Slack for Notifications + +**The goal:** +Send Slack messages when scans complete or vulnerabilities are found. + +**What you'll need:** +- Slack workspace with admin access +- Slack App with incoming webhook +- Understanding of Slack's Block Kit for rich messages + +**Implementation plan:** + +1. **Create Slack App** at https://api.slack.com/apps +2. **Enable Incoming Webhooks** and get webhook URL +3. **Add to config** in `.env`: +```bash +SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL +``` + +4. **Create notification service** in `backend/services/notification_service.py`: +```python +import requests +from typing import List + +class SlackNotifier: + def __init__(self, webhook_url: str): + self.webhook_url = webhook_url + + def send_scan_complete(self, scan: Scan): + """ + Send notification when scan completes + """ + summary = self._calculate_summary(scan) + + blocks = [ + { + "type": "header", + "text": { + "type": "plain_text", + "text": "🔒 Security Scan Complete" + } + }, + { + "type": "section", + "fields": [ + {"type": "mrkdwn", "text": f"*Target:*\n{scan.target_url}"}, + {"type": "mrkdwn", "text": f"*Date:*\n{scan.scan_date}"}, + ] + }, + { + "type": "section", + "fields": [ + {"type": "mrkdwn", "text": f"*Critical:* {summary.critical}"}, + {"type": "mrkdwn", "text": f"*High:* {summary.high}"}, + {"type": "mrkdwn", "text": f"*Medium:* {summary.medium}"}, + ] + } + ] + + if summary.critical > 0 or summary.high > 0: + blocks.append({ + "type": "section", + "text": { + "type": "mrkdwn", + "text": "⚠️ *Critical vulnerabilities found!* Review immediately." + } + }) + + requests.post(self.webhook_url, json={"blocks": blocks}) +``` + +5. **Hook into scan service** at end of `ScanService.run_scan()`: +```python +# After saving results +if settings.SLACK_WEBHOOK_URL: + notifier = SlackNotifier(settings.SLACK_WEBHOOK_URL) + notifier.send_scan_complete(scan) +``` + +**Watch out for:** +- Rate limits on Slack webhooks (1 message per second) +- Message size limits (3000 characters for text blocks) +- Error handling (webhook URL might be invalid) + +### Deploy to Production (AWS/DigitalOcean) + +**The goal:** +Get this running in production on real infrastructure. + +**What you'll learn:** +- Docker deployment to cloud +- Environment variable management in production +- SSL/TLS certificate setup +- Database backups and maintenance + +**Steps:** + +1. **Provision server** (DigitalOcean Droplet, AWS EC2) + - Ubuntu 24.04 LTS + - 2GB RAM minimum (4GB recommended) + - Open ports: 80 (HTTP), 443 (HTTPS), 22 (SSH) + +2. **Install Docker** on server: +```bash +ssh root@your-server-ip + +curl -fsSL https://get.docker.com -o get-docker.sh +sh get-docker.sh + +apt install docker-compose-plugin +``` + +3. **Clone project** and configure: +```bash +git clone https://github.com/yourusername/api-security-scanner.git +cd api-security-scanner + +cp .env.example .env +nano .env # Edit production settings +``` + +4. **Set production environment variables**: +```bash +# .env for production +SECRET_KEY=$(openssl rand -hex 32) +DEBUG=false +DATABASE_URL=postgresql://... +CORS_ORIGINS=https://yourdomain.com +``` + +5. **Set up SSL with Let's Encrypt**: +```bash +# Install certbot +apt install certbot python3-certbot-nginx + +# Get certificate +certbot --nginx -d yourdomain.com +``` + +6. **Update nginx config** in `conf/nginx/prod.nginx` to use SSL: +```nginx +server { + listen 443 ssl http2; + server_name yourdomain.com; + + ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem; + + # ... rest of config +} +``` + +7. **Deploy**: +```bash +docker compose -f compose.yml up -d --build +``` + +**Production checklist:** +- [ ] Changed `SECRET_KEY` to random value +- [ ] Set `DEBUG=false` +- [ ] Configured real database (not localhost) +- [ ] Set up SSL certificates +- [ ] Configured firewall (ufw allow 80,443,22) +- [ ] Set up backups (database dumps to S3/Spaces) +- [ ] Configure monitoring (Prometheus, Grafana, or cloud provider) +- [ ] Set up logging (centralized logs, log rotation) + +## Performance Challenges + +### Challenge: Handle 1000 concurrent scans + +**The goal:** +Scale the system to handle 1000 scans running simultaneously without crashing or slowing down. + +**Current bottleneck:** +Scans run synchronously in request handler. Database connections limited to 5 by default. Memory usage grows linearly with concurrent scans. + +**Optimization approaches:** + +**Approach 1: Task Queue with Celery** +- How: Move scans to background workers with Celery + Redis +- Gain: Non-blocking API, horizontal scaling, retry logic +- Tradeoff: Added complexity, need Redis infrastructure + +Implementation: +```bash +# Install Celery +pip install celery redis + +# backend/celery_app.py +from celery import Celery + +celery = Celery( + "scanner", + broker="redis://localhost:6379/0", + backend="redis://localhost:6379/0" +) + +@celery.task +def run_scan_async(scan_id: int): + """Run scan in background""" + db = SessionLocal() + scan = ScanRepository.get_by_id(db, scan_id) + # ... execute scan +``` + +**Approach 2: Increase Database Connections** +- How: Configure connection pooling in SQLAlchemy +- Gain: More concurrent database operations +- Tradeoff: Higher memory usage, Postgres connection limits + +```python +# core/database.py +engine = create_engine( + settings.DATABASE_URL, + pool_size=20, # Increased from 5 + max_overflow=40, # Up to 60 total connections + pool_pre_ping=True, +) +``` + +**Benchmark it:** +```bash +# Load testing with Apache Bench +ab -n 1000 -c 100 http://localhost:8000/scans/ \ + -H "Authorization: Bearer TOKEN" \ + -p scan_request.json +``` + +Target metrics: +- Throughput: 50+ requests/second +- Latency p95: <2 seconds (for scan creation, not execution) +- Error rate: <1% + +### Challenge: Reduce Scanner Memory Usage + +**The goal:** +Cut memory usage by 50% when running 100 concurrent scans. + +**Profile first:** +```python +# Add memory profiling +import tracemalloc + +tracemalloc.start() + +# Run scan +scanner.scan() + +current, peak = tracemalloc.get_traced_memory() +print(f"Current: {current / 1024 / 1024:.2f} MB") +print(f"Peak: {peak / 1024 / 1024:.2f} MB") + +tracemalloc.stop() +``` + +**Common optimization areas:** +- Store response bodies in memory (large responses eat RAM) +- Session objects not being cleaned up +- Evidence JSON storing entire responses + +Fix: +```python +# Instead of storing full response +evidence = {"response": response.text} # 100KB+ + +# Store summary +evidence = { + "status_code": response.status_code, + "length": len(response.text), + "excerpt": response.text[:500], # Just first 500 chars +} +``` + +## Security Challenges + +### Challenge: Add Webhook Signature Verification + +**What to implement:** +When sending scan results to webhooks, sign the payload so receivers can verify it's from your scanner. + +**Threat model:** +This protects against: +- Attacker sending fake scan results to webhook +- Man-in-the-middle tampering with webhook data + +**Implementation:** + +```python +import hmac +import hashlib + +class WebhookSigner: + def __init__(self, secret: str): + self.secret = secret.encode() + + def sign(self, payload: str) -> str: + """ + Create HMAC-SHA256 signature of payload + """ + signature = hmac.new( + self.secret, + payload.encode(), + hashlib.sha256 + ).hexdigest() + + return signature + + def verify(self, payload: str, signature: str) -> bool: + """ + Verify signature is valid + """ + expected = self.sign(payload) + return hmac.compare_digest(expected, signature) +``` + +Usage: +```python +# When sending webhook +signer = WebhookSigner(settings.WEBHOOK_SECRET) +payload_json = json.dumps(scan_data) +signature = signer.sign(payload_json) + +requests.post( + webhook_url, + json=scan_data, + headers={"X-Signature": signature} +) + +# Receiver verifies +received_signature = request.headers["X-Signature"] +if not signer.verify(payload_json, received_signature): + raise HTTPException(401, "Invalid signature") +``` + +### Challenge: Pass OWASP Top 10 Compliance + +**The goal:** +Make this project compliant with OWASP Top 10 2021. + +**Current gaps:** +- A01:2021-Broken Access Control: IDOR checks implemented ✓ +- A02:2021-Cryptographic Failures: Bcrypt for passwords ✓, but no HTTPS enforcement +- A03:2021-Injection: SQL injection detection ✓, but could add command injection +- A07:2021-Identification and Authentication Failures: JWT validation ✓ +- A09:2021-Security Logging and Monitoring Failures: No audit logging ❌ + +**Remediation:** + +Add audit logging: +```python +# models/AuditLog.py +class AuditLog(BaseModel): + __tablename__ = "audit_logs" + + user_id = Column(Integer, ForeignKey("users.id")) + action = Column(String(100)) # "scan_created", "login_success" + ip_address = Column(String(45)) # Support IPv6 + user_agent = Column(String(255)) + details = Column(JSON) + +# Log every important action +def log_action(user_id: int, action: str, request: Request, **details): + log = AuditLog( + user_id=user_id, + action=action, + ip_address=request.client.host, + user_agent=request.headers.get("User-Agent"), + details=details, + ) + db.add(log) + db.commit() +``` + +## Contribution Ideas + +Finished a challenge? Share it back: + +1. **Fork the repo** at github.com/yourusername/api-security-scanner +2. **Implement your extension** in a feature branch (`git checkout -b feature/xxe-scanner`) +3. **Document it** - Add to learn folder, update README +4. **Submit a PR** with: + - Implementation code + - Unit tests (minimum 80% coverage) + - Integration tests + - Documentation (docstrings, README updates) + - Example usage + +Good extensions might get merged into the main project. + +## Challenge Yourself Further + +### Build Something New + +Use the concepts you learned here to build: + +- **GraphQL security scanner** - Test for introspection leaks, query depth limits, batching attacks +- **WebSocket security tester** - Test for injection, authentication, rate limiting in WebSocket connections +- **Cloud API scanner** - Test AWS, Azure, GCP APIs for misconfiguration (public S3 buckets, open databases) + +### Study Real Implementations + +Compare your implementation to production tools: + +- **Burp Suite** - Study how their active scanner detects SQLi (multiple techniques, adaptive testing) +- **OWASP ZAP** - Look at their scanner plugin architecture +- **Nuclei** - Check out their YAML-based template system for custom checks + +Read their code (many are open source), understand their tradeoffs, steal their good ideas. + +### Write About It + +Document your extension: + +- Blog post explaining what you built and why ("Adding XXE Detection to an API Scanner") +- Tutorial for others to follow ("How to Build a CORS Misconfiguration Scanner") +- Comparison with alternative approaches ("Callback-based SSRF Detection vs Timing-based") + +Teaching others is the best way to verify you understand it. + +## Getting Help + +Stuck on a challenge? + +1. **Debug systematically** + - What did you expect to happen? + - What actually happened? + - What's the smallest test case that reproduces it? + - What have you already tried? + +2. **Read the existing code** + - SQLi scanner does time-based detection (similar to SSRF timing) + - Auth scanner does multi-step testing (similar to stored XSS) + - Base scanner has retry logic you can reuse + +3. **Search for similar problems** + - Stack Overflow with tags: python, fastapi, security-testing + - GitHub issues in similar projects (ZAP, Nuclei, SQLMap) + - InfoSec forums like /r/netsec, /r/AskNetsec + +4. **Ask for help with context** + - Post in project discussions or issues + - Include: what you're trying to build, what you tried, what happened, what you expected + - Provide code snippets and error messages + - Don't just paste error messages without explanation + +## Challenge Completion Tracker + +Track your progress: + +- [ ] Easy Challenge 1: CORS Detection +- [ ] Easy Challenge 2: Password Strength +- [ ] Easy Challenge 3: Response Time Monitoring +- [ ] Intermediate Challenge 4: Stored XSS +- [ ] Intermediate Challenge 5: Advanced Rate Limit Bypass +- [ ] Intermediate Challenge 6: XXE Detection +- [ ] Advanced Challenge 7: Plugin System +- [ ] Advanced Challenge 8: Blind SSRF +- [ ] Expert Challenge 9: Report Generation + +**Bonus challenges:** +- [ ] Slack Integration +- [ ] Production Deployment +- [ ] Performance: 1000 Concurrent Scans +- [ ] Security: Webhook Signatures +- [ ] Compliance: OWASP Top 10 + +Completed all of them? You've mastered API security testing. Time to build something new, contribute to open source security tools, or apply these skills professionally in penetration testing or security engineering roles. diff --git a/TEMPLATES/fullstack-template b/TEMPLATES/fullstack-template index 0eec274f..ecbb534e 160000 --- a/TEMPLATES/fullstack-template +++ b/TEMPLATES/fullstack-template @@ -1 +1 @@ -Subproject commit 0eec274fba59ddd373080ae961dd4497eaf2e454 +Subproject commit ecbb534e85e8e381e6e89aece4b786db8f7ad172