RomanNum3ral
/
Cybersecurity-Projects
mirror of https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Revise overview for API Rate Limiter project 

Replaced security considerations with an overview of the API Rate Limiter project, detailing its features, algorithms, and importance in preventing attacks.
This commit is contained in:
Carter Perez 2026-02-04 02:24:14 -05:00 committed by GitHub
parent 5e9213569b
commit 945dcb4571
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 137 additions and 325 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

137
PROJECTS/advanced/api-rate-limiter/learn/00-OVERVIEW.md Normal file
View File

@ -0,0 +1,137 @@
# API Rate Limiter (fastapi-420)
## What This Is
A production-ready rate limiting library for FastAPI that implements three battle-tested algorithms (sliding window, token bucket, fixed window) with Redis or in-memory storage, advanced client fingerprinting, and a three-layer defense system that scales from per-user limits to global DDoS protection.
## Why This Matters
On February 28, 2018, GitHub experienced a 1.35 Tbps DDoS attack, the largest recorded at the time. The attack lasted only 10 minutes because their rate limiting and traffic filtering systems kicked in. Without proper rate limiting, even legitimate traffic spikes can take down APIs. When the iPhone 15 launched, Apple's activation servers struggled under load from millions of simultaneous requests. Rate limiting isn't just about security, it's about availability.
**Real world scenarios where this applies:**
- **Preventing credential stuffing** - In 2019, Dunkin' Donuts suffered a credential stuffing attack where attackers made millions of login attempts using stolen credentials. Rate limiting login endpoints to 3-5 attempts per minute blocks this attack pattern completely.
- **API cost control** - If you're calling expensive third-party APIs or running ML inference, a single misbehaving client can cost you thousands of dollars in a few hours. Rate limits protect your budget.
- **Fair resource distribution** - In 2020, scalpers used bots to buy all the PS5 inventory within seconds of release. E-commerce sites now use rate limiting on checkout endpoints to give real customers a fair chance.
## What You'll Learn
This project teaches you how production rate limiting actually works. By building it yourself, you'll understand:
**Security Concepts:**
- **Three-layer defense architecture** - Why you need per-user limits (stop individual abuse), per-endpoint limits (prevent endpoint-specific attacks), and global limits (DDoS protection). One layer isn't enough. The 2016 Dyn DNS attack bypassed per-user limits by using millions of IoT devices.
- **Client fingerprinting** - How to identify clients reliably when IPs aren't unique (NAT, proxies, mobile networks). You'll learn why fingerprinting just IP addresses is broken, especially for IPv6 where users control entire /64 blocks (18 quintillion addresses).
- **Algorithm tradeoffs** - Why sliding window is recommended for production (accurate with constant memory), when token bucket makes sense (burst tolerance), and why fixed window has a boundary problem that attackers exploit.
**Technical Skills:**
- **Atomic operations with Lua scripts** - Rate limiting requires atomic read-modify-write operations. You'll see how Redis Lua scripts (stored in `src/fastapi_420/storage/lua/`) guarantee correctness even under heavy concurrent load.
- **ASGI middleware integration** - How to hook into FastAPI's request lifecycle at `src/fastapi_420/middleware.py:37-106` to apply rate limits globally without modifying every endpoint.
- **Async Python patterns** - Managing concurrent requests with asyncio locks (`src/fastapi_420/storage/memory.py:44`), background cleanup tasks, and graceful shutdown.
**Tools and Techniques:**
- **Redis for distributed systems** - The in-memory storage at `src/fastapi_420/storage/memory.py` works for single instances, but production systems need Redis (`src/fastapi_420/storage/redis_backend.py`) to share state across multiple servers.
- **Pydantic settings validation** - The config system at `src/fastapi_420/config.py:120-164` validates environment variables at startup, failing fast if misconfigured instead of causing runtime errors.
- **Testing rate limiters** - The test suite shows patterns for testing time-dependent code using explicit timestamps (`tests/test_algorithms.py:143`) rather than sleeping in tests.
## Prerequisites
Before starting, you should understand:
**Required knowledge:**
- **Python async/await** - The entire codebase is async. You need to know the difference between `async def` and `def`, when to use `await`, and what `asyncio.create_task()` does.
- **FastAPI basics** - Understand middleware, dependency injection with `Depends()`, and how request/response flow works. If you haven't built a FastAPI app before, do that first.
- **HTTP headers** - Know what `X-Forwarded-For`, `User-Agent`, and `Authorization` headers are for. The fingerprinting system relies heavily on header analysis.
**Tools you'll need:**
- **Python 3.12+** - Uses new type hints like `list[str]` instead of `List[str]`
- **Redis (optional)** - For production use. Install with Docker: `docker run -d -p 6379:6379 redis:7-alpine`
- **httpx or curl** - For testing API endpoints
**Helpful but not required:**
- **Redis Lua scripting** - The project includes pre-written Lua scripts, but understanding them helps
- **OAuth/JWT basics** - The auth extractor (`src/fastapi_420/fingerprinting/auth.py`) can parse JWT tokens
## Quick Start
Get the project running locally:
```bash
# Clone and navigate
cd PROJECTS/advanced/api-rate-limiter
# Install dependencies
pip install -e . --break-system-packages
# Optional: Start Redis for production-like testing
docker-compose -f examples/docker-compose.yml up -d
# Run the example app
python examples/app.py
```
Expected output: Server starts on http://0.0.0.0:8000 with multiple rate-limited endpoints. Try hitting the strict login endpoint:
```bash
# First 3 requests work
curl http://localhost:8000/auth/login -X POST -d "username=test&password=test"
# 4th request gets HTTP 420
curl -i http://localhost:8000/auth/login -X POST -d "username=test&password=test"
```
You'll see `HTTP/1.1 420 Enhance Your Calm` with `Retry-After` and `RateLimit-*` headers.
## Project Structure
```
api-rate-limiter/
├── src/fastapi_420/
│ ├── algorithms/ # Rate limiting algorithms
│ │ ├── sliding_window.py # Recommended default (99.997% accurate)
│ │ ├── token_bucket.py # For burst tolerance
│ │ └── fixed_window.py # Simple but has boundary issues
│ ├── storage/ # Storage backends
│ │ ├── memory.py # In-memory (single instance)
│ │ ├── redis_backend.py # Redis (distributed)
│ │ └── lua/ # Atomic Lua scripts
│ ├── fingerprinting/ # Client identification
│ │ ├── ip.py # IP extraction with IPv6 /64 normalization
│ │ ├── headers.py # User-Agent, Accept-* headers
│ │ ├── auth.py # JWT, API keys, sessions
│ │ └── composite.py # Combines all methods
│ ├── defense/ # Multi-layer protection
│ │ ├── layers.py # User/Endpoint/Global limits
│ │ └── circuit_breaker.py# DDoS protection
│ ├── limiter.py # Main RateLimiter class
│ ├── middleware.py # ASGI middleware
│ ├── dependencies.py # FastAPI dependency injection
│ ├── config.py # Settings with Pydantic
│ └── types.py # Data structures
├── examples/
│ └── app.py # Full working example
└── tests/ # Comprehensive test suite
```
## Next Steps
1. **Understand the concepts** - Read [01-CONCEPTS.md](./01-CONCEPTS.md) to learn rate limiting fundamentals, algorithm differences, and why fingerprinting matters
2. **Study the architecture** - Read [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) to see the three-layer defense system and data flow
3. **Walk through the code** - Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) for line-by-line implementation details
4. **Extend the project** - Read [04-CHALLENGES.md](./04-CHALLENGES.md) for ideas like adding geolocation blocking, CAPTCHA integration, or custom algorithms
## Common Issues
**"Redis connection failed" but I want to use memory storage**
```
REDIS_URL not set, fallback to memory storage enabled
```
Solution: This is fine for development. The `FALLBACK_TO_MEMORY=True` setting (default) automatically switches to in-memory storage when Redis is unavailable. For production, set `REDIS_URL` environment variable.
**"Rate limit not working for requests from different IPs"**
Solution: Check if you're behind a proxy. If using nginx or cloudflare, set `TRUST_X_FORWARDED_FOR=True` in your settings so the IP extractor reads `X-Forwarded-For` header instead of the proxy IP. See `src/fastapi_420/fingerprinting/ip.py:52-68` for proxy handling logic.
**"Getting HTTP 420 immediately on first request"**
Check the circuit breaker threshold. If `CIRCUIT_THRESHOLD` is too low (like 100) and you're load testing, the global circuit breaker might be triggering. See `src/fastapi_420/defense/circuit_breaker.py:45-55` for threshold logic.
## Related Projects
If you found this interesting, check out:
- **network-traffic-analyzer** - Builds on packet inspection to detect DDoS attacks before they hit your API
- **docker-security-audit** - Shows how to audit rate limiting configurations in containerized deployments
- **bug-bounty-platform** - Implements rate limiting for submission endpoints to prevent spam

325
PROJECTS/advanced/api-rate-limiter/learn/security.md
View File

@ -1,325 +0,0 @@
# Security Considerations and Common Mistakes
Rate limiting seems simple until you realize how many ways it can fail. This doc covers real vulnerabilities, attack vectors, and the mistakes that show up in CVEs over and over again.
## Real CVEs: What Goes Wrong
### CVE-2023-2531 (AzuraCast) - X-Forwarded-For Spoofing
CVSS 7.5. The `getIp()` method trusted the X-Forwarded-For header without validation.
```python
# What they did (broken)
def get_ip(request):
if 'X-Forwarded-For' in request.headers:
return request.headers['X-Forwarded-For'].split(',')[0]
return request.remote_addr
```
An attacker just sends:
```
X-Forwarded-For: 1.2.3.4
```
Now every request looks like it comes from a different IP. Rate limiting completely bypassed.
The fix is to only trust X-Forwarded-For from known proxy IPs:
```python
TRUSTED_PROXIES = {'10.0.0.1', '10.0.0.2'}
def get_ip(request):
if request.remote_addr in TRUSTED_PROXIES:
forwarded = request.headers.get('X-Forwarded-For', '')
if forwarded:
return forwarded.split(',')[0].strip()
return request.remote_addr
```
This package has `TRUST_X_FORWARDED_FOR=False` by default. You have to explicitly enable it and should also configure `TRUSTED_PROXIES`.
### CVE-2023-46745 (LibreNMS) - Inconsistent Limit Application
Rate limiting was applied to POST `/login` but not GET `/login`. The login form could be submitted via GET with query parameters, bypassing limits entirely.
```
POST /login <- rate limited
GET /login?username=x&password=y <- not rate limited
```
The lesson: rate limit the action, not just a specific method. This package includes the HTTP method in the endpoint key by default (`GET:/api/data` vs `POST:/api/data`), but you need to make sure all paths to sensitive operations are covered.
### GHSA-984p-xq9m-4rjw (express-brute) - Race Conditions
Non-atomic counter updates allowed concurrent requests to bypass limits. Classic TOCTOU (time of check to time of use) bug.
```javascript
// Broken: gap between read and write
const count = await store.get(key);
if (count < limit) {
await store.set(key, count + 1); // Race condition here
next();
}
```
Between the get and set, dozens of requests can sneak through. Under load, this completely defeats rate limiting.
This is why we use Redis Lua scripts. The entire check-and-increment runs atomically.
## Attack Vectors to Understand
### Header Spoofing
Any header can be spoofed. X-Forwarded-For, X-Real-IP, CF-Connecting-IP. Attackers know which headers your reverse proxy sets.
The only safe approach:
1. Know exactly which proxies sit in front of your app
2. Only trust forwarded headers from those specific IPs
3. Take the rightmost IP added by your trusted proxy, not the leftmost (which the client controls)
```
X-Forwarded-For: attacker-spoofed, real-proxy-added
^ ^
client controls proxy controls
```
### GraphQL Batching
This one catches people off guard. GraphQL lets you batch multiple operations in one HTTP request:
```json
[
{"query": "mutation { login(user: \"x\", pass: \"a\") }"},
{"query": "mutation { login(user: \"x\", pass: \"b\") }"},
{"query": "mutation { login(user: \"x\", pass: \"c\") }"}
]
```
That is one HTTP request but three login attempts. If you rate limit by request, you are letting 100 password guesses through per "request."
GraphQL endpoints need operation-level limiting:
- Count mutations, not requests
- Limit query depth and complexity
- Cap batch sizes (5-10 operations max)
### Slowloris and Connection Exhaustion
Rate limiting usually focuses on completed requests. But what about requests that never complete?
Slowloris attacks open many connections and send data very slowly, tying up server resources without triggering request-based limits.
This is outside the scope of application-level rate limiting. You need:
- Connection timeouts at the reverse proxy level
- Per-IP connection limits in nginx/haproxy
- Request body size limits
### IPv6 Address Rotation
Covered in the architecture doc, but worth repeating: a single residential IPv6 user controls 18 quintillion addresses in their /64 prefix.
```
2001:db8:abcd:1234::1
2001:db8:abcd:1234::2
2001:db8:abcd:1234::ffff:ffff:ffff:ffff
```
All controlled by the same person. If you rate limit per IP, they just rotate.
Normalize to /64 prefix before rate limiting:
```python
2001:db8:abcd:1234::1 -> 2001:db8:abcd:1234::
```
## Redis Security
Redis is a common target. Default installations often have no authentication and listen on all interfaces.
### CVE-2025-49844 (RediShell) - CVSS 10.0
Remote code execution through Lua sandbox escape. Affected around 330,000 exposed instances.
Protection:
1. Always require authentication (`requirepass`)
2. Bind to localhost only unless you specifically need network access
3. Use ACLs to restrict what the rate limiter can do
4. Disable dangerous commands
```redis
# Create a restricted user for rate limiting
ACL SETUSER ratelimiter on >strongpassword ~rl:* +EVALSHA +GET +SET +INCR +EXPIRE +TTL
```
This user can only:
- Run EVALSHA (our Lua scripts)
- Access keys starting with `rl:`
- Do basic counter operations
They cannot FLUSHALL, CONFIG, DEBUG, or do anything administrative.
### Key Expiration
Every rate limit key must have a TTL. Without expiration:
- Memory grows forever
- Old keys from users who never return pile up
- Attackers can create millions of keys by rotating identifiers
We set TTL to 2x the window size. A 60 second window gets a 120 second TTL. This ensures the key lives long enough for the sliding window to work, but expires after.
## Common Implementation Mistakes
### 1. Not Testing Under Load
Unit tests pass. Integration tests pass. Production gets slammed and requests slip through.
Race conditions only appear under concurrent load. You need to actually test with hundreds of simultaneous requests:
```python
async def test_concurrent_limit():
tasks = [make_request() for _ in range(200)]
results = await asyncio.gather(*tasks)
allowed = sum(1 for r in results if r.status == 200)
blocked = sum(1 for r in results if r.status == 420)
# With limit of 100, should see ~100 allowed, ~100 blocked
assert 95 <= allowed <= 105
```
### 2. Blocking Logging
```python
# Bad: blocks the event loop
logger.info(f"Rate limited {ip}") # Synchronous I/O
```
Under attack, you might rate limit thousands of requests per second. Synchronous logging blocks the event loop and slows everything down.
Use async logging, batch writes, or background tasks:
```python
# Better: non-blocking
background_tasks.add_task(log_violation, ip, endpoint)
```
### 3. Forgetting TTLs
```python
# Bad: key lives forever
await redis.incr(key)
# Good: key expires
await redis.incr(key)
await redis.expire(key, window_seconds * 2)
```
Actually even better, do it atomically in Lua so there is no gap where the key exists without a TTL.
### 4. Integer Overflow
At extreme scale, counters can overflow. Python handles big integers fine, but Redis stores numbers as 64-bit signed integers. Max value is about 9.2 quintillion.
You probably will not hit this, but if you are doing high volume with long windows, consider it.
### 5. Clock Drift
Distributed systems with multiple app servers can have clock skew. Server A thinks it is 10:00:00, Server B thinks it is 10:00:02.
For sliding window, use Redis server time instead of local time:
```python
server_time = await redis.time() # Returns (seconds, microseconds)
```
Now all servers agree on what time it is.
### 6. Fail Closed Without Monitoring
If you choose to fail closed (block all requests when Redis is down), you absolutely need monitoring and alerting on Redis availability.
Otherwise Redis goes down at 3am, your API becomes completely unavailable, and you do not find out until customers complain.
### 7. Overly Strict Fingerprinting
STRICT fingerprinting mode includes Accept headers and auth tokens. But this can cause problems:
- Same user on different browsers gets different fingerprints
- API clients with slightly different Accept headers look like different users
- Token rotation creates new fingerprints
Start with NORMAL (IP + User-Agent) and only go stricter if you have a specific problem to solve.
## OWASP API4:2023 - Unrestricted Resource Consumption
Rate limiting directly addresses OWASP API4, which identifies these failure modes:
- Missing or inadequate rate limiting
- Lack of payload size limits
- No execution timeouts
- Unlimited resource allocation per request
A complete defense includes:
- Request rate limits (what this package does)
- Payload size limits (configure in your web server)
- Execution timeouts (configure in your ASGI server)
- Memory limits (configure at the container/process level)
Rate limiting is one layer. You need the others too.
## Testing Your Implementation
### Verify limits actually work
```bash
# Send 110 requests, expect ~100 to succeed
for i in {1..110}; do
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/api/test
done | sort | uniq -c
```
Should see roughly 100 responses of 200 and 10 responses of 420.
### Verify headers are present
```bash
curl -i http://localhost:8000/api/test
```
Look for:
```
RateLimit-Limit: 100
RateLimit-Remaining: 99
RateLimit-Reset: 58
```
### Test concurrent requests
```python
import asyncio
import httpx
async def blast():
async with httpx.AsyncClient() as client:
tasks = [client.get("http://localhost:8000/api/test") for _ in range(200)]
responses = await asyncio.gather(*tasks)
codes = [r.status_code for r in responses]
print(f"200: {codes.count(200)}, 420: {codes.count(420)}")
asyncio.run(blast())
```
### Test failover
```bash
# Stop Redis
docker stop redis
# Requests should still work (fail open)
curl http://localhost:8000/api/test
# Check logs for fallback warning
```
## Reporting Security Issues
If you find a security vulnerability in this package, do not open a public issue. Email the maintainer directly so it can be fixed before disclosure.