siem dashbaord learn folder
This commit is contained in:
CarterPerez-dev
parent
d135197348
commit
539dd725b9
|
|
@ -12,19 +12,18 @@ A full-stack Security Information and Event Management dashboard with a built-in
|
|||
|
||||
Real-time overview of ingested events, active alerts, severity distribution, and top source IPs.
|
||||
|
||||
|
||||
|
||||
|
||||
## Log Viewer
|
||||
|
||||
Paginated, filterable log table showing every event ingested by the system — firewall, auth, IDS, endpoint, DNS, and proxy logs. Click any row to inspect the full normalized payload.
|
||||
|
||||
|
||||
|
||||
|
||||
## Alerts
|
||||
|
||||
Alerts fire when correlation rules detect suspicious patterns in the event stream. Each alert shows the matched rule, severity, status, grouped source, and the specific events that triggered it. Analysts can acknowledge, investigate, resolve, or mark as false positive.
|
||||
|
||||
|
||||
|
||||
|
||||
## Correlation Rules
|
||||
|
||||
|
|
@ -34,7 +33,7 @@ Define detection logic using three rule types:
|
|||
- **Sequence** — fire when an ordered series of event patterns appears (e.g. failed logins followed by a success)
|
||||
- **Aggregation** — fire when distinct values of a field exceed a threshold (e.g. one IP hitting 10+ ports)
|
||||
|
||||
|
||||
|
||||
|
||||
## Scenario Engine
|
||||
|
||||
|
|
@ -88,9 +87,9 @@ YAML Playbook ──→ Scenario Thread (replays events with timing)
|
|||
## Quick Start
|
||||
|
||||
```bash
|
||||
git clone <repo-url>
|
||||
cd siem-dashboard
|
||||
docker compose -f dev.compose.yml up --build
|
||||
git clone https://github.com/CarterPerez-dev/Cybersecurity-Projects
|
||||
cd PROJECTS/intermediate/siem-dashboard
|
||||
Just dev-up
|
||||
```
|
||||
|
||||
- **App:** http://localhost:8431
|
||||
|
|
|
|||
|
Before Width: | Height: | Size: 85 KiB After Width: | Height: | Size: 85 KiB |
|
Before Width: | Height: | Size: 132 KiB After Width: | Height: | Size: 132 KiB |
|
Before Width: | Height: | Size: 59 KiB After Width: | Height: | Size: 59 KiB |
|
Before Width: | Height: | Size: 62 KiB After Width: | Height: | Size: 62 KiB |
|
Before Width: | Height: | Size: 125 KiB After Width: | Height: | Size: 125 KiB |
|
|
@ -1,70 +0,0 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
fix_mypy.py
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def remove_unused_ignores(file_path: Path, lines_with_unused: list[int]) -> None:
|
||||
"""Remove unused type: ignore comments from specific lines"""
|
||||
with open(file_path) as f:
|
||||
lines = f.readlines()
|
||||
|
||||
for line_num in lines_with_unused:
|
||||
idx = line_num - 1
|
||||
if idx < len(lines):
|
||||
line = lines[idx]
|
||||
line = re.sub(r' # type: ignore\[no-untyped-call\]', '', line)
|
||||
line = re.sub(r' # type: ignore\[no-any-return\]', '', line)
|
||||
line = re.sub(r' # type: ignore\[attr-defined\]', '', line)
|
||||
line = re.sub(r' # type: ignore\[no-untyped-call, no-any-return\]', '', line)
|
||||
line = re.sub(r' # type: ignore', '', line)
|
||||
lines[idx] = line
|
||||
|
||||
with open(file_path, 'w') as f:
|
||||
f.writelines(lines)
|
||||
|
||||
# Clean up User.py
|
||||
remove_unused_ignores(
|
||||
Path("app/models/User.py"),
|
||||
[57, 64, 71, 78, 97, 109, 117, 127, 135, 142, 149, 155]
|
||||
)
|
||||
|
||||
# Clean up ScenarioRun.py
|
||||
remove_unused_ignores(
|
||||
Path("app/models/ScenarioRun.py"),
|
||||
[63, 71, 80, 81, 89, 97, 104, 111, 120, 127]
|
||||
)
|
||||
|
||||
# Clean up LogEvent.py
|
||||
remove_unused_ignores(
|
||||
Path("app/models/LogEvent.py"),
|
||||
[100, 113, 156, 163, 170, 217, 241, 280]
|
||||
)
|
||||
|
||||
# Clean up CorrelationRule.py
|
||||
remove_unused_ignores(
|
||||
Path("app/models/CorrelationRule.py"),
|
||||
[58]
|
||||
)
|
||||
|
||||
# Clean up Alert.py
|
||||
remove_unused_ignores(
|
||||
Path("app/models/Alert.py"),
|
||||
[88, 125, 132]
|
||||
)
|
||||
|
||||
# Clean up scenario_ctrl.py
|
||||
remove_unused_ignores(
|
||||
Path("app/controllers/scenario_ctrl.py"),
|
||||
[43, 53, 63, 74]
|
||||
)
|
||||
|
||||
# Clean up rule_ctrl.py
|
||||
remove_unused_ignores(
|
||||
Path("app/controllers/rule_ctrl.py"),
|
||||
[23, 48, 61, 70]
|
||||
)
|
||||
|
||||
print("Removed unused type: ignore comments")
|
||||
|
|
@ -1,9 +1,41 @@
|
|||
"""
|
||||
©AngelaMos | 2026
|
||||
wsgi.py
|
||||
|
||||
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠁⣰⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠿⣿⣿⣿⣿⣁⣭⣝⢿⡋⠽⢯⣝⡻⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣮⠻⣿⣿⣿⣷⡝⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
|
||||
⣿⣿⣿⣿⣿⣿⣿⣿⣿⠏⢠⣾⣿⣿⣿⣿⣿⣿⣿⣿⡿⣫⢔⣫⣥⣾⣿⣿⣿⣏⠙⣿⣧⢻⣿⣿⣶⣭⣓⢬⡻⣿⣿⣿⣿⣿⣿⣿⣿⣷⡙⣿⣿⣿⣿⣌⢿⣿⣿⣿⣿⣿⣿⣿⣿
|
||||
⣿⣿⣿⣿⣿⣿⣿⣿⠃⣴⣿⣿⣿⣿⣿⣿⣿⣿⣿⠟⢔⣵⣿⣿⣿⣿⣿⣿⣿⣿⣧⡈⣿⣧⢻⣿⣿⣿⣿⣷⣄⡈⠻⣿⣿⣿⣿⣿⣿⣿⣷⡘⣿⣿⣿⣿⣆⢿⣿⣿⣿⣿⣿⣿⣿
|
||||
⣿⣿⣿⣿⣿⣿⡿⠃⣼⣿⣿⣿⣿⣿⣿⣿⣿⣿⠏⢢⣿⣿⣿⣿⣿⣿⣿⢛⣿⣿⣿⣿⢹⣿⡘⣿⣿⣿⣿⣿⣿⣿⣦⡙⢿⣿⣿⣿⣿⣿⣿⣷⡸⣿⣿⣿⣿⡎⢿⣿⣿⣿⣿⣿⣿
|
||||
⣿⣿⣿⣿⣿⣿⢇⣼⣿⣿⣿⣿⣿⣿⣿⣿⣿⡏⣰⣿⣿⣿⣿⣿⡿⢻⠏⣾⣿⣿⣿⣿⡇⣿⡇⡹⣿⣿⣿⣿⣿⣿⣿⣿⣌⢿⡟⢻⣿⢈⢧⠙⣧⢹⡟⢿⣿⣿⡼⣿⣿⣿⣿⣿⣿
|
||||
⣿⣿⣿⣿⣿⡏⣼⡟⢩⣿⣿⢛⣿⣿⣿⣿⣿⢡⣿⣿⣿⣿⣿⡟⢡⣿⣸⣿⣿⣿⣿⣿⣷⣿⣇⣳⡿⣿⣿⠻⣿⡉⢻⣿⣿⡎⢿⡴⣝⣋⡌⣎⠸⡏⢿⡌⣿⣿⣧⢹⣿⣿⣿⣿⣿
|
||||
⣿⣿⣿⣿⡟⣈⣎⠆⢃⣿⢇⡾⢻⣿⣿⣿⡏⣾⡟⣿⠿⣿⡿⠁⣼⡇⣿⣿⣿⣿⣿⣿⣿⣿⣿⢰⡸⣌⣿⣦⢊⠧⢱⡹⣿⣿⡌⣷⣹⣿⡟⠘⡦⢹⡸⣿⡘⣿⣿⡞⣿⣿⣿⣿⣿
|
||||
⣿⣿⣿⡿⡘⣼⠘⡼⣼⡟⡞⣰⣿⣿⣿⣿⠸⣋⢶⡿⢰⣿⢣⣧⣿⢡⣿⢣⢻⢏⣿⣿⣿⣿⡇⣼⡇⡑⣎⠛⣧⡳⠺⣟⠜⣿⣿⡸⣧⢿⣏⣆⢱⡘⡧⣿⣿⣿⣿⡇⢻⣿⣿⣿⣿
|
||||
⣿⣿⣿⢡⢱⣇⣸⢣⣿⢰⣿⣿⣸⣿⣿⡏⣆⣿⢸⡇⢸⠏⡞⢸⣿⢸⣿⣼⢰⣾⣏⣿⡟⡿⠇⣿⣿⢰⠹⡨⡜⢷⡁⢬⣧⣸⣿⣇⢻⣼⣿⣿⣷⣿⣧⢸⣿⣿⣿⣿⢸⣿⣿⣿⣿
|
||||
⣿⣿⡟⣾⣿⣿⡟⣸⣿⢋⣿⣿⣿⣿⣿⢡⡏⡼⢸⡷⣿⠘⢸⢸⣿⣬⣿⣿⢘⣥⢻⡟⠇⣧⢳⣿⣿⠈⣷⠳⣿⣎⢿⣿⣿⣿⣿⣿⡜⣿⣿⣿⣿⣿⣷⡜⣿⣿⣿⣿⡎⣿⣿⣿⣿
|
||||
⣿⣿⢡⣿⣿⣿⡇⣿⣿⣼⣿⣿⣿⣿⣿⢸⣿⡇⣿⣷⠇⠀⡌⢸⣿⢻⣿⣿⢸⣿⡼⢡⢸⡟⣸⣿⣿⡎⣌⢷⡹⣿⣦⠻⣿⣿⣿⣿⣧⢿⣿⣿⣿⣿⣿⡇⣿⣿⣿⣿⡇⣿⣿⣿⣿
|
||||
⣿⣿⣾⣿⣿⣿⢷⣿⣿⣿⣿⣿⣿⣿⡿⣿⣿⠇⣿⣿⠀⣼⣇⢸⣿⢸⣿⣿⢸⣿⣷⡏⣿⢱⣿⢻⣿⡇⣎⢣⡱⡜⢿⣷⡈⠻⣿⣿⣿⢸⣿⣿⣿⣿⣿⣧⣿⣿⣿⣿⣷⣿⣿⣿⣿
|
||||
⣿⣟⣿⣿⣿⣏⣼⣿⣿⣿⣿⣿⣿⣿⡇⣿⡏⠀⣿⣿⠀⣿⣿⢸⣿⡆⣿⣿⣸⣿⡟⡼⢃⣿⣿⢸⣿⢱⣿⣷⡕⢌⠢⡻⣿⣮⡪⡙⢿⠀⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
|
||||
⣿⣯⣿⣿⡿⢸⣿⣿⣿⣿⣿⣿⣿⣿⡇⣿⠉⣴⢹⣿⣸⣿⣿⡏⣿⣷⢿⣿⡏⡿⠑⠁⢸⣿⡿⢸⣿⢸⣿⣿⣿⣷⣕⠌⠪⣿⣿⡌⠳⡆⣿⣿⢻⣿⣿⣿⡙⣿⣿⣿⣿⣿⣿⣿⣿
|
||||
⣿⣿⣿⣿⠃⢳⣼⣿⣿⣿⣿⣿⣿⣿⡇⡏⢸⣿⢸⣿⣿⣿⣿⣷⣸⣿⠈⣿⠇⡀⢀⣾⢸⣿⡇⠼⠟⣸⠿⠿⠿⢛⣛⣥⣤⡀⠐⠶⣷⡄⣿⡿⣸⣿⣿⣿⣼⠸⣿⣿⣿⣿⣿⣿⣿
|
||||
⡏⢸⣿⣿⠀⠰⣾⣿⣿⣿⡇⣿⣿⣿⡇⠃⣾⣿⡆⢍⢛⠻⠿⣿⣧⢻⡌⡁⠞⠁⢿⣿⢸⣿⣧⢸⣆⣐⡤⢞⣛⠻⠿⠍⠉⠙⠁⠈⠀⠀⣚⡃⢿⣿⣿⣿⣿⡇⣿⣿⣿⣿⣿⣿⣿
|
||||
⣸⢸⣿⡇⠀⡗⣼⣿⣿⣿⣧⢹⣿⣿⣇⠀⣿⣿⣿⡘⠸⡿⢷⣶⣶⣦⠲⢲⡐⢷⡈⡻⣼⡿⣸⠘⣸⣿⣞⠉⢀⡀⡀⠀⠀⠀⢰⣾⣿⠂⢉⢀⣿⣿⣿⣿⣿⢣⣿⣿⣿⣿⣿⣿⣿
|
||||
⢧⢸⣿⡇⠃⡇⠟⣿⣿⣿⣿⡞⣿⣿⣿⠀⡿⠇⠉⢁⠀⢀⠀⣀⠀⠀⡀⠀⠈⢎⠳⡄⣿⢡⠇⣰⣿⣿⣧⣾⣿⣇⠑⠮⠔⣁⣾⣿⠛⣠⡾⢸⣿⣿⣿⡏⢣⣿⣿⣿⣿⢿⣿⣿⣿
|
||||
⣾⣾⣿⡇⠀⡇⠀⣯⡻⣿⣿⣷⡸⣿⣿⡆⣶⣄⠐⢿⣷⣿⡄⠱⣄⡠⠆⣼⣌⣿⡷⡰⠃⠋⣴⣿⣿⣿⣿⣿⣿⣿⣯⣭⣭⣤⣴⣿⢣⣿⢡⢸⣿⣿⣿⡇⣿⣿⣿⣿⣿⣸⣿⣿⣿
|
||||
⣿⡿⣿⡇⡇⡇⢀⢹⣇⣿⣿⣿⣷⡹⣿⣿⡸⣿⣷⣤⠛⣿⠿⠶⠶⢒⣺⣿⣿⡿⢁⣵⣿⣶⣬⣹⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⢇⣾⢃⣸⣼⣿⣿⣿⢠⣿⣿⣿⣿⣿⢿⣿⣿⣿
|
||||
⣿⠁⣿⡇⠃⡇⢸⠌⡸⡘⣿⣿⣿⣷⡘⣿⣷⡹⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣴⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡟⡼⣣⡟⡇⣿⣿⣿⢁⣸⣿⣿⣿⣿⣿⢸⣿⣿⣿
|
||||
⡛⢰⢻⣧⠀⡇⡸⢀⢣⣷⢻⣿⣿⣿⣿⡌⠻⣷⡽⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠏⢈⣴⣿⢃⢸⣿⢏⡇⡆⠋⣿⣿⣿⣿⢻⢸⣿⣿⣿
|
||||
⣬⠘⡌⣿⢀⡇⡇⢸⡆⡜⣧⢻⣿⣿⣿⣧⢢⣜⢻⣌⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣡⣶⣿⣿⡟⢄⣿⡟⡼⡙⣼⢸⣿⣿⣿⣿⢸⢸⣿⣿⣿
|
||||
⣿⡗⡆⠹⡇⢡⡇⢸⡇⡘⣽⣇⢻⣿⣿⣿⡌⢿⡇⢮⡓⠙⠿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣏⡸⠃⣼⣿⣱⠳⢱⣿⢸⣿⣿⣿⣿⢸⡆⣿⣿⣿
|
||||
⣿⡇⡄⣷⠹⡼⡇⢸⡇⣷⡙⣿⡄⠻⣿⣿⣿⡌⣿⢸⣷⢕⢦⣬⣛⣿⣿⣿⣿⣿⣿⣿⣿⣿⣟⣛⠿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠟⡅⣼⣿⢡⡇⣰⢸⣿⣸⣿⣿⣿⣿⠈⡇⣿⣿⣿
|
||||
⣿⢸⠇⣿⢰⠀⠁⣾⠃⣿⢱⡜⢷⡱⡹⣿⣿⣷⡸⢸⣿⢸⣷⣍⡻⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⢛⢡⡆⣸⡿⣡⢏⣾⣿⢸⣿⣿⣿⣿⣿⣿⠀⣧⣿⣿⣿
|
||||
⣿⢸⢰⣿⢸⢸⡆⡙⠀⣿⢸⣗⣌⢧⠱⣜⢿⣿⣷⡘⠟⣼⣿⣿⣿⣷⣮⣝⠻⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠟⣩⣶⣿⠘⣱⠟⢕⣵⣿⣯⢹⣼⣿⢹⢸⣿⣿⣿⠈⢻⢹⣿⣿
|
||||
⡟⣼⢸⣿⠸⣿⢻⡟⡇⣿⢸⠛⣿⣷⡁⠻⣦⡹⣿⣿⣄⢻⣿⢟⣿⣿⣿⣿⠀⢿⡄⣭⣟⡛⠿⣿⣿⣿⠿⣛⣵⣾⣿⣿⠟⡜⢁⡐⣿⣿⣿⣿⡸⡟⣿⢸⢸⣿⣿⣿⠀⡸⡸⣿⣿
|
||||
⡇⣿⢸⣿⡇⡏⣼⢡⠇⡿⡾⠀⣿⣿⡿⠆⠈⠵⢎⡻⣿⣦⡁⣾⣿⣿⣿⡏⣼⢸⣧⢻⣿⣿⣿⣶⣶⣶⣾⣿⣿⣿⡿⢋⣨⡆⢿⣇⢻⣿⣿⣿⣇⠃⢻⠘⡈⣿⣿⣿⡇⠃⡇⣿⣿
|
||||
⡇⣿⢸⣿⡇⢣⡏⣾⢸⡇⢇⡇⡟⡩⢊⠀⣠⣶⢸⣿⣦⠙⠷⣜⢻⣿⠟⣼⣿⢸⠟⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣯⣴⣿⢛⣼⢸⣿⡜⣿⣿⣿⣿⠎⠘⡇⡇⣿⣿⣿⡇⡈⢻⢹⣿
|
||||
⡇⣿⢸⣿⡇⡼⢹⠟⢾⣘⣘⡓⡘⠰⢁⣾⣿⣿⢸⡿⣿⢸⣿⣶⠅⣁⣚⣭⣵⣶⠏⡘⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠟⣡⣿⣿⣇⠻⣧⢹⣿⠏⢱⣿⣆⢧⢧⣿⣿⣿⣇⣧⡜⡎⣿
|
||||
⣷⢸⠸⡿⢘⣥⣶⣿⣿⣿⣿⡇⠇⢶⣾⢿⣿⣿⣸⡇⣿⢺⢏⡔⣹⣿⣿⣿⣿⣿⣿⣿⣮⡙⢿⣿⣿⣿⣿⣿⠟⣡⣾⣿⣿⣿⣿⣧⣌⠃⢿⣧⢃⢻⣿⣎⢸⢹⣿⣿⣿⢻⠰⢷⢹
|
||||
⣿⡦⢅⣴⣿⣿⣿⣿⣿⣿⣿⣿⡸⡌⢿⣇⣿⣿⡟⡇⢿⠠⡿⢱⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣮⣝⣛⣫⣵⣾⣿⣿⣿⣿⣿⣿⣿⣿⣷⡄⢿⣇⢸⣏⠿⡄⠜⣿⣿⣿⢸⣇⠘⡞
|
||||
"""
|
||||
|
||||
from app import create_app
|
||||
|
||||
|
||||
app = create_app()
|
||||
|
|
|
|||
|
|
@ -0,0 +1,185 @@
|
|||
# SIEM Dashboard
|
||||
|
||||
## What This Is
|
||||
|
||||
A full stack Security Information and Event Management (SIEM) platform that ingests log events from multiple source types, normalizes them into a common schema, classifies severity using pattern matching, and runs a real time correlation engine to generate alerts. It includes a React dashboard for monitoring, investigation, and attack scenario playback.
|
||||
|
||||
The backend is Flask with MongoDB for persistence and Redis Streams for real time event delivery. The frontend is React with TypeScript, Zustand for state, and Server-Sent Events for live updates.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Every organization with more than a handful of servers needs centralized log monitoring. Without it, you're blind to attacks until the damage is done. Commercial SIEMs like Splunk or Microsoft Sentinel cost tens of thousands per year, and most security teams still struggle with alert fatigue, missed correlations, and slow investigation workflows.
|
||||
|
||||
Building a SIEM from scratch teaches you how these systems actually work under the hood. You'll understand why correlation rules matter, how event normalization prevents data silos, and why real time streaming is critical for incident response.
|
||||
|
||||
**Real world scenarios where this applies:**
|
||||
- A SOC analyst needs to correlate 20 failed SSH logins from one IP with a successful login 30 seconds later. That's a brute force followed by compromise, and the correlation engine in this project detects exactly that pattern.
|
||||
- An incident responder pivots from a suspicious source IP to find all related events across firewall, auth, and endpoint logs. The pivot API (`/v1/logs/pivot`) enables this workflow.
|
||||
- A security team wants to test detection rules against historical data before deploying them. The rule test endpoint (`/v1/rules/<id>/test`) replays events through a rule without creating real alerts.
|
||||
|
||||
## What You'll Learn
|
||||
|
||||
This project teaches you how SIEM systems process, correlate, and surface security events. By building it yourself, you'll understand:
|
||||
|
||||
**Security Concepts:**
|
||||
- Log normalization across different source formats (firewall, IDS, auth, endpoint, DNS, proxy). Each source type has different fields and the normalizer in `app/engine/normalizer.py` maps them to a common schema.
|
||||
- Severity classification using regex pattern matching against known attack indicators like "brute force", "lateral movement", and "data exfiltration".
|
||||
- Correlation rule evaluation with sliding windows, including threshold counting, ordered sequence detection, and distinct-value aggregation.
|
||||
- Alert lifecycle management from initial detection through acknowledgment, investigation, resolution, or false positive classification.
|
||||
|
||||
**Technical Skills:**
|
||||
- Building a Flask application factory with MongoEngine, Redis, and structured error handling
|
||||
- Implementing JWT authentication with Argon2id password hashing and timing safe verification to prevent user enumeration
|
||||
- Using Redis Streams with consumer groups for reliable event delivery and Server-Sent Events for real time browser updates
|
||||
- Writing a threaded correlation engine that evaluates rules against a continuous event stream
|
||||
- Building a React frontend with Zustand state management, TanStack Query for data fetching, and SSE integration
|
||||
|
||||
**Tools and Techniques:**
|
||||
- Docker Compose for local development with MongoDB, Redis, Nginx, Flask, and Vite running together
|
||||
- YAML-based attack scenario playbooks that simulate real MITRE ATT&CK techniques like brute force (T1110.001) and DNS tunneling (T1048.003)
|
||||
- Pydantic schemas for request validation with automatic error formatting
|
||||
- MongoDB aggregation pipelines for timeline bucketing, severity breakdowns, and top source analysis
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, you should understand:
|
||||
|
||||
**Required knowledge:**
|
||||
- Python basics including classes, decorators, and type hints. You'll see decorators like `@endpoint(roles=ADMIN)` and `@_register(SourceType.FIREWALL)` throughout the codebase.
|
||||
- Basic understanding of REST APIs and HTTP. The backend exposes around 30 endpoints under `/v1/`.
|
||||
- Familiarity with JSON and basic database concepts. MongoDB stores documents as JSON-like structures and you'll work with MongoEngine's ORM.
|
||||
|
||||
**Tools you'll need:**
|
||||
- Docker and Docker Compose, to run the full stack locally
|
||||
- Python 3.14+ (the project uses modern type hints like `str | None`)
|
||||
- Node.js 24+ and pnpm for the frontend
|
||||
- A tool for making HTTP requests (curl, httpie, or Postman) to test the API directly
|
||||
|
||||
**Helpful but not required:**
|
||||
- Experience with MongoDB queries and aggregation pipelines
|
||||
- Familiarity with Redis data structures, particularly Streams
|
||||
- Basic React and TypeScript knowledge for understanding the frontend
|
||||
|
||||
## Quick Start
|
||||
|
||||
Get the project running locally:
|
||||
|
||||
```bash
|
||||
# Navigate to the project
|
||||
cd PROJECTS/intermediate/siem-dashboard
|
||||
|
||||
# Start development environment
|
||||
docker compose -f dev.compose.yml up --build
|
||||
|
||||
# The app will be available at http://localhost:8431
|
||||
# Backend API is at http://localhost:8431/api/v1/
|
||||
# Direct backend access at http://localhost:5113
|
||||
# Direct frontend dev server at http://localhost:3959
|
||||
```
|
||||
|
||||
Create an admin account:
|
||||
|
||||
```bash
|
||||
docker exec -it siem-backend-dev flask admin create \
|
||||
--username admin \
|
||||
--email admin@example.com
|
||||
```
|
||||
|
||||
Expected output: `Admin account 'admin' created successfully.`
|
||||
|
||||
Test the API:
|
||||
|
||||
```bash
|
||||
# Register a regular user
|
||||
curl -X POST http://localhost:8431/api/v1/auth/register \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"username":"analyst1","email":"analyst@test.com","password":"testpass123"}'
|
||||
|
||||
# Ingest a test log event
|
||||
curl -X POST http://localhost:8431/api/v1/logs/ingest \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"source_type":"auth","event_type":"login_failure","source_ip":"10.0.0.1","username":"root"}'
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
siem-dashboard/
|
||||
├── backend/
|
||||
│ ├── app/
|
||||
│ │ ├── __init__.py # Flask app factory
|
||||
│ │ ├── config.py # Pydantic settings (60+ config values)
|
||||
│ │ ├── extensions.py # MongoDB and Redis initialization
|
||||
│ │ ├── cli.py # Flask CLI for admin management
|
||||
│ │ ├── core/
|
||||
│ │ │ ├── auth.py # JWT + Argon2id authentication
|
||||
│ │ │ ├── streaming.py # Redis Streams + SSE generator
|
||||
│ │ │ ├── errors.py # Error hierarchy and handlers
|
||||
│ │ │ ├── rate_limiting.py # Flask-Limiter setup
|
||||
│ │ │ ├── serialization.py # MongoEngine to JSON conversion
|
||||
│ │ │ └── decorators/ # @endpoint, @S, @R composable decorators
|
||||
│ │ ├── engine/
|
||||
│ │ │ ├── normalizer.py # Multi-format log normalizer
|
||||
│ │ │ ├── severity.py # Pattern-based severity classifier
|
||||
│ │ │ └── correlation.py # Sliding window correlation engine
|
||||
│ │ ├── models/ # MongoEngine documents (User, LogEvent, Alert, etc.)
|
||||
│ │ ├── routes/ # Flask blueprints for each resource
|
||||
│ │ ├── controllers/ # Business logic handlers
|
||||
│ │ ├── schemas/ # Pydantic request/response schemas
|
||||
│ │ └── scenarios/
|
||||
│ │ ├── playbook.py # YAML playbook parser
|
||||
│ │ ├── runner.py # Threaded scenario executor
|
||||
│ │ └── playbooks/ # Attack scenario YAML files
|
||||
│ ├── pyproject.toml # Python dependencies and tool config
|
||||
│ └── wsgi.py # Gunicorn entry point
|
||||
├── frontend/
|
||||
│ ├── src/
|
||||
│ │ ├── api/hooks/ # TanStack Query hooks for each resource
|
||||
│ │ ├── api/types/ # Zod schemas and TypeScript types
|
||||
│ │ ├── core/app/ # Shell layout, routing, protected routes
|
||||
│ │ ├── core/lib/ # Axios client, error handling, query config
|
||||
│ │ ├── core/stores/ # Zustand stores (auth, stream, UI)
|
||||
│ │ ├── core/charts/ # visx chart theme and color constants
|
||||
│ │ ├── routes/ # Page components (lazy loaded)
|
||||
│ │ └── config.ts # API endpoints, query keys, constants
|
||||
│ └── package.json
|
||||
├── conf/
|
||||
│ ├── docker/ # Dockerfiles for dev and prod
|
||||
│ └── nginx/ # Nginx configs for proxying and SSE
|
||||
├── compose.yml # Production Docker Compose
|
||||
└── dev.compose.yml # Development Docker Compose
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Understand the concepts** - Read [01-CONCEPTS.md](./01-CONCEPTS.md) to learn about log correlation, event normalization, and SIEM architecture principles
|
||||
2. **Study the architecture** - Read [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) to see how the backend engine, streaming layer, and frontend connect
|
||||
3. **Walk through the code** - Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) for a detailed walkthrough of the correlation engine, normalizer, and streaming pipeline
|
||||
4. **Extend the project** - Read [04-CHALLENGES.md](./04-CHALLENGES.md) for ideas like adding Sigma rule support, building a threat intelligence feed, or implementing SOAR playbooks
|
||||
|
||||
## Common Issues
|
||||
|
||||
**Backend container keeps restarting**
|
||||
```
|
||||
siem-backend-dev | Connection refused: mongodb://mongo:27017/siem
|
||||
```
|
||||
Solution: MongoDB takes 10-30 seconds to initialize. The healthcheck in `dev.compose.yml` has a `start_period: 30s` but sometimes this isn't enough on slower machines. Wait and check `docker compose logs mongo`.
|
||||
|
||||
**SSE streams disconnect immediately**
|
||||
The Nginx config at `conf/nginx/dev.nginx` has special handling for SSE endpoints. If you see instant disconnects, verify the regex location block `location ~ ^/api(/v1/(logs|alerts)/stream.*)` is matching your request. The SSE endpoints need `proxy_buffering off` and long read timeouts.
|
||||
|
||||
**Redis Stream errors on startup**
|
||||
```
|
||||
BUSYGROUP Consumer Group name already exists
|
||||
```
|
||||
This is normal. The `ensure_consumer_group()` function in `app/core/streaming.py` wraps the creation in `contextlib.suppress(Exception)` because it's idempotent. The consumer group already exists from a previous run.
|
||||
|
||||
**Rate limiting blocks development requests**
|
||||
Auth endpoints are limited to 10 requests per minute (`RATELIMIT_AUTH` in `config.py`). General endpoints allow 200 per minute. If you're hammering the login endpoint during testing, wait a minute or temporarily increase the limit in your `.env` file.
|
||||
|
||||
## Related Projects
|
||||
|
||||
If you found this interesting, check out:
|
||||
- **API Rate Limiter** (advanced) - Deep dive into the rate limiting algorithms this project uses via Flask-Limiter, including sliding window and token bucket strategies
|
||||
- **Network Traffic Analyzer** - Packet-level analysis using Scapy, complementing the higher level log analysis this SIEM performs
|
||||
- **Docker Security Auditor** - Container security scanning that could feed events into this SIEM's ingestion pipeline
|
||||
|
|
@ -0,0 +1,321 @@
|
|||
# Core Security Concepts
|
||||
|
||||
This document explains the security concepts you'll encounter while building this SIEM dashboard. These aren't just definitions. We'll dig into why they matter, how attacks actually work, and how this project implements defenses.
|
||||
|
||||
## Log Normalization
|
||||
|
||||
### What It Is
|
||||
|
||||
Different security tools produce logs in completely different formats. A firewall log has fields like `action`, `protocol`, and `bytes_sent`. An IDS log has `signature_id`, `classification`, and `priority`. An authentication log has `auth_method`, `result`, and `failure_reason`. Raw, these logs are incomparable.
|
||||
|
||||
Log normalization transforms all these different formats into a common schema so you can query, correlate, and analyze them together. In this project, the normalizer (`app/engine/normalizer.py`) takes raw events from seven source types and maps them into a unified `LogEvent` document with consistent fields like `source_ip`, `dest_ip`, `severity`, `event_type`, and a `normalized` dict for source-specific extras.
|
||||
|
||||
### Why It Matters
|
||||
|
||||
Without normalization, a SOC analyst investigating a suspicious IP would need to manually search each tool's interface separately. With normalized data, a single pivot query returns every firewall connection, every auth attempt, every DNS query, and every endpoint process execution from that IP.
|
||||
|
||||
The 2013 Target breach is a textbook case. Their FireEye IDS generated alerts about malware beaconing, but the team couldn't correlate those alerts with the POS system logs showing data exfiltration. The tools existed in silos. A functioning SIEM with proper normalization would have connected those dots.
|
||||
|
||||
### How It Works
|
||||
|
||||
The normalizer uses a registry pattern. Each source type registers a normalizer function via a decorator:
|
||||
|
||||
```
|
||||
Source Type Normalizer Function Extracted Fields
|
||||
───────────── ───────────────────────── ──────────────────────────
|
||||
firewall _normalize_firewall() action, protocol, bytes_sent/received
|
||||
ids _normalize_ids() signature_id, signature_name, priority
|
||||
auth _normalize_auth() auth_method, result, failure_reason
|
||||
endpoint _normalize_endpoint() process_name, command_line, file_path
|
||||
dns _normalize_dns() query, query_type, response_code
|
||||
proxy _normalize_proxy() url, method, status_code, user_agent
|
||||
generic _normalize_generic() message (fallback)
|
||||
```
|
||||
|
||||
The dispatch is straightforward. `normalize()` extracts common fields (IP addresses, ports, timestamps, hostnames), then calls the source-specific normalizer to extract fields unique to that log type. Both get merged into a single document.
|
||||
|
||||
### Common Pitfalls
|
||||
|
||||
**Mistake 1: Assuming all sources have the same fields**
|
||||
```python
|
||||
# Bad - will throw KeyError on auth logs
|
||||
def process_event(event):
|
||||
bytes = event["bytes_sent"] # Only firewall events have this
|
||||
|
||||
# Good - use .get() with defaults and check source_type
|
||||
def process_event(event):
|
||||
if event.get("source_type") == "firewall":
|
||||
bytes = event.get("bytes_sent", 0)
|
||||
```
|
||||
|
||||
**Mistake 2: Losing the raw data**
|
||||
|
||||
The normalizer in this project preserves the original event in the `raw` DictField on `LogEvent`. This is critical for forensics. Normalization is lossy by definition, and during an investigation you'll often need the original log line.
|
||||
|
||||
## Event Correlation
|
||||
|
||||
### What It Is
|
||||
|
||||
Correlation is the process of connecting multiple individual events into a higher level detection. A single failed login is noise. Twenty failed logins from the same IP in 60 seconds is a brute force attack. A brute force followed by a successful login is a compromise.
|
||||
|
||||
This project implements three correlation strategies in `app/engine/correlation.py`:
|
||||
|
||||
**Threshold** rules fire when the count of matching events for a group key exceeds a limit within a time window. Example: more than 10 `login_failure` events from the same `source_ip` in 300 seconds.
|
||||
|
||||
**Sequence** rules fire when a specific ordered set of event types occurs within a window. Example: `login_failure` (count >= 5) followed by `login_success` from the same `source_ip`.
|
||||
|
||||
**Aggregation** rules fire when the number of distinct values of a field exceeds a threshold. Example: one `source_ip` connects to more than 20 distinct `dest_ip` values in 60 seconds (port scan behavior).
|
||||
|
||||
### Why It Matters
|
||||
|
||||
Without correlation, a SIEM is just a log search engine. Correlation is what turns data into detections. The difference between a SIEM that generates useful alerts and one that drowns analysts in noise comes down to the quality of its correlation rules.
|
||||
|
||||
The 2020 SolarWinds attack went undetected for months partly because existing detection tools were looking at individual events in isolation. The attackers used valid credentials, standard protocols, and legitimate tools. Only correlating multiple weak signals across time (unusual service account behavior + new federation trust + anomalous DNS queries) could have surfaced the intrusion earlier.
|
||||
|
||||
### How It Works
|
||||
|
||||
The correlation engine runs as a daemon thread that consumes events from a Redis Stream:
|
||||
|
||||
```
|
||||
Redis Stream (siem:logs)
|
||||
│
|
||||
▼
|
||||
┌───────────────────┐
|
||||
│ CorrelationEngine │ Daemon thread, reads via XREADGROUP
|
||||
│ ._run() loop │
|
||||
└────────┬──────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────┐
|
||||
│ _process_event() │ Iterates all enabled rules
|
||||
└────────┬──────────┘
|
||||
│
|
||||
┌────┴────┬──────────┐
|
||||
▼ ▼ ▼
|
||||
Threshold Sequence Aggregation
|
||||
evaluator evaluator evaluator
|
||||
│ │ │
|
||||
└────┬────┘──────────┘
|
||||
│
|
||||
Fire? ──Yes──▶ Alert.create_from_rule()
|
||||
│
|
||||
▼
|
||||
Redis Stream (siem:alerts)
|
||||
│
|
||||
▼
|
||||
SSE to browser
|
||||
```
|
||||
|
||||
Each evaluator uses a `CorrelationState` object that maintains sliding windows in memory. The state is thread-safe (uses `threading.Lock`) and tracks per-rule, per-group-key windows of events. When evaluating, it prunes expired entries based on `window_seconds` and checks whether the firing condition is met.
|
||||
|
||||
A cooldown mechanism (`CORRELATION_COOLDOWN_SECONDS`, default 300) prevents the same rule from firing repeatedly for the same group key. Without this, a sustained brute force would generate hundreds of identical alerts.
|
||||
|
||||
### Common Attacks This Detects
|
||||
|
||||
1. **Brute Force (T1110.001)** - Threshold rule: count `login_failure` events grouped by `source_ip`, fire when count > 10 in 300 seconds
|
||||
2. **Lateral Movement (T1021.004)** - Sequence rule: `login_success` from external IP followed by internal SSH connections from the compromised host
|
||||
3. **Port Scanning (T1046)** - Aggregation rule: count distinct `dest_ip` values per `source_ip`, fire when > 20 unique destinations in 60 seconds
|
||||
4. **DNS Tunneling (T1048.003)** - Threshold rule on high-frequency TXT queries to a single domain, or aggregation on distinct query subdomains
|
||||
|
||||
### Defense Strategies
|
||||
|
||||
The correlation engine itself is a defense mechanism. But it needs well-tuned rules. This project ships with scenario playbooks that demonstrate attacks and the rules that detect them. The `brute_force_lateral.yml` playbook, for example, simulates 20 SSH login failures followed by a successful login and lateral movement. A threshold rule on `login_failure` grouped by `source_ip` with threshold 10 and window 300 would catch the brute force phase.
|
||||
|
||||
## Severity Classification
|
||||
|
||||
### What It Is
|
||||
|
||||
Not all security events are equally urgent. A firewall allowing an outbound HTTPS connection is informational. A process executing `cat /etc/shadow` is high severity. Severity classification assigns a priority level to each event so analysts focus on what matters.
|
||||
|
||||
This project classifies events into five levels: `critical`, `high`, `medium`, `low`, and `info`. The classifier in `app/engine/severity.py` uses two approaches: event type lookup (fast, for known event types) and regex pattern matching (flexible, for content-based classification).
|
||||
|
||||
### How It Works
|
||||
|
||||
The classifier checks event types first against frozen sets:
|
||||
|
||||
```
|
||||
HIGH_SEVERITY_EVENT_TYPES: privilege_escalation, data_exfiltration,
|
||||
c2_communication, reverse_shell
|
||||
|
||||
MEDIUM_SEVERITY_EVENT_TYPES: login_failure, port_scan,
|
||||
firewall_deny, ids_alert
|
||||
```
|
||||
|
||||
If the event type doesn't match, it falls through to regex matching. The classifier concatenates relevant text fields (event_type, message, normalized fields like signature_name and command_line) into a searchable string, then runs regex patterns from critical down to low. First match wins.
|
||||
|
||||
Critical patterns include things like `privilege.?escalat`, `ransomware`, and `c2.?beacon`. High patterns include `brute.?force`, `lateral.?movement`, and `reverse.?shell`. The regex uses `re.IGNORECASE` for case insensitive matching and `?` for optional separators (so "privilege escalation" and "privilege_escalation" both match).
|
||||
|
||||
### Common Pitfalls
|
||||
|
||||
**Mistake: Classifying everything as high severity**
|
||||
|
||||
This leads to alert fatigue. If everything is critical, nothing is. The classifier in this project defaults to `info` for events that don't match any pattern. This is intentional. An unknown event type with no suspicious keywords is probably routine. Better to miss a low-confidence detection than to bury real alerts in noise.
|
||||
|
||||
## Authentication and Authorization Security
|
||||
|
||||
### What It Is
|
||||
|
||||
The SIEM itself needs to be secured. If an attacker compromises the SIEM, they can suppress alerts, manipulate rules, and cover their tracks. This project implements JWT-based authentication with Argon2id password hashing and role-based access control.
|
||||
|
||||
### How It Works
|
||||
|
||||
The auth system in `app/core/auth.py` uses `pwdlib` with Argon2id, the winner of the 2015 Password Hashing Competition. Argon2id is memory-hard, meaning it's resistant to GPU-based cracking attacks that make bcrypt and SHA-based hashes vulnerable.
|
||||
|
||||
A critical detail: the `verify_password_timing_safe()` function provides constant-time behavior for failed lookups. When a login attempt uses a username that doesn't exist, the system still performs a hash comparison against a dummy hash (`DUMMY_HASH`). This prevents timing attacks that could enumerate valid usernames by measuring response time differences.
|
||||
|
||||
```
|
||||
Login attempt with valid username:
|
||||
1. Look up user → found
|
||||
2. Verify password against stored hash → ~150ms (Argon2id)
|
||||
3. Return result
|
||||
|
||||
Login attempt with invalid username:
|
||||
1. Look up user → not found
|
||||
2. Verify password against DUMMY_HASH → ~150ms (same timing)
|
||||
3. Return "Invalid username or password"
|
||||
```
|
||||
|
||||
Without this, an attacker could distinguish "user doesn't exist" (fast response) from "wrong password" (slow response, because Argon2id is intentionally expensive).
|
||||
|
||||
### RBAC Model
|
||||
|
||||
Two roles: `analyst` (default) and `admin`. The `@endpoint(roles=ADMIN)` decorator on admin routes checks the JWT claims. Admins can manage users, change roles, and deactivate accounts. The system protects against demoting the last admin (`User.count_admins()` check in `admin_ctrl.py`).
|
||||
|
||||
## Real-Time Event Streaming
|
||||
|
||||
### What It Is
|
||||
|
||||
A SIEM that only shows data on page refresh is nearly useless during an active incident. Analysts need to see events as they happen. This project uses Redis Streams for internal event delivery and Server-Sent Events (SSE) for browser updates.
|
||||
|
||||
### How It Works
|
||||
|
||||
When a log event is ingested or an alert is created, it's published to a Redis Stream via `XADD`. The correlation engine reads from the log stream using `XREADGROUP` with consumer groups (reliable delivery with acknowledgment). The SSE endpoints use `XREAD` to tail the stream and emit events to connected browsers.
|
||||
|
||||
```
|
||||
Log Ingestion ──XADD──▶ Redis Stream (siem:logs)
|
||||
│
|
||||
┌───────────────┼───────────────┐
|
||||
│ │ │
|
||||
XREADGROUP XREADGROUP XREAD
|
||||
│ │ │
|
||||
Correlation (Future SSE Generator
|
||||
Engine consumers) ──▶ Browser
|
||||
│
|
||||
Alert Created ──XADD──▶ Redis Stream (siem:alerts)
|
||||
│
|
||||
XREAD
|
||||
│
|
||||
SSE Generator
|
||||
──▶ Browser
|
||||
```
|
||||
|
||||
The SSE generator in `app/core/streaming.py` sends keepalive comments (`: keepalive\n\n`) when no events arrive within the block timeout. This prevents proxy timeouts and lets the browser detect dropped connections. The Nginx config disables buffering and sets `proxy_read_timeout 3600s` for SSE endpoints.
|
||||
|
||||
## How These Concepts Relate
|
||||
|
||||
```
|
||||
Raw Log Event
|
||||
│
|
||||
▼
|
||||
Normalization (common schema)
|
||||
│
|
||||
▼
|
||||
Severity Classification (priority label)
|
||||
│
|
||||
▼
|
||||
Persistence (MongoDB) + Publish (Redis Stream)
|
||||
│
|
||||
├──▶ SSE to browser (real-time log viewer)
|
||||
│
|
||||
▼
|
||||
Correlation Engine (pattern matching across events)
|
||||
│
|
||||
▼
|
||||
Alert Generation
|
||||
│
|
||||
├──▶ SSE to browser (real-time alert feed)
|
||||
│
|
||||
▼
|
||||
Alert Lifecycle (new → acknowledged → investigating → resolved)
|
||||
│
|
||||
▼
|
||||
Forensic Investigation (pivot queries, matched event drill-down)
|
||||
```
|
||||
|
||||
Each concept builds on the previous one. Without normalization, correlation can't compare events across sources. Without severity classification, analysts can't prioritize. Without streaming, response time suffers. Without correlation, individual events are meaningless noise.
|
||||
|
||||
## Industry Standards and Frameworks
|
||||
|
||||
### OWASP
|
||||
|
||||
While OWASP focuses on application security, several categories apply to the SIEM itself:
|
||||
- **A01:2021 Broken Access Control** - The RBAC system with `@endpoint(roles=ADMIN)` prevents unauthorized access to admin functions. The `_prevent_self_action()` helper blocks admins from deactivating their own accounts.
|
||||
- **A02:2021 Cryptographic Failures** - Argon2id for password hashing, HS256 JWTs with configurable expiration. The `SECRET_KEY` default of "change-me-in-production" is intentionally obvious.
|
||||
- **A07:2021 Identification and Authentication Failures** - Timing-safe password verification, rate limiting on auth endpoints (10/minute), account deactivation support.
|
||||
|
||||
### MITRE ATT&CK
|
||||
|
||||
The scenario playbooks map directly to MITRE techniques:
|
||||
|
||||
- **T1110.001 (Brute Force: Password Guessing)** - `brute_force_lateral.yml` simulates 20 SSH password attempts across common usernames
|
||||
- **T1021.004 (Remote Services: SSH)** - Lateral movement via SSH key reuse after initial compromise
|
||||
- **T1048.003 (Exfiltration Over Alternative Protocol: DNS)** - `data_exfiltration.yml` simulates DNS tunneling with high-entropy TXT queries
|
||||
- **T1059.001 (Command and Scripting Interpreter: PowerShell)** - `phishing_c2.yml` includes encoded PowerShell download cradle execution
|
||||
- **T1068 (Exploitation for Privilege Escalation)** - `privilege_escalation.yml` simulates kernel exploit compilation and execution
|
||||
- **T1071.001 (Application Layer Protocol: Web)** - C2 beaconing over HTTPS with regular intervals
|
||||
|
||||
Correlation rules store `mitre_tactic` and `mitre_technique` fields, and these propagate to generated alerts for analyst context.
|
||||
|
||||
### CWE
|
||||
|
||||
Common weakness enumerations relevant to what this project teaches:
|
||||
- **CWE-778 (Insufficient Logging)** - The entire project is about solving this. Seven source types with format-specific normalization.
|
||||
- **CWE-223 (Omission of Security-Relevant Information)** - The normalizer preserves raw event data alongside normalized fields.
|
||||
- **CWE-779 (Logging of Excessive Data)** - Redis Streams use `maxlen` with approximate trimming (`STREAM_MAXLEN = 10000`) to prevent unbounded growth.
|
||||
- **CWE-307 (Improper Restriction of Excessive Authentication Attempts)** - Rate limiting on auth endpoints plus correlation rules for brute force detection.
|
||||
|
||||
## Real World Examples
|
||||
|
||||
### Case Study 1: Target Corporation Breach (2013)
|
||||
|
||||
Target's network was breached via a phishing email sent to an HVAC vendor. The attackers moved laterally to POS systems and installed RAM-scraping malware. Target's FireEye IDS detected the malware and generated alerts, but the SOC team failed to correlate those alerts with the broader attack chain.
|
||||
|
||||
This is exactly the kind of failure a SIEM with proper correlation prevents. A sequence rule looking for "external access → credential theft → lateral movement → data access" would have connected the dots. The scenario playbook `brute_force_lateral.yml` in this project simulates a similar kill chain: external brute force → credential harvest → SSH lateral movement → database dump.
|
||||
|
||||
### Case Study 2: DNS Tunneling in the APT34/OilRig Campaign
|
||||
|
||||
APT34 (also known as OilRig) used DNS tunneling to exfiltrate data from compromised networks. They encoded stolen data in DNS query subdomains, sending it to attacker-controlled nameservers. The technique works because most firewalls allow outbound DNS traffic without deep inspection.
|
||||
|
||||
The `data_exfiltration.yml` playbook simulates this exact technique. It generates high-entropy TXT queries to subdomains like `4d5a90000300000004000000ffff0000b800.data.exfil-tunnel.tk`. A threshold correlation rule on DNS query frequency, or an aggregation rule on distinct subdomains per destination domain, would detect this activity.
|
||||
|
||||
### Case Study 3: Emotet/TrickBot Delivery Chain
|
||||
|
||||
TA542's campaigns follow a consistent pattern: spear-phishing email → macro-enabled document → PowerShell download cradle → second-stage payload → C2 beaconing. The `phishing_c2.yml` playbook reproduces this chain event by event, from the initial proxy log of the phishing link click through to periodic C2 beacon heartbeats.
|
||||
|
||||
This demonstrates why sequence correlation matters. Any individual event in the chain could be benign. Downloading an Excel file is normal. Running PowerShell is normal. Outbound HTTPS is normal. But the sequence of Excel → cmd.exe → PowerShell → download → C2 beaconing is not normal, and a sequence rule can catch it.
|
||||
|
||||
## Testing Your Understanding
|
||||
|
||||
Before moving to the architecture, make sure you can answer:
|
||||
|
||||
1. Why does the normalizer preserve the raw event data in addition to normalized fields? What investigation scenario would require the raw data?
|
||||
2. What's the difference between a threshold rule and an aggregation rule? Give an example attack that each would detect but the other wouldn't.
|
||||
3. Why does `verify_password_timing_safe()` hash against a dummy value when the user doesn't exist? What information leak does this prevent?
|
||||
4. If the correlation engine's cooldown is set to 300 seconds, what happens during a sustained 10-minute brute force? How many alerts would fire?
|
||||
5. Why does the SSE generator send keepalive comments? What would happen in a production deployment behind Nginx without them?
|
||||
|
||||
If these questions feel unclear, re-read the relevant sections. The implementation details will make more sense once these fundamentals click.
|
||||
|
||||
## Further Reading
|
||||
|
||||
**Essential:**
|
||||
- [MITRE ATT&CK Framework](https://attack.mitre.org/) - The taxonomy this project's scenarios and rules are built around. Start with the "Enterprise" matrix.
|
||||
- [Redis Streams documentation](https://redis.io/docs/data-types/streams/) - Understanding XADD, XREADGROUP, consumer groups, and acknowledgment is critical for understanding the streaming layer.
|
||||
|
||||
**Deep dives:**
|
||||
- [Sigma Rules](https://github.com/SigmaHQ/sigma) - An open standard for SIEM detection rules. Challenge 6 in the challenges doc suggests implementing Sigma rule support.
|
||||
- [OWASP Password Storage Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html) - Why Argon2id was chosen over bcrypt, and the recommended parameter tuning.
|
||||
|
||||
**Historical context:**
|
||||
- Mandiant's APT1 Report (2013) - One of the first public threat intelligence reports, and a great example of the kind of investigation workflow a SIEM enables.
|
||||
- NIST SP 800-92 (Guide to Computer Security Log Management) - The foundational document on why and how to collect, store, and analyze security logs.
|
||||
|
|
@ -0,0 +1,598 @@
|
|||
# System Architecture
|
||||
|
||||
How the SIEM dashboard is designed, why each component exists, and how data moves through the system.
|
||||
|
||||
## High Level Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Browser │
|
||||
│ React 19 + TypeScript + Zustand + TanStack Query │
|
||||
│ SSE Listeners (EventSource) │
|
||||
└──────────────┬────────────────────────────┬──────────────────┘
|
||||
│ HTTP/JSON │ SSE
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Nginx Reverse Proxy │
|
||||
│ Rate limiting · Gzip · Static assets · SSE passthrough │
|
||||
│ proxy_buffering off (for /stream endpoints) │
|
||||
└──────────────┬────────────────────────────┬──────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Flask Backend │
|
||||
│ │
|
||||
│ ┌─────────┐ ┌──────────────┐ ┌────────────────────┐ │
|
||||
│ │ Routes │→ │ Controllers │→ │ Engine │ │
|
||||
│ │ (auth, │ │ (validation, │ │ (normalizer, │ │
|
||||
│ │ logs, │ │ business │ │ severity, │ │
|
||||
│ │ alerts │ │ logic) │ │ correlation) │ │
|
||||
│ │ rules) │ │ │ │ │ │
|
||||
│ └─────────┘ └──────────────┘ └────────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────┐ ┌──────────────────────────┐ │
|
||||
│ │ Scenario Runner │ │ Correlation Engine │ │
|
||||
│ │ (daemon threads, │ │ (daemon thread, │ │
|
||||
│ │ playbook playback) │ │ XREADGROUP consumer) │ │
|
||||
│ └──────────────────────┘ └──────────────────────────┘ │
|
||||
└──────────┬──────────────────────────┬────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌────────────────────┐ ┌────────────────────┐
|
||||
│ MongoDB 8.0 │ │ Redis 7 Alpine │
|
||||
│ │ │ │
|
||||
│ log_events │ │ siem:logs stream │
|
||||
│ alerts │ │ siem:alerts stream│
|
||||
│ correlation_rules │ │ rate limit keys │
|
||||
│ users │ │ │
|
||||
│ scenario_runs │ │ │
|
||||
└────────────────────┘ └────────────────────┘
|
||||
```
|
||||
|
||||
### Component Breakdown
|
||||
|
||||
**Nginx Reverse Proxy**
|
||||
Sits in front of everything. Handles TLS termination in production, rate limiting at the edge (`limit_req_zone` in `conf/nginx/nginx.conf:34-36`), gzip compression, and static asset caching. The critical detail is the SSE passthrough configuration. Regular API endpoints use buffered proxying, but `/stream` endpoints need `proxy_buffering off` and a 3600s read timeout (`conf/nginx/prod.nginx:30-43`). Without this, Nginx buffers SSE events and the browser gets nothing until the buffer fills.
|
||||
|
||||
**Flask Backend**
|
||||
Application factory pattern in `backend/app/__init__.py`. Creates the Flask app, wires up MongoDB and Redis connections, registers blueprints, initializes rate limiting, creates Redis consumer groups, cleans up orphaned scenario runs, and starts the correlation engine. That startup order matters. The consumer groups must exist before the correlation engine tries to read from them.
|
||||
|
||||
**MongoDB**
|
||||
Primary data store for all persistent documents. MongoEngine ODM maps Python classes to collections. Five main collections: `log_events`, `alerts`, `correlation_rules`, `users`, `scenario_runs`. Each has specific indexes defined in the model `meta` dict for query performance.
|
||||
|
||||
**Redis**
|
||||
Two roles. First, it powers the streaming pipeline. Two Redis Streams (`siem:logs` and `siem:alerts`) handle the pub/sub fanout between log ingestion, the correlation engine, and SSE endpoints. Second, it backs the rate limiter (`flask-limiter` uses Redis as its storage backend via the `REDIS_URL` config).
|
||||
|
||||
**React Frontend**
|
||||
Single page app with React 19, TypeScript, and a clean separation between data fetching (TanStack Query hooks in `frontend/src/api/hooks/`) and UI state (Zustand stores in `frontend/src/core/stores/`). Real time updates come through two SSE connections managed by `useLogStream` and `useAlertStream` hooks.
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Log Ingestion Pipeline
|
||||
|
||||
This is the core pipeline. Every log event, whether from the scenario runner or an external POST, follows the same path:
|
||||
|
||||
```
|
||||
1. Raw event arrives
|
||||
POST /v1/logs/ingest (or ScenarioRunner._emit_event)
|
||||
│
|
||||
2. Normalize (app/engine/normalizer.py)
|
||||
│ Dispatches to source_type-specific normalizer
|
||||
│ Extracts common fields, preserves raw payload
|
||||
│
|
||||
3. Classify severity (app/engine/severity.py)
|
||||
│ Event type lookup → regex pattern matching → default to info
|
||||
│
|
||||
4. Persist to MongoDB (app/models/LogEvent.py)
|
||||
│ LogEvent.create_event() → document saved with all fields
|
||||
│
|
||||
5. Publish to Redis Stream (app/core/streaming.py)
|
||||
│ XADD siem:logs {payload: JSON}
|
||||
│ Maxlen ~10000 (approximate trim)
|
||||
│
|
||||
├──→ 6a. Correlation Engine reads via XREADGROUP
|
||||
│ │ Evaluates all enabled rules against event
|
||||
│ │ If rule fires → create Alert → XADD siem:alerts
|
||||
│ │ ACK message after processing
|
||||
│ │
|
||||
│ └──→ SSE /v1/alerts/stream (browser picks up new alerts)
|
||||
│
|
||||
└──→ 6b. SSE /v1/logs/stream
|
||||
│ XREAD with blocking, yields to EventSource
|
||||
└── Browser pushes to Zustand StreamStore
|
||||
```
|
||||
|
||||
The important thing here is that steps 4 and 5 happen atomically from the controller's perspective. The event is persisted before it hits the stream. If Redis is down, the event is still in MongoDB. If MongoDB is down, nothing gets published because `create_event()` would throw first.
|
||||
|
||||
### Alert Lifecycle
|
||||
|
||||
Alerts have a state machine with five statuses:
|
||||
|
||||
```
|
||||
new → acknowledged → investigating → resolved
|
||||
→ false_positive
|
||||
```
|
||||
|
||||
When the correlation engine fires a rule (`app/engine/correlation.py:_process_event`), it calls `Alert.create_from_rule()` which both saves the alert to MongoDB and publishes it to the `siem:alerts` Redis Stream. The alert document references the matched event IDs so analysts can drill into what triggered the alert.
|
||||
|
||||
Status transitions happen through `PATCH /v1/alerts/<id>/status` and are tracked with timestamps. The `acknowledged_by` field records which analyst claimed the alert, and `resolved_at` marks closure time.
|
||||
|
||||
### Authentication Flow
|
||||
|
||||
```
|
||||
Register/Login
|
||||
│
|
||||
▼
|
||||
POST /v1/auth/register or POST /v1/auth/login
|
||||
│ │
|
||||
▼ ▼
|
||||
hash_password() verify_password_timing_safe()
|
||||
(Argon2id) (constant time, dummy hash for missing users)
|
||||
│ │
|
||||
▼ ▼
|
||||
User.create_user() Validate credentials
|
||||
│ │
|
||||
└───────┬───────────────────┘
|
||||
▼
|
||||
create_access_token()
|
||||
JWT with sub=user_id, username, role, exp
|
||||
│
|
||||
▼
|
||||
Return {access_token, token_type: "bearer"}
|
||||
│
|
||||
▼
|
||||
Frontend stores in Zustand (persisted to localStorage)
|
||||
Axios interceptor attaches to every request
|
||||
```
|
||||
|
||||
Each subsequent request hits the `endpoint()` decorator which extracts the Bearer token, decodes the JWT, loads the user from MongoDB, and attaches it to Flask's `g.current_user`. The decorator also handles role gating. Pass `roles=["admin"]` and non-admins get a 403.
|
||||
|
||||
## Design Patterns
|
||||
|
||||
### Application Factory
|
||||
|
||||
**Where it lives:** `backend/app/__init__.py`
|
||||
|
||||
The `create_app()` function builds the Flask app from scratch every time it's called. This isn't just a Flask convention. It solves real problems: test isolation (each test gets a fresh app), configuration flexibility (swap `.env` files between dev/prod), and import order issues (extensions initialize after app config is set).
|
||||
|
||||
The initialization order is deliberate:
|
||||
|
||||
1. Config loading (from env vars via Pydantic)
|
||||
2. CORS setup
|
||||
3. MongoDB and Redis connections
|
||||
4. Error handlers
|
||||
5. Rate limiter
|
||||
6. Blueprint registration
|
||||
7. Consumer group creation
|
||||
8. Orphan scenario cleanup
|
||||
9. Correlation engine start
|
||||
|
||||
If you move step 7 after step 9, the correlation engine will crash trying to read from a consumer group that doesn't exist yet.
|
||||
|
||||
### Decorator Stack Pattern
|
||||
|
||||
**Where it lives:** `backend/app/core/decorators/`
|
||||
|
||||
Every route handler uses the same decorator stack: `@endpoint` → `@S` → `@R`. This is a composable pipeline:
|
||||
|
||||
```python
|
||||
# backend/app/routes/logs.py:32-39
|
||||
@logs_bp.post("/ingest")
|
||||
@endpoint(auth_required=False)
|
||||
@S(LogIngestRequest)
|
||||
@R(status=201)
|
||||
def ingest_log() -> Any:
|
||||
return log_ctrl.ingest_log()
|
||||
```
|
||||
|
||||
`@endpoint` handles JWT extraction, user loading, role enforcement, and error boundaries. `@S` (Schema) validates request data with Pydantic and stores the result on `g.validated`. `@R` (Response) auto-serializes the return value into JSON with the right status code.
|
||||
|
||||
The decorators execute outside-in: endpoint runs first (auth check), then S (validation), then the function body, then R (serialization). If auth fails, validation never runs. If validation fails, the controller never executes. This fail-fast approach keeps controller code clean.
|
||||
|
||||
**Trade-offs:**
|
||||
The decorator stack is concise but can be confusing to debug. Stack traces go through multiple wrapper layers. If you add a new decorator, ordering matters and gets it wrong silently.
|
||||
|
||||
### Registry Pattern for Normalizers
|
||||
|
||||
**Where it lives:** `backend/app/engine/normalizer.py`
|
||||
|
||||
Each source type (firewall, IDS, auth, endpoint, DNS, proxy, generic) has its own normalizer function registered via the `@_register` decorator:
|
||||
|
||||
```python
|
||||
# backend/app/engine/normalizer.py:16-22
|
||||
def _register(source_type: SourceType) -> Callable[[NormalizerFn], NormalizerFn]:
|
||||
def decorator(fn: NormalizerFn) -> NormalizerFn:
|
||||
NORMALIZERS[source_type.value] = fn
|
||||
return fn
|
||||
return decorator
|
||||
```
|
||||
|
||||
The `normalize()` dispatcher looks up the right function from the `NORMALIZERS` dict and falls back to `_normalize_generic` for unknown types. Adding a new source type means writing one function and adding the decorator. No switch statements, no if/elif chains, no modification of the dispatch logic.
|
||||
|
||||
### Thread-safe State with Locks
|
||||
|
||||
**Where it lives:** `backend/app/engine/correlation.py` (`CorrelationState` class)
|
||||
|
||||
The correlation engine runs on a daemon thread but shares state structures (sliding windows, cooldown timestamps) that could be accessed during rule testing from the main Flask thread. Every method on `CorrelationState` acquires `self._lock` before touching `_windows` or `_cooldowns`. This prevents data races but means the correlation engine can't process two events in parallel. For the throughput this project targets, that's fine.
|
||||
|
||||
## Layer Separation
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────┐
|
||||
│ Routes Layer (app/routes/) │
|
||||
│ HTTP concerns only: URL mapping, rate limits │
|
||||
│ Does NOT: query databases, process data │
|
||||
└───────────────────────┬────────────────────────────┘
|
||||
▼
|
||||
┌────────────────────────────────────────────────────┐
|
||||
│ Controller Layer (app/controllers/) │
|
||||
│ Business logic: orchestrate models and engine │
|
||||
│ Does NOT: parse requests, format responses │
|
||||
└───────────────────────┬────────────────────────────┘
|
||||
▼
|
||||
┌────────────────────────────────────────────────────┐
|
||||
│ Model Layer (app/models/) │
|
||||
│ Data access: MongoEngine documents, queries │
|
||||
│ Does NOT: know about HTTP, Flask request/response│
|
||||
└───────────────────────┬────────────────────────────┘
|
||||
▼
|
||||
┌────────────────────────────────────────────────────┐
|
||||
│ Engine Layer (app/engine/) │
|
||||
│ Domain logic: normalization, severity, correlation│
|
||||
│ Does NOT: persist data, know about Flask │
|
||||
└────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### What Lives Where
|
||||
|
||||
**Routes** (`app/routes/`): Blueprint definitions, URL patterns, decorator stacks. Each route function is a thin wrapper that calls the corresponding controller function. Routes import from controllers and schemas, never from models directly.
|
||||
|
||||
**Controllers** (`app/controllers/`): Business logic coordination. The controller for log ingestion (`log_ctrl.ingest_log`) calls the normalizer, the severity classifier, the model's `create_event`, and the streaming publisher. Controllers access `g.validated` for input and `g.current_user` for auth context.
|
||||
|
||||
**Models** (`app/models/`): MongoEngine document definitions with query methods. `BaseDocument` provides shared functionality like `get_by_id`, `paginate`, and auto-updating `updated_at` timestamps. Models can call other models (Alert references LogEvent) but never import from routes or controllers.
|
||||
|
||||
**Engine** (`app/engine/`): Pure domain logic. The normalizer, severity classifier, and correlation engine live here. The correlation engine is the only component that reaches into both the streaming layer (to read events) and the model layer (to create alerts). This is a pragmatic trade-off to keep the daemon thread self-contained.
|
||||
|
||||
## Data Models
|
||||
|
||||
### LogEvent
|
||||
|
||||
```python
|
||||
# backend/app/models/LogEvent.py
|
||||
class LogEvent(BaseDocument):
|
||||
meta = {
|
||||
"collection": "log_events",
|
||||
"ordering": ["-timestamp"],
|
||||
"indexes": [
|
||||
"timestamp", "source_type", "severity",
|
||||
"source_ip", "dest_ip", "username",
|
||||
"hostname", "event_type", "scenario_run_id",
|
||||
],
|
||||
}
|
||||
timestamp = DateTimeField()
|
||||
source_type = StringField(required=True) # firewall, ids, auth, etc.
|
||||
source_ip = StringField()
|
||||
dest_ip = StringField()
|
||||
source_port = IntField()
|
||||
dest_port = IntField()
|
||||
severity = StringField(default="info")
|
||||
event_type = StringField() # login_failure, port_scan, etc.
|
||||
raw = DictField() # original payload, untouched
|
||||
normalized = DictField() # source-type-specific fields
|
||||
hostname = StringField()
|
||||
username = StringField()
|
||||
scenario_run_id = ObjectIdField() # links to ScenarioRun if simulated
|
||||
```
|
||||
|
||||
Nine indexes cover the query patterns used by the log viewer, pivot searches, and dashboard aggregations. The `raw` field preserves the original event exactly as submitted. The `normalized` field holds source-type-specific fields extracted by the normalizer. This dual storage means you can always go back to the original data if the normalizer had a bug.
|
||||
|
||||
### Alert
|
||||
|
||||
```python
|
||||
# backend/app/models/Alert.py
|
||||
class Alert(BaseDocument):
|
||||
rule_id = ObjectIdField(required=True)
|
||||
rule_name = StringField(required=True)
|
||||
severity = StringField(required=True)
|
||||
title = StringField(required=True) # "{rule_name} [{group_value}]"
|
||||
matched_event_ids = ListField(ObjectIdField()) # references to LogEvent docs
|
||||
matched_event_count = IntField(default=0)
|
||||
group_value = StringField() # the IP, username, etc. that grouped
|
||||
status = StringField(default="new")
|
||||
mitre_tactic = StringField()
|
||||
mitre_technique = StringField()
|
||||
acknowledged_by = StringField()
|
||||
acknowledged_at = DateTimeField()
|
||||
resolved_at = DateTimeField()
|
||||
```
|
||||
|
||||
Alerts link back to both the rule that generated them (`rule_id`) and the specific events that matched (`matched_event_ids`). The `get_with_events()` method loads the referenced LogEvent documents for the alert detail view. This is a manual join since MongoDB doesn't do relational joins, but the list of IDs is bounded by the correlation window size, so it's never thousands of documents.
|
||||
|
||||
### CorrelationRule
|
||||
|
||||
```python
|
||||
# backend/app/models/CorrelationRule.py
|
||||
class CorrelationRule(BaseDocument):
|
||||
name = StringField(required=True, unique=True)
|
||||
rule_type = StringField(required=True) # threshold, sequence, aggregation
|
||||
conditions = DictField(required=True) # type-specific config
|
||||
severity = StringField(required=True)
|
||||
enabled = BooleanField(default=True)
|
||||
mitre_tactic = StringField()
|
||||
mitre_technique = StringField()
|
||||
```
|
||||
|
||||
The `conditions` field is a flexible dict whose shape depends on `rule_type`. For threshold rules it contains `event_filter`, `threshold`, `window_seconds`, and `group_by`. For sequence rules it has `steps` (an ordered list of event filters). For aggregation rules it adds `aggregation_field` for counting distinct values. Validation of these shapes happens in the Pydantic schemas (`backend/app/schemas/rule.py:60-82`) using a `@model_validator` that dispatches to the correct condition schema based on `rule_type`.
|
||||
|
||||
## Security Architecture
|
||||
|
||||
### Threat Model
|
||||
|
||||
What the platform defends against:
|
||||
|
||||
1. **Credential stuffing on the login endpoint.** Rate limiting at both Nginx (3r/s for auth endpoints) and Flask (10/minute via `flask-limiter`) makes brute force impractical. The `verify_password_timing_safe` function prevents username enumeration through timing differences.
|
||||
|
||||
2. **Unauthorized access to SIEM data.** JWT-based authentication on every API endpoint (except `/v1/logs/ingest` and public auth routes). Role-based access control gates admin operations. The `endpoint()` decorator enforces this uniformly.
|
||||
|
||||
3. **Privilege escalation via role manipulation.** The `update_role` controller in `admin_ctrl.py:36-43` checks `User.count_admins()` before allowing demotion. You can't demote the last admin. Admins also can't deactivate or delete their own accounts.
|
||||
|
||||
4. **SSE token leakage.** SSE connections can't use the Authorization header (browser limitation with EventSource), so tokens go in the query string. The `extract_bearer_token()` function in `app/core/auth.py:97-102` checks the header first, then falls back to `request.args.get("token")`. This is a known trade-off. The token appears in server access logs but is transmitted over HTTPS.
|
||||
|
||||
What's out of scope:
|
||||
Network-level attacks (handled by infrastructure), database injection (MongoEngine parameterizes queries), XSS (React escapes by default, plus security headers from Nginx).
|
||||
|
||||
### Defense in Depth
|
||||
|
||||
```
|
||||
Nginx Layer
|
||||
├── Rate limiting (10r/s API, 3r/s auth)
|
||||
├── Connection limits (50 per IP in prod)
|
||||
├── Security headers (X-Frame-Options, X-Content-Type-Options, etc.)
|
||||
└── Request size limits (10MB max body)
|
||||
│
|
||||
Flask Layer
|
||||
├── flask-limiter (moving window, 200/min default, 10/min auth)
|
||||
├── Pydantic validation (all inputs validated before processing)
|
||||
├── JWT verification (expiry, required claims)
|
||||
└── Role enforcement (decorator-based RBAC)
|
||||
│
|
||||
Application Layer
|
||||
├── Argon2id password hashing
|
||||
├── Constant-time password verification
|
||||
├── Last-admin protection
|
||||
└── Self-action prevention (can't deactivate your own account)
|
||||
```
|
||||
|
||||
## Storage Strategy
|
||||
|
||||
### MongoDB: Persistent State
|
||||
|
||||
All documents that need to survive restarts. Log events, alerts, correlation rules, users, scenario runs. MongoEngine provides the ODM layer with type coercion and validation.
|
||||
|
||||
The `BaseDocument` class (`app/models/Base.py`) adds `created_at` and `updated_at` to every document and overrides `save()` to auto-update the timestamp. The `paginate()` class method provides consistent offset-based pagination across all collections.
|
||||
|
||||
Index strategy: every field used in a filter or sort gets an index. LogEvent has nine indexes, which is aggressive but appropriate for a query-heavy dashboard. Write performance takes a small hit, but reads (which dominate in a SIEM) stay fast.
|
||||
|
||||
### Redis: Ephemeral Streams and Rate Limits
|
||||
|
||||
Two Redis Streams with approximate maxlen of 10,000 entries each. The streams are ephemeral. Restarting Redis loses unprocessed messages, but since events are persisted in MongoDB first, the only impact is that the correlation engine might miss some events during the restart window. For a learning project, this is an acceptable trade-off. Production SIEMs would use persistent queues.
|
||||
|
||||
Rate limit counters also live in Redis. The `moving-window` strategy in `flask-limiter` stores sliding window counters keyed by client IP.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
All configuration flows through `backend/app/config.py` using Pydantic Settings:
|
||||
|
||||
```python
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=".env",
|
||||
env_file_encoding="utf-8",
|
||||
case_sensitive=False,
|
||||
)
|
||||
MONGO_URI: str = "mongodb://mongo:27017/siem"
|
||||
REDIS_URL: str = "redis://redis:6379/0"
|
||||
SECRET_KEY: str = "change-me-in-production"
|
||||
JWT_EXPIRATION_HOURS: int = 24
|
||||
CORRELATION_COOLDOWN_SECONDS: int = 300
|
||||
RATELIMIT_DEFAULT: str = "200/minute"
|
||||
RATELIMIT_AUTH: str = "10/minute"
|
||||
# ... ~40 more settings with defaults
|
||||
```
|
||||
|
||||
Every setting has a sensible default, so the project works out of the box with just `docker compose up`. But the defaults are explicitly not secure for production. The `SECRET_KEY` default is `"change-me-in-production"` and Redis has no password in dev mode.
|
||||
|
||||
### Development vs Production
|
||||
|
||||
**Development** (`dev.compose.yml`): Hot reload on both frontend and backend. No Nginx auth rate limiting. Redis without a password. MongoDB port exposed on host. Frontend runs as a Vite dev server with HMR.
|
||||
|
||||
**Production** (`compose.yml`): Gunicorn with 4 workers. Frontend pre-built and served as static files from Nginx. Redis requires `REDIS_PASSWORD`. Nginx adds security headers, connection limits, and aggressive caching for static assets. Docker resource limits are set (backend gets 2 CPUs, 1GB RAM).
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Bottlenecks
|
||||
|
||||
**Correlation engine is single-threaded.** The `CorrelationEngine._run()` loop processes events sequentially. Under heavy load (hundreds of events per second), the consumer group will accumulate a backlog. The `STREAM_READ_COUNT` setting (default 10) and `STREAM_BLOCK_MS` (default 2000) control the batch size and blocking behavior.
|
||||
|
||||
**MongoDB aggregations for the dashboard.** The `timeline_aggregation`, `severity_breakdown`, and `top_sources` methods all run aggregation pipelines. With millions of log events, these become slow. The `$dateTrunc` grouping in the timeline pipeline is particularly expensive without a compound index on `(timestamp, severity)`.
|
||||
|
||||
**SSE connections hold threads.** Each SSE client holds a Gunicorn worker thread open for the entire connection duration. With 4 workers and 3 SSE clients, only 1 worker is available for regular API requests. Production would need either more workers or an async SSE solution.
|
||||
|
||||
### Optimizations Already Present
|
||||
|
||||
**Rule caching.** The correlation engine caches enabled rules for `CORRELATION_RULE_CACHE_SECONDS` (default 30s) to avoid hitting MongoDB on every event. See `CorrelationEngine._get_rules()`.
|
||||
|
||||
**Approximate stream trimming.** `XADD` with `approximate=True` lets Redis trim the stream lazily rather than on every write. Slightly exceeds the 10,000 maxlen but avoids the per-write trim overhead.
|
||||
|
||||
**Sliding window cleanup.** `CorrelationState.get_window()` evicts expired entries every time a window is read, keeping memory bounded without a separate cleanup thread.
|
||||
|
||||
## Deployment Architecture
|
||||
|
||||
### Docker Compose Topology
|
||||
|
||||
```
|
||||
┌──────────┐
|
||||
:8431 ────→│ Nginx │
|
||||
└────┬─────┘
|
||||
│
|
||||
┌─────────┴─────────┐
|
||||
│ │
|
||||
┌────▼────┐ ┌────▼────┐
|
||||
│ Backend │ │ Static │
|
||||
│ (Flask) │ │ (built │
|
||||
│ :5000 │ │ React) │
|
||||
└────┬────┘ └─────────┘
|
||||
│
|
||||
┌────────┴────────┐
|
||||
│ │
|
||||
┌────▼────┐ ┌──────▼─────┐
|
||||
│ MongoDB │ │ Redis │
|
||||
│ :27017 │ │ :6379 │
|
||||
└─────────┘ └────────────┘
|
||||
```
|
||||
|
||||
Two Docker networks isolate traffic. The `frontend` network connects Nginx to the backend. The `backend` network connects the backend to MongoDB and Redis. MongoDB and Redis are not directly accessible from the frontend network.
|
||||
|
||||
Resource limits in production:
|
||||
- Nginx: 1 CPU, 256MB RAM
|
||||
- Backend: 2 CPUs, 1GB RAM
|
||||
- MongoDB: 1 CPU, 512MB RAM
|
||||
- Redis: 0.5 CPU, 256MB RAM
|
||||
|
||||
### Health Checks
|
||||
|
||||
Every service has a health check. The backend exposes `/health` which returns `"1"` (defined in `app/__init__.py`). MongoDB uses `mongosh --eval "db.adminCommand('ping')"`. Redis uses `redis-cli ping`. Nginx depends on the backend being healthy before it starts accepting traffic.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### MongoDB over PostgreSQL
|
||||
|
||||
MongoDB was chosen because log events are semi-structured. Different source types have different fields (firewall events have `action` and `protocol`, DNS events have `query` and `query_type`). Storing this in a relational schema would mean either a wide sparse table or a separate table per source type. MongoDB's flexible documents with the `DictField` for `raw` and `normalized` handle this naturally.
|
||||
|
||||
Trade-off: no transactional joins. The alert detail view requires a manual lookup of matched events by ID. With PostgreSQL, this would be a single JOIN query.
|
||||
|
||||
### Redis Streams over Kafka or RabbitMQ
|
||||
|
||||
Redis Streams provide enough pub/sub for this use case without adding another infrastructure dependency. The consumer group semantics (`XREADGROUP`, `XACK`) give exactly-once processing for the correlation engine. SSE endpoints use plain `XREAD` (no consumer group) since they're just tailing the stream for display.
|
||||
|
||||
Trade-off: no persistence guarantees, no multi-node replication, limited backpressure. A production SIEM would need something more robust.
|
||||
|
||||
### JWT over Sessions
|
||||
|
||||
Stateless auth means the backend doesn't need to store sessions in Redis or MongoDB. Each request carries its own proof of identity. The frontend stores the token in Zustand (persisted to localStorage via the `persist` middleware in `frontend/src/core/stores/auth.store.ts`).
|
||||
|
||||
Trade-off: you can't invalidate a JWT before it expires. If a user's account is deactivated, they can still make requests until the token's `exp` claim passes. The `endpoint()` decorator mitigates this by loading the user from MongoDB on every request and checking `is_active`.
|
||||
|
||||
### Pydantic for Request Validation
|
||||
|
||||
Every endpoint validates input through Pydantic schemas via the `@S` decorator. This catches bad data at the boundary before it reaches controller logic. The correlation rule schemas are particularly interesting. The `RuleCreateRequest` uses a `@model_validator` to dispatch condition validation based on `rule_type`, so threshold rules validate differently from sequence rules.
|
||||
|
||||
Trade-off: Pydantic adds import time and a layer of indirection. Simple endpoints that just take an ID don't need validation but still go through the decorator stack.
|
||||
|
||||
## Error Handling Strategy
|
||||
|
||||
### Error Hierarchy
|
||||
|
||||
```python
|
||||
# backend/app/core/errors.py
|
||||
AppError (500)
|
||||
├── NotFoundError (404)
|
||||
├── ValidationError (422, includes field-level details)
|
||||
├── AuthenticationError (401)
|
||||
├── ForbiddenError (403)
|
||||
└── ConflictError (409)
|
||||
```
|
||||
|
||||
All custom errors extend `AppError`, which carries a `status_code` and `message`. The `register_error_handlers()` function in `errors.py` attaches a single Flask error handler for `AppError` that serializes any subclass into a consistent JSON response: `{"error": "ErrorClassName", "message": "..."}`.
|
||||
|
||||
The `endpoint()` decorator adds a catch-all for unexpected exceptions. If a controller raises something that isn't an `AppError`, the decorator logs the traceback via structlog and returns a generic 500. This prevents stack traces from leaking to the client.
|
||||
|
||||
### Frontend Error Handling
|
||||
|
||||
The frontend mirrors this with `ApiError` in `frontend/src/core/lib/errors.ts`. The Axios response interceptor transforms HTTP errors into typed `ApiError` instances with codes like `AUTHENTICATION_ERROR`, `VALIDATION_ERROR`, etc. The React Query cache has global error handlers that show toast notifications for background query failures.
|
||||
|
||||
## Extensibility
|
||||
|
||||
### Adding a New Log Source Type
|
||||
|
||||
1. Add the source type to `SourceType` enum in `backend/app/models/LogEvent.py`
|
||||
2. Write a normalizer function in `backend/app/engine/normalizer.py`:
|
||||
|
||||
```python
|
||||
@_register(SourceType.YOUR_NEW_TYPE)
|
||||
def _normalize_your_type(raw: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"your_field": raw.get("your_field"),
|
||||
# ... extract source-specific fields
|
||||
}
|
||||
```
|
||||
|
||||
3. Optionally add severity patterns in `backend/app/engine/severity.py`
|
||||
|
||||
That's it. The registry pattern means no other code changes are needed. The ingest endpoint, SSE streaming, correlation engine, and frontend log viewer all work with any source type.
|
||||
|
||||
### Adding a New Correlation Rule Type
|
||||
|
||||
1. Add the type to `RuleType` enum in `backend/app/models/CorrelationRule.py`
|
||||
2. Add a Pydantic conditions schema in `backend/app/schemas/rule.py`
|
||||
3. Write the evaluator function in `backend/app/engine/correlation.py`:
|
||||
|
||||
```python
|
||||
def _evaluate_your_type(rule, event_data, state, rule_id, group_key):
|
||||
# Your evaluation logic
|
||||
# Return EvaluationResult if fired, None otherwise
|
||||
```
|
||||
|
||||
4. Register it in the `evaluators` dict inside `evaluate_rule()`
|
||||
|
||||
### Adding a New API Endpoint
|
||||
|
||||
1. Create or update a schema in `backend/app/schemas/`
|
||||
2. Write the controller function in `backend/app/controllers/`
|
||||
3. Add the route in `backend/app/routes/` with the decorator stack
|
||||
4. Add the frontend hook in `frontend/src/api/hooks/`
|
||||
5. Add the endpoint path to `frontend/src/config.ts`
|
||||
|
||||
## Limitations
|
||||
|
||||
**No horizontal scaling.** The correlation engine uses in-memory state (`CorrelationState`). Running multiple backend instances means each instance has its own sliding windows and cooldowns. Events would be split across instances via the consumer group, but the state wouldn't be shared. Fixing this would require moving correlation state to Redis.
|
||||
|
||||
**No event deduplication.** If the same event is ingested twice (network retry, for example), it gets stored twice and may trigger correlation rules twice. A production system would hash the raw event and check for duplicates before persisting.
|
||||
|
||||
**No log retention policy.** Events accumulate in MongoDB forever. The dashboard aggregations will slow down as the collection grows. A TTL index on `timestamp` or a periodic cleanup job would fix this.
|
||||
|
||||
**Single consumer for correlation.** The `CONSUMER_NAME` is hardcoded to `"engine-1"`. Adding more consumers would require partitioning logic to prevent duplicate alert generation.
|
||||
|
||||
## Key Files Reference
|
||||
|
||||
Backend core:
|
||||
- `backend/app/__init__.py` - Application factory, startup sequence
|
||||
- `backend/app/config.py` - All settings via Pydantic
|
||||
- `backend/app/core/auth.py` - JWT and password operations
|
||||
- `backend/app/core/streaming.py` - Redis Streams pub/sub and SSE
|
||||
- `backend/app/core/decorators/` - endpoint, S, R decorator stack
|
||||
- `backend/app/engine/correlation.py` - Correlation engine and rule evaluation
|
||||
- `backend/app/engine/normalizer.py` - Log normalization registry
|
||||
- `backend/app/engine/severity.py` - Severity classification
|
||||
- `backend/app/scenarios/runner.py` - Scenario playback threads
|
||||
|
||||
Frontend core:
|
||||
- `frontend/src/config.ts` - API endpoints, query keys, routes
|
||||
- `frontend/src/core/lib/api.ts` - Axios client with interceptors
|
||||
- `frontend/src/core/stores/auth.store.ts` - Auth state with persistence
|
||||
- `frontend/src/core/stores/stream.store.ts` - SSE event buffer
|
||||
- `frontend/src/api/hooks/useEventStream.ts` - SSE connection management
|
||||
|
||||
Infrastructure:
|
||||
- `dev.compose.yml` - Development Docker setup
|
||||
- `compose.yml` - Production Docker setup
|
||||
- `conf/nginx/nginx.conf` - Nginx main config with rate limits
|
||||
- `conf/nginx/prod.nginx` - Production server block with SSE passthrough
|
||||
|
||||
## Next Steps
|
||||
|
||||
Now that you understand the architecture:
|
||||
1. Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) for a code-level walkthrough
|
||||
2. Try modifying the normalizer to add a new source type and see the registry pattern in action
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Loading…
Reference in New Issue