diff --git a/docs/plans/2026-02-20-secrets-scanner-design.md b/docs/plans/2026-02-20-secrets-scanner-design.md new file mode 100644 index 00000000..03480a2d --- /dev/null +++ b/docs/plans/2026-02-20-secrets-scanner-design.md @@ -0,0 +1,334 @@ +# Portia — Secrets Scanner Design Document + +**Date**: 2026-02-20 +**Project**: Cybersecurity-Projects / intermediate / secrets-scanner +**Replaces**: Container Security Scanner (overlap with Docker Security Audit) +**Module**: `github.com/CarterPerez-dev/portia` + +--- + +## Overview + +Portia is a Go CLI tool that scans git repositories and directories for leaked API keys, passwords, tokens, and other secrets. It combines keyword pre-filtering, regex pattern matching, Shannon entropy analysis, and Have I Been Pwned breach verification into a pipeline architecture modeled after production scanners like Gitleaks and TruffleHog. + +**Scan targets**: Git history (via go-git in-process) and filesystem directories. +**Not in scope**: Docker image layer extraction (covered by the existing Docker Security Audit project). + +--- + +## CLI Structure + +### Commands + +``` +portia scan [path] Scan a directory/file for secrets +portia git [path] Scan git history for secrets +portia init Generate .portia.toml config +portia config rules List all built-in detection rules +portia config test Test a specific rule against sample input +``` + +### Global Flags + +``` +--format string Output format: terminal (default), json, sarif +--rules string Path to custom rules TOML file +--verbose/-v Show full finding details (line context, entropy scores) +--no-color Disable colored output +--exit-code int Exit code when secrets found (default: 1, 0 for CI soft-fail) +``` + +### Scan Flags + +``` +portia scan: + --exclude []string Glob patterns to skip (e.g., "vendor/**", "*.lock") + --max-size string Skip files larger than (default: "1MB") + --entropy float Override entropy threshold (default: per-rule) + --hibp Enable HIBP password breach checking + +portia git: + --since string Only scan commits after date (e.g., "7 days ago") + --branch string Scan specific branch (default: all) + --depth int Max commit depth (default: unlimited) + --staged-only Only scan currently staged changes (pre-commit hook mode) + --hibp Enable HIBP password breach checking +``` + +--- + +## Pipeline Architecture + +``` +Source --> Chunker --> Keyword Filter --> Detector Pool --> Verifier (HIBP) --> Reporter +``` + +### Stage 1: Source + +Two implementations behind a common interface: +- **DirSource**: Walks filesystem, respects .gitignore + .portiaignore, skips binaries and large files. +- **GitSource**: Uses go-git to iterate commits, produces diffs (added lines only). + +### Stage 2: Chunker + +Splits source output into Chunk structs containing content, file path, line number, and optional git metadata (commit SHA, author, date). + +### Stage 3: Keyword Filter + +The critical performance optimization. Each rule declares case-insensitive keywords. Before any regex runs, the engine checks whether the chunk contains at least one keyword. This eliminates ~95% of chunks. Only rules whose keywords matched are forwarded to the detector pool for that chunk. + +### Stage 4: Detector Pool + +Bounded concurrency via errgroup.SetLimit(runtime.NumCPU()). For each chunk that passed keyword filtering, matched rules execute: +1. Regex match on captured group +2. Entropy check on captured value (if rule has a threshold) +3. Stopword/allowlist filtering +4. Structural validation (assignment operator present, not a template variable) + +### Stage 5: Verifier (HIBP) + +Post-processing enrichment, only when --hibp flag is set. Classifies each finding by secret type, routes passwords to HIBP k-anonymity API, marks results as verified_breach, not_breached, or skipped. + +### Stage 6: Reporter + +Three implementations behind a Reporter interface: +- **TerminalReporter**: Angela-style colored output with severity colors, spinners, summary. +- **JSONReporter**: Structured JSON to stdout. +- **SARIFReporter**: SARIF v2.1.0 for GitHub/GitLab integration. + +### Concurrency Model + +- Channels buffered at 256 to prevent backpressure stalls +- errgroup.WithContext for detector pool — first fatal error cancels everything +- HIBP lookups run as a separate bounded pool (20 concurrent, 50 req/s rate limit) after scanning completes +- Context propagation throughout for clean Ctrl+C cancellation + +--- + +## Core Types + +### Finding + +``` +Finding { + RuleID string + Description string + Severity string // critical, high, medium, low + Match string + Secret string // The captured secret value (redactable) + Entropy float64 + FilePath string + LineNumber int + LineContent string + CommitSHA string // Empty for dir scan + Author string // Empty for dir scan + CommitDate time.Time // Empty for dir scan + HIBPStatus string // "breached", "clean", "skipped", "unchecked" + BreachCount int +} +``` + +### Rule + +``` +Rule { + ID string + Description string + Severity string + Keywords []string + Pattern *regexp.Regexp + SecretGroup int + Entropy *float64 // nil = skip entropy check + Allowlist Allowlist + SecretType SecretType // Password, APIKey, Token, PrivateKey, etc. +} +``` + +### Chunk + +``` +Chunk { + Content string + FilePath string + LineStart int + CommitSHA string + Author string + CommitDate time.Time +} +``` + +--- + +## Rule Engine + +### 50+ Built-in Rules + +| Category | Count | Examples | Severity | +|---|---|---|---| +| Cloud providers | ~10 | AWS Access Key, AWS Secret Key, GCP API Key, Azure Client Secret, DigitalOcean PAT | Critical | +| Source control | ~8 | GitHub PAT (classic), Fine-grained PAT, OAuth, GitLab PAT, Bitbucket App Password | Critical | +| Payment/SaaS | ~8 | Stripe Secret Key, Stripe Restricted Key, SendGrid, Twilio, Shopify | Critical | +| Communication | ~5 | Slack Bot/User/App Token, Slack Webhook, Discord Bot Token | High | +| AI/ML services | ~4 | OpenAI, Anthropic, HuggingFace, Replicate | High | +| Infrastructure | ~5 | Heroku, Terraform Cloud, Vault Token, PlanetScale, Supabase | Critical | +| Cryptographic | ~3 | Private Keys (RSA/EC/DSA/OPENSSH), PGP, X.509 | Critical | +| Authentication | ~4 | JWT, Bearer Token, Basic Auth, OAuth Client Secret | High | +| Database | ~5 | PostgreSQL URI, MySQL URI, MongoDB URI, Redis URI, Generic DB URI | Critical | +| Generic | ~3 | Generic API Key (entropy 3.7), Generic Secret (entropy 3.7), Generic Password | Medium | + +### False Positive Defenses (5 Layers) + +**Layer 1 — Keyword pre-filter**: Part of the pipeline. No keyword match = chunk skipped. + +**Layer 2 — Structural validation**: Generic rules require an assignment operator between keyword and value (=, :, =>, :=, ||). + +**Layer 3 — Stopwords**: ~1,500 common programming words. If captured secret contains a stopword, finding is dropped. + +**Layer 4 — Allowlists**: +- Global path allowlist: go.sum, package-lock.json, pnpm-lock.yaml, *.min.js, vendor/, node_modules/, .git/, dist-info/ +- Global value allowlist: placeholder values (EXAMPLE_KEY, your-api-key-here, xxxx, ${VAR}, {{template}}, os.Getenv(...)) +- Per-rule allowlists for rule-specific exclusions + +**Layer 5 — Entropy validation**: Post-filter on captured secret value. +- Base64 charset: 4.5 threshold +- Hex charset: 3.0 threshold +- Alphanumeric (generic rules): 3.7 threshold +- Provider-specific rules with known prefixes: 3.0 threshold + +### TOML Override Format (.portia.toml) + +Users can disable built-in rules, add path excludes, allowlist values, and define custom rules. Custom rules use the same structure as built-in rules (id, description, severity, keywords, pattern, secret-group, entropy). + +--- + +## HIBP Integration + +### K-Anonymity Protocol + +1. SHA-1 hash the plaintext password locally +2. Send only the 5-character hex prefix to api.pwnedpasswords.com/range/{prefix} +3. Receive ~800 suffixes with breach counts +4. Match locally — the API never learns the password + +### Architecture + +- **HTTP Client**: hashicorp/go-retryablehttp with custom transport (MaxIdleConnsPerHost: 50, HTTP/2, TLS 1.2+) +- **Cache**: hashicorp/golang-lru/v2/expirable — LRU cache keyed by 5-char prefix, 24-hour TTL +- **Circuit Breaker**: sony/gobreaker — trips after 5 consecutive failures, 30s recovery timeout +- **Rate Limiter**: golang.org/x/time/rate at 50 req/s with burst of 10 +- **Concurrency**: errgroup.SetLimit(20) for parallel lookups + +### Secret Classification Router + +Only human-chosen passwords are sent to HIBP. Machine-generated secrets (API keys, tokens, private keys) are skipped based on: +- Known format prefixes (AKIA, ghp_, sk_live_, xoxb-, SG., sk-) +- Keyword context (password, passwd, pwd in surrounding code) +- Entropy heuristic (< 3.5 bits + < 32 chars = likely human-chosen) + +### Privacy Safeguards + +- Never transmit full hash or plaintext +- Logger set to nil to prevent logging of /range/{prefix} URLs +- Results reported as boolean compromised/not-compromised (breach count available in verbose mode only) +- HTTPS exclusively, TLS 1.2+ enforced + +--- + +## Terminal Output Design + +### Color Mapping + +- CRITICAL: red +- HIGH: bold red +- MEDIUM: yellow +- LOW: cyan +- File paths: hi-cyan +- Rule IDs: blue +- Secret values: dim (partially redacted) +- Line numbers: dim italic +- HIBP breach warnings: bold yellow + +### Output Sections + +1. Banner (alternating red/blue ASCII art) +2. Spinner during scan (cyan braille frame + magenta message) +3. Findings grouped by file, sorted by severity +4. HIBP breach check section (when --hibp used) +5. Summary (files scanned, rules evaluated, secrets found, duration) + +### Redaction + +Secret values are partially redacted by default: first 6 characters shown, rest replaced with "...". Full values shown only with --verbose flag. + +--- + +## Project Structure + +``` +PROJECTS/intermediate/secrets-scanner/ + cmd/portia/main.go + internal/ + cli/ root.go, scan.go, git.go, init.go, config.go + engine/ pipeline.go, chunk.go, detector.go, filter.go + source/ source.go, directory.go, git.go + rules/ registry.go, builtin.go, custom.go, entropy.go + hibp/ client.go, cache.go, breaker.go, classify.go + report/ reporter.go, terminal.go, json.go, sarif.go + ui/ banner.go, color.go, symbol.go, spinner.go + config/ config.go + pkg/types/ types.go + testdata/ repos/, fixtures/, golden/ + learn/ 00-OVERVIEW.md through 04-CHALLENGES.md + .portia.toml, .golangci.yml, .gitignore, Justfile, go.mod, LICENSE, README.md +``` + +### Dependencies + +``` +github.com/spf13/cobra +github.com/fatih/color +github.com/go-git/go-git/v5 +github.com/pelletier/go-toml/v2 +github.com/hashicorp/golang-lru/v2 +github.com/sony/gobreaker/v2 +github.com/hashicorp/go-retryablehttp +golang.org/x/sync +golang.org/x/time +``` + +--- + +## Testing Strategy + +### Unit Tests + +- **rules/**: Table-driven tests for each of the 50+ rules with true positives, true negatives, and known false positive patterns. +- **engine/**: Pipeline stages tested in isolation (chunker, keyword filter, detector, structural validation). +- **rules/entropy.go**: Shannon entropy tested against known values from detect-secrets' test suite. +- **hibp/**: HTTP test server mocks, cache behavior, circuit breaker states, classification routing. +- **source/**: Directory walker respects gitignore, skips binaries, respects max-size. Git source produces correct diffs. +- **report/**: JSON and SARIF output compared against golden files. +- **config/**: TOML parsing, rule merging, disable/override behavior. + +### Integration Tests + +- testdata/fixtures/ with planted secrets across Python, YAML, .env, JSON, Go, JS files. +- testdata/repos/ with small git repos containing commits with secrets added then deleted. +- Golden file tests for JSON and SARIF output. + +### CI + +- go test -race ./... (mandatory) +- golangci-lint v2 with Tier 1/2/3 linters +- gofumpt + golines (80-char max) + +--- + +## Learn Folder + +Grounded in three real-world incidents: +- **Uber 2022**: Hardcoded HCP Vault credentials in PowerShell script on GitHub +- **Samsung 2022**: Leaked SmartThings secrets via accidental git push +- **CircleCI 2023**: Rotated all customer secrets after infrastructure breach + +Documentation follows the standard learn/ template: 00-OVERVIEW, 01-CONCEPTS, 02-ARCHITECTURE, 03-IMPLEMENTATION, 04-CHALLENGES.