docs: add Portia secrets scanner design document
Design for intermediate-level secrets scanner replacing the Container Security Scanner project. Covers pipeline architecture, 50+ detection rules, HIBP integration, CLI structure, and terminal output design.
This commit is contained in:
CarterPerez-dev
parent
7d81b29646
commit
c6c844db47
|
|
@ -0,0 +1,334 @@
|
|||
# Portia — Secrets Scanner Design Document
|
||||
|
||||
**Date**: 2026-02-20
|
||||
**Project**: Cybersecurity-Projects / intermediate / secrets-scanner
|
||||
**Replaces**: Container Security Scanner (overlap with Docker Security Audit)
|
||||
**Module**: `github.com/CarterPerez-dev/portia`
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Portia is a Go CLI tool that scans git repositories and directories for leaked API keys, passwords, tokens, and other secrets. It combines keyword pre-filtering, regex pattern matching, Shannon entropy analysis, and Have I Been Pwned breach verification into a pipeline architecture modeled after production scanners like Gitleaks and TruffleHog.
|
||||
|
||||
**Scan targets**: Git history (via go-git in-process) and filesystem directories.
|
||||
**Not in scope**: Docker image layer extraction (covered by the existing Docker Security Audit project).
|
||||
|
||||
---
|
||||
|
||||
## CLI Structure
|
||||
|
||||
### Commands
|
||||
|
||||
```
|
||||
portia scan [path] Scan a directory/file for secrets
|
||||
portia git [path] Scan git history for secrets
|
||||
portia init Generate .portia.toml config
|
||||
portia config rules List all built-in detection rules
|
||||
portia config test <rule> Test a specific rule against sample input
|
||||
```
|
||||
|
||||
### Global Flags
|
||||
|
||||
```
|
||||
--format string Output format: terminal (default), json, sarif
|
||||
--rules string Path to custom rules TOML file
|
||||
--verbose/-v Show full finding details (line context, entropy scores)
|
||||
--no-color Disable colored output
|
||||
--exit-code int Exit code when secrets found (default: 1, 0 for CI soft-fail)
|
||||
```
|
||||
|
||||
### Scan Flags
|
||||
|
||||
```
|
||||
portia scan:
|
||||
--exclude []string Glob patterns to skip (e.g., "vendor/**", "*.lock")
|
||||
--max-size string Skip files larger than (default: "1MB")
|
||||
--entropy float Override entropy threshold (default: per-rule)
|
||||
--hibp Enable HIBP password breach checking
|
||||
|
||||
portia git:
|
||||
--since string Only scan commits after date (e.g., "7 days ago")
|
||||
--branch string Scan specific branch (default: all)
|
||||
--depth int Max commit depth (default: unlimited)
|
||||
--staged-only Only scan currently staged changes (pre-commit hook mode)
|
||||
--hibp Enable HIBP password breach checking
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Architecture
|
||||
|
||||
```
|
||||
Source --> Chunker --> Keyword Filter --> Detector Pool --> Verifier (HIBP) --> Reporter
|
||||
```
|
||||
|
||||
### Stage 1: Source
|
||||
|
||||
Two implementations behind a common interface:
|
||||
- **DirSource**: Walks filesystem, respects .gitignore + .portiaignore, skips binaries and large files.
|
||||
- **GitSource**: Uses go-git to iterate commits, produces diffs (added lines only).
|
||||
|
||||
### Stage 2: Chunker
|
||||
|
||||
Splits source output into Chunk structs containing content, file path, line number, and optional git metadata (commit SHA, author, date).
|
||||
|
||||
### Stage 3: Keyword Filter
|
||||
|
||||
The critical performance optimization. Each rule declares case-insensitive keywords. Before any regex runs, the engine checks whether the chunk contains at least one keyword. This eliminates ~95% of chunks. Only rules whose keywords matched are forwarded to the detector pool for that chunk.
|
||||
|
||||
### Stage 4: Detector Pool
|
||||
|
||||
Bounded concurrency via errgroup.SetLimit(runtime.NumCPU()). For each chunk that passed keyword filtering, matched rules execute:
|
||||
1. Regex match on captured group
|
||||
2. Entropy check on captured value (if rule has a threshold)
|
||||
3. Stopword/allowlist filtering
|
||||
4. Structural validation (assignment operator present, not a template variable)
|
||||
|
||||
### Stage 5: Verifier (HIBP)
|
||||
|
||||
Post-processing enrichment, only when --hibp flag is set. Classifies each finding by secret type, routes passwords to HIBP k-anonymity API, marks results as verified_breach, not_breached, or skipped.
|
||||
|
||||
### Stage 6: Reporter
|
||||
|
||||
Three implementations behind a Reporter interface:
|
||||
- **TerminalReporter**: Angela-style colored output with severity colors, spinners, summary.
|
||||
- **JSONReporter**: Structured JSON to stdout.
|
||||
- **SARIFReporter**: SARIF v2.1.0 for GitHub/GitLab integration.
|
||||
|
||||
### Concurrency Model
|
||||
|
||||
- Channels buffered at 256 to prevent backpressure stalls
|
||||
- errgroup.WithContext for detector pool — first fatal error cancels everything
|
||||
- HIBP lookups run as a separate bounded pool (20 concurrent, 50 req/s rate limit) after scanning completes
|
||||
- Context propagation throughout for clean Ctrl+C cancellation
|
||||
|
||||
---
|
||||
|
||||
## Core Types
|
||||
|
||||
### Finding
|
||||
|
||||
```
|
||||
Finding {
|
||||
RuleID string
|
||||
Description string
|
||||
Severity string // critical, high, medium, low
|
||||
Match string
|
||||
Secret string // The captured secret value (redactable)
|
||||
Entropy float64
|
||||
FilePath string
|
||||
LineNumber int
|
||||
LineContent string
|
||||
CommitSHA string // Empty for dir scan
|
||||
Author string // Empty for dir scan
|
||||
CommitDate time.Time // Empty for dir scan
|
||||
HIBPStatus string // "breached", "clean", "skipped", "unchecked"
|
||||
BreachCount int
|
||||
}
|
||||
```
|
||||
|
||||
### Rule
|
||||
|
||||
```
|
||||
Rule {
|
||||
ID string
|
||||
Description string
|
||||
Severity string
|
||||
Keywords []string
|
||||
Pattern *regexp.Regexp
|
||||
SecretGroup int
|
||||
Entropy *float64 // nil = skip entropy check
|
||||
Allowlist Allowlist
|
||||
SecretType SecretType // Password, APIKey, Token, PrivateKey, etc.
|
||||
}
|
||||
```
|
||||
|
||||
### Chunk
|
||||
|
||||
```
|
||||
Chunk {
|
||||
Content string
|
||||
FilePath string
|
||||
LineStart int
|
||||
CommitSHA string
|
||||
Author string
|
||||
CommitDate time.Time
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rule Engine
|
||||
|
||||
### 50+ Built-in Rules
|
||||
|
||||
| Category | Count | Examples | Severity |
|
||||
|---|---|---|---|
|
||||
| Cloud providers | ~10 | AWS Access Key, AWS Secret Key, GCP API Key, Azure Client Secret, DigitalOcean PAT | Critical |
|
||||
| Source control | ~8 | GitHub PAT (classic), Fine-grained PAT, OAuth, GitLab PAT, Bitbucket App Password | Critical |
|
||||
| Payment/SaaS | ~8 | Stripe Secret Key, Stripe Restricted Key, SendGrid, Twilio, Shopify | Critical |
|
||||
| Communication | ~5 | Slack Bot/User/App Token, Slack Webhook, Discord Bot Token | High |
|
||||
| AI/ML services | ~4 | OpenAI, Anthropic, HuggingFace, Replicate | High |
|
||||
| Infrastructure | ~5 | Heroku, Terraform Cloud, Vault Token, PlanetScale, Supabase | Critical |
|
||||
| Cryptographic | ~3 | Private Keys (RSA/EC/DSA/OPENSSH), PGP, X.509 | Critical |
|
||||
| Authentication | ~4 | JWT, Bearer Token, Basic Auth, OAuth Client Secret | High |
|
||||
| Database | ~5 | PostgreSQL URI, MySQL URI, MongoDB URI, Redis URI, Generic DB URI | Critical |
|
||||
| Generic | ~3 | Generic API Key (entropy 3.7), Generic Secret (entropy 3.7), Generic Password | Medium |
|
||||
|
||||
### False Positive Defenses (5 Layers)
|
||||
|
||||
**Layer 1 — Keyword pre-filter**: Part of the pipeline. No keyword match = chunk skipped.
|
||||
|
||||
**Layer 2 — Structural validation**: Generic rules require an assignment operator between keyword and value (=, :, =>, :=, ||).
|
||||
|
||||
**Layer 3 — Stopwords**: ~1,500 common programming words. If captured secret contains a stopword, finding is dropped.
|
||||
|
||||
**Layer 4 — Allowlists**:
|
||||
- Global path allowlist: go.sum, package-lock.json, pnpm-lock.yaml, *.min.js, vendor/, node_modules/, .git/, dist-info/
|
||||
- Global value allowlist: placeholder values (EXAMPLE_KEY, your-api-key-here, xxxx, ${VAR}, {{template}}, os.Getenv(...))
|
||||
- Per-rule allowlists for rule-specific exclusions
|
||||
|
||||
**Layer 5 — Entropy validation**: Post-filter on captured secret value.
|
||||
- Base64 charset: 4.5 threshold
|
||||
- Hex charset: 3.0 threshold
|
||||
- Alphanumeric (generic rules): 3.7 threshold
|
||||
- Provider-specific rules with known prefixes: 3.0 threshold
|
||||
|
||||
### TOML Override Format (.portia.toml)
|
||||
|
||||
Users can disable built-in rules, add path excludes, allowlist values, and define custom rules. Custom rules use the same structure as built-in rules (id, description, severity, keywords, pattern, secret-group, entropy).
|
||||
|
||||
---
|
||||
|
||||
## HIBP Integration
|
||||
|
||||
### K-Anonymity Protocol
|
||||
|
||||
1. SHA-1 hash the plaintext password locally
|
||||
2. Send only the 5-character hex prefix to api.pwnedpasswords.com/range/{prefix}
|
||||
3. Receive ~800 suffixes with breach counts
|
||||
4. Match locally — the API never learns the password
|
||||
|
||||
### Architecture
|
||||
|
||||
- **HTTP Client**: hashicorp/go-retryablehttp with custom transport (MaxIdleConnsPerHost: 50, HTTP/2, TLS 1.2+)
|
||||
- **Cache**: hashicorp/golang-lru/v2/expirable — LRU cache keyed by 5-char prefix, 24-hour TTL
|
||||
- **Circuit Breaker**: sony/gobreaker — trips after 5 consecutive failures, 30s recovery timeout
|
||||
- **Rate Limiter**: golang.org/x/time/rate at 50 req/s with burst of 10
|
||||
- **Concurrency**: errgroup.SetLimit(20) for parallel lookups
|
||||
|
||||
### Secret Classification Router
|
||||
|
||||
Only human-chosen passwords are sent to HIBP. Machine-generated secrets (API keys, tokens, private keys) are skipped based on:
|
||||
- Known format prefixes (AKIA, ghp_, sk_live_, xoxb-, SG., sk-)
|
||||
- Keyword context (password, passwd, pwd in surrounding code)
|
||||
- Entropy heuristic (< 3.5 bits + < 32 chars = likely human-chosen)
|
||||
|
||||
### Privacy Safeguards
|
||||
|
||||
- Never transmit full hash or plaintext
|
||||
- Logger set to nil to prevent logging of /range/{prefix} URLs
|
||||
- Results reported as boolean compromised/not-compromised (breach count available in verbose mode only)
|
||||
- HTTPS exclusively, TLS 1.2+ enforced
|
||||
|
||||
---
|
||||
|
||||
## Terminal Output Design
|
||||
|
||||
### Color Mapping
|
||||
|
||||
- CRITICAL: red
|
||||
- HIGH: bold red
|
||||
- MEDIUM: yellow
|
||||
- LOW: cyan
|
||||
- File paths: hi-cyan
|
||||
- Rule IDs: blue
|
||||
- Secret values: dim (partially redacted)
|
||||
- Line numbers: dim italic
|
||||
- HIBP breach warnings: bold yellow
|
||||
|
||||
### Output Sections
|
||||
|
||||
1. Banner (alternating red/blue ASCII art)
|
||||
2. Spinner during scan (cyan braille frame + magenta message)
|
||||
3. Findings grouped by file, sorted by severity
|
||||
4. HIBP breach check section (when --hibp used)
|
||||
5. Summary (files scanned, rules evaluated, secrets found, duration)
|
||||
|
||||
### Redaction
|
||||
|
||||
Secret values are partially redacted by default: first 6 characters shown, rest replaced with "...". Full values shown only with --verbose flag.
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
PROJECTS/intermediate/secrets-scanner/
|
||||
cmd/portia/main.go
|
||||
internal/
|
||||
cli/ root.go, scan.go, git.go, init.go, config.go
|
||||
engine/ pipeline.go, chunk.go, detector.go, filter.go
|
||||
source/ source.go, directory.go, git.go
|
||||
rules/ registry.go, builtin.go, custom.go, entropy.go
|
||||
hibp/ client.go, cache.go, breaker.go, classify.go
|
||||
report/ reporter.go, terminal.go, json.go, sarif.go
|
||||
ui/ banner.go, color.go, symbol.go, spinner.go
|
||||
config/ config.go
|
||||
pkg/types/ types.go
|
||||
testdata/ repos/, fixtures/, golden/
|
||||
learn/ 00-OVERVIEW.md through 04-CHALLENGES.md
|
||||
.portia.toml, .golangci.yml, .gitignore, Justfile, go.mod, LICENSE, README.md
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
```
|
||||
github.com/spf13/cobra
|
||||
github.com/fatih/color
|
||||
github.com/go-git/go-git/v5
|
||||
github.com/pelletier/go-toml/v2
|
||||
github.com/hashicorp/golang-lru/v2
|
||||
github.com/sony/gobreaker/v2
|
||||
github.com/hashicorp/go-retryablehttp
|
||||
golang.org/x/sync
|
||||
golang.org/x/time
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- **rules/**: Table-driven tests for each of the 50+ rules with true positives, true negatives, and known false positive patterns.
|
||||
- **engine/**: Pipeline stages tested in isolation (chunker, keyword filter, detector, structural validation).
|
||||
- **rules/entropy.go**: Shannon entropy tested against known values from detect-secrets' test suite.
|
||||
- **hibp/**: HTTP test server mocks, cache behavior, circuit breaker states, classification routing.
|
||||
- **source/**: Directory walker respects gitignore, skips binaries, respects max-size. Git source produces correct diffs.
|
||||
- **report/**: JSON and SARIF output compared against golden files.
|
||||
- **config/**: TOML parsing, rule merging, disable/override behavior.
|
||||
|
||||
### Integration Tests
|
||||
|
||||
- testdata/fixtures/ with planted secrets across Python, YAML, .env, JSON, Go, JS files.
|
||||
- testdata/repos/ with small git repos containing commits with secrets added then deleted.
|
||||
- Golden file tests for JSON and SARIF output.
|
||||
|
||||
### CI
|
||||
|
||||
- go test -race ./... (mandatory)
|
||||
- golangci-lint v2 with Tier 1/2/3 linters
|
||||
- gofumpt + golines (80-char max)
|
||||
|
||||
---
|
||||
|
||||
## Learn Folder
|
||||
|
||||
Grounded in three real-world incidents:
|
||||
- **Uber 2022**: Hardcoded HCP Vault credentials in PowerShell script on GitHub
|
||||
- **Samsung 2022**: Leaked SmartThings secrets via accidental git push
|
||||
- **CircleCI 2023**: Rotated all customer secrets after infrastructure breach
|
||||
|
||||
Documentation follows the standard learn/ template: 00-OVERVIEW, 01-CONCEPTS, 02-ARCHITECTURE, 03-IMPLEMENTATION, 04-CHALLENGES.
|
||||
Loading…
Reference in New Issue