Cybersecurity-Projects/PROJECTS/intermediate/secrets-scanner/learn/02-ARCHITECTURE.md

System Architecture

This document breaks down how Portia is designed and why certain architectural decisions were made. We'll trace requests through the system and explain the tradeoffs.

High Level Architecture

┌──────────────────────────────────────────────────────┐
│                       CLI                            │
│           root.go, scan.go, git.go                   │
└───────────────────────┬──────────────────────────────┘
                        │
              ┌─────────┴─────────┐
              ▼                   ▼
      ┌──────────────┐   ┌──────────────┐
      │   Directory   │   │     Git      │
      │    Source     │   │    Source     │
      │ directory.go  │   │   git.go     │
      └──────┬───────┘   └──────┬───────┘
             │                   │
             └─────────┬─────────┘
                       │
                       ▼ chan types.Chunk
              ┌─────────────────┐
              │    Pipeline     │
              │  pipeline.go    │
              ├─────────────────┤
              │                 │
              │  ┌───────────┐  │
              │  │ Worker 1  │  │
              │  │ detector  │  │
              │  └───────────┘  │
              │  ┌───────────┐  │
              │  │ Worker 2  │  │
              │  │ detector  │  │
              │  └───────────┘  │
              │  ┌───────────┐  │
              │  │ Worker N  │  │
              │  │ detector  │  │
              │  └───────────┘  │
              │                 │
              └────────┬────────┘
                       │
                       ▼ chan types.Finding
              ┌─────────────────┐
              │   Collector     │
              │  dedup + merge  │
              └────────┬────────┘
                       │
              ┌────────┴────────┐
              │                 │
              ▼                 ▼
      ┌──────────────┐  ┌──────────────┐
      │  HIBP Check  │  │   Reporter   │
      │  (optional)  │  │  term/json/  │
      │  client.go   │  │    sarif     │
      └──────┬───────┘  └──────────────┘
             │                 ▲
             └─────────────────┘

Component Breakdown

CLI Layer (internal/cli/)

Purpose: Parse command line arguments and orchestrate the scan workflow.

Responsibilities:

• Route scan, git, init, pyproject, config commands to their handlers
• Merge CLI flags with TOML config file values (CLI flags take precedence)
• Create the appropriate Source, run the Pipeline, optionally check HIBP, produce output

Interfaces: Uses Cobra for argument parsing. Each command is a cobra.Command with a RunE function. The executeScan function in scan.go is shared between scan and git commands.

Config Loader (internal/config/)

Purpose: Load and merge configuration from .portia.toml files.

Responsibilities:

• Search for config files in three locations: current directory, .portia/config.toml, ~/.config/portia/config.toml
• Fall back to pyproject.toml ([tool.portia] table) when no .portia.toml is found
• Parse TOML into a Config struct with sections for Rules, Scan, Output, HIBP, Allowlist
• Provide default templates for portia init and portia pyproject

Interfaces: Load(path string) (*Config, error) returns a config or error. Empty path triggers auto-discovery.

Source Interface (internal/source/)

Purpose: Produce chunks of text from various inputs (directories, git history).

Responsibilities:

• Walk filesystem or git object tree
• Skip binary files, excluded paths, oversized files
• Split content into 50-line chunks with file path and line number metadata
• Send chunks into a channel for pipeline consumption

Interfaces:

type Source interface {
    Chunks(ctx context.Context, out chan<- types.Chunk) error
    String() string
}

Directory source (directory.go):

• Uses filepath.WalkDir for filesystem traversal
• Skips .git, node_modules, vendor, __pycache__, .venv
• Checks file size against configurable max (default 1MB)
• Chunks files into 50-line segments using a buffered scanner

Git source (git.go):

• Uses go-git v5 for in-process git operations
• scanHistory: walks commit log backwards, extracts file content from each commit's tree
• scanStaged: reads git index entries for staged-only scanning
• Supports --branch, --since, --depth filters

Rule Registry (internal/rules/)

Purpose: Store detection rules and provide fast keyword-based lookup.

Responsibilities:

• Store rules in a map keyed by rule ID
• Provide MatchKeywords(content) that returns only rules whose keywords appear in the content
• Support enabling/disabling rules
• Maintain global path and value allowlists

Interfaces: Register(rule), Get(id), All(), MatchKeywords(content), Disable(ids...), Len()

Detection Engine (internal/engine/)

Purpose: Apply rules to chunks and produce findings.

Detector (detector.go):

• Takes a chunk, runs keyword pre-filter via registry
• For each matched rule, scans line by line with regex
• Extracts secret from capture group
• Validates entropy if the rule has an entropy threshold
• Runs through FilterFinding for false positive reduction

Filter (filter.go):

• IsPlaceholder - checks against GlobalValueAllowlist patterns
• IsTemplated - checks for ${...}, {{...}}, os.getenv(), process.env.
• IsStopword - splits secret on _-./ delimiters, checks parts against 700+ stopwords
• IsAllowedPath - checks file path against GlobalPathAllowlist
• FilterFinding - orchestrates all checks, returns true if finding is real

Pipeline (pipeline.go):

• Creates errgroup with source goroutine + N worker goroutines + collector goroutine
• Workers pull chunks from channel, run detector, push findings to findings channel
• Collector merges all findings, deduplicates by ruleID+filePath+secret+commitSHA

HIBP Client (internal/hibp/)

Purpose: Check detected secrets against the Have I Been Pwned breach database.

Responsibilities:

• SHA-1 hash computation
• k-anonymity API queries (5-char prefix)
• LRU cache (10,000 entries) for repeated lookups
• Circuit breaker (5 failures = 60s cooldown) Interfaces: Check(ctx, secret) (Result, error)

Reporters (internal/reporter/)

Purpose: Format scan results for output.

Terminal (terminal.go): Colored output with severity-based colors (red for CRITICAL, yellow for MEDIUM), secret masking (show first/last few characters), SHA truncation for git commits, HIBP breach status.

JSON (json.go): Structured JSON with findings array and summary object. Secrets are masked in output.

SARIF (sarif.go): SARIF v2.1.0 compliant output with tool metadata, rule definitions, results with locations and properties.

Interfaces: Reporter interface with Report(w io.Writer, result *types.ScanResult) error. Factory function New(format) Reporter returns the appropriate implementation.

Data Flow

Tracing: portia scan ./myproject

Step-by-step walkthrough of what happens when you run a directory scan:

1. CLI parses arguments
   root.go:init() → cobra.OnInitialize(initConfig)
   scan.go:runScan() receives path="./myproject"

2. Config loading
   root.go:initConfig() → config.Load(cfgFile)
   Merges CLI flags with TOML config
   Format defaults to "terminal", maxSize defaults to 1MB

3. Registry setup
   scan.go:runScan() → rules.NewRegistry() + rules.RegisterBuiltins(reg)
   Loads 150 rules into the registry map
   Applies disabled rules from config: reg.Disable(cfg.Rules.Disable...)

4. Source creation
   scan.go:runScan() → source.NewDirectory(path, maxSize, excludes)
   Creates Directory struct with path, max file size, exclude patterns

5. Pipeline execution
   scan.go:executeScan() → engine.NewPipeline(reg).Run(ctx, src)

   5a. Source goroutine starts
       Calls src.Chunks(ctx, chunks)
       WalkDir traverses ./myproject
       Skips .git, node_modules, vendor, binary extensions
       Splits each file into 50-line chunks
       Sends each chunk into the chunks channel

   5b. Worker goroutines start (2-16 based on NumCPU)
       Each pulls chunks from the channel
       Calls detector.Detect(chunk):
         - reg.MatchKeywords(chunk.Content) → only rules with matching keywords
         - For each matched rule, scan each line with rule.Pattern regex
         - Extract secret from capture group
         - If rule has entropy threshold, compute Shannon entropy and compare
         - Run FilterFinding: IsPlaceholder → IsTemplated → IsStopword → path allowlist
         - If all checks pass, create Finding and send to findings channel

   5c. Collector goroutine
       Pulls findings from findings channel
       Appends to allFindings slice (mutex-protected)

   5d. Wait for all goroutines (errgroup.Wait)
       Dedup findings by ruleID+filePath+secret+commitSHA

6. HIBP verification (if --hibp flag)
   scan.go:checkHIBP(ctx, result)
   For each finding, calls client.Check(ctx, finding.Secret)
   Updates finding.HIBPStatus and finding.BreachCount

7. Reporter output
   scan.go:executeScan() → reporter.New(format).Report(os.Stdout, result)
   Terminal: colored table with severity, rule, file:line, masked secret
   JSON: structured JSON to stdout
   SARIF: SARIF v2.1.0 JSON to stdout

Concurrency Model

The pipeline uses Go's errgroup pattern for structured concurrency:

                    errgroup
                   ┌────────────────────────────────┐
                   │                                │
  Source goroutine │  ──chunks──▶  Worker 1          │
                   │              Worker 2          │
                   │              ...               │
                   │              Worker N          │
                   │              ──findings──▶     │
                   │                    Collector   │
                   │                                │
                   └────────────────────────────────┘

Why bounded workers? CPU-bound regex matching doesn't benefit from unbounded parallelism. Too many goroutines competing for CPU time causes context switching overhead. The formula min(max(NumCPU, 2), 16) gives 2 workers on single-core machines and caps at 16 on large servers.

Why errgroup? It provides two things: (1) if any goroutine returns an error, the context is cancelled and all goroutines wind down cleanly, and (2) g.Wait() blocks until all goroutines complete, giving you a single point to check for errors.

Channel sizing: Channels are buffered at workers * 4. This allows the source to stay ahead of workers (avoiding blocking on sends) without unbounded memory growth. If workers are slow, the source will block once the buffer fills, providing natural backpressure.

The detectWg dance: Workers share a separate sync.WaitGroup so we know when all detection is done. The collector goroutine runs in the same errgroup but only closes after all workers finish. This prevents the collector from exiting early while findings are still being produced. See pipeline.go:52-77.

Configuration Resolution

Configuration is resolved in this order (later overrides earlier):

1. Defaults (hardcoded)
   Format: "terminal"
   MaxSize: 1MB (1 << 20)
   Workers: min(max(NumCPU, 2), 16)
   HIBP: disabled
   Verbose: false
   NoColor: false

2. Config file (.portia.toml)
   Searched in order:
     .portia.toml (current directory)
     .portia/config.toml
     ~/.config/portia/config.toml
   First found is loaded. Later paths are not checked.
   If none found, falls back to pyproject.toml ([tool.portia] table).

3. CLI flags
   --format, --verbose, --no-color, --exclude, --max-size, --hibp, --config
   These always win over config file values.

This merge logic is in internal/cli/root.go:initConfig(). The pattern is: check if the CLI flag was explicitly set (non-zero/non-empty), and only fall back to config file value if the flag wasn't set.

Rule Matching Strategy

The detection pipeline is optimized for speed. Regex matching is expensive, so the goal is to avoid running regex against content that will never match.

Content chunk (50 lines of code)
        │
        ▼
┌───────────────────┐
│  Keyword Filter   │  ← O(rules * keywords) string.Contains
│  ~95% eliminated  │
└────────┬──────────┘
         │ Only rules whose keywords appear in this chunk
         ▼
┌───────────────────┐
│  Line-by-Line     │  ← O(lines * matched_rules) regex
│  Regex Matching   │
└────────┬──────────┘
         │ Raw matches with capture groups
         ▼
┌───────────────────┐
│  Secret Extract   │  ← Extract from capture group
│  + Entropy Check  │     Discard if below threshold
└────────┬──────────┘
         │ Validated candidates
         ▼
┌───────────────────┐
│  Filter Chain     │  ← IsPlaceholder → IsTemplated
│  5-layer check    │     → IsStopword → Allowlists
└────────┬──────────┘
         │ Real findings only
         ▼
      Finding

The keyword filter is the key performance optimization. If a 50-line chunk of HTML doesn't contain any strings like password, secret, key, token, AKIA, ghp_, sk_live, etc., then zero rules will match and zero regex patterns need to run against it. In practice, this eliminates the vast majority of chunks.