Closes 36 findings from the project audit. The headline win is correctness:
the daemon now actually does what the README claims, all three audit-log
integrity layers verify, and the rotation control loop terminates.
Critical fixes:
- Orchestrator bumps Credential.last_rotated_at on success and writes a
sealed credential_versions row, so the policy evaluator stops re-firing
the same rotation every tick
- Envelope encryption (AES-256-GCM, KEK-wrapped DEK, AAD-bound) is now
wired into the rotation success path; the credential_versions table is
no longer dead schema
- cre audit verify checks all three layers (hash chain + HMAC ratchet +
Merkle batches) instead of only the chain
- BatchSealerScheduler runs every 300s and on shutdown so audit_batches
actually fills with signed Merkle roots
- Compliance bundle exports the real audit_batches list and covers
public_key.pem under the manifest's checksum
- cre run/watch hard-fail without CRE_HMAC_KEY_HEX + CRE_KEK_HEX rather
than silently using zero defaults
Quality fixes:
- HTTP::Client wrapper with connect/read timeouts + bounded retry-with-
jitter on 408/429/5xx; Telegram error logs redact the bot token
- EventBus dispatch isolates Block subscribers behind select+timeout so
one stuck subscriber can't pin the bus
- RotationWorker checks rotations.in_flight before dispatching to dedupe
duplicate schedules; PG enforces it at the DB via partial unique index
- Commit-step failures transition rotations to Inconsistent (a terminal
state) and raise a critical alert instead of pretending success
- env_file rotator uses per-PID pending paths plus an exclusive flock on
a sibling .lock file so two daemons can't race on the rename
- Versioned migration runner replaces the soup of CREATE IF NOT EXISTS;
SQLite gains BEFORE UPDATE / BEFORE DELETE triggers on audit_events
- AuditSubscriber publishes a critical alert + panics by default when
log writes fail, instead of silently dropping
- Engine.stop drains via an ack channel with a 2s deadline rather than
the previous magic 50ms sleep
- Policy DSL moved into CRE::Policy::Dsl module (consumers `include` it);
multiple matching policies raise PolicyConflictError
- Evaluator uses credentials.overdue() per-max_age group instead of
loading every credential each tick
- AuditRepo gains each_in_range streaming and all_batches enumeration;
bundle export streams instead of buffering
- New cre verify-bundle <zip> CLI re-runs every check the bundle README
documents; new cre tui-demo for synthetic event preview
Tests: 179 unit (up from 159), all passing. New rotation_worker_spec,
new SigV4 regression vector, new orchestrator success-path coverage
(envelope write + last_rotated_at bump + Inconsistent state).
Docs: README + learn/00..04 aligned with the post-audit behavior;
removed /snooze (was a stub) and the AwsIamKey/Database CredentialKind
variants that had no rotators. Added required-env-var documentation.
Justfile mirrors the haskell-reverse-proxy pattern (grouped recipes:
build, run, demo, test, quality, db, util). 33 recipes covering full
lifecycle - including parameterized recipes (cre rotate ID, cre policy
show NAME) that read more naturally as 'just rotate ...' than as
make targets.
User-edited docker-compose.yml port mappings preserved
(postgres:18 on 6022, localstack on 45666, vault on 18201,
fake-github on 7115).
Tier 1 (cre demo): in-memory SQLite + tempfile .env rotator. Walks
through inventory -> before -> 4-step rotation -> after -> audit chain
verify. Narrates every step event live with green/red glyphs. Runs in
under 1 second on commodity hardware, no external deps.
Tier 2 (docker/docker-compose.yml): postgres:16, localstack
(secretsmanager), hashicorp/vault dev mode, plus a fake-GitHub Flask
app that mocks /user, POST/DELETE /user/personal-access-tokens.
config/demo-full.cr.example documents env vars for the full stack.
Also fixed env_file rotator newline preservation (lines(chomp: true)
+ explicit \n join) so the rotated file is properly delimited.
Stack:
- Ansi: stdlib-only escape helpers (move, colorize, strip)
- State: rolling view of active rotations + recent events; trims to N
- Renderer: paints header / status / active / recent panels; works against
any IO so tests assert against IO::Memory
- Tui: ties State + Renderer to event bus; coalesces repaints to a tick
interval (default 200ms) to avoid flicker under burst events
Active rotation rows show step progress as a filled-bar (▰▰▰▱). Recent
events list latest 8 with timestamp + severity glyph (✓ / ⚠ / !). The
renderer's pad/center helpers correctly account for ANSI escape widths
when computing visible column width.
9 specs cover ANSI helpers (color codes, cursor move, strip), State
event-to-row mapping with completion bookkeeping and ring-buffer trim,
and Renderer output assertions across empty + active + populated states.
LogNotifier subscribes (Drop overflow) and emits stdlib Log lines with
structured kwargs (credential_id, rotation_id, severity, etc.) suitable
for downstream ingestion by journald/vector/fluentd.
Telegram is a thin HTTP::Client wrapper for sendMessage and getUpdates.
TelegramSubscriber sends emoji-prefixed alerts on RotationFailed,
DriftDetected, PolicyViolation, AlertRaised; success notifications gated
by the notify_on_success flag. Errors are swallowed so a flaky bot never
blocks the engine.
TelegramBot does long-polling getUpdates and dispatches commands:
- viewer tier: /status /queue /history /alerts /help
- operator tier: viewer + /rotate + /snooze
ACL is by chat_id allowlist (bot token + chat IDs in env vars).
/rotate publishes RotationScheduled to the bus.
11 unit specs cover send_message + error handling + getUpdates parsing,
subscriber dispatch, success-suppression flag, and bot ACL enforcement
(unauthorized / viewer-blocked-from-mutations / operator-can-rotate).
Initial tick fires immediately on start so first policy evaluation
doesn't wait for the full interval on boot. The PolicyEvaluator
subscribes to SchedulerTick events and invokes evaluate_all -> overdue
discovery -> RotationScheduled fan-out. 3 specs verify boot-tick,
periodic cadence, and clean stop().
Github::Client wraps the bearer-auth REST API with required
'X-GitHub-Api-Version: 2022-11-28' header. Implements me, create_pat
(POST /user/personal-access-tokens), delete_pat (DELETE by id).
GithubPatRotator's 4-step contract:
- generate: create_pat returns new id + token value (fresh PAT alongside old)
- apply: no-op (create_pat already active)
- verify: probe-client uses the NEW PAT to GET /user; success means it works
- commit: delete OLD PAT by id (tracked via old_pat_id tag)
- rollback_apply: delete NEW PAT
7 unit specs verify create/delete/me round-trips, full rotation path,
verify-on-401, and rollback deletion.
Rotator base exposes the four lifecycle methods as abstract; subclasses
register at compile time via 'register_as :kind' macro. REGISTRY is
populated as soon as the rotator file is required - drop a new file in
src/cre/rotators/ and the orchestrator can dispatch to it.
EnvFileRotator implements the simplest rotation:
- generate: 32 random bytes -> base64-urlsafe (no padding)
- apply: write to PATH.pending atomically (in-place line replace, 0600)
- verify: parse pending file, confirm key=value present and non-empty
- commit: rename PATH.pending -> PATH (atomic on POSIX)
- rollback_apply: unlink PATH.pending
RotationOrchestrator runs the 4-step contract with full event publishing,
state machine transitions in DB (generating -> applying -> verifying ->
committing -> completed | failed), and rollback_apply on apply/verify
failure. 8 specs cover happy path + raised-during-apply + rollback verification.
- Policy struct: matcher Proc, max_age, warn_at, enforce_action, notify_channels, triggers
- Builder validates all required fields at .build, raises BuilderError with the policy name
- DSL exposes top-level 'policy' method via 'with builder yield' so all
builder methods (description, match, max_age, enforce, notify_via, etc.)
are receiver-less inside the block
- Symbol literals autocast to Action/Channel/Trigger enums on direct calls;
Symbol overload for splat parameters (notify_via :telegram, :email)
- Evaluator subscribes to bus, fires PolicyViolation + RotationScheduled
(rotate_immediately) or AlertRaised (notify_only/quarantine) when overdue
- 15 unit specs cover Policy.matches?/overdue?/in_warning_window?,
Builder validation, DSL syntax with closures, evaluator action dispatch
Events form a Crystal class hierarchy so subscribers pattern-match
exhaustively. EventBus delivers to subscribers via Crystal channels with
per-subscriber overflow policy (Block for audit, Drop for best-effort).
AuditSubscriber listens for all rotation/policy/drift/alert events and
serializes them into the hash-chained audit log via AuditLog.append.
Engine wires Persistence + AuditLog + AuditSubscriber + EventBus together
so callers just .start/.stop. 7 unit specs verify boot, fanout to N
subscribers, drop-on-overflow semantics, and end-to-end event -> audit row.
Three layers of integrity:
- Hash chain (SHA-256 chained) catches silent edits
- HMAC ratchet rotates keys every N entries with HKDF-style derivation
and zeroizes old key bytes from memory
- Merkle batch sealing builds tree over content_hashes, signs root
with Ed25519 (FFI to libcrypto EVP_PKEY_ED25519 directly since stdlib
lacks the high-level OpenSSL::PKey wrapper)
AuditLog.append is mutex-guarded for serial writes; verify_chain detects
both single-row tampering and structural inconsistency. BatchSealer's
pack_message format (BE start_seq || BE end_seq || merkle_root) is shared
between signer and verifier so external auditors can validate offline.
- Random with constant-time-equal helper
- KEK loader from env (Tier 1) with InvalidKekError on bad/missing var
- AEAD via direct LibCrypto FFI bindings (Crystal 1.20 stdlib OpenSSL::Cipher
doesn't expose GCM auth_data/auth_tag, so we extend LibCrypto with
EVP_CIPHER_CTX_ctrl + EVP_aes_256_gcm and call them directly)
- Envelope: per-row DEK wrapped by KEK, AAD-bound, algorithm_id reserved
for crypto agility (0x01 = AES-256-GCM today, 0x02/0x03 reserved)
- 17 unit specs including a 100-iteration random property test
- Tag-tampered ciphertext / wrong-AAD / wrong-key all raise Aead::Error
- All 4 repos mirror the SQLite contracts using PG-native types
(UUID, BYTEA, JSONB, BIGSERIAL, TIMESTAMPTZ, SMALLINT)
- audit_no_modify() trigger refuses UPDATE/DELETE/TRUNCATE on audit_events
- pg_advisory_xact_lock for cross-process row locking
- Integration tests verify trigger fires correctly on tamper attempt
- Test cleanup helper temporarily disables trigger to reset between specs
- 4 integration specs all green against PostgreSQL 16
All four repos implement their abstract contracts. Migrations create
tables idempotently. Bytes round-trip through SQLite BLOB columns
correctly. RotatorKind and RotationState enums replace Symbol fields
in record types so they round-trip safely through DB string columns.
8 unit specs verify CRUD, find_by_external, in_flight filtering,
revocation, and audit append/range/latest_hash genesis behavior.
CarterPerez-dev