Closes 36 findings from the project audit. The headline win is correctness:
the daemon now actually does what the README claims, all three audit-log
integrity layers verify, and the rotation control loop terminates.
Critical fixes:
- Orchestrator bumps Credential.last_rotated_at on success and writes a
sealed credential_versions row, so the policy evaluator stops re-firing
the same rotation every tick
- Envelope encryption (AES-256-GCM, KEK-wrapped DEK, AAD-bound) is now
wired into the rotation success path; the credential_versions table is
no longer dead schema
- cre audit verify checks all three layers (hash chain + HMAC ratchet +
Merkle batches) instead of only the chain
- BatchSealerScheduler runs every 300s and on shutdown so audit_batches
actually fills with signed Merkle roots
- Compliance bundle exports the real audit_batches list and covers
public_key.pem under the manifest's checksum
- cre run/watch hard-fail without CRE_HMAC_KEY_HEX + CRE_KEK_HEX rather
than silently using zero defaults
Quality fixes:
- HTTP::Client wrapper with connect/read timeouts + bounded retry-with-
jitter on 408/429/5xx; Telegram error logs redact the bot token
- EventBus dispatch isolates Block subscribers behind select+timeout so
one stuck subscriber can't pin the bus
- RotationWorker checks rotations.in_flight before dispatching to dedupe
duplicate schedules; PG enforces it at the DB via partial unique index
- Commit-step failures transition rotations to Inconsistent (a terminal
state) and raise a critical alert instead of pretending success
- env_file rotator uses per-PID pending paths plus an exclusive flock on
a sibling .lock file so two daemons can't race on the rename
- Versioned migration runner replaces the soup of CREATE IF NOT EXISTS;
SQLite gains BEFORE UPDATE / BEFORE DELETE triggers on audit_events
- AuditSubscriber publishes a critical alert + panics by default when
log writes fail, instead of silently dropping
- Engine.stop drains via an ack channel with a 2s deadline rather than
the previous magic 50ms sleep
- Policy DSL moved into CRE::Policy::Dsl module (consumers `include` it);
multiple matching policies raise PolicyConflictError
- Evaluator uses credentials.overdue() per-max_age group instead of
loading every credential each tick
- AuditRepo gains each_in_range streaming and all_batches enumeration;
bundle export streams instead of buffering
- New cre verify-bundle <zip> CLI re-runs every check the bundle README
documents; new cre tui-demo for synthetic event preview
Tests: 179 unit (up from 159), all passing. New rotation_worker_spec,
new SigV4 regression vector, new orchestrator success-path coverage
(envelope write + last_rotated_at bump + Inconsistent state).
Docs: README + learn/00..04 aligned with the post-audit behavior;
removed /snooze (was a stub) and the AwsIamKey/Database CredentialKind
variants that had no rotators. Added required-env-var documentation.
Two changes that go together:
1. RotationWorker (src/cre/engine/rotation_worker.cr) is the missing
subscriber that turns RotationScheduled events into actual 4-step
rotations. Without it, the daemon ticked, evaluated, fired
RotationScheduled, and... nothing happened. Now it dispatches via
the orchestrator using a kind -> Rotator dispatch table populated
at boot.
2. cre run now wires:
- All 4 rotators based on env vars (AWS_ACCESS_KEY_ID, VAULT_ADDR/TOKEN,
GITHUB_TOKEN). EnvFile is always available.
- Telegram subscriber + bot if TELEGRAM_TOKEN + chat-id env vars set.
Boot output reports which rotators wired and whether telegram is on.
3. CONFIGURATION.md: full operator setup guide. 13 sections covering
required env vars, Postgres setup with role hardening, key generation,
policy authoring, inventory seeding (per-rotator tag schemas), AWS IAM
policy, Vault token policy.hcl, GitHub admin PAT acquisition, Telegram
bot creation + chat ID discovery, systemd unit with hardening
directives, audit chain verification, KEK rotation procedure, and a
13-item production security checklist.
Initial tick fires immediately on start so first policy evaluation
doesn't wait for the full interval on boot. The PolicyEvaluator
subscribes to SchedulerTick events and invokes evaluate_all -> overdue
discovery -> RotationScheduled fan-out. 3 specs verify boot-tick,
periodic cadence, and clean stop().
Rotator base exposes the four lifecycle methods as abstract; subclasses
register at compile time via 'register_as :kind' macro. REGISTRY is
populated as soon as the rotator file is required - drop a new file in
src/cre/rotators/ and the orchestrator can dispatch to it.
EnvFileRotator implements the simplest rotation:
- generate: 32 random bytes -> base64-urlsafe (no padding)
- apply: write to PATH.pending atomically (in-place line replace, 0600)
- verify: parse pending file, confirm key=value present and non-empty
- commit: rename PATH.pending -> PATH (atomic on POSIX)
- rollback_apply: unlink PATH.pending
RotationOrchestrator runs the 4-step contract with full event publishing,
state machine transitions in DB (generating -> applying -> verifying ->
committing -> completed | failed), and rollback_apply on apply/verify
failure. 8 specs cover happy path + raised-during-apply + rollback verification.
Events form a Crystal class hierarchy so subscribers pattern-match
exhaustively. EventBus delivers to subscribers via Crystal channels with
per-subscriber overflow policy (Block for audit, Drop for best-effort).
AuditSubscriber listens for all rotation/policy/drift/alert events and
serializes them into the hash-chained audit log via AuditLog.append.
Engine wires Persistence + AuditLog + AuditSubscriber + EventBus together
so callers just .start/.stop. 7 unit specs verify boot, fanout to N
subscribers, drop-on-overflow semantics, and end-to-end event -> audit row.
CarterPerez-dev