## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Every agent run streams its stdout/stderr/system output into the
run-log store (`server/src/services/run-log-store.ts`), and the run-log
API serves those logs back for review and debugging
> - The only store implementation is `local_file`: logs live on the
server pod's filesystem under `PAPERCLIP_HOME`
> - In hardened / ephemeral deployments, `PAPERCLIP_HOME` is an
`emptyDir` with no persistent volume, so every pod restart wipes the log
files while the DB row still references them — the run-log API then
returns "Run log not found" for every completed run after any redeploy
> - Run logs are the primary audit/debugging trail for agent work;
losing them on routine redeploys undermines trust in the platform
> - This pull request adds transparent durability: when
`RUN_LOG_S3_BUCKET` is set, the store mirrors each completed log to
object storage on `finalize` (same `logRef` key) and falls back to it on
`read` when the local file is gone; live append/tail stays on the fast
pod-local file
> - The benefit is that completed run logs survive pod restarts and
redeploys with zero changes for existing deployments (unset bucket =
today's behaviour) and zero downstream changes (store id stays
`local_file`)
## Linked Issues or Issue Description
No existing public issue — inline description following the bug report
template:
**What happened?** After any server pod restart/redeploy, the run-log
API returns "Run log not found" for all previously completed runs. The
DB still references the log file, but the file is gone because run logs
are written only to the pod-local filesystem.
**Expected behavior:** Completed run logs remain readable across pod
restarts and redeploys.
**Steps to reproduce:**
1. Deploy the server with `PAPERCLIP_HOME` on an `emptyDir` (no
persistent volume — common in hardened/ephemeral Kubernetes
deployments).
2. Complete an agent run and confirm its log is readable via the run-log
API.
3. Restart or redeploy the server pod.
4. Request the same run's log — the API throws "Run log not found".
**Paperclip version or commit:** reproducible on current `master`.
**Deployment mode:** Kubernetes (server pod without persistent volume).
**Agent adapter(s) involved:** Not adapter-specific (core bug).
Supersedes #8795.
## What Changed
- `server/src/services/run-log-store.ts`: the local-file store becomes a
durable store with an optional object-storage mirror
- `finalize` mirrors the completed NDJSON log to S3-compatible object
storage (keyed by the same `logRef`), best-effort so a failed upload can
never break run finalization; upload failures are logged via
`console.warn` so operators can detect a persistently broken mirror
before a pod roll makes logs unreadable
- `read` serves the pod-local file when present and falls back to a
ranged object-storage read (with correct `nextOffset`) when the local
file is gone
- Live `append`/tail stays on the pod-local file — fast path unchanged,
no per-chunk PUT
- Store id stays `local_file`, so nothing downstream changes (feedback
pipeline, read casts, fixtures untouched)
- New optional config, all read at store construction:
`RUN_LOG_S3_BUCKET`, `RUN_LOG_S3_ENDPOINT`, `RUN_LOG_S3_REGION` (default
`us-east-1`), `RUN_LOG_S3_PREFIX` (default `run-logs`),
`RUN_LOG_S3_FORCE_PATH_STYLE` (default `true`); credentials via the
standard AWS env chain; works with any S3-compatible endpoint
- Reuses the existing `createS3StorageProvider`; deliberately
independent from `PAPERCLIP_STORAGE_PROVIDER` so enabling durable logs
does not redirect workspace/file storage
- `server/src/services/run-log-store.test.ts` (new): 7 tests with an
in-memory `StorageProvider` mock
## Verification
- `npx vitest run src/services/run-log-store.test.ts` in `server/` — 7/7
pass locally:
- store id stays `local_file`
- live read served from the local file (no S3 round-trip)
- `finalize` uploads the completed log to the mirror
- read falls back to S3 after a simulated pod roll (local file deleted)
- ranged S3 read returns correct slice + `nextOffset`
- not-found when neither local nor mirror has the log
- local-only safe degrade when no bucket is configured
- `npx tsc --noEmit -p server` — clean for the touched files
- Manual: set `RUN_LOG_S3_*` against any S3-compatible endpoint (e.g.
MinIO), complete a run, delete the local `.ndjson` file, and re-request
the log via the run-log API — it is served from the mirror
## Risks
- Low risk: with `RUN_LOG_S3_BUCKET` unset (the default), behaviour is
byte-for-byte today's local-only store
- Mirror upload is best-effort by design — a misconfigured bucket loses
durability (not correctness) for affected runs; failures are now
surfaced via a `console.warn` per failed upload
- No DB migration, no API shape change, no change to the persisted
`store`/`logRef` handle format
## Model Used
- Claude (Anthropic), model ID `claude-fable-5` (Fable 5), via Claude
Code with extended thinking and tool use (code execution, file editing).
Original implementation TDD-authored with the same tooling.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
b4e7ba5143