## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The run-log store records each agent run's output and can mirror
completed logs to S3-compatible object storage
> - The mirror uploads only on finalize, so a server restart mid-run
loses the whole in-flight log
> - Deployments and crashes are routine on ephemeral hosts, and lost run
output makes failed runs impossible to debug
> - This pull request adds an opt-in throttled mirror for still-running
logs plus a graceful-shutdown flush
> - The benefit is that a restart mid-run keeps the log tail up to the
last mirror interval, and an orderly restart keeps everything
## Linked Issues or Issue Description
No public issue exists — describing the feature inline (per the feature
request template).
**Subsystem affected**
server/ — REST API & orchestration services
**Problem or motivation**
`RUN_LOG_S3_BUCKET` gives finished run logs durability, but the mirror
uploads only on finalize. A run that is still writing when the server
restarts leaves nothing in object storage. On hosts with ephemeral disks
the local file is gone too, so the run's output is lost end to end and
failed runs cannot be debugged.
**Proposed solution**
Mirror the in-flight log to the same object key on a throttled cadence
(`RUN_LOG_S3_INFLIGHT_MIRROR_SECONDS`), and flush dirty tails during
graceful shutdown. Keep it opt-in so existing deployments see zero new
upload traffic unless they ask for it.
**Alternatives considered**
Per-append uploads (rejected: one PUT per output chunk is hostile to S3
endpoints and run latency). Chunked part objects with read-time
stitching (rejected: complicates the read path, and S3 multipart minimum
part sizes do not fit small tails). Persistent volumes (rejected
upstream already: the data dir is deliberately an emptyDir in hardened
cloud_tenant deployments).
**Roadmap alignment**
Not on ROADMAP.md; extends the existing run-log durability mirror
without changing any default behavior.
**Additional context**
Ranged reads already serve partial objects like a live tail, so the read
path needs no change; finalize overwrites the mirror with the complete
file.
**Related PRs (dedup search):** the finalize-only S3 mirror landed
previously and this extends it; no duplicate or competing PR found for
in-flight run-log mirroring.
## What Changed
- `server/src/services/run-log-store.ts`: new opt-in `inflightMirrorMs`
on the S3 options (`RUN_LOG_S3_INFLIGHT_MIRROR_SECONDS` env). When set,
appends schedule at most one upload of the current file per interval, to
the same key finalize uses. Ranged reads already serve that key, so a
partial object behaves like a live tail and needs no read-path change.
Finalize retires the in-flight bookkeeping and waits out an upload
already on the wire, so a stale partial can never overwrite a finalized
log. Upload failures warn, re-mark the tail dirty, and retry at most
once per interval.
- `server/src/services/run-log-store.ts`: new `flushInflightMirrors()`
on the store and a module-level `flushInFlightRunLogMirrors()` for the
shutdown path. Both are no-ops when the mirror is off.
- `server/src/index.ts`: graceful shutdown flushes dirty in-flight tails
after the heartbeat run drain, so runs the drain did not finalize
(timeouts, the hot-restart skip path) still persist their output.
- `server/src/services/run-log-store.test.ts`: five new tests —
off-by-default (no uploads before finalize), tail preserved after a wipe
without finalize, throttle coalescing with a single flush upload,
finalize superseding the in-flight mirror and retiring its timer, and
upload failures never breaking appends with recovery on the next flush.
## Verification
- `pnpm vitest run server/src/services/run-log-store.test.ts` — 13
passed (8 existing + 5 new).
- `pnpm vitest run server/src/__tests__/heartbeat-run-log.test.ts
server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts` — 21
passed (consumers of the store, unchanged behavior).
- `pnpm -C server run typecheck` — clean.
- Self-hosted behavior is unchanged unless
`RUN_LOG_S3_INFLIGHT_MIRROR_SECONDS` is set: with the variable unset
there are zero new uploads and the finalize-only mirroring is
byte-identical (asserted by the off-by-default test).
## Risks
- Low. The feature is opt-in; unset env preserves today's behavior
exactly. When enabled, worst case is one extra PUT per interval per
active run, and every upload is best-effort — a failing endpoint warns
and never breaks appends, finalization, or shutdown. The finalize path
awaits any in-flight upload before writing the complete file, closing
the only overwrite race the design introduces. Timers are `unref`ed so
the mirror never keeps the process alive.
## Model Used
Claude Fable 5 (`claude-fable-5`, Anthropic; Claude Code CLI with
extended thinking and tool use; tests executed locally via Vitest).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (no duplicates found for in-flight run-log mirroring; the
finalize-only mirror landed previously and this extends it)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Every agent run streams its stdout/stderr/system output into the
run-log store (`server/src/services/run-log-store.ts`), and the run-log
API serves those logs back for review and debugging
> - The only store implementation is `local_file`: logs live on the
server pod's filesystem under `PAPERCLIP_HOME`
> - In hardened / ephemeral deployments, `PAPERCLIP_HOME` is an
`emptyDir` with no persistent volume, so every pod restart wipes the log
files while the DB row still references them — the run-log API then
returns "Run log not found" for every completed run after any redeploy
> - Run logs are the primary audit/debugging trail for agent work;
losing them on routine redeploys undermines trust in the platform
> - This pull request adds transparent durability: when
`RUN_LOG_S3_BUCKET` is set, the store mirrors each completed log to
object storage on `finalize` (same `logRef` key) and falls back to it on
`read` when the local file is gone; live append/tail stays on the fast
pod-local file
> - The benefit is that completed run logs survive pod restarts and
redeploys with zero changes for existing deployments (unset bucket =
today's behaviour) and zero downstream changes (store id stays
`local_file`)
## Linked Issues or Issue Description
No existing public issue — inline description following the bug report
template:
**What happened?** After any server pod restart/redeploy, the run-log
API returns "Run log not found" for all previously completed runs. The
DB still references the log file, but the file is gone because run logs
are written only to the pod-local filesystem.
**Expected behavior:** Completed run logs remain readable across pod
restarts and redeploys.
**Steps to reproduce:**
1. Deploy the server with `PAPERCLIP_HOME` on an `emptyDir` (no
persistent volume — common in hardened/ephemeral Kubernetes
deployments).
2. Complete an agent run and confirm its log is readable via the run-log
API.
3. Restart or redeploy the server pod.
4. Request the same run's log — the API throws "Run log not found".
**Paperclip version or commit:** reproducible on current `master`.
**Deployment mode:** Kubernetes (server pod without persistent volume).
**Agent adapter(s) involved:** Not adapter-specific (core bug).
Supersedes #8795.
## What Changed
- `server/src/services/run-log-store.ts`: the local-file store becomes a
durable store with an optional object-storage mirror
- `finalize` mirrors the completed NDJSON log to S3-compatible object
storage (keyed by the same `logRef`), best-effort so a failed upload can
never break run finalization; upload failures are logged via
`console.warn` so operators can detect a persistently broken mirror
before a pod roll makes logs unreadable
- `read` serves the pod-local file when present and falls back to a
ranged object-storage read (with correct `nextOffset`) when the local
file is gone
- Live `append`/tail stays on the pod-local file — fast path unchanged,
no per-chunk PUT
- Store id stays `local_file`, so nothing downstream changes (feedback
pipeline, read casts, fixtures untouched)
- New optional config, all read at store construction:
`RUN_LOG_S3_BUCKET`, `RUN_LOG_S3_ENDPOINT`, `RUN_LOG_S3_REGION` (default
`us-east-1`), `RUN_LOG_S3_PREFIX` (default `run-logs`),
`RUN_LOG_S3_FORCE_PATH_STYLE` (default `true`); credentials via the
standard AWS env chain; works with any S3-compatible endpoint
- Reuses the existing `createS3StorageProvider`; deliberately
independent from `PAPERCLIP_STORAGE_PROVIDER` so enabling durable logs
does not redirect workspace/file storage
- `server/src/services/run-log-store.test.ts` (new): 7 tests with an
in-memory `StorageProvider` mock
## Verification
- `npx vitest run src/services/run-log-store.test.ts` in `server/` — 7/7
pass locally:
- store id stays `local_file`
- live read served from the local file (no S3 round-trip)
- `finalize` uploads the completed log to the mirror
- read falls back to S3 after a simulated pod roll (local file deleted)
- ranged S3 read returns correct slice + `nextOffset`
- not-found when neither local nor mirror has the log
- local-only safe degrade when no bucket is configured
- `npx tsc --noEmit -p server` — clean for the touched files
- Manual: set `RUN_LOG_S3_*` against any S3-compatible endpoint (e.g.
MinIO), complete a run, delete the local `.ndjson` file, and re-request
the log via the run-log API — it is served from the mirror
## Risks
- Low risk: with `RUN_LOG_S3_BUCKET` unset (the default), behaviour is
byte-for-byte today's local-only store
- Mirror upload is best-effort by design — a misconfigured bucket loses
durability (not correctness) for affected runs; failures are now
surfaced via a `console.warn` per failed upload
- No DB migration, no API shape change, no change to the persisted
`store`/`logRef` handle format
## Model Used
- Claude (Anthropic), model ID `claude-fable-5` (Fable 5), via Claude
Code with extended thinking and tool use (code execution, file editing).
Original implementation TDD-authored with the same tooling.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The issue thread UI renders live agent output from adapter run logs
and transcript parsing.
> - Some adapter streams emit many small or repeated token chunks, and
live UI updates can expose partial words, duplicated slices, or
transient markdown placeholders.
> - That makes active run updates look like gibberish even when the
underlying agent output is valid.
> - The fix needs to preserve raw logs while making the live thread view
stable, readable, and ordered.
> - This pull request adds monotonic run-log sequencing, safer live
transcript dedupe/order handling, markdown placeholder hiding, and
readable live text stabilization.
> - The benefit is a live issue thread that updates smoothly without
showing confusing partial parser artifacts.
## Linked Issues or Issue Description
No public GitHub issue exists yet, so this PR includes the bug details
inline.
### What happened?
Live run updates in the issue thread can show confusing repeated or
partial text while an adapter is streaming. The visible text appears to
lose parsing boundaries during active updates, especially with ACP-style
token deltas, so the live output can briefly render duplicated chunks,
incomplete words, or HTML-comment placeholders.
### Expected behavior
Live text should remain readable while preserving the underlying run
output for raw inspection.
### Steps to reproduce
1. Start a live agent run whose adapter emits small stdout token deltas.
2. Watch the issue thread while the run is still active.
3. Observe transient duplicated chunks, incomplete words, or markdown
placeholder artifacts in the live rendered text.
### Paperclip version or commit
Reproduced against current `master` before this PR branch.
### Deployment mode
Local dev issue-thread UI with live local adapter runs.
### Additional context
GitHub PR search for `live run streaming text markdown transcript` found
one broad merged PR, `#252` (“Dotta updates - sorry it's so large”), but
no targeted duplicate for this live streaming readability bug.
## What Changed
- Added per-run monotonic sequence numbers to persisted and live run-log
chunks.
- Dedupe and order live transcript chunks by sequence before falling
back to timestamp ordering.
- Hide markdown HTML comment placeholder text from rendered markdown
output.
- Smooth live issue-thread text updates so partial additions reveal at
readable word boundaries and sliding-window removals do not produce
gibberish.
- Added coverage for run-log ordering/deduping, markdown comment hiding,
live issue-thread stabilization, and Greptile-reviewed edge cases where
overlap rewrites could synthesize text or no-boundary additions could
stay hidden.
## Verification
- `pnpm --filter @paperclipai/ui exec vitest run
src/lib/issue-chat-messages.test.ts src/components/MarkdownBody.test.tsx
src/components/transcript/useLiveRunTranscripts.test.tsx` passed before
the review fix: 3 files, 86 tests.
- `pnpm --filter @paperclipai/ui exec vitest run
src/lib/issue-chat-messages.test.ts` passed after the review fix: 1
file, 30 tests.
- `pnpm --filter @paperclipai/ui exec vitest run
src/lib/issue-chat-messages.test.ts
src/components/transcript/useLiveRunTranscripts.test.tsx` passed after
the final Greptile overlap fix: 2 files, 40 tests.
- `pnpm check:token-gates` passed.
- Local PII/secret scan of touched files found only expected code/test
words such as `secret`, `token`, and redaction-related strings; no
literal credentials found.
- `pnpm -r typecheck` passed after restoring declared dependencies with
`CI=1 pnpm install --frozen-lockfile` and running with a short `TMPDIR`
because `tsx` IPC sockets fail under the long sandbox temp path.
- `pnpm build` passed with existing Vite CSS/font/chunk warnings.
- GitHub PR checks passed on head
`4c052dfe86aecb5feb73504e6b48843f68fce813`: build, typecheck/release
registry, server and workspace test shards, serialized server suites,
e2e, canary dry run, policy, review, Socket, Superagent, Snyk, and
verify.
- Greptile review passed on head
`4c052dfe86aecb5feb73504e6b48843f68fce813` with confidence score 5/5 and
no blocking issues found.
- `pnpm test:run` failed in unrelated server workspace tests on this
macOS local environment:
- `server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
two assertions compare `/tmp/...` with `/private/tmp/...`.
- `server/src/__tests__/heartbeat-worktree-suppression.test.ts`:
expected one heartbeat run but observed two, followed by cleanup fallout
in the full run.
- Isolated rerun of those two server suites reproduced the same three
failures.
## Risks
- Low product risk for the UI changes: the readable smoothing only
affects active live-run display stabilization, not stored comments or
raw run logs.
- Moderate verification risk: local full Vitest did not pass because of
unrelated server workspace tests. Targeted tests for this change,
typecheck, token gates, and build passed.
- Run-log sequence fields are optional for compatibility with older log
rows that do not include `seq`.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex, GPT-5-based coding agent, tool-using local workspace
execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I searched the GitHub PR list for similar PRs and confirmed this
is not a duplicate
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Run logs defaulted to process.cwd()/data/run-logs, placing logs in
unexpected locations when launched from non-home directories. Now
defaults to ~/.paperclip/instances/<id>/data/run-logs/.
Fixes#89
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The append method created a new WriteStream for every log chunk and resolved
the promise on the end callback (data flushed) rather than the close event
(fd released). Over many agent runs the leaked fds corrupted the fd table,
causing child_process.spawn to fail with EBADF.
Replace with fs.appendFile which properly opens, writes, and closes the fd
before resolving.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Expand heartbeat service with full run executor, wakeup coordinator,
and adapter lifecycle. Add run-log-store for pluggable log persistence.
Add live-events service and WebSocket handler for realtime updates.
Expand agent and issue routes with runtime operations. Add ws dependency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Devin Foley
Jannes Stubbemann
Dotta
Matt Van Horn
Dotta
Forgotten