Commit Graph

20 Commits

Author SHA1 Message Date
Nicky Leach fdb9a4880d
fix(security): route paperclipai CLI guidance through safe npx form (CWE-78) (#11400)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip provides CLI commands and guidance for operators and
agents
> - The `pnpm paperclipai` script can pass argument values through a
shell
> - Shell re-parsing can execute command substitutions inside quoted
values
> - This pull request routes guidance through inert-argv `npx
paperclipai` commands and adds regression coverage
> - The benefit is safer operator guidance across documentation and
runtime hints

## Linked Issues or Issue Description

This pull request fixes a command-injection-class defect in Paperclip
CLI guidance.

**What happened?**

The `pnpm paperclipai <sub> --flag "$VALUE"` form can re-parse argument
values through a shell. A command substitution inside a quoted value can
execute on the host.

**Expected behavior**

Paperclip guidance must pass CLI values as inert argument values.
Host-derived values must not appear in copyable commands.

**Steps to reproduce**

1. Run a Paperclip guidance command that uses the `pnpm paperclipai`
script.
2. Provide a quoted value that contains a command substitution.
3. Observe that the shell can evaluate the substitution before the CLI
starts.
4. Compare the result with the `npx paperclipai` form.

**Paperclip version or commit**

`5670984b75d109950c968542a0111ebb6967f4da`

**Deployment mode**

All deployment modes that show or use the affected CLI guidance.

**Installation method**

Built from source and installed CLI guidance.

**Agent adapter(s) involved**

Not adapter-specific (core bug).

**Database mode**

Not database-related.

**Access context**

Both.

**Additional context**

The earlier merged PR
[#11343](https://github.com/paperclipai/paperclip/pull/11343) used the
unsafe `pnpm exec paperclipai` form. This fresh PR replaces that
guidance with the safe `npx paperclipai` form.

## What Changed

- Standardize documentation and runtime hints on `npx paperclipai`.
- Remove the broken `pnpm exec paperclipai` guidance.
- Use a static `<host>` placeholder in private-hostname guidance.
- Add regression tests for unsafe forms, continued lines, static hosts,
and offline guidance.

## Verification

- `git diff --check
origin/master...origin/fix/paperclipai-cli-npx-safe-invocation` passes.
- The branch adds `server/src/__tests__/cli-invocation-safety.test.ts`
and updates private-hostname tests.
- CI must run the new tests, typecheck, lint, and build checks.
- Local Vitest execution was not available because this worktree has no
installed Vitest binary.

## Risks

- The change affects operator and agent documentation text.
- The runtime hints now show `<host>` instead of a request-derived host
value.
- No database schema or migration changes exist.
- CI will detect any missed unsafe invocation or type error.

## Model Used

OpenAI GPT-5, exact model ID `gpt-5`, with tool use and code-review
assistance. The model used repository inspection, Git operations, and PR
preparation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] CI ran the test suites and they pass; local test execution was
unavailable in this worktree
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I addressed all Greptile and reviewer comments before requesting
merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-14 22:11:16 -07:00
Nicky Leach 5ca7b4c1fe
fix(security): standardize paperclipai CLI guidance on safe npx path (#11343)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip provides CLI guidance to agents and operators through
documentation and runtime messages.
> - Content-bearing `pnpm paperclipai` examples send arguments through a
shell.
> - Shell evaluation can execute command substitutions in untrusted
argument content.
> - Runtime hostname guidance can also place request-derived content
inside a shell command.
> - This pull request uses `npx paperclipai` for content-bearing
guidance and uses a static hostname placeholder.
> - The benefit is safer copy-paste guidance for agents and operators.

## Linked Issues or Issue Description

**Issue type**
Incorrect information

**Where is the issue?**
CLI guidance in `doc/CLI.md`, `skills/paperclip/SKILL.md`,
documentation, and runtime-generated hints.

**What's wrong?**
Content-bearing `pnpm paperclipai` commands can pass argument text
through `/bin/sh`. Shell command substitution in an argument can execute
before the CLI receives the value.

**Suggested fix**
Use `npx paperclipai` for content-bearing commands. Use a static
`<host>` placeholder when runtime guidance displays the allowed-hostname
command.

## What Changed

- Replace content-bearing `pnpm paperclipai` examples with `npx
paperclipai` across the documentation and agent-facing guidance.
- Update runtime-generated CLI hints to use a static `<host>`
placeholder.
- Add safety notes to `doc/CLI.md` and `skills/paperclip/SKILL.md`.
- Add scans and regression tests for unsafe invocation and hostile
hostname headers.
- Keep fixed lifecycle commands and `pnpm --filter @paperclipai/*` build
commands unchanged.

## Verification

- Run `tsc --noEmit` for the changed server files.
- Run `cli-invocation-safety.test.ts`.
- Run `private-hostname-guard.test.ts`.
- Confirm that hostile hostname headers do not enter shown shell
commands.
- Confirm that the three commits contain the required Paperclip
co-author trailer.

## Risks

- This change updates documentation and diagnostic text across many
surfaces.
- Fixed lifecycle and setup commands remain unchanged.
- The tests fail if content-bearing `pnpm paperclipai` guidance returns.
- The change does not alter the CLI argument parser.

## Model Used

OpenAI Codex, GPT-5, tool use, code execution, and repository review
assistance.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-13 16:43:21 -07:00
Devin Foley cb52f0b750
db: env-configurable client options; parallelize attention feed queries (#10795)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server stores all state in PostgreSQL through Drizzle and the
postgres.js driver
> - Self-hosted installs run Postgres on localhost, so per-query latency
is near zero; hosted installs often attach Postgres over a network,
sometimes through a transaction-mode pooler
> - The DB client passes no options to the driver, so operators cannot
disable prepared statements or tune the pool without a source edit, and
the deploy docs told them to edit `client.ts`
> - The attention feed also runs its related-data lookups one after
another, so its latency grows as queries × network round trip
> - This pull request adds optional environment configuration for the DB
client and batches the independent attention-feed lookups with
`Promise.all`
> - The benefit is that network-attached deployments get correct pooler
support and a much faster attention feed, while self-hosted behavior
does not change

## Linked Issues or Issue Description

No public issue exists for this; description follows the bug report
template:

**What happened?**

On deployments where PostgreSQL is network-attached (managed providers,
pooled endpoints), the attention feed endpoint is slow:
`attentionService.list()` awaits ~15–20 queries strictly in sequence, so
a 70ms round trip turns into more than one second of pure network wait
per call. Separately, connecting through a transaction-mode pooler
(pgbouncer, Supavisor port 6543, Neon `-pooler` hosts) requires
disabling prepared statements, and the only documented way was to
hand-edit `packages/db/src/client.ts` — which `doc/DATABASE.md` itself
tells operators not to do.

**Expected behavior**

The DB client is configurable from the environment (prepared statements,
pool size, timeouts) with driver defaults when unset, and hot read paths
do not multiply network latency by issuing independent queries
sequentially.

**Steps to reproduce**

1. Run the server with `DATABASE_URL` pointing at a Postgres instance
with ~70ms round-trip latency.
2. Open the attention feed (`GET /companies/:companyId/attention`) and
measure response time — it exceeds one second even with little data.
3. Try to connect through a transaction-mode pooler: there is no
supported configuration to disable prepared statements.

## What Changed

- `packages/db/src/client.ts`: `createDb` accepts a
`DatabaseClientOptions` argument and reads optional env config —
`DATABASE_PREPARED_STATEMENTS`, `DATABASE_POOL_MAX`,
`DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`.
When nothing is set, no option is passed to the driver and behavior is
identical to the previous bare `postgres(url)`.
- `packages/db/src/client-options.test.ts` (new): env parsing and
driver-option mapping tests, including malformed-value rejection.
- `server/src/services/attention.ts`: the independent related-data
lookups in each feed section now run under `Promise.all` (issue
summary/image/plan-document maps, decision bundle titles, blocked-issue
maps, the newer-runs scan). Section order, item assembly, and query
shapes are unchanged.
- `doc/DATABASE.md` and `docs/deploy/database.md`: the edit-source
pooling instruction is replaced with the env toggle, plus a short
client-tuning reference.

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/client-options.test.ts` — 6 tests pass.
- `pnpm --filter server exec vitest run
src/__tests__/attention-service.test.ts` — 22 tests pass.
- `pnpm --filter server exec vitest run
src/__tests__/decisions-service.test.ts
src/__tests__/decision-training.test.ts` — 45 tests pass; this covers
the call path that runs `attentionService.list()` inside
`db.transaction`, where postgres.js serializes queries on the reserved
connection.
- `tsc` reports no errors in the changed files.

## Risks

- Low risk for self-hosted installs: with no env vars set,
`postgres(url, {})` receives an empty options object, which postgres.js
treats the same as no options — driver defaults throughout.
- The `Promise.all` batches only group queries that had no data
dependency on each other; on the transaction call path the driver still
executes them one at a time on the reserved connection, so transactional
semantics are unchanged.
- Malformed env values now fail fast at startup with a clear message
instead of being silently ignored; this is intentional and only affects
operators who set the new variables.

## Model Used

Claude Fable 5 (`claude-fable-5`), Anthropic — via Claude Code CLI,
extended thinking enabled, tool use (test execution, live latency
measurement against a network-attached Postgres to size the problem).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (searched "prepared statements", "pgbouncer", "pool",
"attention feed", "lockfile" — closest matches are #10573/#10787
lockfile chores, unrelated to this change)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-04 06:30:36 -07:00
Devin Foley e0c2448267
feat(server): authenticate server-side git clone and fetch with a company-secret GitHub token (#10720)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Repo-only project workspaces are materialized by a server-side `git
clone`, and isolated `git_worktree` runs refresh their base ref with
server-side `git fetch`
> - Both operations run outside the agent process with no credentials,
so private GitHub repositories can never be cloned or refreshed —
agent-scoped credential env bindings do not reach them
> - The company secret store already has a well-known GitHub token
convention (`GITHUB_TOKEN` / `GH_TOKEN` / `PAPERCLIP_GITHUB_TOKEN`,
consumed by the external-object provider for API reads), but nothing
server-side consults it for git
> - This pull request resolves that token per run and authenticates the
managed clone and every base-ref refresh with it through an ephemeral
credential helper
> - The benefit is that isolated workspaces work on private repositories
with one company secret, while public repositories and self-hosted
ambient git configuration keep working unchanged

## Linked Issues or Issue Description

**Subsystem affected**

Server workspace materialization (`server/src/services/heartbeat.ts`)
and execution-workspace realization
(`server/src/services/workspace-runtime.ts`).

**Problem or motivation**

A project workspace configured with only a private GitHub `repoUrl`
cannot be used for isolated `git_worktree` runs: the managed `git clone`
runs with a sanitized, credential-less environment, and plain git cannot
consume a bare token env variable without a credential helper. There is
no way to give the server a git credential — storing a `GH_TOKEN`
company secret has no effect on server-side git, and a credential-less
private clone hangs on a terminal prompt until the ten-minute clone
timeout. Base-ref refreshes (`git fetch`) during worktree realization
have the same gap.

**Proposed solution**

A `git-credentials` module resolves a token per run — company secret by
well-known name (`GITHUB_TOKEN`, `GH_TOKEN`, `PAPERCLIP_GITHUB_TOKEN`),
then `GITHUB_TOKEN`/`GH_TOKEN` in the server process environment for
self-hosted deployments, then none — and builds a git invocation that
authenticates via an inline credential helper. The token travels in an
env variable; it never appears in argv, URLs, or on disk. Only
`https://github.com` remotes are authenticated; everything else keeps
ambient behavior. The provider is a single factory seam so a future
brokered credential source can replace it without touching call sites.

**Alternatives considered**

- A GitHub OAuth "connect your account" flow: heavier product surface,
needs app registration and callback custody; out of scope for a server
credential and better served by a dedicated connector later. The
provider seam keeps that path open.
- `gh auth setup-git`: writes helper configuration to disk and requires
a global token env; rejected in favor of per-invocation config with no
persistent state.
- Embedding the token in the clone URL: leaks into argv, error messages,
and `.git/config`; rejected.

## What Changed

- New `server/src/services/git-credentials.ts`:
`createGitRemoteAuthProvider` (memoized per run, one secret resolution
and one audit event), `buildGitAuthInvocation` (helper-reset + inline
helper, `x-access-token` username, `GIT_TERMINAL_PROMPT=0`),
`isGitHubHttpsRemoteUrl` host gating (rejects ssh/GHES/http/other
hosts/userinfo URLs), `describeGitAuthFailure`, and the canonical
`scrubGitCredentialText`. Secret resolutions pass a `system` consumer
access context so they are recorded as secret access events.
- `ensureManagedProjectWorkspace` (now exported) accepts an optional
auth provider; the clone env spreads the token after
`sanitizeRuntimeServiceBaseEnv` (which strips `PAPERCLIP_*`), always
sets `GIT_TERMINAL_PROMPT=0`, distinguishes "credential rejected" from
"no credential configured — add a GITHUB_TOKEN or GH_TOKEN company
secret" in the error, and removes the partially created directory on
clone failure so a timeout-killed clone cannot be adopted as a broken
checkout by the next run.
- `refreshRemoteTrackingBaseRef` (now exported) captures the remote URL
it already looked up, asks the provider for an invocation, and
attributes failed authenticated fetches to the credential in a scrubbed
warning. The optional provider threads through `detectDefaultBranch`,
`resolveAuthoritativeBaseRef`, `inspectExecutionWorkspaceBaseDrift`,
`realizeExecutionWorkspace`, and
`ensurePersistedExecutionWorkspaceAvailable`; heartbeat builds one
provider per run for both the anchor-resolution clone path and workspace
realization/restore.
- `github-external-object-provider.ts` imports the shared secret-name
list; `isGitHubDotCom` is exported from `github-fetch.ts`.
- Docs: "Private repositories and repo-only project workspaces" section
in the execution-workspaces guide, cross-linked from the secrets deploy
doc.

## Verification

- `cd server && npx vitest run src/__tests__/git-credentials.test.ts` —
resolution chain order and precedence, env fallback, memoization,
audited access context, host-gating matrix, invocation shape (token
absent from argv), scrubber, failure descriptions, and a real-git `git
credential fill` round trip that proves the helper executes and answers
with the env-carried token (no network).
- `cd server && npx vitest run
src/__tests__/heartbeat-managed-clone-credentials.test.ts` — clones
behave byte-identically with no provider or a null-returning provider
(local repos, no network), authenticated-failure errors name the
credential, non-auth failures do not mention credentials, partial clone
directories are removed, pre-existing non-git directories keep the
"Using it as-is" path, and the sanitizer spread order keeps the token
env alive.
- `cd server && npx vitest run src/__tests__/workspace-runtime.test.ts`
— new `refreshRemoteTrackingBaseRef` cases: provider offered the remote
URL and null keeps behavior identical; failed authenticated fetch
warning names the credential; unauthenticated failure warning stays
credential-free.
- `pnpm --filter @paperclipai/server typecheck` is clean.
- Manual (optional, networked): store a `GH_TOKEN` company secret,
configure a repo-only project workspace pointing at a private GitHub
repository, run an isolated-workspace issue — the managed clone succeeds
and the worktree run proceeds.

## Risks

- Every new parameter is optional; with no provider the git invocations
are byte-identical to before. Public repos and ambient credential
helpers keep working whenever no token resolves.
- Precedence change when a token exists: a stored company secret now
wins over ambient helpers for `https://github.com` remotes (the helper
list is reset for that invocation). The rejected-credential error names
the secret so an operator can fix or remove it.
- `GIT_TERMINAL_PROMPT=0` on the managed clone is the one always-on
change: a credential-less private clone now fails fast with a clear
message instead of hanging until the ten-minute timeout (it could only
ever "succeed" interactively on a TTY dev server).
- The token is scoped to the git process env for one invocation; it is
never written to agent env, run context, disk, or logs, and error text
is scrubbed of URL userinfo.
- No migrations, no image changes (git ships in the image).

## Model Used

Claude Fable 5 (`claude-fable-5`, extended thinking, agentic tool use
via Claude Code CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-02 20:27:09 -07:00
Granis87 bbb3e19c6f
docs: add safe local worktree bootstrap guidance (#9195)
## Thinking Path

> - Parallel local agent experiments should not reuse the primary
Paperclip instance.
> - Paperclip already creates isolated worktree instances with generated
names and environment files.
> - The local development guide lacked a complete bootstrap and recovery
path.
> - This pull request adds that path using the registered CLI forms and
generated selector.
> - The benefit is safer setup, recovery, and cleanup for local
experiments.

## Linked Issues or Issue Description

No public issue was found. The local development guide did not connect
worktree creation, environment loading, startup, repair/reseed, and
cleanup into one safe sequence.

## What Changed

- Added an isolated worktree bootstrap example.
- Documented bash/zsh environment loading without presenting invalid
PowerShell syntax.
- Used registered repair/reseed commands and the generated
paperclip-local-lab selector.
- Added explicit cleanup guidance.

## Verification

- git diff --check upstream/master...HEAD
- Verified command registration and generated naming in
cli/src/commands/worktree.ts.

## Risks

Low. Documentation-only; reviewed command and selector mismatches are
corrected.

## Model Used

OpenAI GPT-5.3 Codex Spark for initial branch work; OpenAI GPT-5 Codex
for review and follow-up fixes, with repository and GitHub tool use.
Context-window sizes were not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this docs PR does not
duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked or
described the problem above
- [x] I have either linked an existing issue or described the issue
in-PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run focused validation locally
- [x] I have added or updated tests where applicable
(documentation-only; no runtime tests needed)
- [x] I have updated the relevant documentation
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green on the latest commit
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on the latest commit
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: RobinALG87 <RobinALG87@users.noreply.github.com>
2026-07-15 15:51:53 -05:00
Dotta 555391fed7
fix: run restart recovery, workspace self-heal, quota-aware retries, failed-run metrics (#9183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in heartbeat runs orchestrated by the server; run
lifecycle, retry scheduling, and the dashboard's run-activity metrics
are the subsystems involved
> - A spike in "failed" tasks traced to three causes: server restarts
killing in-flight runs and mislabeling them as failures, deterministic
workspace-validation loops when a worktree's branch diverged, and
provider quota/usage-limit errors being classified as generic transient
failures (putting agents into error state and polluting metrics)
> - Killed-then-recovered runs and quota waits are not product failures,
so both the runtime behavior and the reporting needed to distinguish
them
> - This pull request drains runs gracefully on shutdown with idempotent
restart retries, self-heals workspace branch mismatches, adds a
quota-aware failure class with reset-time retry, separates recovered
restart kills from true failures on the dashboard, and documents restart
hygiene for operators
> - The benefit is fewer spurious failures, automatic recovery instead
of manual repair, and dashboard metrics that reflect real failure rates

## Linked Issues or Issue Description

No public GitHub issue exists; describing the bug inline per the
bug-report template:

**What happened?**

In-flight heartbeat runs are marked `failed` when the server restarts,
even though a retry later succeeds. Worktrees whose checked-out branch
diverges from the issue branch fail workspace validation on every
subsequent run with no recovery path. Provider quota/usage-limit
responses are treated as generic transient upstream errors, putting
agents into an error state and retrying before the quota window resets.
The dashboard counts all of these as true failures, inflating failure
metrics.

**Expected behavior**

Graceful shutdown should interrupt (not fail) running runs and chain
exactly one recovery retry. Workspace validation should repair
recoverable branch mismatches automatically. Quota errors should get
their own error class with the retry scheduled at the provider reset
time and the agent left idle. The dashboard should report recovered
restart kills separately from true failures.

**Steps to reproduce**

1. Start a heartbeat run, then restart the server (SIGTERM) while it is
in flight — the run lands as `failed` with a process-loss error code
even when its retry succeeds
2. Check out an issue whose worktree branch has diverged (e.g. after a
force-moved branch) — every subsequent run fails
`workspace_validation_failed` deterministically
3. Drive an agent into a provider usage-limit window — the run fails as
a generic transient upstream error and the agent enters an error state
instead of idling until the reset time

**Paperclip version or commit**

master (base c07e650cd)

**Deployment mode**

Self-hosted dev plane (Linux, node server + embedded Postgres)

## What Changed

- Graceful shutdown (SIGTERM hook) now marks in-flight runs
`interrupted` instead of `failed` and enqueues an idempotent
process-loss retry (pre-insert existence check on `retryOfRunId`
prevents duplicates; bursts chain exactly one retry per interrupted run)
- Run-liveness classification routes `interrupted` to `needs_followup`
rather than `failed`
- Workspace validation self-heals branch mismatch / missing-branch
states instead of failing deterministically on every run
- New `provider_quota` error class: session/usage-limit responses
schedule the retry at the provider reset time and leave the agent idle
(not errored); fixes a case where a quota-terminated run with subtype
`success` was misclassified as failed; HTTP 529 remains transient
- Dashboard run-activity query separates recovered restart kills from
true failures via a recursive CTE over `retry_of_run_id` (ancestors of a
succeeded retry count as recovered), adds a per-day failed-by-error-code
breakdown, and binds the window start as a timestamptz string
- Activity charts UI: amber "Recovered" segment with legend and per-day
error-code tooltip; success-rate chart counts recovered runs as
successes
- New ops runbook: `docs/deploy/dev-plane-restart-hygiene.md`

## Verification

- Greptile follow-up fixes on `992705edd`: `pnpm exec vitest run
server/src/__tests__/server-startup-feedback-export.test.ts
server/src/__tests__/heartbeat-workspace-branch-containment.test.ts
server/src/__tests__/heartbeat-workspace-session.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts` (209
tests); `pnpm --filter @paperclipai/server typecheck`; `git diff
--check`
- Post-rebase CI fixes: `pnpm exec vitest run
server/src/__tests__/heartbeat-retry-scheduling.test.ts`; `pnpm exec
vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts
server/src/__tests__/workspace-runtime.test.ts
server/src/__tests__/heartbeat-workspace-branch-containment.test.ts
server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts
server/src/__tests__/heartbeat-workspace-session.test.ts` (227 tests);
`pnpm --filter @paperclipai/server typecheck`
- `npm test` server suites covering the changes:
`heartbeat-process-recovery`, `heartbeat-stop-metadata`,
`heartbeat-retry-scheduling` (95 tests), quota parse +
execute/retry-scheduling suites (100 tests), workspace self-heal suites
(200 tests), dashboard run-activity tests (3 tests) — all green,
typecheck exit 0
- Dashboard CTE cross-checked against a real development database: two
restart-burst days moved from 17 to 8 and 19 to 6 true failures once
recovered kills were separated, matching manual retry-chain inspection
- Screenshot verification of the real ActivityCharts component
(recovered segment + tooltip) during QA

## Risks

- Behavioral shift: runs killed by a restart no longer surface as
`failed`; anyone consuming raw run statuses will see `interrupted` (new
status value) — dashboards/queries in this repo were updated accordingly
- Retry chaining on repeated restarts is bounded (one chained retry per
interruption) but a pathological restart loop still delays work rather
than failing it; the runbook covers operator hygiene for that case
- Dashboard query adds a recursive CTE; cost is bounded by the
day-window row count and was verified against production-sized data
- No schema migrations; low migration risk

## Model Used

- Claude (Anthropic) — claude-fable-5 via Claude Code / Paperclip agent
harness, extended thinking with tool use; implementation commits also
produced with Codex CLI (GPT-5 class) agents under the same harness

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-07 20:30:09 -05:00
Dotta ad961227f5
feat(secrets): add user-specific runtime secrets (#8825)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent runs often need provider credentials, API tokens, and other
environment-bound secrets.
> - Company-level secrets work for shared credentials, but they do not
model values that should differ by human operator.
> - Without a user-scoped model, a run can dispatch without knowing
whether the responsible human has supplied the needed value.
> - Paperclip also needs run attribution to make those user-scoped
runtime checks deterministic and auditable.
> - This pull request adds user-specific secret definitions, per-user
values, environment bindings, responsible-user attribution, and runtime
resolution gates.
> - The benefit is that teams can define the secret once, let each user
provide their own value, and block runs before dispatch when required
user secrets or active definitions are unavailable.

## Linked Issues or Issue Description

Refs #224
Refs #6057

This PR implements user-specific secret support as a core
secret-management capability rather than a one-off adapter setting. It
is related to existing public work on company secrets UI and runtime
secret refs, but is distinct because the value is owned by the
responsible user and resolved at run dispatch time.

Related PR search before opening found existing secrets work such as
#1550, #8256, #8614, #8634, and #8647; none of those add the full
user-secret definition/value/runtime gate covered here.

## What Changed

- Added user-secret definitions and per-user "My secrets" values,
keeping stored values out of access metadata.
- Added `user_secret_ref` environment bindings and UI affordances to
pick them alongside existing secret refs.
- Added responsible-user runtime resolution so user-secret refs resolve
against the human responsible for the run.
- Added pre-dispatch missing-secret gates so runs fail before adapter
dispatch when required user values are absent or definitions are
inactive.
- Added low-trust allowlist hardening for user-secret runtime access.
- Added issue, routine, run, and agent API key responsible-user
attribution and fail-closed dispatch behavior when attribution cannot be
resolved.
- Added denial-copy mapping so responsible-user authorization failures
surface as actionable run outcomes instead of opaque setup failures.
- Added OpenAPI documentation for the user-secret routes.
- Rebases cleanly on current `master`; migrations were renumbered
incrementally as `0128_user_specific_secrets`,
`0129_agent_api_key_responsible_user`, and
`0130_run_responsible_user_invariant` after upstream `0126`/`0127`
migrations.
- Removed previously committed local design screenshots so the PR
contains code/docs/tests only.

## Verification

- PASS: PR head `2527febd106bcf3ca264ca0da7fca491084192d6` is based on
`paperclipai/paperclip:master`.
- PASS: `git diff --check`
- PASS: `git diff --name-only public/master...HEAD | rg
'^(pnpm-lock\\.yaml|\\.github/workflows/|screenshots/)' || true`
produced no files.
- PASS: migration journal audit confirmed unique indexes through `130`
with tail entries `0126_issue_comment_derived_attribution`,
`0127_environment_custom_images_instance_scoped`,
`0128_user_specific_secrets`, `0129_agent_api_key_responsible_user`, and
`0130_run_responsible_user_invariant`.
- PASS: `pnpm --filter @paperclipai/ui typecheck`
- PASS: `pnpm --filter @paperclipai/server typecheck`
- PASS: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-responsible-user-invariant.test.ts`
- PASS: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-active-run-output-watchdog.test.ts
src/__tests__/heartbeat-stale-queue-invalidation.test.ts
src/__tests__/heartbeat-workspace-finalize-branch.test.ts
src/__tests__/issue-monitor-scheduler.test.ts`
- PASS: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts
src/__tests__/heartbeat-plugin-environment.test.ts`
- PASS: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/low-trust-red-team-routes.test.ts`
- PASS: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/secrets-service.test.ts` (55 tests)
- PASS: `pnpm vitest run server/src/__tests__/secrets-routes.test.ts
server/src/__tests__/secrets-service.test.ts` (89 tests after final
Greptile cleanup fixes)
- PASS: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-issue-liveness-escalation.test.ts` (17 tests
after the final rebase CI fix)
- PASS: focused server Vitest batches covering heartbeat recovery,
project env, plugin env, routines, low-trust, pipelines, monitors,
watchdog, and stale queue paths.
- PASS: GitHub checks are green on
`2527febd106bcf3ca264ca0da7fca491084192d6`, including Typecheck +
Release Registry, Build, General tests, serialized server suites, e2e,
Canary Dry Run, verify, security checks, and Greptile Review.
- PASS: Greptile Review completed successfully on
`2527febd106bcf3ca264ca0da7fca491084192d6` with Confidence Score 5/5,
and GraphQL review-thread audit returned zero unresolved non-outdated
threads.

## Risks

- Runtime behavior now depends on a run having a correct responsible
user; missing or incorrect responsibility assignment can block runs
before adapter dispatch.
- `user_secret_ref` bindings intentionally expose metadata without
values, but UI/API callers may need to handle the new binding kind
explicitly.
- External secret providers and IAM policies are not automatically
provisioned by this PR; operators still need to configure provider-side
access for non-local vaults.
- The PR is broad across db/shared/server/UI/runtime paths, so release
validation should include both API and UI secret workflows before merge.
- The migration renumbering is intentionally incremental after upstream
migrations; the branch migrations use guarded
column/table/index/constraint creation so users who tested the older
draft numbering should not hit duplicate DDL for the existing objects.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex, GPT-5-based coding agent (`gpt-5`), Codex local adapter
with shell/tool use and code execution. Context window and internal
reasoning mode are not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 05:58:20 -05:00
Devin Foley bf982c8c83
Normalize adapter display labels (#8913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Adapter names are part of the board-facing agent setup and
management experience.
> - The product now treats adapters as harnesses, while execution
environments are modeled separately.
> - Several built-in adapter labels still carried legacy local wording
from the older harness-by-environment model.
> - That wording makes the UI noisier and implies a distinction users no
longer need to reason about.
> - This pull request normalizes adapter display labels while keeping
persisted adapter type identifiers unchanged.
> - The benefit is clearer adapter selection and management copy without
a database migration.

## Linked Issues or Issue Description

No public GitHub issue was found for this exact cleanup.

Related public PRs:

- Supersedes #8910, an earlier branch for the same cleanup that did not
include the later docs/gateway/Cursor alignment.
- Refs #8819, which is related display-registry work for external
multi-segment adapter labels, but not a duplicate of this built-in label
cleanup.

Feature request details:

- Subsystem affected: Cross-cutting (`ui/`, `packages/adapters`, and
docs).
- Problem or motivation: user-facing adapter names include legacy local
qualifiers even though adapters map to harnesses and environments are
first-class elsewhere.
- Proposed solution: remove the legacy local wording from built-in
display labels, keep machine-readable adapter type ids unchanged, and
keep gateway disambiguation where it is useful.
- Alternatives considered: changing persisted adapter type ids was ruled
out because it would create migration and compatibility risk; one-off UI
replacements were ruled out because the display registry is already the
correct central label boundary.
- Roadmap alignment: this is small adapter UX polish, not a new
roadmap-level core feature.

## What Changed

- Updated the adapter display registry so known adapter labels are final
and no built-in local adapter renders a legacy local suffix.
- Preserved clean derived labels for unknown plugin local types while
keeping gateway disambiguation for unknown gateway types.
- Updated `AdapterManager` to prefer registry labels when the server
reports raw adapter type ids for built-ins.
- Removed legacy local wording from built-in adapter metadata labels in
UI and adapter packages.
- Aligned Cursor adapter metadata with the central display registry
label.
- Updated adapter docs and Storybook fixtures to match the new display
names.
- Added focused registry coverage for built-in labels and unknown plugin
suffix behavior.

## Verification

- `pnpm check:tokens`
- `git diff --check origin/master...fix/adapter-display-labels`
- Patch-addition scan for added secrets, private paths, and internal
links: no matches.
- GitHub duplicate search for open adapter-label/local-suffix issues and
PRs; #8910 was identified as the older superseded public PR.
- `pnpm exec vitest run
ui/src/adapters/adapter-display-registry.test.ts`
- `pnpm --filter @paperclipai/ui typecheck`
- Stale-label scan found no remaining user-facing display-label
suffixes; remaining local wording is operational/test terminology such
as adapter ids, docs about running locally, and test descriptions.

## Risks

Low risk. The change is display-label and documentation focused, and
adapter type ids remain unchanged. The main risk is ambiguous gateway
naming, mitigated by keeping explicit gateway labels where variants need
disambiguation.

## Model Used

OpenAI GPT-5 via Codex, tool-enabled coding agent in a local repository
workspace. Context window size is not exposed by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-02 21:32:50 -07:00
Darren Davison 9802636be3
build(docker): bundle Gemini CLI in image for gemini_local adapter (#7693)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work, and it runs agents through pluggable adapters.
> - One of those adapters, `gemini_local`
(`packages/adapters/gemini-local/`), runs Google's Gemini CLI on the
same host as the server.
> - For local (in-container) execution, the adapter only probes `PATH`
for the binary (`packages/adapter-utils/src/execution-target.ts`); it
does **not** auto-install — only `sandbox` transport targets install on
demand via `SANDBOX_INSTALL_COMMAND`.
> - The production Docker image bakes in `claude`, `codex`, and
`opencode` so their `*_local` adapters work out of the box, but `gemini`
was never added — so `gemini_local` fails inside the container with a
missing-binary error.
> - This PR adds `@google/gemini-cli@latest` to the image's global
install so `gemini_local` works locally like the other bundled CLIs,
sets `GEMINI_SANDBOX=false` for safe in-container CLI use, and documents
all bundled CLIs.
> - The benefit is plug-and-play Gemini support in Docker with no
per-deployment CLI install step, plus accurate docs (including a
previously-undocumented `opencode` bundle) and a heads-up about Google's
imminent unrestricted-API-key block.

## Linked Issues or Issue Description

No existing issue. Following the **adapter/feature** template fields:

- **Capability:** Run the existing `gemini_local` adapter inside the
official Docker image without a manual CLI install.
- **Problem/motivation:** The image pre-installs `claude`, `codex`, and
`opencode`, but not `gemini`. Because local execution probes `PATH` and
never auto-installs (only sandbox targets do), `gemini_local` runs fail
in-container with a missing-binary error. Operators currently have to
maintain a forked image.
- **Proposed solution:** Add `@google/gemini-cli@latest` to the existing
global `npm install` line; set `GEMINI_SANDBOX=false` in `ENV`.
- **Alternatives considered:** Runtime install at container start
(slower, network-dependent, inconsistent with how the other three CLIs
are handled).

**Related / superseded PRs** (both make the same Dockerfile change and
can be closed if this merges):
- #5912 — `fix(docker): install gemini-cli and normalize persistent
volume permissions`. Same Dockerfile line, but also bundles unrelated
volume-permission and heartbeat-test changes, and a maintainer noted it
now has a merge conflict. I corrected Greptile's `GEMINI_SANDBOX`
reasoning there and it was re-scored 5/5 (see [resolved
thread](https://github.com/paperclipai/paperclip/pull/5912#discussion_r3367332785))
— this PR is the rebased, docs-complete, single-purpose version of that
change.
- #2731 — `Added @google/gemini-cli to the Dockerfile`. One-liner, now
stale and merge-conflicting (predates the `opencode-ai` addition).

> **Note for Greptile:** the `--sandbox=none` reasoning for the Gemini
CLI sandbox concern was already validated on #5912's `Dockerfile:59`
thread and accepted (re-scored 5/5). This PR additionally bakes
`GEMINI_SANDBOX=false` into `ENV` as defense-in-depth, so the concern is
covered both at the adapter layer and the image layer.

## What Changed

- **Dockerfile:** add `@google/gemini-cli@latest` to the production
global npm install (after `opencode-ai`).
- **Dockerfile:** add `GEMINI_SANDBOX=false` to the `ENV` block (matches
the existing `OPENCODE_ALLOW_ALL_MODELS=true` precedent) so a
manually-invoked `gemini` inside the container does not attempt a
Docker-in-Docker sandbox.
- **docs/deploy/docker.md:** rename the section to "Local Adapter CLIs
in Docker"; list all four bundled CLIs mapped to their adapter type keys
(fixes a pre-existing gap — `opencode` was bundled but undocumented);
add `GEMINI_API_KEY` to the example; document per-provider credentials,
the `GEMINI_SANDBOX=false` default, and Google's 2026-06-19
unrestricted-key block with the `gemini auth login` (OAuth) alternative.

## Verification

- `npm view @google/gemini-cli` confirms the package exists, provides
the `gemini` bin, and requires Node `>=20` (the base image is Node 22
LTS). 
- Adapter already disables the CLI sandbox per run:
`packages/adapters/gemini-local/src/server/execute.ts` pushes
`--sandbox=none` whenever `config.sandbox` is false (the default). 
- Confirmed neither file was modified on `upstream/master`, so this
rebases cleanly with no conflicts. 
- Full image build is exercised by CI. (I did not run the multi-stage
`docker build` locally; the change adds one package to an existing,
working `npm install` line.)
- Reviewer manual check: `docker build -t paperclip-local . && docker
run --rm paperclip-local gemini --version` should print the CLI version.

## Risks

- **Low risk.** Adds one npm package to an existing global install and
one inert env var; no application code paths change.
- Minor image-size increase from the additional CLI (consistent with the
three already bundled).
- `@latest` is unpinned — intentionally consistent with the sibling
`@anthropic-ai/claude-code@latest` / `@openai/codex@latest` on the same
line; pinning all of them is a separate decision out of scope here.

## Model Used

Claude Opus 4.8 (model ID `claude-opus-4-8`, 1M-context variant), via
Claude Code with tool use / agentic file editing and web research.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues OR (b) described the
issue in-PR following the relevant issue template
- [ ] I have run tests locally and they pass (no unit tests cover the
Dockerfile; package/bin/engine verified via `npm view`, full build runs
in CI)
- [ ] I have added or updated tests where applicable (N/A — Docker image
+ docs change)
- [ ] If this change affects the UI, I have included before/after
screenshots (N/A)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (pending CI run on this PR)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending review)
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 10:42:07 -07:00
Dotta 778e775c35
Add secrets provider vaults and remote import (#5429)
## Thinking Path

> - Paperclip orchestrates AI-agent companies and needs secrets handling
to work across local development, hosted operators, and governed agent
execution.
> - The affected subsystem is the company-scoped secrets control plane:
database schema, server services/routes, CLI workflows, and the Secrets
settings UI.
> - The gap was that secrets were local-only and operators could not
manage provider vaults or import existing remote references without
exposing plaintext.
> - This branch adds provider vault configuration plus an AWS Secrets
Manager remote-import path while preserving company boundaries, binding
context, and audit trails.
> - I kept the PR to a single branch PR, removed unrelated
lockfile/package drift, rebased the full branch onto the current
`public-gh/master`, and addressed fresh Greptile findings.
> - The benefit is a reviewable implementation of provider-backed
secrets with focused tests covering provider selection, import
conflicts, deleted secret reuse, rotation guards, and AWS signing
behavior.

## What Changed

- Added provider vault support for company secrets, including provider
config storage, default vault handling, health checks, binding usage,
access events, and remote import preview/commit.
- Added an AWS Secrets Manager provider using SigV4 request signing,
bounded request timeouts, namespace guardrails, cached runtime
credential resolution, and external-reference linking without plaintext
reads.
- Added Secrets UI surfaces for vault management and remote import, plus
CLI/API documentation for setup and operations.
- Stabilized routine webhook secret binding paths and SSH
environment-driver fixture bindings discovered during verification.
- Addressed Greptile and CI findings: no lockfile/package drift,
monotonic migration metadata, disabled-vault default races, soft-deleted
secret hiding/recreate behavior, remove behavior with disabled vaults,
soft-deleted external-reference re-import, non-active rotation guards,
managed-secret soft deletion through PATCH, and per-call AWS SDK
credential client churn.
- Rebased this branch onto `public-gh/master` at `0e1a5828` and
force-pushed with lease to keep this as the single PR for the branch.

## Verification

- `git fetch public-gh master`
- `git rebase public-gh/master`
- `git diff --name-only public-gh/master...HEAD | grep
'^pnpm-lock\.yaml$' || true` confirmed `pnpm-lock.yaml` is not in the PR
diff.
- Confirmed migration ordering: master ends at `0081_optimal_dormammu`;
this PR adds `0082_dry_vision` and
`0083_company_secret_provider_configs`.
- Inspected migrations for repeat safety: new tables/indexes use `IF NOT
EXISTS`; foreign keys are guarded by `DO $$ ... IF NOT EXISTS`; column
additions use `ADD COLUMN IF NOT EXISTS`.
- `pnpm -r typecheck` passed before the Greptile follow-up commits.
- `pnpm test:run` ran the full stable Vitest path before the Greptile
follow-up commits; it completed with 3 timing-related failures under
parallel load: `codex-local-execute.test.ts`,
`cursor-local-execute.test.ts`, and `environment-service.test.ts`.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/codex-local-execute.test.ts
src/__tests__/cursor-local-execute.test.ts
src/__tests__/environment-service.test.ts` passed on targeted rerun
(`24/24`).
- `pnpm build` passed before the Greptile follow-up commits. Vite
reported existing chunk-size/dynamic-import warnings.
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/secrets-service.test.ts` passed (`26/26`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/aws-secrets-manager-provider.test.ts
src/__tests__/secrets-service.test.ts` passed (`39/39`).
- After Greptile follow-up commits: `pnpm --filter @paperclipai/server
typecheck` passed.
- Captured Storybook screenshots from `ui/storybook-static` for visual
review.
- Latest PR checks on `5ca3a5cf`: `policy`, serialized server suites
1/4-4/4, `Canary Dry Run`, `e2e`, `security/snyk`, and `Greptile Review`
pass; aggregate `verify` is still registering the completed child
checks.
- Greptile review loop continued through the latest requested pass; all
Greptile review threads are resolved and the latest `Greptile Review`
check on `5ca3a5cf` passed with 0 comments added.

## Screenshots

Before: the provider-vault and remote-import surfaces did not exist on
`master`; these are after-state screenshots from the Storybook fixtures.

![Secrets
inventory](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secrets-inventory.png)

![Secret binding
picker](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/secret-binding-picker.png)

![Environment editor with
secrets](https://raw.githubusercontent.com/paperclipai/paperclip/PAP-2339-secrets-make-a-plan/doc/pr/5429/env-editor-with-secrets.png)

## Risks

- Migration risk: this adds new secret provider tables and extends
existing secret rows. The migrations were checked for monotonic ordering
and idempotent guards, but reviewers should still inspect upgrade
behavior carefully.
- Provider risk: AWS support uses direct SigV4 requests. Automated tests
cover signing, request timeouts, vault-config selection, namespace
guardrails, pending-version archival, sanitized provider errors, and
service-level cleanup paths. A real-vault AWS smoke test remains
deployment validation for an operator with AWS credentials rather than
an unverified merge blocker in this local branch.
- UI risk: the Secrets page and import dialog are large new surfaces;
screenshots are included above for reviewer inspection.
- Verification risk: the full local stable test command hit
parallel-load timing failures, although the exact failed files passed
when rerun directly.
- Operational risk: remote import intentionally avoids plaintext reads;
operators must understand that imported external references resolve at
runtime and may fail if AWS permissions change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5 coding agent with local shell/tool use in the
Paperclip worktree. Exact context-window size was not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 18:22:17 -05:00
Neeraj Kumar Singh B f0f9460d1d
docs: AWS ECS Fargate deployment runbook (#3897)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies and ships
a
>   "local-first, cloud-ready" deployment model
> - The deploy docs currently cover local/Docker but not a production
> cloud target, so teams asking "how do I put this behind a real domain"
>   have no canonical path
> - We already support Docker images, RDS-compatible Postgres, and an
EFS
>   storage profile, so AWS ECS Fargate is a natural fit
> - Without a runbook, each team reinvents VPC, security groups, TLS,
and
>   secrets wiring and usually gets at least one step wrong
> - This pull request adds `docs/deploy/aws-ecs.md`, an ECS
task-definition
> template, and an `.env.aws.example`, cross-linked from the deploy
overview
> - The benefit is a single, reproducible ~$110/mo path to a production
>   deployment, plus a full teardown for throwaway environments

## What Changed

- New `docs/deploy/aws-ecs.md` — an 11-step ECS Fargate runbook covering
ECR,
  VPC, RDS, EFS, Secrets Manager, IAM, ALB, and ECS service with the
  deployment circuit breaker enabled
- New `docker/ecs-task-definition.json` — Fargate-ready task definition
with
  `<ACCOUNT_ID>`, `<REGION>`, `<EFS_ID>`, `<DOMAIN>` placeholder tokens
- New `docker/.env.aws.example` — documents every non-secret env var the
  ECS deployment needs
- `docs/deploy/overview.md` — one-line cross-reference to the new guide
- Greptile feedback addressed in follow-up commits:
  - `containerName` in the service-create call now matches
    `paperclip-server` in the task definition
  - HTTP :80 listener added that 301-redirects to :443
  - Dedicated RDS DB subnet group created before `create-db-instance`
  - EFS teardown polls on mount-target deletion instead of `sleep 30`

## Verification

- Walked every step of the runbook against the task definition to
confirm
  variable names (`$ALB_SG`, `$ECS_SG`, `$RDS_SG`, `$EFS_SG`, `$TG_ARN`,
`$LISTENER_ARN`, `$HTTP_LISTENER_ARN`, `$EFS_ID`, `$RDS_ENDPOINT`, etc.)
are
  defined before they are referenced
- Confirmed the `containerName` in Step 10 (`paperclip-server`) matches
  `docker/ecs-task-definition.json` line 11
- Confirmed the `sed` placeholder substitution in Step 8 matches the
tokens
  in the task definition template
- Teardown order was checked in reverse-dependency order: ECS service →
  listeners → target group → ALB → RDS (waits for deletion) → DB subnet
  group → EFS mount targets (polled) → EFS → secrets → SGs → ECR → IAM →
  log group

## Risks

- **Low risk for the repo.** Docs-only change plus two template files
under
`docker/`; no runtime code paths are touched and nothing is imported by
  the build.
- **Risk for users who follow the runbook:** AWS bills accrue
immediately
  once RDS/ALB/EFS exist. The runbook calls this out and includes a full
  teardown procedure. Placeholder tokens (`<ACCOUNT_ID>`, `<REGION>`,
`<EFS_ID>`, `<DOMAIN>`) are documented so nothing is silently
hard-coded.

## Model Used

- Claude (Anthropic), model `claude-opus-4-6`, ~200K context window,
extended thinking mode on, used with tool access (file edit, shell) via
Claude Code. The Greptile follow-up commits were authored the same way.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have run tests locally and they pass — N/A for docs/config
templates; validated by reading
- [x] I have added or updated tests where applicable — N/A for docs
- [x] If this change affects the UI, I have included before/after
screenshots — N/A, no UI
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-27 08:41:47 -05:00
Chris Farhood b816809a1e
fix(server): respect externally set PAPERCLIP_API_URL env var (#3472)
## Thinking Path

> - Paperclip server starts up and sets internal `PAPERCLIP_API_URL` for
downstream services and adapters
> - The server startup code was unconditionally overwriting
`PAPERCLIP_API_URL` with `http://localhost:3100` (or equivalent based on
`config.host`)
> - In Kubernetes deployments, `PAPERCLIP_API_URL` is set via a
ConfigMap to the externally accessible load balancer URL (e.g.
`https://paperclip.example.com`)
> - Because the env var was unconditionally set after loading the
ConfigMap value, the ConfigMap-provided URL was ignored and replaced
with the internal localhost address
> - This caused downstream services (adapter env building) to use the
wrong URL, breaking external access
> - This pull request makes the assignment conditional — only set if not
already provided by the environment
> - External deployments can now supply `PAPERCLIP_API_URL` and it will
be respected; local development continues to work without setting it

## What Changed

- `server/src/index.ts`: Wrapped `PAPERCLIP_API_URL` assignment in `if
(!process.env.PAPERCLIP_API_URL)` guard so externally provided values
are preserved
- `server/src/__tests__/server-startup-feedback-export.test.ts`: Added
tests verifying external `PAPERCLIP_API_URL` is respected and fallback
behavior is correct
- `docs/deploy/environment-variables.md`: Updated `PAPERCLIP_API_URL`
description to clarify it can be externally provided and the load
balancer/reverse proxy use case

## Verification

- Run the existing test suite: `pnpm test:run
server/src/__tests__/server-startup-feedback-export.test.ts` — all 3
tests pass
- Manual verification: Set `PAPERCLIP_API_URL` to a custom value before
starting the server and confirm it is not overwritten

## Risks

- Low risk — purely additive conditional check; existing behavior for
unset env var is unchanged

## Model Used

MiniMax M2.7 — reasoning-assisted for tracing the root cause through the
startup chain (`buildPaperclipEnv` → `startServer` → `config.host` →
`HOST` env var)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Pawla Abdul (Bot) <pawla@groombook.dev>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-04-15 06:43:48 -05:00
dotta 2a84e53c1b Introduce bind presets for deployment setup
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-04-11 07:09:07 -05:00
Cody (Radius Red) 420cd4fd8d chore(docker): improve base image and organize docker files
- Add wget, ripgrep, python3, and GitHub CLI (gh) to base image
- Add OPENCODE_ALLOW_ALL_MODELS=true to production ENV
- Move compose files, onboard-smoke Dockerfile to docker/
- Move entrypoint script to scripts/docker-entrypoint.sh
- Add Podman Quadlet unit files (pod, app, db containers)
- Add docker/README.md with build, compose, and quadlet docs
- Add scripts/docker-build-test.sh for local build validation
- Update all doc references for new file locations
- Keep main Dockerfile at project root (no .dockerignore changes needed)

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-04-01 11:36:27 +00:00
Dotta e9ffde610b docs: add tailscale private access guide 2026-03-05 14:21:47 -06:00
Dotta 09d2ef1a37 fix: restore docs deleted in v0.2.3 release, add Paperclip branding
- Restored docs/ directory that was accidentally deleted by `git add -A`
  in the v0.2.3 release script
- Replaced generic "P" favicon with actual paperclip icon using brand
  primary color (#2563EB)
- Added light/dark logo SVGs for Mintlify navbar (paperclip icon + wordmark)
- Updated docs.json with logo configuration for dark/light mode
- Fixed release.sh to stage only release-related files instead of `git add -A`
  to prevent sweeping unrelated changes into release commits

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 15:49:43 -06:00
Dotta 28bf5e9e9b chore: release v0.2.3 2026-03-03 15:39:13 -06:00
Dotta f60c1001ec refactor: rename packages to @paperclipai and CLI binary to paperclipai
Rename all workspace packages from @paperclip/* to @paperclipai/* and
the CLI binary from `paperclip` to `paperclipai` in preparation for
npm publishing. Bump CLI version to 0.1.0 and add package metadata
(description, keywords, license, repository, files). Update all
imports, documentation, user-facing messages, and tests accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 08:45:26 -06:00
Dotta cabd16bc70 docs: sync docs and skills updates from backup branch 2026-03-02 16:44:10 -06:00
Forgotten 02dc46e782 docs: add external documentation site content
Add structured documentation covering quickstart, architecture, core
concepts, API reference, adapter guides, CLI commands, deployment
options, and operator/developer guides.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 16:33:55 -06:00