paperclip/packages/db/src
Devin Foley 916c13501f
Replace host-to-host Cloud Sync with full-fidelity company Import/Export (#10507)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A company accumulates real state — issues, labels, blockers,
documents, work products, monitors, attachments, agents, routines — and
people need to move that state between instances: self-hosted to cloud,
cloud back to self-hosted, or plain backups
> - The experimental, flag-gated Cloud Sync transport (#6548) tried to
solve this host-to-host: the source pushed into a receiver over HTTPS
with a cross-instance consent/token handshake, which required the
destination to be publicly reachable and broke for common self-hosted
topologies (plain-HTTP LAN/VPN origins); the receiver half never landed
upstream at all
> - Meanwhile the portability bundle and the existing export/import
pages already move companies offline with none of those networking
constraints — but silently dropped labels, blockers, issue documents,
work products, monitors, and every attachment
> - This pull request removes the host-to-host transport and makes
Import/Export the single data-movement path: the pages become
first-class company-settings destinations, exports declare exactly what
they do not carry, and bundle schemaVersion 6 now carries all of the
above, with attachments as content-addressed sha256 blobs verified
before a single row is written
> - The benefit is a migration and backup flow that works between any
two instances with no reachability requirements, no cross-instance auth,
and no silent data loss

## Linked Issues or Issue Description

- Refs #6548 — the original Cloud Sync sender this PR supersedes and
removes.
- Related, not duplicates: #1697 (goals in the portability manifest —
orthogonal field addition), #954 (an earlier import/export +
skill-visibility proposal predating the current portability bundle).
- No open issue describes this directly, so in brief (feature-request
shape): **Problem** — moving a company between instances silently lost
labels (imports with label references actually hard-failed), blocker
relations, issue documents, work products, monitor state, and all
attachments, and the alternative Cloud Sync transport required the
destination to be publicly reachable over HTTPS plus a consent
handshake, which failed for typical self-hosted setups. **Desired
behavior** — one Import/Export flow in company settings that produces a
portable bundle carrying all of that data, tells the operator up front
what it cannot carry, imports with automations paused, and offers real
one-click activation afterwards.

## What Changed

- New export fidelity report (`GET
/api/companies/:companyId/export/fidelity`) + an "Export fidelity" panel
on the Export page listing anything a bundle will not include (now only:
approvals, cost history, activity history)
- Imports accept `pauseAutomations`; imported agents and routines land
paused, the import result reports created routines, and the Import page
ends in an activation panel that actually resumes selected
agents/activates routines
- Export and Import pages promoted into the company-settings nav; the
Cloud Upstream wizard, ux-lab page, and API client removed; the old
settings route redirects to Export
- Host-to-host transport removed: upstream-sync/receiver-client routes
and services, CLI `cloud connect`/`cloud push` + keypair store, the
shared upstream transfer contract, and the `enableCloudSync` flag;
migration `0196` drops the two experimental `cloud_upstream_*` sender
tables
- Bundle schemaVersion 6: labels (definitions + per-task names, remapped
by name on import), blocker relations (`blockedBy` slugs,
cycle-tolerant), issue documents (`tasks/<slug>/documents/<key>.md`),
work products (system refs nulled), monitors (notes/scheduledBy
restored, imported un-armed)
- Attachments travel as content-addressed `blobs/<sha256>` entries
(deduped; comment-scoped attachments re-link via comment index); every
blob is hash-verified **before any write**, so a corrupted bundle cannot
leave a partially imported company; both zip codecs now round-trip
extensionless/binary entries byte-exactly; the Import page preflights
the inline body limit and offers continue-without-attachments
- v5 (and older) bundles still import, with an informational warning;
bundles newer than v6 are rejected cleanly
- Docs: board-operator import/export guide, CLI README, README/ROADMAP
updated

## Verification

- `pnpm -r` typechecks (shared, db incl. migration numbering/safety
checks, server, ui, cli) and `pnpm check:token-gates` — clean
- Vitest: full server + shared sweep 4,888 passed / 1 skipped, with the
only 3 failures being pre-existing on `master` (2×
heartbeat-workspace-branch-containment, 1× workspace-runtime auto-port;
reproduced identically with this change stashed); ui + cli suites green;
the embedded-Postgres export-fidelity suite applies the full migration
chain including the new `0196` against a fresh database
- Live end-to-end on a scratch instance: seeded a company with labels, a
blocker pair, an issue document, a work product, a monitor, an agent, a
routine, and two binary attachments (one comment-scoped) → export →
import into a fresh company → labels remapped to new ids, blocker edge
and document restored, monitor un-armed with notes intact, attachments
byte-identical (sha256-compared through the API), agents/routines paused
→ activation panel resumed them; a v5-shaped bundle imported with only
the info warning; flipping one byte in a blob made the import 422 with
**zero** rows created
- Reviewer repro: create a company with a labeled issue + attachment →
Settings → Export → download → Settings → Import on another
company/instance → watch the preview, apply with "start paused", then
activate

## Risks

- Migration `0196` drops
`cloud_upstream_connections`/`cloud_upstream_runs` — experimental tables
behind a default-off flag; their connection/run history is intentionally
discarded
- Breaking removals are all of experimental, flag-gated surface:
`/api/upstream-sync/*` + `/api/cloud-upstreams/*` routes, `paperclipai
cloud connect|push`, and the `enableCloudSync` flag (stale keys in
stored instance settings parse harmlessly)
- Import remains non-atomic on mid-apply errors generally (pre-existing
behavior); the new blob verification specifically moved ahead of all
writes so tampered bundles cannot create partial state
- GitHub-sourced imports do not fetch `blobs/*` and skip attachments
with a warning

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic), via Claude Code CLI with
extended thinking, tool use, and subagent orchestration; implementation
and review split across Fable 5 subagents, with live end-to-end
verification against a running instance

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-07-30 11:37:00 -07:00
..
migrations Replace host-to-host Cloud Sync with full-fidelity company Import/Export (#10507) 2026-07-30 11:37:00 -07:00
schema Replace host-to-host Cloud Sync with full-fidelity company Import/Export (#10507) 2026-07-30 11:37:00 -07:00
backup-lib.test.ts fix(db): use calendar month retention for backups (#3718) 2026-07-24 13:12:43 -05:00
backup-lib.ts fix(db): use calendar month retention for backups (#3718) 2026-07-24 13:12:43 -05:00
backup.ts [codex] Add LLM Wiki plugin host support (#5597) 2026-05-10 07:34:12 -05:00
built-in-agent-unique-marker-migration.test.ts fix: prevent duplicate built-in agents and self-heal reconciliation (#10223) 2026-07-28 11:12:58 -07:00
check-migration-numbering.ts feat(routines): add workspace-aware routine runs 2026-04-02 11:38:57 -05:00
check-migration-safety.test.ts feat(db): add migration safety lint 2026-07-07 01:43:12 +00:00
check-migration-safety.ts feat(db): add migration safety lint 2026-07-07 01:43:12 +00:00
client.test.ts feat(skills): import skills from projects (#9620) 2026-07-15 18:01:44 -05:00
client.ts feat(mcp) [split 2/8]: add governed access contracts (#9557) 2026-07-14 12:57:20 -05:00
connections-v3-schema-core-migration.test.ts feat(connections): add v3 schema core (#9958) 2026-07-21 15:16:26 -05:00
embedded-postgres-error.test.ts Improve embedded Postgres bootstrap errors 2026-03-26 12:35:19 -05:00
embedded-postgres-error.ts Improve embedded Postgres bootstrap errors 2026-03-26 12:35:19 -05:00
embedded-postgres-native.test.ts [codex] Improve runtime and import reliability (#6549) 2026-05-22 09:57:22 -05:00
embedded-postgres-native.ts [codex] Improve runtime and import reliability (#6549) 2026-05-22 09:57:22 -05:00
environment-custom-images-schema.test.ts Scope environment custom images to instance environments (#8850) 2026-07-01 23:57:31 -07:00
external-objects-schema.test.ts External object references across issue surfaces (#8512) 2026-06-23 08:27:19 -05:00
inbox-archive-agent-policies-migration.test.ts feat(authz): govern agent inbox archive access (#9658) 2026-07-16 09:51:48 -05:00
index.ts feat(mcp) [split 2/8]: add governed access contracts (#9557) 2026-07-14 12:57:20 -05:00
issue-comment-derived-attribution-migration.test.ts fix(db): relocate slow 0126 issue-comment attribution backfill to fast idempotent 0132 (#9108) 2026-07-06 11:41:24 -07:00
migrate.ts Fix dev migration prompt and embedded db:migrate 2026-03-10 15:31:05 -05:00
migration-runtime.ts [codex] Improve runtime and import reliability (#6549) 2026-05-22 09:57:22 -05:00
migration-safety-baseline.ts feat(db): add migration safety lint 2026-07-07 01:43:12 +00:00
migration-status.ts feat(costs): add billing, quota, and budget control plane 2026-03-16 15:11:01 -05:00
nested-skill-folders-migration.test.ts feat: organize skills with nested folders and My Skills (#9633) 2026-07-16 15:50:45 -05:00
pipelines-schema.test.ts Add pipeline workflow primitives and operator UI (#7903) 2026-06-26 12:02:44 -05:00
runtime-config.test.ts [codex] Add LLM Wiki plugin host support (#5597) 2026-05-10 07:34:12 -05:00
runtime-config.ts [codex] Add LLM Wiki plugin host support (#5597) 2026-05-10 07:34:12 -05:00
seed.ts Expand data model with companies, approvals, costs, and heartbeats 2026-02-17 09:07:22 -06:00
status-card-migrations.test.ts feat(status-cards): add experimental status card update view (#10101) 2026-07-24 12:26:43 -05:00
summary-slots-schema.test.ts feat: add built-in summarizer and summary slots (#9713) 2026-07-17 11:03:07 -05:00
table-size-estimates.ts Replace host-to-host Cloud Sync with full-fidelity company Import/Export (#10507) 2026-07-30 11:37:00 -07:00
test-embedded-postgres.ts fix(test): stop flaky server-suite afterAll hook timeouts (#10024) 2026-07-22 11:51:30 -07:00