From 5ec7ce76e580c1e515fe811f566df928eb1e08dd Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 30 Jul 2026 18:48:22 -0700 Subject: [PATCH] Upload company import packages as compressed zip uploads (fix large-company imports through Cloud) (#10531) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Company Import (#10507, hardened in #10523) lets a user upload a company package on the Import page > - The page expanded the user's `.zip` into a files map and POSTed it as ONE inline JSON body — ~40MB for a real company because attachment blobs get base64-inflated > - On Paperclip Cloud that body travels browser → harness proxy → tenant, where it truncated in transit → body-parser 400 → the browser saw "Failed to fetch", and nothing imported > - Two compounding causes: the giant inline body itself, and the board async opt-in riding an `x-paperclip-cloud-*` header that the Cloud harness strips as anti-spoofing (so async never engaged and the import held one fragile synchronous connection) > - This pull request uploads the raw compressed `.zip` as a multipart request (about a third the size, already compressed) parsed server-side into the same bundle the importer consumes, and moves the async opt-in to a proxy-safe `?async=1` > - The benefit is that a large-company import actually completes through Cloud: a small compressed upload, a real async job that survives dropped connections ## Linked Issues or Issue Description - Refs #10507 / #10523 (Import/Export and its hardening). No open issue; problem described above (large-company browser import through a proxy: inline JSON body truncates → 400 → "Failed to fetch"; async opt-in header stripped by the front door → async never engages). ## What Changed - **Multipart zip transport.** The Import page uploads the raw `File` as `multipart/form-data` (field `package`, import options in a JSON `meta` field); the server unzips it into `{ rootPath, files }` and runs the exact existing preview/import logic. The `application/json` inline path is byte-identical for CLI/programmatic callers. Bare `application/zip` (meta via `?meta=`) is also accepted for programmatic use. - **Shared node zip reader.** `packages/shared/src/portability-zip.ts` (node-only subpath, not re-exported to the browser bundle — same pattern as `portability-hash.ts`); the CLI's `zip.ts` becomes a thin re-export. Identical codec (STORE + DEFLATE via `inflateRawSync`, rejects data descriptors/zip64). - **Proxy-safe async signal.** `wantsAsyncImport` = `?async=1` (board browsers, survives the harness) OR the existing `x-paperclip-cloud-async-import` header (cloud tenants, set server-side). The UI async client now uses `?async=1`. Backward compatible. - **Size + preflight.** New `PORTABLE_ZIP_UPLOAD_LIMIT_BYTES = 128MB`; the inline 56MB preflight no longer gates the zip path (it shows the compressed size instead). Async submit/poll/resume, the duplicate-guard fingerprint (now over the resolved bundle), pause-on-import, progress/error panels, and activation all apply to the multipart path. - OpenAPI documents json + multipart + zip bodies and the `async` query param. ## Verification - Full typecheck chain (shared, server, ui, cli) clean. - 152 tests across 8 files: new `portability-zip.test.ts` (STORE/DEFLATE/base64-blob byte-exact round-trip, truncation throws, data-descriptor rejection); `company-portability-routes.test.ts` +7 (multipart import+preview equals the inline bundle; async multipart 202→poll→success; board async via `?async=1` with no cloud header; cloud-tenant async via header; sync fallback with neither; truncated-zip 400, nothing imported); `CompanyImport.test.tsx` asserts the local zip sends the raw File and the inline preflight no longer blocks; `openapi-routes.test.ts` green. - NOT yet measured: the end-to-end browser upload through the live Cloud harness — verified on staging after deploy before closing out. ## Risks - Import semantics unchanged — only transport changed; the JSON inline path is byte-identical, the cloud-tenant header async path untouched. Multipart parsing is server-side (memory-bound: a ~13MB zip → ~30MB files map, fine on the server). - The bare `application/zip` path is programmatic-only and covered by content-type dispatch but not a dedicated route test (the multipart path is). ## Model Used - Claude Fable 5 (`claude-fable-5`, Anthropic), Claude Code CLI, extended thinking + tool use; root-caused against live logs/DB and the harness proxy source. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- cli/src/commands/client/zip.ts | 162 +-------- packages/shared/src/portability-zip.test.ts | 303 +++++++++++++++++ packages/shared/src/portability-zip.ts | 308 ++++++++++++++++++ .../company-portability-routes.test.ts | 205 ++++++++++++ server/src/http/body-limits.ts | 6 + server/src/routes/companies.ts | 158 ++++++++- server/src/routes/openapi.ts | 68 +++- ui/src/api/client.ts | 9 +- ui/src/api/companies.ts | 31 +- ui/src/pages/CompanyImport.test.tsx | 52 +-- ui/src/pages/CompanyImport.tsx | 144 ++++---- 11 files changed, 1172 insertions(+), 274 deletions(-) create mode 100644 packages/shared/src/portability-zip.test.ts create mode 100644 packages/shared/src/portability-zip.ts diff --git a/cli/src/commands/client/zip.ts b/cli/src/commands/client/zip.ts index 18265fcdb8..8f7b4db4ab 100644 --- a/cli/src/commands/client/zip.ts +++ b/cli/src/commands/client/zip.ts @@ -1,152 +1,10 @@ -import { inflateRawSync } from "node:zlib"; -import path from "node:path"; -import type { CompanyPortabilityFileEntry } from "@paperclipai/shared"; - -const textDecoder = new TextDecoder(); -// ignoreBOM keeps a leading BOM in the decoded text so text entries -// re-encode to their original bytes; fatal surfaces invalid UTF-8. -const strictTextDecoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); - -export const binaryContentTypeByExtension: Record = { - ".gif": "image/gif", - ".jpeg": "image/jpeg", - ".jpg": "image/jpeg", - ".png": "image/png", - ".svg": "image/svg+xml", - ".webp": "image/webp", -}; - -function normalizeArchivePath(pathValue: string) { - return pathValue - .replace(/\\/g, "/") - .split("/") - .filter(Boolean) - .join("/"); -} - -function readUint16(source: Uint8Array, offset: number) { - return source[offset]! | (source[offset + 1]! << 8); -} - -function readUint32(source: Uint8Array, offset: number) { - return ( - source[offset]! | - (source[offset + 1]! << 8) | - (source[offset + 2]! << 16) | - (source[offset + 3]! << 24) - ) >>> 0; -} - -function sharedArchiveRoot(paths: string[]) { - if (paths.length === 0) return null; - const firstSegments = paths - .map((entry) => normalizeArchivePath(entry).split("/").filter(Boolean)) - .filter((parts) => parts.length > 0); - if (firstSegments.length === 0) return null; - const candidate = firstSegments[0]![0]!; - return firstSegments.every((parts) => parts.length > 1 && parts[0] === candidate) - ? candidate - : null; -} - -export function isBlobStorePath(pathValue: string) { - return /(^|\/)blobs\/[^/]+$/.test(normalizeArchivePath(pathValue)); -} - -function decodeStrictUtf8(bytes: Uint8Array): string | null { - let text: string; - try { - text = strictTextDecoder.decode(bytes); - } catch { - return null; - } - return Buffer.from(text, "utf8").equals(Buffer.from(bytes)) ? text : null; -} - -export function bytesToPortableFileEntry(pathValue: string, bytes: Uint8Array): CompanyPortabilityFileEntry { - // Content-addressed blob entries are opaque bytes regardless of extension. - if (isBlobStorePath(pathValue)) { - return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType: "application/octet-stream" }; - } - const contentType = binaryContentTypeByExtension[path.extname(pathValue).toLowerCase()]; - if (contentType) { - return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType }; - } - const text = decodeStrictUtf8(bytes); - if (text !== null) return text; - // Bytes that are not valid UTF-8 must not be decoded lossily; fall back - // to base64 so they round-trip exactly. - return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType: "application/octet-stream" }; -} - -async function inflateZipEntry(compressionMethod: number, bytes: Uint8Array) { - if (compressionMethod === 0) return bytes; - if (compressionMethod !== 8) { - throw new Error("Unsupported zip archive: only STORE and DEFLATE entries are supported."); - } - return new Uint8Array(inflateRawSync(bytes)); -} - -export async function readZipArchive(source: ArrayBuffer | Uint8Array): Promise<{ - rootPath: string | null; - files: Record; -}> { - const bytes = source instanceof Uint8Array ? source : new Uint8Array(source); - const entries: Array<{ path: string; body: CompanyPortabilityFileEntry }> = []; - let offset = 0; - - while (offset + 4 <= bytes.length) { - const signature = readUint32(bytes, offset); - if (signature === 0x02014b50 || signature === 0x06054b50) break; - if (signature !== 0x04034b50) { - throw new Error("Invalid zip archive: unsupported local file header."); - } - - if (offset + 30 > bytes.length) { - throw new Error("Invalid zip archive: truncated local file header."); - } - - const generalPurposeFlag = readUint16(bytes, offset + 6); - const compressionMethod = readUint16(bytes, offset + 8); - const compressedSize = readUint32(bytes, offset + 18); - const fileNameLength = readUint16(bytes, offset + 26); - const extraFieldLength = readUint16(bytes, offset + 28); - - if ((generalPurposeFlag & 0x0008) !== 0) { - throw new Error("Unsupported zip archive: data descriptors are not supported."); - } - - const nameOffset = offset + 30; - const bodyOffset = nameOffset + fileNameLength + extraFieldLength; - const bodyEnd = bodyOffset + compressedSize; - if (bodyEnd > bytes.length) { - throw new Error("Invalid zip archive: truncated file contents."); - } - - const rawArchivePath = textDecoder.decode(bytes.slice(nameOffset, nameOffset + fileNameLength)); - const archivePath = normalizeArchivePath(rawArchivePath); - const isDirectoryEntry = /\/$/.test(rawArchivePath.replace(/\\/g, "/")); - if (archivePath && !isDirectoryEntry) { - const entryBytes = await inflateZipEntry(compressionMethod, bytes.slice(bodyOffset, bodyEnd)); - entries.push({ - path: archivePath, - body: bytesToPortableFileEntry(archivePath, entryBytes), - }); - } - - offset = bodyEnd; - } - - const rootPath = sharedArchiveRoot(entries.map((entry) => entry.path)); - const files: Record = {}; - for (const entry of entries) { - const normalizedPath = - rootPath && entry.path.startsWith(`${rootPath}/`) - ? entry.path.slice(rootPath.length + 1) - : entry.path; - if (!normalizedPath) continue; - files[normalizedPath] = entry.body; - } - - return { rootPath, files }; -} +// The node-side portability zip reader lives in @paperclipai/shared so the +// server can consume the same codec (a raw uploaded zip is unzipped into the +// exact `{ rootPath, files }` bundle the inline import source carries). This +// module re-exports it to keep the CLI's existing import paths stable. +export { + binaryContentTypeByExtension, + bytesToPortableFileEntry, + isBlobStorePath, + readZipArchive, +} from "@paperclipai/shared/portability-zip"; diff --git a/packages/shared/src/portability-zip.test.ts b/packages/shared/src/portability-zip.test.ts new file mode 100644 index 0000000000..f128b701b8 --- /dev/null +++ b/packages/shared/src/portability-zip.test.ts @@ -0,0 +1,303 @@ +import { deflateRawSync } from "node:zlib"; +import { describe, expect, it } from "vitest"; +import { + bytesToPortableFileEntry, + isBlobStorePath, + readZipArchive, +} from "./portability-zip.js"; + +// A minimal, faithful zip writer so the node reader can be round-tripped +// against both STORE (method 0) and DEFLATE (method 8) entries. The layout +// matches the browser writer in ui/src/lib/zip.ts: local file headers, then a +// central directory, then the end-of-central-directory record. +const crcTable = (() => { + const table = new Uint32Array(256); + for (let i = 0; i < 256; i += 1) { + let crc = i; + for (let bit = 0; bit < 8; bit += 1) { + crc = (crc & 1) === 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1; + } + table[i] = crc >>> 0; + } + return table; +})(); + +function crc32(bytes: Uint8Array) { + let crc = 0xffffffff; + for (const byte of bytes) crc = (crc >>> 8) ^ crcTable[(crc ^ byte) & 0xff]!; + return (crc ^ 0xffffffff) >>> 0; +} + +interface ZipInput { + path: string; + bytes: Uint8Array; + method?: 0 | 8; +} + +function buildZip(entries: ZipInput[], rootPath: string): Uint8Array { + const encoder = new TextEncoder(); + const localChunks: Buffer[] = []; + const centralChunks: Buffer[] = []; + let localOffset = 0; + + for (const entry of entries) { + const method = entry.method ?? 0; + const fileName = encoder.encode(`${rootPath}/${entry.path}`); + const checksum = crc32(entry.bytes); + const body = method === 8 ? deflateRawSync(Buffer.from(entry.bytes)) : Buffer.from(entry.bytes); + + const localHeader = Buffer.alloc(30 + fileName.length); + localHeader.writeUInt32LE(0x04034b50, 0); + localHeader.writeUInt16LE(20, 4); + localHeader.writeUInt16LE(0x0800, 6); + localHeader.writeUInt16LE(method, 8); + localHeader.writeUInt32LE(checksum, 14); + localHeader.writeUInt32LE(body.length, 18); + localHeader.writeUInt32LE(entry.bytes.length, 22); + localHeader.writeUInt16LE(fileName.length, 26); + Buffer.from(fileName).copy(localHeader, 30); + + const centralHeader = Buffer.alloc(46 + fileName.length); + centralHeader.writeUInt32LE(0x02014b50, 0); + centralHeader.writeUInt16LE(20, 4); + centralHeader.writeUInt16LE(20, 6); + centralHeader.writeUInt16LE(0x0800, 8); + centralHeader.writeUInt16LE(method, 10); + centralHeader.writeUInt32LE(checksum, 16); + centralHeader.writeUInt32LE(body.length, 20); + centralHeader.writeUInt32LE(entry.bytes.length, 24); + centralHeader.writeUInt16LE(fileName.length, 28); + centralHeader.writeUInt32LE(localOffset, 42); + Buffer.from(fileName).copy(centralHeader, 46); + + localChunks.push(localHeader, body); + centralChunks.push(centralHeader); + localOffset += localHeader.length + body.length; + } + + const centralDirectory = Buffer.concat(centralChunks); + const eocd = Buffer.alloc(22); + eocd.writeUInt32LE(0x06054b50, 0); + eocd.writeUInt16LE(entries.length, 8); + eocd.writeUInt16LE(entries.length, 10); + eocd.writeUInt32LE(centralDirectory.length, 12); + eocd.writeUInt32LE(localOffset, 16); + + return new Uint8Array(Buffer.concat([...localChunks, centralDirectory, eocd])); +} + +describe("isBlobStorePath", () => { + it("matches blobs/ entries at the archive root and under a package root", () => { + expect(isBlobStorePath("blobs/4f2d1c9a")).toBe(true); + expect(isBlobStorePath("paperclip-demo/blobs/4f2d1c9a")).toBe(true); + expect(isBlobStorePath("tasks/pap-1/TASK.md")).toBe(false); + }); +}); + +describe("bytesToPortableFileEntry", () => { + it("keeps blobs/ entries as base64 octet streams regardless of extension", () => { + const bytes = new Uint8Array([0x00, 0x01, 0x80, 0xfe, 0xff]); + expect(bytesToPortableFileEntry("blobs/4f2d1c9a", bytes)).toEqual({ + encoding: "base64", + data: Buffer.from(bytes).toString("base64"), + contentType: "application/octet-stream", + }); + }); + + it("decodes valid UTF-8 entries to text and falls back to base64 for invalid bytes", () => { + const text = new TextEncoder().encode("# Notes\n\ncafé ✅\n"); + expect(bytesToPortableFileEntry("tasks/pap-1/TASK.md", text)).toBe("# Notes\n\ncafé ✅\n"); + const invalid = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]); + expect(bytesToPortableFileEntry("tasks/pap-1/raw", invalid)).toEqual({ + encoding: "base64", + data: Buffer.from(invalid).toString("base64"), + contentType: "application/octet-stream", + }); + }); +}); + +describe("readZipArchive", () => { + it("round-trips STORE, DEFLATE, and base64 blob entries byte-exactly and strips the shared root", async () => { + const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff, 0x13, 0x37]); + // A text body large and repetitive enough that DEFLATE actually shrinks it, + // so the DEFLATE decode path is exercised, not just written. + const deflated = `# Weekly report\n${"paperclip ".repeat(512)}\n`; + + const archive = buildZip( + [ + { path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n"), method: 0 }, + { path: "reports/weekly.md", bytes: new TextEncoder().encode(deflated), method: 8 }, + { path: "blobs/4f2d1c9a", bytes: blobBytes, method: 0 }, + ], + "paperclip-demo", + ); + + await expect(readZipArchive(archive)).resolves.toEqual({ + rootPath: "paperclip-demo", + files: { + "COMPANY.md": "---\nname: Demo\n---\n", + "reports/weekly.md": deflated, + "blobs/4f2d1c9a": { + encoding: "base64", + data: Buffer.from(blobBytes).toString("base64"), + contentType: "application/octet-stream", + }, + }, + }); + }); + + it("throws on a truncated archive so a partial upload fails closed", async () => { + const archive = buildZip( + [{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }], + "paperclip-demo", + ); + // Chop the tail so a declared entry body runs past the end of the buffer. + const truncated = archive.slice(0, 40); + await expect(readZipArchive(truncated)).rejects.toThrow(/truncated|Invalid zip/i); + }); + + it("rejects data-descriptor entries the writer never emits", async () => { + const archive = buildZip( + [{ path: "COMPANY.md", bytes: new TextEncoder().encode("hi") }], + "paperclip-demo", + ); + // Flip bit 0x0008 in the local header's general-purpose flag (offset 6). + archive[6] = archive[6]! | 0x08; + await expect(readZipArchive(archive)).rejects.toThrow(/data descriptors/i); + }); + + // Locate the first central-directory file header so a test can lop off the + // whole directory + EOCD, leaving only intact local entries. + function centralDirectoryOffset(archive: Uint8Array): number { + for (let i = 0; i + 4 <= archive.length; i += 1) { + if (archive[i] === 0x50 && archive[i + 1] === 0x4b && archive[i + 2] === 0x01 && archive[i + 3] === 0x02) { + return i; + } + } + return -1; + } + + it("rejects an archive truncated before the central directory", async () => { + const archive = buildZip( + [ + { path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }, + { path: "agents/ceo/AGENTS.md", bytes: new TextEncoder().encode("---\nname: CEO\n---\n") }, + ], + "paperclip-demo", + ); + // Keep every local entry intact but drop the directory + EOCD, mimicking an + // upload cut at a record boundary. The reader must not import the fragment. + const withoutDirectory = archive.slice(0, centralDirectoryOffset(archive)); + await expect(readZipArchive(withoutDirectory)).rejects.toThrow(/truncated before the central directory/i); + }); + + it("rejects an archive whose end-of-central-directory record is missing", async () => { + const archive = buildZip( + [{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }], + "paperclip-demo", + ); + // The central directory survives but the trailing 22-byte EOCD is gone. + await expect(readZipArchive(archive.slice(0, archive.length - 22))).rejects.toThrow( + /end-of-central-directory/i, + ); + }); + + it("rejects an archive whose central directory count does not match the entries read", async () => { + const archive = buildZip( + [{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }], + "paperclip-demo", + ); + // Overstate the total-entries field (EOCD offset +10 → last 12 bytes in) so a + // silently-dropped central-directory record is caught. + archive[archive.length - 12] = 5; + archive[archive.length - 11] = 0; + await expect(readZipArchive(archive)).rejects.toThrow(/central directory declares 5 entries/i); + }); + + it("rejects a truncated archive re-terminated with a forged EOCD that matches the surviving entries", async () => { + // The exact silent-partial-import this reader guards against: an archive is + // cut after a complete leading local entry (losing the real central directory + // and EOCD), then re-terminated with a hand-forged 22-byte EOCD whose entry + // count matches the surviving local entry. An entry-count-only check would + // wave it through; validating the central directory it points at must not. + const archive = buildZip( + [{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }], + "paperclip-demo", + ); + // Keep only the intact local entry (everything before the first central record). + const localOnly = archive.slice(0, centralDirectoryOffset(archive)); + + const forgedEocd = Buffer.alloc(22); + forgedEocd.writeUInt32LE(0x06054b50, 0); + forgedEocd.writeUInt16LE(1, 8); // entries on this disk + forgedEocd.writeUInt16LE(1, 10); // total entries — matches the one surviving local entry + forgedEocd.writeUInt32LE(0, 12); // central directory size (forged) + forgedEocd.writeUInt32LE(0, 16); // central directory offset (forged) + const forged = new Uint8Array(Buffer.concat([Buffer.from(localOnly), forgedEocd])); + + await expect(readZipArchive(forged)).rejects.toThrow(/central directory location is inconsistent/i); + }); + + it("rejects a forged EOCD whose central directory offset points at non-directory bytes", async () => { + const archive = buildZip( + [{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }], + "paperclip-demo", + ); + const firstEntryOnly = archive.slice(0, centralDirectoryOffset(archive)); + // A forged EOCD whose offset+size abut the record (passing the location + // check) but point into the local entry, which carries no directory signature. + const forgedEocd = Buffer.alloc(22); + forgedEocd.writeUInt32LE(0x06054b50, 0); + forgedEocd.writeUInt16LE(1, 8); + forgedEocd.writeUInt16LE(1, 10); + forgedEocd.writeUInt32LE(46, 12); // size + forgedEocd.writeUInt32LE(firstEntryOnly.length - 46, 16); // start = eocdOffset - size + const forged = new Uint8Array(Buffer.concat([Buffer.from(firstEntryOnly), forgedEocd])); + + await expect(readZipArchive(forged)).rejects.toThrow(/malformed central directory record/i); + }); + + it("rejects two entries that normalize to the same path instead of silently overwriting", async () => { + const archive = buildZip( + [ + { path: "docs/x.md", bytes: new TextEncoder().encode("first") }, + { path: "docs//x.md", bytes: new TextEncoder().encode("second") }, + ], + "paperclip-demo", + ); + await expect(readZipArchive(archive)).rejects.toThrow(/duplicate entry path "docs\/x\.md"/i); + }); + + it("bounds a highly compressible DEFLATE entry at the per-entry decompressed limit", async () => { + // Compresses tiny but expands to 8 KiB; a 1 KiB cap must reject it before it + // materializes. Real packages sit far under the 256 MB production default. + const bomb = new TextEncoder().encode("a".repeat(8 * 1024)); + const archive = buildZip([{ path: "bomb.txt", bytes: bomb, method: 8 }], "paperclip-demo"); + await expect( + readZipArchive(archive, { maxEntryDecompressedBytes: 1024, maxTotalDecompressedBytes: 1 << 30 }), + ).rejects.toThrow(/per-entry limit/i); + }); + + it("bounds a stored entry at the per-entry decompressed limit", async () => { + const stored = new TextEncoder().encode("b".repeat(4 * 1024)); + const archive = buildZip([{ path: "big.bin", bytes: stored, method: 0 }], "paperclip-demo"); + await expect( + readZipArchive(archive, { maxEntryDecompressedBytes: 1024, maxTotalDecompressedBytes: 1 << 30 }), + ).rejects.toThrow(/per-entry limit/i); + }); + + it("bounds the aggregate decompressed size across many entries", async () => { + const chunk = new TextEncoder().encode("c".repeat(600)); + const archive = buildZip( + [ + { path: "a.txt", bytes: chunk, method: 0 }, + { path: "b.txt", bytes: chunk, method: 0 }, + ], + "paperclip-demo", + ); + // Each entry is under the per-entry cap, but together they cross the total. + await expect( + readZipArchive(archive, { maxEntryDecompressedBytes: 4096, maxTotalDecompressedBytes: 1000 }), + ).rejects.toThrow(/exceed the 1000-byte limit/i); + }); +}); diff --git a/packages/shared/src/portability-zip.ts b/packages/shared/src/portability-zip.ts new file mode 100644 index 0000000000..ea531512ed --- /dev/null +++ b/packages/shared/src/portability-zip.ts @@ -0,0 +1,308 @@ +import { inflateRawSync } from "node:zlib"; +import path from "node:path"; +import type { CompanyPortabilityFileEntry } from "./types/company-portability.js"; + +// Node-side reader for the company portability zip package. It produces the +// exact `{ rootPath, files }` shape the inline import source carries, so the +// server can accept a raw uploaded zip and feed the importer the same bundle +// the browser used to expand and post as inline JSON. The browser has its own +// reader in `ui/src/lib/zip.ts` (DecompressionStream); this one uses node zlib. +// Both must stay byte-compatible with the writer in `ui/src/lib/zip.ts`. + +const textDecoder = new TextDecoder(); +// ignoreBOM keeps a leading BOM in the decoded text so text entries +// re-encode to their original bytes; fatal surfaces invalid UTF-8. +const strictTextDecoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); + +// Decompression-bomb guards. The compressed upload is already capped upstream +// (multer/express.raw at PORTABLE_ZIP_UPLOAD_LIMIT_BYTES), but DEFLATE lets a +// small compressed archive expand to gigabytes. Bound the expansion so a +// malicious highly-compressible package cannot exhaust server memory: a +// per-entry ceiling (passed to zlib as maxOutputLength, so it fails before +// over-allocating) plus an aggregate ceiling across all entries. Both sit far +// above any real company package (inline JSON was historically capped at 64MB) +// yet far below what a bomb would need. +export const MAX_ZIP_ENTRY_DECOMPRESSED_BYTES = 256 * 1024 * 1024; +export const MAX_ZIP_TOTAL_DECOMPRESSED_BYTES = 512 * 1024 * 1024; + +export const binaryContentTypeByExtension: Record = { + ".gif": "image/gif", + ".jpeg": "image/jpeg", + ".jpg": "image/jpeg", + ".png": "image/png", + ".svg": "image/svg+xml", + ".webp": "image/webp", +}; + +function normalizeArchivePath(pathValue: string) { + return pathValue + .replace(/\\/g, "/") + .split("/") + .filter(Boolean) + .join("/"); +} + +function readUint16(source: Uint8Array, offset: number) { + return source[offset]! | (source[offset + 1]! << 8); +} + +function readUint32(source: Uint8Array, offset: number) { + return ( + source[offset]! | + (source[offset + 1]! << 8) | + (source[offset + 2]! << 16) | + (source[offset + 3]! << 24) + ) >>> 0; +} + +const LOCAL_FILE_SIGNATURE = 0x04034b50; +const CENTRAL_DIRECTORY_SIGNATURE = 0x02014b50; +const EOCD_SIGNATURE = 0x06054b50; + +// Locate the end-of-central-directory record by scanning back from the tail. +// The record is 22 bytes plus an optional trailing comment (max 0xffff), so the +// signature lives within the last 22 + 0xffff bytes; returns -1 when absent +// (a truncated or non-zip upload), which the reader treats as fail-closed. +function findEndOfCentralDirectoryOffset(bytes: Uint8Array): number { + const minOffset = Math.max(0, bytes.length - (22 + 0xffff)); + for (let offset = bytes.length - 22; offset >= minOffset; offset -= 1) { + if (readUint32(bytes, offset) === EOCD_SIGNATURE) return offset; + } + return -1; +} + +// Fully validate the central directory the EOCD advertises against the local +// entries actually read from the archive body. Trusting only the EOCD's entry +// count is not enough: a truncated archive with a forged 22-byte EOCD whose +// count matches the surviving local entries would otherwise be accepted, and the +// importer would silently process a partial company package (dropping agents, +// issues, and so on). This walks the real central directory and requires that: +// • it is fully present and sits immediately before the EOCD (no gap, no +// pointer past the buffer — a truncated tail fails here); +// • every record carries the central-directory signature and its lengths sum +// to exactly the declared directory size; and +// • every record references a real local file header, and the record count +// equals both the EOCD's declared count and the local entries parsed. +function validateCentralDirectory(bytes: Uint8Array, eocdOffset: number, localHeaderCount: number) { + const declaredEntryCount = readUint16(bytes, eocdOffset + 10); + const centralDirectorySize = readUint32(bytes, eocdOffset + 12); + const centralDirectoryStart = readUint32(bytes, eocdOffset + 16); + if (centralDirectoryStart > eocdOffset || centralDirectoryStart + centralDirectorySize !== eocdOffset) { + throw new Error( + "Invalid zip archive: central directory location is inconsistent (truncated or forged).", + ); + } + + const directoryEnd = centralDirectoryStart + centralDirectorySize; + let cursor = centralDirectoryStart; + let recordCount = 0; + while (cursor < directoryEnd) { + if (cursor + 46 > directoryEnd || readUint32(bytes, cursor) !== CENTRAL_DIRECTORY_SIGNATURE) { + throw new Error("Invalid zip archive: malformed central directory record."); + } + const fileNameLength = readUint16(bytes, cursor + 28); + const extraFieldLength = readUint16(bytes, cursor + 30); + const commentLength = readUint16(bytes, cursor + 32); + const localHeaderOffset = readUint32(bytes, cursor + 42); + if (localHeaderOffset + 4 > bytes.length || readUint32(bytes, localHeaderOffset) !== LOCAL_FILE_SIGNATURE) { + throw new Error("Invalid zip archive: central directory references a missing local entry."); + } + cursor += 46 + fileNameLength + extraFieldLength + commentLength; + recordCount += 1; + } + + if (cursor !== directoryEnd) { + throw new Error("Invalid zip archive: central directory size does not match its records."); + } + if (recordCount !== declaredEntryCount || recordCount !== localHeaderCount) { + throw new Error( + `Invalid zip archive: central directory declares ${declaredEntryCount} entries but ${localHeaderCount} local entries were read (truncated or corrupt).`, + ); + } +} + +function sharedArchiveRoot(paths: string[]) { + if (paths.length === 0) return null; + const firstSegments = paths + .map((entry) => normalizeArchivePath(entry).split("/").filter(Boolean)) + .filter((parts) => parts.length > 0); + if (firstSegments.length === 0) return null; + const candidate = firstSegments[0]![0]!; + return firstSegments.every((parts) => parts.length > 1 && parts[0] === candidate) + ? candidate + : null; +} + +export function isBlobStorePath(pathValue: string) { + return /(^|\/)blobs\/[^/]+$/.test(normalizeArchivePath(pathValue)); +} + +function decodeStrictUtf8(bytes: Uint8Array): string | null { + let text: string; + try { + text = strictTextDecoder.decode(bytes); + } catch { + return null; + } + return Buffer.from(text, "utf8").equals(Buffer.from(bytes)) ? text : null; +} + +export function bytesToPortableFileEntry(pathValue: string, bytes: Uint8Array): CompanyPortabilityFileEntry { + // Content-addressed blob entries are opaque bytes regardless of extension. + if (isBlobStorePath(pathValue)) { + return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType: "application/octet-stream" }; + } + const contentType = binaryContentTypeByExtension[path.extname(pathValue).toLowerCase()]; + if (contentType) { + return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType }; + } + const text = decodeStrictUtf8(bytes); + if (text !== null) return text; + // Bytes that are not valid UTF-8 must not be decoded lossily; fall back + // to base64 so they round-trip exactly. + return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType: "application/octet-stream" }; +} + +// Size caps applied while expanding an archive. Callers use the module defaults; +// the limits are parameterizable so the guard can be exercised in tests without +// allocating hundreds of megabytes. +export interface ReadZipArchiveLimits { + maxEntryDecompressedBytes: number; + maxTotalDecompressedBytes: number; +} + +const DEFAULT_ZIP_LIMITS: ReadZipArchiveLimits = { + maxEntryDecompressedBytes: MAX_ZIP_ENTRY_DECOMPRESSED_BYTES, + maxTotalDecompressedBytes: MAX_ZIP_TOTAL_DECOMPRESSED_BYTES, +}; + +function inflateZipEntry(compressionMethod: number, bytes: Uint8Array, maxEntryDecompressedBytes: number) { + if (compressionMethod === 0) { + if (bytes.length > maxEntryDecompressedBytes) { + throw new Error( + `Unsupported zip archive: a stored entry exceeds the ${maxEntryDecompressedBytes}-byte per-entry limit.`, + ); + } + return bytes; + } + if (compressionMethod !== 8) { + throw new Error("Unsupported zip archive: only STORE and DEFLATE entries are supported."); + } + try { + // maxOutputLength makes zlib throw (ERR_BUFFER_TOO_LARGE) before it allocates + // past the per-entry ceiling, so a bomb entry never materializes in memory. + return new Uint8Array(inflateRawSync(bytes, { maxOutputLength: maxEntryDecompressedBytes })); + } catch (error) { + if ((error as NodeJS.ErrnoException | undefined)?.code === "ERR_BUFFER_TOO_LARGE") { + throw new Error( + `Unsupported zip archive: a compressed entry expands beyond the ${maxEntryDecompressedBytes}-byte per-entry limit.`, + ); + } + throw error; + } +} + +export async function readZipArchive( + source: ArrayBuffer | Uint8Array, + limits: ReadZipArchiveLimits = DEFAULT_ZIP_LIMITS, +): Promise<{ + rootPath: string | null; + files: Record; +}> { + const { maxEntryDecompressedBytes, maxTotalDecompressedBytes } = limits; + const bytes = source instanceof Uint8Array ? source : new Uint8Array(source); + const entries: Array<{ path: string; body: CompanyPortabilityFileEntry }> = []; + let offset = 0; + // Count every local file header (including directory entries) so the tally can + // be reconciled against the central directory below; guard total expansion so + // a bomb split across many entries is still bounded. + let localHeaderCount = 0; + let totalDecompressedBytes = 0; + let reachedCentralDirectory = false; + + while (offset + 4 <= bytes.length) { + const signature = readUint32(bytes, offset); + if (signature === CENTRAL_DIRECTORY_SIGNATURE || signature === EOCD_SIGNATURE) { + reachedCentralDirectory = true; + break; + } + if (signature !== LOCAL_FILE_SIGNATURE) { + throw new Error("Invalid zip archive: unsupported local file header."); + } + + if (offset + 30 > bytes.length) { + throw new Error("Invalid zip archive: truncated local file header."); + } + + const generalPurposeFlag = readUint16(bytes, offset + 6); + const compressionMethod = readUint16(bytes, offset + 8); + const compressedSize = readUint32(bytes, offset + 18); + const fileNameLength = readUint16(bytes, offset + 26); + const extraFieldLength = readUint16(bytes, offset + 28); + + if ((generalPurposeFlag & 0x0008) !== 0) { + throw new Error("Unsupported zip archive: data descriptors are not supported."); + } + + const nameOffset = offset + 30; + const bodyOffset = nameOffset + fileNameLength + extraFieldLength; + const bodyEnd = bodyOffset + compressedSize; + if (bodyEnd > bytes.length) { + throw new Error("Invalid zip archive: truncated file contents."); + } + + localHeaderCount += 1; + const rawArchivePath = textDecoder.decode(bytes.slice(nameOffset, nameOffset + fileNameLength)); + const archivePath = normalizeArchivePath(rawArchivePath); + const isDirectoryEntry = /\/$/.test(rawArchivePath.replace(/\\/g, "/")); + if (archivePath && !isDirectoryEntry) { + const entryBytes = inflateZipEntry(compressionMethod, bytes.slice(bodyOffset, bodyEnd), maxEntryDecompressedBytes); + totalDecompressedBytes += entryBytes.length; + if (totalDecompressedBytes > maxTotalDecompressedBytes) { + throw new Error( + `Unsupported zip archive: decompressed contents exceed the ${maxTotalDecompressedBytes}-byte limit.`, + ); + } + entries.push({ + path: archivePath, + body: bytesToPortableFileEntry(archivePath, entryBytes), + }); + } + + offset = bodyEnd; + } + + // A complete archive always ends with a central directory after its local + // entries. If the scan ran off the end of the buffer without reaching one, the + // upload was truncated at a record boundary — fail closed rather than import a + // leading fragment. Then fully validate the central directory the EOCD points + // at so a truncated tail with a forged EOCD (whose count happens to match the + // surviving entries) cannot smuggle in a partial import. + if (!reachedCentralDirectory) { + throw new Error("Invalid zip archive: truncated before the central directory."); + } + const eocdOffset = findEndOfCentralDirectoryOffset(bytes); + if (eocdOffset === -1) { + throw new Error("Invalid zip archive: missing end-of-central-directory record."); + } + validateCentralDirectory(bytes, eocdOffset, localHeaderCount); + + const rootPath = sharedArchiveRoot(entries.map((entry) => entry.path)); + const files: Record = {}; + for (const entry of entries) { + const normalizedPath = + rootPath && entry.path.startsWith(`${rootPath}/`) + ? entry.path.slice(rootPath.length + 1) + : entry.path; + if (!normalizedPath) continue; + // Two entries that normalize to the same path (e.g. `a/b` and `a//b`) make + // the package ambiguous; reject it rather than silently letting the later + // entry's contents win over the earlier one. + if (Object.prototype.hasOwnProperty.call(files, normalizedPath)) { + throw new Error(`Invalid zip archive: duplicate entry path "${normalizedPath}".`); + } + files[normalizedPath] = entry.body; + } + + return { rootPath, files }; +} diff --git a/server/src/__tests__/company-portability-routes.test.ts b/server/src/__tests__/company-portability-routes.test.ts index 8899620086..2de71aba9d 100644 --- a/server/src/__tests__/company-portability-routes.test.ts +++ b/server/src/__tests__/company-portability-routes.test.ts @@ -260,6 +260,79 @@ async function waitForImportJobStatusAs( throw new Error(`Timed out waiting for import job to reach ${status}`); } +// A minimal STORE-only zip writer so the multipart/zip upload path can be +// exercised end-to-end: the route unzips this into the same inline bundle an +// application/json caller would send. Layout matches ui/src/lib/zip.ts (local +// file headers, central directory, end-of-central-directory). +function crc32(bytes: Uint8Array) { + let crc = 0xffffffff; + for (const byte of bytes) { + crc ^= byte; + for (let bit = 0; bit < 8; bit += 1) { + crc = (crc & 1) === 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1; + } + } + return (crc ^ 0xffffffff) >>> 0; +} + +function buildStoreZip(files: Record, rootPath: string): Buffer { + const encoder = new TextEncoder(); + const localChunks: Buffer[] = []; + const centralChunks: Buffer[] = []; + let localOffset = 0; + const entries = Object.entries(files); + + for (const [relativePath, content] of entries) { + const fileName = encoder.encode(`${rootPath}/${relativePath}`); + const body = Buffer.from(encoder.encode(content)); + const checksum = crc32(body); + + const localHeader = Buffer.alloc(30 + fileName.length); + localHeader.writeUInt32LE(0x04034b50, 0); + localHeader.writeUInt16LE(20, 4); + localHeader.writeUInt16LE(0x0800, 6); + localHeader.writeUInt32LE(checksum, 14); + localHeader.writeUInt32LE(body.length, 18); + localHeader.writeUInt32LE(body.length, 22); + localHeader.writeUInt16LE(fileName.length, 26); + Buffer.from(fileName).copy(localHeader, 30); + + const centralHeader = Buffer.alloc(46 + fileName.length); + centralHeader.writeUInt32LE(0x02014b50, 0); + centralHeader.writeUInt16LE(20, 4); + centralHeader.writeUInt16LE(20, 6); + centralHeader.writeUInt16LE(0x0800, 8); + centralHeader.writeUInt32LE(checksum, 16); + centralHeader.writeUInt32LE(body.length, 20); + centralHeader.writeUInt32LE(body.length, 24); + centralHeader.writeUInt16LE(fileName.length, 28); + centralHeader.writeUInt32LE(localOffset, 42); + Buffer.from(fileName).copy(centralHeader, 46); + + localChunks.push(localHeader, body); + centralChunks.push(centralHeader); + localOffset += localHeader.length + body.length; + } + + const centralDirectory = Buffer.concat(centralChunks); + const eocd = Buffer.alloc(22); + eocd.writeUInt32LE(0x06054b50, 0); + eocd.writeUInt16LE(entries.length, 8); + eocd.writeUInt16LE(entries.length, 10); + eocd.writeUInt32LE(centralDirectory.length, 12); + eocd.writeUInt32LE(localOffset, 16); + + return Buffer.concat([...localChunks, centralDirectory, eocd]); +} + +// The import fields a multipart caller ships as the JSON `meta` field: the same +// object an inline caller sends, minus `source` (the source is the uploaded zip). +const importMeta = { + include: importRequest.include, + target: importRequest.target, + collisionStrategy: importRequest.collisionStrategy, +}; + describe.sequential("company portability routes", () => { beforeEach(() => { vi.clearAllMocks(); @@ -1007,4 +1080,136 @@ describe.sequential("company portability routes", () => { { mode: "agent_safe", sourceCompanyId: companyId, pauseAutomations: true }, ); }); + + it.sequential("imports a company from a multipart zip upload, unzipping into the same inline bundle", async () => { + const app = await createBoardApp(); + const files = { "COMPANY.md": "---\nname: Test\n---\n", "agents/ceo/AGENTS.md": "---\nname: CEO\n---\n" }; + const zip = buildStoreZip(files, "paperclip"); + + const res = await request(app) + .post("/api/companies/import") + .set(TEST_USER_HEADER, "board-user-a") + .field("meta", JSON.stringify(importMeta)) + .attach("package", zip, "paperclip-demo.zip"); + + expect(res.status).toBe(200); + expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledTimes(1); + const call = mockCompanyPortabilityService.importBundle.mock.calls[0]!; + // The uploaded zip is unzipped into the exact inline source the importer + // consumes; the other import fields ride along from the JSON meta field. + expect(call[0]).toEqual({ ...importMeta, source: { type: "inline", rootPath: "paperclip", files } }); + expect(call[1]).toBe("board-user-a"); + expect(call[2]).toEqual({ pauseAutomations: false }); + }); + + it.sequential("previews a company from a multipart zip upload", async () => { + const app = await createBoardApp(); + const files = { "COMPANY.md": "---\nname: Test\n---\n" }; + const zip = buildStoreZip(files, "paperclip"); + + const res = await request(app) + .post("/api/companies/import/preview") + .set(TEST_USER_HEADER, "board-user-a") + .field("meta", JSON.stringify(importMeta)) + .attach("package", zip, "paperclip-demo.zip"); + + expect(res.status).toBe(200); + expect(mockCompanyPortabilityService.previewImport).toHaveBeenCalledTimes(1); + const call = mockCompanyPortabilityService.previewImport.mock.calls[0]!; + expect(call[0]).toEqual({ ...importMeta, source: { type: "inline", rootPath: "paperclip", files } }); + }); + + it.sequential("runs a multipart zip import as an async board job via ?async=1", async () => { + let resolveImport: (value: ReturnType) => void = () => undefined; + const pendingImport = new Promise>((resolve) => { + resolveImport = resolve; + }); + mockCompanyPortabilityService.importBundle.mockReturnValueOnce(pendingImport); + const app = await createBoardApp(); + const files = { "COMPANY.md": "---\nname: Test\n---\n" }; + const zip = buildStoreZip(files, "paperclip"); + + const accepted = await request(app) + .post("/api/companies/import?async=1") + .set(TEST_USER_HEADER, "board-user-a") + .field("meta", JSON.stringify(importMeta)) + .attach("package", zip, "paperclip-demo.zip"); + + expect(accepted.status).toBe(202); + expect(accepted.body.job.status).toBe("running"); + expect(accepted.body.statusUrl).toMatch(/^\/api\/companies\/import\/jobs\/import-/); + await waitForCondition( + () => mockCompanyPortabilityService.importBundle.mock.calls.length === 1, + "multipart async import start", + ); + expect(mockCompanyPortabilityService.importBundle.mock.calls[0]![0]).toEqual({ + ...importMeta, + source: { type: "inline", rootPath: "paperclip", files }, + }); + + const fullResult = createImportResult("created"); + resolveImport(fullResult); + const succeeded = await waitForImportJobStatusAs(app, accepted.body.statusUrl, "succeeded", { + [TEST_USER_HEADER]: "board-user-a", + }); + expect(succeeded.body.job.status).toBe("succeeded"); + expect(succeeded.body.job.importResult).toEqual(fullResult); + }); + + it.sequential("engages the async path for board sessions via ?async=1, not the stripped cloud header", async () => { + mockCompanyPortabilityService.importBundle.mockReturnValueOnce(new Promise(() => undefined)); + const app = await createBoardApp(); + + // The Cloud harness strips inbound x-paperclip-cloud-* headers, so a browser + // can only opt into async with the proxy-safe query parameter. + const accepted = await request(app) + .post("/api/companies/import?async=1") + .set(TEST_USER_HEADER, "board-user-a") + .send(importRequest); + + expect(accepted.status).toBe(202); + expect(accepted.body.job.status).toBe("running"); + expect(accepted.body.statusUrl).toMatch(/^\/api\/companies\/import\/jobs\/import-/); + }); + + it.sequential("keeps a board import synchronous when neither async signal is present", async () => { + const app = await createBoardApp(); + + const res = await request(app) + .post("/api/companies/import") + .set(TEST_USER_HEADER, "board-user-a") + .send(importRequest); + + expect(res.status).toBe(200); + expect(res.body.company.id).toBe(companyId); + }); + + it.sequential("still engages the async path for cloud tenants via the x-paperclip-cloud-async-import header", async () => { + mockCompanyPortabilityService.importBundle.mockReturnValueOnce(new Promise(() => undefined)); + const app = await createApp(cloudTenantActor()); + + const accepted = await request(app) + .post("/api/companies/import") + .set("x-paperclip-cloud-async-import", "1") + .set(cloudHeaders) + .send(importRequest); + + expect(accepted.status).toBe(202); + expect(accepted.body.statusUrl).toMatch(/^\/api\/companies\/import\/jobs\/tenant-import-/); + }); + + it.sequential("rejects a truncated zip upload without importing anything", async () => { + const app = await createBoardApp(); + const zip = buildStoreZip({ "COMPANY.md": "---\nname: Test\n---\n" }, "paperclip"); + const truncated = zip.subarray(0, 40); + + const res = await request(app) + .post("/api/companies/import") + .set(TEST_USER_HEADER, "board-user-a") + .field("meta", JSON.stringify(importMeta)) + .attach("package", truncated, "paperclip-demo.zip"); + + expect(res.status).toBe(400); + expect(mockCompanyPortabilityService.importBundle).not.toHaveBeenCalled(); + }); }); diff --git a/server/src/http/body-limits.ts b/server/src/http/body-limits.ts index 12c45705b4..29c92b2baf 100644 --- a/server/src/http/body-limits.ts +++ b/server/src/http/body-limits.ts @@ -1,3 +1,9 @@ export const DEFAULT_JSON_BODY_LIMIT = "10mb"; export const PORTABLE_JSON_BODY_LIMIT = "64mb"; export const PORTABLE_JSON_BODY_LIMIT_BYTES = 64 * 1024 * 1024; + +// A company import can also be uploaded as its raw compressed zip (multipart or +// application/zip) instead of an inflated inline JSON body. The compressed zip +// is roughly a third of the inline size, but the limit is kept generous so a +// large company package uploads in one request rather than truncating in transit. +export const PORTABLE_ZIP_UPLOAD_LIMIT_BYTES = 128 * 1024 * 1024; diff --git a/server/src/routes/companies.ts b/server/src/routes/companies.ts index 3ac4003bb9..8a9b38df8d 100644 --- a/server/src/routes/companies.ts +++ b/server/src/routes/companies.ts @@ -1,10 +1,12 @@ import { createHash, randomUUID } from "node:crypto"; -import { Router, type Request } from "express"; +import express, { Router, type Request, type Response } from "express"; +import multer from "multer"; import { and, count as countFn, eq } from "drizzle-orm"; import { z } from "zod"; import type { Db } from "@paperclipai/db"; import { agents as agentsTable } from "@paperclipai/db"; import type { CompanyPortabilityImportResult } from "@paperclipai/shared"; +import { readZipArchive } from "@paperclipai/shared/portability-zip"; import { DEFAULT_FEEDBACK_DATA_SHARING_TERMS_VERSION, companyArtifactsQuerySchema, @@ -18,7 +20,8 @@ import { updateCompanyBrandingSchema, updateCompanySchema, } from "@paperclipai/shared"; -import { badRequest, forbidden } from "../errors.js"; +import { badRequest, forbidden, unprocessable } from "../errors.js"; +import { PORTABLE_ZIP_UPLOAD_LIMIT_BYTES } from "../http/body-limits.js"; import { validate } from "../middleware/validate.js"; import { accessService, @@ -37,6 +40,147 @@ import type { StorageService } from "../storage/types.js"; import { assertBoard, assertCompanyAccess, assertInstanceAdmin, getActorInfo } from "./authz.js"; import { COMPANY_IMPORT_ROUTE_PATH } from "./company-import-paths.js"; +// A company import can arrive one of two ways on the import + preview routes: +// • application/json — the original inline body `{ source, target, ... }`, +// kept byte-identical for CLI and programmatic callers; or +// • the raw compressed zip, either as multipart/form-data (a `package` file +// field plus a JSON `meta` field) or a bare application/zip body. The zip +// is a third of the inline size and already gzip-friendly, so it survives +// the browser → edge → harness-proxy → tenant chain that truncated the +// inflated inline JSON on large companies. +// The zip is unzipped server-side into the exact `{ rootPath, files }` bundle +// the inline source carries, then fed through the unchanged preview/import +// logic, so import semantics are identical regardless of transport. +const PORTABLE_ZIP_CONTENT_TYPES = ["application/zip", "application/x-zip-compressed"] as const; + +const zipPackageUpload = multer({ + storage: multer.memoryStorage(), + limits: { fileSize: PORTABLE_ZIP_UPLOAD_LIMIT_BYTES, files: 1 }, +}); + +const rawZipBodyParser = express.raw({ + type: [...PORTABLE_ZIP_CONTENT_TYPES], + limit: PORTABLE_ZIP_UPLOAD_LIMIT_BYTES, +}); + +function requestContentType(req: Request) { + return (req.header("content-type") ?? "").toLowerCase(); +} + +function isMultipartImport(req: Request) { + return requestContentType(req).includes("multipart/form-data"); +} + +function isZipImport(req: Request) { + const contentType = requestContentType(req); + return PORTABLE_ZIP_CONTENT_TYPES.some((type) => contentType.includes(type)); +} + +function runMiddleware( + middleware: (req: Request, res: Response, next: (err?: unknown) => void) => void, + req: Request, + res: Response, +) { + return new Promise((resolve, reject) => { + middleware(req, res, (err?: unknown) => (err ? reject(err) : resolve())); + }); +} + +/** + * Parse the `meta` form/query field into the object the import schemas expect. + * A client-declared `source` is ignored — the source is always the uploaded + * zip — so a multipart caller only ships the other import fields (include, + * target, collisionStrategy, nameOverrides, selectedFiles, adapterOverrides, + * pauseAutomations, ...). + */ +function parseImportMeta(metaRaw: string | undefined): Record { + if (metaRaw === undefined || metaRaw.trim().length === 0) return {}; + let parsed: unknown; + try { + parsed = JSON.parse(metaRaw); + } catch { + throw badRequest("Import package metadata was not valid JSON."); + } + if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) { + throw badRequest("Import package metadata must be a JSON object."); + } + const { source: _ignoredSource, ...rest } = parsed as Record; + return rest; +} + +/** + * Resolve the object to hand the preview/import zod schemas. For a JSON request + * this is the inline body unchanged. For a multipart or application/zip request + * the uploaded zip is read into `{ rootPath, files }` and combined with the + * `meta` fields as `source = { type: "inline", rootPath, files }`. A truncated, + * corrupt, or empty zip fails closed with a 400/422 before anything is imported. + */ +async function resolveImportPayload(req: Request, res: Response): Promise { + const multipart = isMultipartImport(req); + const rawZip = isZipImport(req); + if (!multipart && !rawZip) { + return req.body; + } + + let zipBytes: Buffer | undefined; + let metaRaw: string | undefined; + if (multipart) { + try { + await runMiddleware(zipPackageUpload.single("package"), req, res); + } catch (error) { + if (error instanceof multer.MulterError) { + if (error.code === "LIMIT_FILE_SIZE") { + throw unprocessable(`Import package exceeds ${PORTABLE_ZIP_UPLOAD_LIMIT_BYTES} bytes`); + } + throw badRequest(error.message); + } + throw error; + } + const file = (req as Request & { file?: { buffer?: Buffer } }).file; + zipBytes = file?.buffer; + const metaField = (req.body as { meta?: unknown } | undefined)?.meta; + metaRaw = typeof metaField === "string" ? metaField : undefined; + } else { + try { + await runMiddleware(rawZipBodyParser, req, res); + } catch (error) { + throw badRequest(error instanceof Error ? error.message : "Invalid zip upload"); + } + zipBytes = Buffer.isBuffer(req.body) ? req.body : undefined; + const metaQuery = req.query.meta; + metaRaw = typeof metaQuery === "string" ? metaQuery : undefined; + } + + if (!zipBytes || zipBytes.length === 0) { + throw badRequest("Import package upload was empty."); + } + let archive: Awaited>; + try { + archive = await readZipArchive(zipBytes); + } catch (error) { + throw badRequest(`Import package could not be read: ${errorMessage(error)}`); + } + if (Object.keys(archive.files).length === 0) { + throw badRequest("Import package contained no files."); + } + return { + ...parseImportMeta(metaRaw), + source: { type: "inline", rootPath: archive.rootPath, files: archive.files }, + }; +} + +/** + * Async job opt-in. Cloud tenants set the `x-paperclip-cloud-async-import` + * header server-side (they are not a browser, so it is never stripped). Board + * browsers cannot use that header — the Cloud harness proxy strips every + * inbound `x-paperclip-cloud-*` header as anti-spoofing — so they opt in with + * the proxy-safe `?async=1` query parameter instead. Either signal enters the + * async path. + */ +function wantsAsyncImport(req: Request) { + return req.query.async === "1" || req.header("x-paperclip-cloud-async-import") === "1"; +} + export function companyRoutes(db: Db, storage?: StorageService) { const router = Router(); const svc = companyService(db); @@ -264,7 +408,7 @@ export function companyRoutes(db: Db, storage?: StorageService) { router.post("/import/preview", async (req, res) => { assertBoard(req); - const body = companyPortabilityPreviewSchema.parse(req.body); + const body = companyPortabilityPreviewSchema.parse(await resolveImportPayload(req, res)); assertImportTargetAccess(req, body.target); const preview = await portability.previewImport(body); res.json(preview); @@ -288,10 +432,14 @@ export function companyRoutes(db: Db, storage?: StorageService) { router.post(COMPANY_IMPORT_ROUTE_PATH, async (req, res) => { assertBoard(req); - const rawImportBody: unknown = req.body; + // Resolve the request body up front: a JSON caller's inline body is used + // unchanged; a multipart/application-zip caller's uploaded zip is read into + // the same `{ source: { type: "inline", rootPath, files } }` bundle here, + // fast and in-memory, so the async job machinery below is transport-agnostic. + const rawImportBody: unknown = await resolveImportPayload(req, res); const actor = getActorInfo(req); const boardUserId = req.actor.type === "board" ? req.actor.userId : null; - if (req.header("x-paperclip-cloud-async-import") === "1") { + if (wantsAsyncImport(req)) { // Async job path. Two kinds of callers opt in: // - trusted Cloud tenants (original behavior, kept byte-identical), // keyed by their tenant identity headers; diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index f06316a3ff..9129f96fd8 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -523,6 +523,40 @@ const jsonBody = (schema: z.ZodTypeAny) => ({ required: true as const, }); +// The company import + preview routes accept the inline JSON body or the raw +// company package as a compressed zip upload. Document both content types: the +// JSON variant keeps its zod schema; the multipart variant carries the zip in a +// `package` file field plus the other import fields as a JSON `meta` field. +const importRequestBody = (schema: z.ZodTypeAny) => ({ + content: { + "application/json": { schema }, + "multipart/form-data": { + schema: { + type: "object", + properties: { + package: { + type: "string", + format: "binary", + description: "The company package as a compressed .zip.", + }, + meta: { + type: "string", + description: + "JSON-encoded import fields (include, target, collisionStrategy, and, for " + + "the apply route, nameOverrides / selectedFiles / adapterOverrides / " + + "pauseAutomations). A `source` here is ignored — the source is the zip.", + }, + }, + required: ["package"], + }, + }, + "application/zip": { + schema: { type: "string", format: "binary" }, + }, + }, + required: true as const, +}); + const r = responses; const externalObjectSummariesBodySchema = z.object({ @@ -5239,8 +5273,15 @@ registry.registerPath({ path: "/api/companies/import/preview", tags: ["companies"], summary: "Preview a company import (legacy route)", - request: { body: jsonBody(companyPortabilityPreviewSchema) }, - responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized }, + description: + "Accepts either the inline JSON body (`application/json`) or the raw company " + + "package uploaded as a compressed zip (`multipart/form-data` with a `package` " + + "file field plus a JSON `meta` field carrying the other import fields, or a bare " + + "`application/zip` body with the `meta` JSON in the `meta` query parameter). The " + + "zip is unzipped server-side into the same `{ source: { type: \"inline\", ... } }` " + + "bundle the JSON body carries.", + request: { body: importRequestBody(companyPortabilityPreviewSchema) }, + responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 422: r.unprocessable }, }); registry.registerPath({ @@ -5249,17 +5290,28 @@ registry.registerPath({ tags: ["companies"], summary: "Apply a company import (legacy route)", description: - "Board sessions and trusted Cloud tenants can opt into asynchronous processing with the " + - "`x-paperclip-cloud-async-import: 1` header: the server responds 202 with a job id and status " + - "URL instead of holding the connection open for the whole import. While a board actor already " + - "has an async job running, a resubmit returns 409 carrying the running job's id and status URL. " + - "Jobs are held in memory and are lost on restart.", - request: { body: jsonBody(companyPortabilityImportSchema) }, + "Accepts either the inline JSON body (`application/json`) or the raw company package " + + "uploaded as a compressed zip (`multipart/form-data` with a `package` file field plus a " + + "JSON `meta` field, or a bare `application/zip` body with the `meta` JSON in the `meta` " + + "query parameter); the zip is unzipped server-side into the same import bundle. " + + "Callers can opt into asynchronous processing: trusted Cloud tenants set the " + + "`x-paperclip-cloud-async-import: 1` header (browsers cannot — the Cloud harness proxy " + + "strips inbound `x-paperclip-cloud-*` headers), while board sessions use the proxy-safe " + + "`?async=1` query parameter. Either way the server responds 202 with a job id and status " + + "URL instead of holding the connection open for the whole import. While a board actor " + + "already has an async job running, a resubmit returns 409 carrying the running job's id " + + "and status URL. Jobs are held in memory and are lost on restart.", + request: { + query: z.object({ async: z.enum(["1"]).optional() }), + body: importRequestBody(companyPortabilityImportSchema), + }, responses: { 200: r.ok(), + 202: { description: "Async import job accepted" }, 400: r.badRequest, 401: r.unauthorized, 409: { description: "An async import job is already running for this actor" }, + 422: r.unprocessable, }, }); diff --git a/ui/src/api/client.ts b/ui/src/api/client.ts index 7505278c20..6ba3f68a6c 100644 --- a/ui/src/api/client.ts +++ b/ui/src/api/client.ts @@ -155,7 +155,14 @@ export const api = { ...(options?.headers ? { headers: options.headers } : {}), }), postForm: (path: string, body: FormData, options?: RequestOptions) => - request(path, { method: "POST", body, signal: options?.signal }), + request(path, { + method: "POST", + body, + signal: options?.signal, + // Never set Content-Type here — the browser sets multipart/form-data with + // the boundary. Extra headers (e.g. an async opt-in) may still ride along. + ...(options?.headers ? { headers: options.headers } : {}), + }), put: (path: string, body: unknown, options?: RequestOptions) => request(path, { method: "PUT", body: JSON.stringify(body), signal: options?.signal }), patch: (path: string, body: unknown, options?: RequestOptions) => diff --git a/ui/src/api/companies.ts b/ui/src/api/companies.ts index 40b334aeb2..04c7d09daa 100644 --- a/ui/src/api/companies.ts +++ b/ui/src/api/companies.ts @@ -14,6 +14,21 @@ import { api } from "./client"; export type CompanyStats = Record; +/** + * Import fields for a zip package upload: everything the JSON request carries + * except `source` — the source is the uploaded zip, read server-side. Sent as a + * JSON-encoded `meta` form field alongside the raw `package` file. + */ +export type CompanyPortabilityPreviewMeta = Omit; +export type CompanyPortabilityImportMeta = Omit; + +function importPackageForm(file: File, meta: CompanyPortabilityPreviewMeta | CompanyPortabilityImportMeta): FormData { + const form = new FormData(); + form.append("package", file); + form.append("meta", JSON.stringify(meta)); + return form; +} + export type CompanyImportJobState = "running" | "succeeded" | "failed"; /** 202 body from the async opt-in on POST /companies/import (and the 409 body when a job is already running). */ @@ -82,13 +97,21 @@ export const companiesApi = { api.get(`/companies/${companyId}/export/fidelity`), importPreview: (data: CompanyPortabilityPreviewRequest) => api.post("/companies/import/preview", data), + /** Preview a local .zip package by uploading the raw compressed zip as multipart. */ + importPreviewPackage: (file: File, meta: CompanyPortabilityPreviewMeta) => + api.postForm("/companies/import/preview", importPackageForm(file, meta)), importBundle: (data: CompanyPortabilityImportRequest) => api.post("/companies/import", data), - /** Submit an import as a server-side job: 202 with a job id to poll, or 409 with the already-running job. */ + // Submit an import as a server-side job: 202 with a job id to poll, or 409 + // with the already-running job. Board sessions opt in with the proxy-safe + // `?async=1` query parameter — the Cloud harness strips the inbound + // `x-paperclip-cloud-*` header a browser would otherwise use, so that header + // never survives to engage the async path. importBundleAsync: (data: CompanyPortabilityImportRequest) => - api.post("/companies/import", data, { - headers: { "x-paperclip-cloud-async-import": "1" }, - }), + api.post("/companies/import?async=1", data), + /** Submit a local .zip package as an async import job by uploading the raw compressed zip as multipart. */ + importBundlePackageAsync: (file: File, meta: CompanyPortabilityImportMeta) => + api.postForm("/companies/import?async=1", importPackageForm(file, meta)), getImportJob: (jobId: string) => api.get(`/companies/import/jobs/${encodeURIComponent(jobId)}`), }; diff --git a/ui/src/pages/CompanyImport.test.tsx b/ui/src/pages/CompanyImport.test.tsx index c02ec36bdf..07ff1abf4c 100644 --- a/ui/src/pages/CompanyImport.test.tsx +++ b/ui/src/pages/CompanyImport.test.tsx @@ -11,8 +11,10 @@ import { CompanyImport } from "./CompanyImport"; const mockCompaniesApi = vi.hoisted(() => ({ importPreview: vi.fn(), + importPreviewPackage: vi.fn(), importBundle: vi.fn(), importBundleAsync: vi.fn(), + importBundlePackageAsync: vi.fn(), getImportJob: vi.fn(), get: vi.fn(), })); @@ -186,9 +188,11 @@ describe("CompanyImport", () => { mockAgentsApi.resume.mockResolvedValue({ id: "agent-1", status: "idle" }); mockRoutinesApi.update.mockResolvedValue({ id: "routine-1", status: "active" }); mockCompaniesApi.importPreview.mockResolvedValue(buildPreviewResult()); + mockCompaniesApi.importPreviewPackage.mockResolvedValue(buildPreviewResult()); // Default async flow: the submit is accepted (202) and the first poll finds // the job already finished with the full result. Individual tests override. mockCompaniesApi.importBundleAsync.mockResolvedValue(buildAccepted()); + mockCompaniesApi.importBundlePackageAsync.mockResolvedValue(buildAccepted()); mockCompaniesApi.getImportJob.mockResolvedValue(buildSucceededJob()); mockCompaniesApi.get.mockResolvedValue({ id: "company-2", name: "Imported Test", issuePrefix: "IMP" }); mockSidebarPreferencesApi.updateProjectOrder.mockResolvedValue(undefined); @@ -303,9 +307,10 @@ describe("CompanyImport", () => { expect(mockPushToast).toHaveBeenCalledWith(expect.objectContaining({ tone: "error" })); }); - it("blocks oversized local packages and sends the declared file count once attachments are dropped", async () => { - // A synthetic parsed package: the base64 blob payload alone exceeds the - // inline import limit, so no real 60MB zip needs to be built. + it("uploads a local .zip as a multipart package and never blocks on inline size", async () => { + // Even a package whose inflated inline JSON would blow past the old browser + // limit uploads fine now: the raw compressed zip goes up as multipart and is + // unzipped server-side, so the inline-size ceiling no longer gates it. mockReadZipArchive.mockResolvedValue({ rootPath: "big-package", files: { @@ -325,7 +330,7 @@ describe("CompanyImport", () => { const fileInput = container.querySelector('input[type="file"]'); expect(fileInput).toBeTruthy(); - const file = new File(["stub"], "big-package.zip", { type: "application/zip" }); + const file = new File(["stub-zip-bytes"], "big-package.zip", { type: "application/zip" }); Object.defineProperty(file, "arrayBuffer", { value: async () => new ArrayBuffer(0) }); Object.defineProperty(fileInput!, "files", { value: [file] }); await act(async () => { @@ -333,32 +338,33 @@ describe("CompanyImport", () => { }); await flushReact(); - expect(container.textContent).toContain("CLI folder import"); - expect(container.textContent).toContain("Package too large for browser import"); - expect(findButton((text) => text === "Preview import")?.disabled).toBe(true); - - await clickButton((text) => text === "Continue without attachments"); - - expect(container.textContent).not.toContain("CLI folder import"); + // The inline preflight no longer blocks the zip path. expect(container.textContent).not.toContain("Package too large for browser import"); + expect(container.textContent).not.toContain("CLI folder import"); expect(findButton((text) => text === "Preview import")?.disabled).toBe(false); await clickButton((text) => text === "Preview import"); - // The button label reflects the preview's file count (3); the sent source - // is the local package, which is down to two files after the blob is - // dropped. + // The preview goes up as a multipart package (the raw File), not inline JSON. + expect(mockCompaniesApi.importPreviewPackage).toHaveBeenCalledTimes(1); + expect(mockCompaniesApi.importPreview).not.toHaveBeenCalled(); + expect(mockCompaniesApi.importPreviewPackage.mock.calls[0]![0]).toBe(file); + await clickButton((text) => text.startsWith("Import 3 file")); await settle(); - expect(mockCompaniesApi.importBundleAsync).toHaveBeenCalledTimes(1); - const request = mockCompaniesApi.importBundleAsync.mock.calls[0]![0] as { - source: { type: string; files: Record; expectedFileCount?: number }; - }; - expect(request.source.type).toBe("inline"); - expect(Object.keys(request.source.files).sort()).toEqual([".paperclip.yaml", "COMPANY.md"]); - // The client declares the file count so the server can reject a truncated - // upload instead of importing a fragment. - expect(request.source.expectedFileCount).toBe(2); + // The apply also uploads the raw File as a multipart async job — never the + // inflated inline body that truncated in transit. + expect(mockCompaniesApi.importBundleAsync).not.toHaveBeenCalled(); + expect(mockCompaniesApi.importBundlePackageAsync).toHaveBeenCalledTimes(1); + const [sentFile, meta] = mockCompaniesApi.importBundlePackageAsync.mock.calls[0]! as [ + File, + { pauseAutomations: boolean; target: { mode: string } }, + ]; + expect(sentFile).toBe(file); + expect(sentFile.name).toBe("big-package.zip"); + expect(meta.pauseAutomations).toBe(true); + // The bundle itself is never expanded into the request; only the raw zip travels. + expect(meta).not.toHaveProperty("source"); }); it("explains the disabled preview button until a package is chosen", async () => { diff --git a/ui/src/pages/CompanyImport.tsx b/ui/src/pages/CompanyImport.tsx index 4972598126..d2921f6d99 100644 --- a/ui/src/pages/CompanyImport.tsx +++ b/ui/src/pages/CompanyImport.tsx @@ -50,13 +50,7 @@ import { FileTree, } from "../components/FileTree"; import { readZipArchive } from "../lib/zip"; -import { - INLINE_IMPORT_MAX_BYTES, - buildInlineImportPreflight, - formatMegabytes, - isBlobStoreFilePath, - stripBlobFiles, -} from "../lib/import-preflight"; +import { formatMegabytes } from "../lib/import-preflight"; import { getPortableFileDataUrl, getPortableFileText, isPortableImageFile } from "../lib/portable-files"; import { clearStoredImportJob, @@ -667,6 +661,14 @@ function AdapterPickerList({ async function readLocalPackageZip(file: File): Promise<{ name: string; + /** + * The raw .zip File itself. Local imports upload this compressed file as + * multipart instead of inflating it into one large inline JSON body (which + * truncated in transit on big companies); the parsed `files` map below is + * kept only for the client-side preflight display and preview affordances. + */ + file: File; + compressedBytes: number; rootPath: string | null; files: Record; }> { @@ -679,6 +681,8 @@ async function readLocalPackageZip(file: File): Promise<{ } return { name: file.name, + file, + compressedBytes: file.size, rootPath: archive.rootPath, files: archive.files, }; @@ -779,6 +783,8 @@ export function CompanyImport() { const [importUrl, setImportUrl] = useState(""); const [localPackage, setLocalPackage] = useState<{ name: string; + file: File; + compressedBytes: number; rootPath: string | null; files: Record; } | null>(null); @@ -845,24 +851,29 @@ export function CompanyImport() { ]); }, [selectedCompany?.name, setBreadcrumbs]); - function buildSource(): CompanyPortabilitySource | null { - if (sourceMode === "local") { - if (!localPackage) return null; - // Declare how many files we are sending so the server can reject a - // truncated upload instead of importing a fragment (see the server-side - // completeness check in importBundle). - return { - type: "inline", - rootPath: localPackage.rootPath, - files: localPackage.files, - expectedFileCount: Object.keys(localPackage.files).length, - }; - } + // The GitHub/URL source still travels inline (it is just a URL, so it never + // hits the inline-size ceiling). The local .zip source uploads its raw + // compressed file as multipart instead — see the preview/import mutations. + function buildGithubSource(): CompanyPortabilitySource | null { const url = importUrl.trim(); if (!url) return null; return { type: "github", url }; } + // Import fields shared by preview and apply, and by both transports. The + // multipart zip upload ships these as a JSON `meta` field; the inline GitHub + // request spreads them alongside its `source`. + function buildImportMetaCommon() { + return { + include: { company: true, agents: true, projects: true, issues: true }, + target: + targetMode === "new" + ? { mode: "new_company" as const, newCompanyName: newCompanyName || null } + : { mode: "existing_company" as const, companyId: selectedCompanyId! }, + collisionStrategy, + }; + } + // Monotonic id for preview requests. Structural configuration changes bump // it, so an in-flight preview they supersede settles silently instead of // publishing a result or error for a package that is no longer selected. @@ -873,17 +884,16 @@ export function CompanyImport() { // Preview mutation const previewMutation = useMutation({ mutationFn: (_generation: number) => { - const source = buildSource(); + const meta = buildImportMetaCommon(); + if (sourceMode === "local") { + if (!localPackage) throw new Error("No source configured."); + // Upload the raw compressed zip; the server unzips it into the same + // inline bundle the importer consumes. + return companiesApi.importPreviewPackage(localPackage.file, meta); + } + const source = buildGithubSource(); if (!source) throw new Error("No source configured."); - return companiesApi.importPreview({ - source, - include: { company: true, agents: true, projects: true, issues: true }, - target: - targetMode === "new" - ? { mode: "new_company", newCompanyName: newCompanyName || null } - : { mode: "existing_company", companyId: selectedCompanyId! }, - collisionStrategy, - }); + return companiesApi.importPreview({ source, ...meta }); }, onSuccess: (result, generation) => { if (generation !== previewGenerationRef.current) return; @@ -1001,24 +1011,24 @@ export function CompanyImport() { if (variables.resume) { return watchImportJob(variables.resume.jobId, variables.resume.storageKey); } - const source = buildSource(); - if (!source) throw new Error("No source configured."); + const meta = { + ...buildImportMetaCommon(), + nameOverrides: buildFinalNameOverrides(), + selectedFiles: buildSelectedFiles(), + adapterOverrides: buildFinalAdapterOverrides(), + pauseAutomations: variables.pauseAutomations, + }; + const localFile = sourceMode === "local" ? localPackage?.file : null; + const githubSource = sourceMode === "local" ? null : buildGithubSource(); + if (sourceMode === "local" ? !localFile : !githubSource) { + throw new Error("No source configured."); + } const storageKey = currentImportJobStorageKey(); let accepted: CompanyImportJobAccepted; try { - accepted = await companiesApi.importBundleAsync({ - source, - include: { company: true, agents: true, projects: true, issues: true }, - target: - targetMode === "new" - ? { mode: "new_company", newCompanyName: newCompanyName || null } - : { mode: "existing_company", companyId: selectedCompanyId! }, - collisionStrategy, - nameOverrides: buildFinalNameOverrides(), - selectedFiles: buildSelectedFiles(), - adapterOverrides: buildFinalAdapterOverrides(), - pauseAutomations: variables.pauseAutomations, - }); + accepted = localFile + ? await companiesApi.importBundlePackageAsync(localFile, meta) + : await companiesApi.importBundleAsync({ source: githubSource!, ...meta }); } catch (err) { // 409: this user's previous import is still running. Adopt that job // and watch it — never fire a second import. @@ -1352,20 +1362,10 @@ export function CompanyImport() { sourceMode === "local" ? !!localPackage : importUrl.trim().length > 0; const hasErrors = importPreview ? importPreview.errors.length > 0 : false; - // Inline imports ship the parsed package as one JSON request; oversized - // local packages are blocked before any request is attempted. - const inlinePreflight = useMemo( - () => (sourceMode === "local" && localPackage ? buildInlineImportPreflight(localPackage.files) : null), - [sourceMode, localPackage], - ); - const inlineImportBlocked = Boolean(inlinePreflight?.tooLarge); - - function handleContinueWithoutAttachments() { - if (!localPackage) return; - resetMutationState(); - setLocalPackage({ ...localPackage, files: stripBlobFiles(localPackage.files) }); - setCheckedFiles((prev) => new Set([...prev].filter((filePath) => !isBlobStoreFilePath(filePath)))); - } + // The local .zip uploads its raw compressed file as multipart and is unzipped + // server-side, so the old inline-size ceiling no longer gates it. Surface the + // compressed upload size instead of the inflated-JSON estimate. + const localCompressedBytes = sourceMode === "local" ? localPackage?.compressedBytes ?? null : null; const previewContent = selectedFile && importPreview ? (() => { @@ -1555,6 +1555,7 @@ export function CompanyImport() { {localPackage.name} with{" "} {Object.keys(localPackage.files).length} file {Object.keys(localPackage.files).length === 1 ? "" : "s"} + {localCompressedBytes !== null ? ` (${formatMegabytes(localCompressedBytes)} zip)` : ""} )} @@ -1563,20 +1564,6 @@ export function CompanyImport() { {localZipHelpText}

)} - {inlinePreflight?.tooLarge && ( -
-

- This package is about {formatMegabytes(inlinePreflight.estimatedBytes)} inline, which is - larger than the {formatMegabytes(INLINE_IMPORT_MAX_BYTES)} browser import limit. Large - packages need the CLI folder import today (paperclip company import). -

- {inlinePreflight.canDropAttachments && ( - - )} -
- )} ) : ( previewMutation.mutate(previewGenerationRef.current)} disabled={ - previewMutation.isPending || importMutation.isPending || !hasSource || inlineImportBlocked + previewMutation.isPending || importMutation.isPending || !hasSource } > {previewMutation.isPending ? "Previewing..." : "Preview import"} @@ -1672,18 +1659,13 @@ export function CompanyImport() { Import in progress — the package and settings unlock when it finishes. )} - {inlineImportBlocked && ( - - Package too large for browser import — see the notice above. - - )} {previewMutation.isPending && (

Uploading and analyzing your package - {inlinePreflight ? ` (about ${formatMegabytes(inlinePreflight.estimatedBytes)})` : ""} — large + {localCompressedBytes !== null ? ` (${formatMegabytes(localCompressedBytes)} zip)` : ""} — large packages can take a few minutes. Keep this page open.

@@ -1767,7 +1749,7 @@ export function CompanyImport() {