Commit Graph

6 Commits

Author SHA1 Message Date
Nicky Leach 64b7dce0ad
refactor(adapter-utils): replace the process-wide byte ledger with route-local byte bounds (#12465)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The adapter layer carries sandbox requests to host processes.
> - The HTTP/2 bridge used one process-wide byte ledger for all routes.
> - One busy route could exhaust that shared budget and move another
route to file transport.
> - This pull request gives each host retention site a fixed byte bound
and limits concurrent HTTP/2 streams.
> - The benefit is local protection: one route cannot consume the byte
budget of another route.

## Linked Issues or Issue Description

**What happened?**

The HTTP/2 bridge used one aggregate byte ledger for retained bytes
across all routes. A busy route could exhaust the shared budget and
force an unrelated route to use file transport.

**Expected behavior**

Each route should protect its own retained bytes. A reset on one HTTP/2
stream should cancel only that stream's host forward.

**Steps to reproduce**

1. Start the HTTP/2 bridge with multiple sandbox routes.
2. Send enough retained data through one route to reach the aggregate
byte limit.
3. Send a request through a sibling route.
4. Observe that the sibling route can fall back to file transport
because the first route used the shared ledger.

**Paperclip version or commit**

`47639e227e78e3c5e0dd1a3c0e2d792fe86895a3`

**Deployment mode**

Built from source with the adapter-utils and server test suites.

## What Changed

- Bound each host retention site with a fixed local byte limit.
- Limited concurrent live HTTP/2 streams with one built-in stream limit.
- Bound each host forward and response-body read to its own HTTP/2
stream lifetime.
- Removed the process-wide byte ledger, its environment override, its
metrics, and its file-transport fallbacks.
- Added tests for the stream limit, host body budget, and sibling-stream
cancellation.

## Verification

- Run `pnpm vitest run --project adapter-utils`.
- Confirm that 996 adapter-utils tests pass.
- Confirm that `test_live_forward_work_never_passes_the_stream_limit`
passes.
- Confirm that `test_the_host_body_budget_matches_the_stream_limit`
passes.
- Confirm that the sibling-stream cancellation test passes.
- Run `pnpm tsc --noEmit`.
- Confirm that all pull request checks pass.

## Risks

The bridge no longer uses a process-wide byte ledger. A local bound or
stream limit that is too low can reject or delay valid work. The tests
cover the new limits and stream cancellation behavior.

## Model Used

OpenAI GPT-5 Codex. Runtime model ID: GPT-5. The model used code
execution and repository tools. The runtime does not expose the context
window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-28 14:36:18 -07:00
Nicky Leach 802f2af154
refactor(adapter-utils): delete the dead duplex body-chunk protocol code (#12186)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The adapter utilities package provides transport code for sandbox
agents
> - The retired `duplex_v1` broker no longer produces or consumes
body-chunk frames
> - Dead protocol code remains in the host codec, gateway copy, bridge
options, and tests
> - This pull request removes that dead code and keeps the READY
handshake unchanged
> - The benefit is a smaller transport surface with fewer unused paths
to maintain

## Linked Issues or Issue Description

**What existing behavior does this improve?**

This change improves the adapter utilities code that supports sandbox
duplex readiness and frame handling.

**Subsystem affected**

`packages/adapter-utils/` — sandbox transport codecs, execution targets,
and callback bridge tests.

**Current behavior**

The repository keeps body-chunk frame types, validators, a body spool,
decoder limits, and tests after the `duplex_v1` broker removal. No live
producer or consumer uses this code.

**Proposed behavior**

Remove the unused body-chunk protocol code and retain the READY
handshake, its strict checks, and its size limits.

**Reason and benefit**

The removal reduces dead code and keeps the host and embedded gateway
paths easier to inspect. It adds no new behavior.

**Breaking changes**

The removed frame types now decode as `unknown_type`. The live readiness
gate already ignores those frames. The READY handshake stays
byte-for-byte compatible.

**Additional context**

This cleanup follows [PR
#12171](https://github.com/paperclipai/paperclip/pull/12171), which
removed the duplex broker.

## What Changed

- Remove `duplex-body-spool.ts` and its test.
- Remove unused body-chunk frame types, validators, decoder code,
vectors, and limits.
- Remove the unused `reassembledBody` option and decoder limit
environment entry.
- Remove the embedded gateway decoder copy and the unused frame type
map.
- Keep the READY handshake and its existing boundary tests unchanged in
behavior.

## Verification

- `pnpm -F @paperclip/adapter-utils typecheck` passes.
- The duplex frame codec test passes with 30 tests.
- The sandbox execution-target test passes with 136 tests.
- The sandbox callback bridge test passes with 46 tests.
- CI must confirm all required checks after it starts.

## Risks

Low risk. The change removes code only. The READY handshake, HTTP/2 body
path, and byte-ledger path remain unchanged.

## Model Used

OpenAI Codex, GPT-5, tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-25 14:33:38 -07:00
Nicky Leach 16b59c9315
feat(adapter-utils): stream duplex bridge bodies as sequenced chunks with receive-side spill (#12006)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent adapters use a duplex bridge to send requests and responses
across an isolated boundary
> - The bridge held each request body and response body in memory on
both ends
> - Large bodies can exhaust memory and reduce the safe size of adapter
traffic
> - This pull request sends receive-side bodies as sequenced chunks and
spills large bodies to disk
> - The benefit is bounded memory use with strict size, order, and
cleanup checks

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The adapter-utils duplex bridge transports request and response bodies
across the sandbox boundary.

**Current behavior**

The bridge stores each complete body in memory on both ends of the
duplex channel.

**Proposed behavior**

The bridge sends body chunks with sequence checks. The receive side
keeps bodies up to 1 MiB in memory and spills larger bodies to a
temporary file.

**Reason and benefit**

This change reduces memory pressure and keeps malformed or oversized
input on a terminal error path.

**Breaking changes**

The duplex frame version changes to version 2. The request and response
envelopes now carry bodyByteCount, and body_chunk frames carry the body
data.

## What Changed

- Add version 2 body_chunk frames with 256 KiB raw slices encoded as
canonical base64 text.
- Add receive-side memory and spill reassembly with per-channel disk and
file limits.
- Reject malformed, reordered, oversized, truncated, and non-canonical
body chunks.
- Stream reassembled request bodies to the host forward handler with a
web stream and half-duplex request.
- Remove spill files on success, failure, channel death, and startup
cleanup.

## Verification

- Run the adapter-utils type-check.
- Run the adapter-utils duplex test suite.
- Run all pull request checks.
- Run the Greptile review and confirm a 5/5 score with no open findings.

## Risks

The wire format changes from version 1 to version 2. Older bridge peers
cannot use this protocol. The receive path adds temporary file
operations and cleanup paths. The implementation fails closed when a
body violates size or sequence rules.

## Model Used

OpenAI GPT-5 Codex, tool-enabled coding agent. The exact context-window
size and reasoning mode are not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with Fixes: # / Closes #
/ Refs # OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
- [x] My branch name describes the change (e.g. docs/..., fix/...) and
contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-23 08:52:31 -07:00
Nicky Leach 141b815294
fix(adapter-utils): enforce the duplex frame size bound on encode in both codec copies (#11983)
## Thinking Path

> - Paperclip uses adapter utilities to move bounded messages between
agent processes.
> - The duplex frame codec encodes and decodes these messages.
> - The decoder rejects frames above the documented byte limit.
> - The encoder did not apply the same limit before it sent a frame.
> - This mismatch let a sender write a frame that the peer rejected
after transmission.
> - This pull request applies the same byte limit to both codec copies
and keeps the broker channel open.
> - The benefit is a local error with stable request telemetry instead
of a channel loss.

## Linked Issues or Issue Description

No public GitHub issue exists for this change. The problem follows the
bug report fields below.

**What happened?**

The duplex encoder could write a frame larger than
`DEFAULT_MAX_DUPLEX_FRAME_BYTES`. The peer decoder then rejected the
frame after transmission. In the WebSocket 1009 case, this closed the
channel and reported a process exit.

**Expected behavior**

The encoder should reject an oversized frame before it writes bytes. The
gateway should return HTTP 413. The broker should return a bounded
terminal response and keep other requests active.

**Steps to reproduce**

1. Encode a duplex frame above `DEFAULT_MAX_DUPLEX_FRAME_BYTES`.
2. Send the frame through the gateway or broker.
3. Observe that the old path writes the frame or drops the channel after
peer rejection.

**Paperclip version or commit**

Reproduced from the `master` development line before this change.

**Deployment mode**

Local dev (`pnpm dev`).

## What Changed

- Add `encodeDuplexFrameChecked` to the host and embedded gateway
codecs.
- Measure encoded JSON bytes without the trailing newline.
- Return a typed `frame_too_large` result without throwing.
- Return HTTP 413 for oversized gateway requests without writing a
frame.
- Share one frame bound between broker decode and encode checks.
- Return a bounded, non-retryable terminal response for oversized broker
responses.
- Add encode vectors to the shared wire-compatibility fixture.

## Verification

- Run `pnpm --filter @paperclipai/adapter-utils typecheck`.
- Run `npx vitest run
packages/adapter-utils/src/duplex-frame-codec.test.ts
packages/adapter-utils/src/duplex-bridge-broker.test.ts
packages/adapter-utils/src/execution-target-sandbox.test.ts`.
- Confirm the oversized-response broker test keeps the channel open and
serves the other in-flight request.
- Confirm the gateway test returns HTTP 413 and keeps the channel open.

## Risks

The encoder now rejects oversized frames before transmission. This
changes an unsafe write into a typed local error. The broker and gateway
keep existing frame limits and affect only oversized frames.

## Model Used

OpenAI Codex, GPT-5, tool use and code execution. The model assisted
with review and repository operations.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-22 16:32:18 -07:00
Nicky Leach 10d2781a29
feat(sandbox): add the duplex bridge broker, gated transport selection, and fixed observability (#11769)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sandbox adapters provide controlled execution for untrusted provider
environments.
> - The sandbox channel needs one persistent duplex transport with
strict host control.
> - The transport must remain off unless the instance setting and
provider capability both allow it.
> - The host must detect loss, bound resource use, and expose only safe
telemetry.
> - This pull request adds the broker, gated selection, kill-switch
wiring, fixed observability, and real-process proof.
> - The benefit is safer sandbox execution with bounded failure behavior
and inspectable transport results.

## Linked Issues or Issue Description

No public issue exists for this change. The related pull requests are
#11738 and #11750.

**Problem or motivation**

The sandbox duplex channel needs a host-controlled broker, strict
transport gates, bounded provider input, and safe loss telemetry.
Without these controls, a provider can cause replay, resource growth,
unsafe endpoint selection, or data exposure through telemetry.

**Proposed solution**

Add a host broker with nested time limits, request limits, one-shot
loss, and per-id deduplication. Select duplex transport only when the
instance setting and provider capability both equal true. Assign the
endpoint and nonce on the host. Reject invalid readiness data and use
the file bridge on failure. Add fixed redacted telemetry and a
real-process end-to-end test harness.

**Alternatives considered**

Keep the file bridge as the only transport. This avoids new channel
behavior but does not provide persistent duplex operation for supported
sandbox providers.

**Roadmap alignment**

This change supports the Cloud / Sandbox agents section in ROADMAP.md.

## What Changed

- Add the duplex bridge broker with bounded forward, response, and
gateway wait budgets.
- Bound concurrent requests, lifetime requests, and request-id bytes
before retention or forwarding.
- Select duplex transport only when both required gates are true.
- Assign the loopback port and nonce on the host and enforce a
liveness-only READY frame.
- Fall back to the file bridge after invalid readiness, contamination,
bind failure, or timeout.
- Carry the kill switch through the server, acpx engine, and six local
adapters.
- Add fixed, redacted duplex telemetry with a provider allowlist.
- Add a real-process end-to-end harness for readiness, round trips,
loss, and teardown.
- Add regression coverage for limits, loss, UTF-8 splits, concurrency,
and telemetry dimensions.

## Verification

- Adapter-utils, server, and Daytona typechecks pass locally.
- Adapter-utils tests pass, including the codec, broker,
execution-target sandbox, and real-process harness.
- Server kill-switch tests pass.
- Live Daytona tests pass with the required provider key and skip
without that key.
- The root pnpm-lock.yaml file has no diff.
- The branch contains ten commits after origin/master.

## Risks

- Duplex transport remains disabled unless both gates equal true.
- A provider remains an untrusted boundary and needs least-privilege
credentials and quotas.
- The server telemetry recorder stays deferred; the default recorder
does nothing.
- A provider that pre-binds the host port causes a fail-closed fallback
to the file bridge.
- The change adds no database migration and changes no root lockfile.

## Model Used

OpenAI GPT-5, exact model family GPT-5, large context window, reasoning,
and tool use. The model assisted with Git handoff validation and PR
preparation. The implementation commits came from the engineering
worktree.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with Fixes: # / Closes #
/ Refs # OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
- [x] My branch name describes the change (e.g. docs/... or fix/...) and
contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
- [x] I searched the GitHub PR list for similar PRs and confirmed this
is not a duplicate

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-22 09:01:31 -07:00
Nicky Leach 8161244284
feat(sandbox): add opt-in duplex command-stream foundation (capability, protocol, bounded host route, frame codec) (#11738)
## Thinking Path

> - Paperclip provides a control plane for companies that run AI agents.
> - Sandboxed agents need a safe execution path for persistent command
streams.
> - The existing callback transport does not provide a bounded, generic
duplex route.
> - The host must control capability access, route identity, protocol
limits, and close behavior.
> - This pull request adds an opt-in duplex command-stream foundation
across the sandbox layers.
> - The feature stays inert because no current provider declares the
capability.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting (multiple of the above)

**Problem or motivation**

Sandbox command execution needs a persistent host-to-sandbox stream. The
current callback bridge uses a file transport and does not provide this
generic route.

**Proposed solution**

Add a fail-closed provider capability, generic worker protocol messages,
a host-owned bounded route, cross-layer service mediation, and a
versioned newline-delimited frame codec.

**Alternatives considered**

Keep the file transport and add feature-specific commands. This does not
provide one reusable duplex contract or host-owned route bounds.

**Roadmap alignment**

This work supports the completed Cloud / Sandbox agents roadmap area and
the safe autonomy goal in the product definition.

**Additional context**

The change passed a two-stage security review. The final code review
verdict was approve after fixes for active-stream bounds and
service-layer capability mediation.

## What Changed

- Add the opt-in `duplexCommandStream` provider capability with
fail-closed narrowing.
- Add duplex open, write, stop, and close requests and data and exit
notifications to the plugin worker protocol.
- Add a host-owned route with bounds for chunk size, cumulative bytes,
lifetime, protocol errors, pending requests, and pre-bind buffering.
- Add close acknowledgement handling with worker retirement when the
close remains unconfirmed.
- Wire `openDuplexChannel` through the execution target, runtime
service, and plugin worker.
- Add a versioned frame codec with shared wire-compatibility vectors and
split UTF-8 handling.

## Verification

- `server/src/__tests__/plugin-worker-manager-duplex.test.ts` passes 18
tests.
- `server/src/__tests__/environment-execution-target-duplex.test.ts`
passes 11 tests.
- `packages/adapter-utils/src/duplex-frame-codec.test.ts` passes 38
tests.
- `server/src/__tests__/sandbox-capability-contract.test.ts` passes 15
tests.
- Setup-token pseudo-terminal regression tests pass 47 tests.
- Server TypeScript check passes.
- Continuous integration will run the full required test, typecheck,
build, and policy checks.

## Risks

- Providers that opt into the capability must implement the complete
worker protocol.
- Route limit defaults can close a stream when a workload exceeds the
configured bounds.
- The capability remains disabled for current providers, so current
production behavior does not change.

## Model Used

OpenAI GPT-5 (`gpt-5`), with tool use and code execution. The model
reviewed and prepared this pull request from the supplied implementation
and verification record.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-19 13:39:13 -07:00