Commit Graph

4 Commits

Author SHA1 Message Date
Nicky Leach 445547c989
feat(duplex): run the Daytona sandbox callback bridge over Node HTTP/2 (#12120)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox providers carry agent work through controlled execution
channels
> - The Daytona callback bridge uses a bespoke line-framed protocol over
its duplex channel
> - The bespoke protocol adds framing work and does not use the Node
transport that already supports multiplexed streams
> - This pull request carries raw bytes across the channel, adds a Node
HTTP/2 bridge, and selects it for Daytona
> - The benefit is one authenticated, multiplexed callback session with
queue_v1 as the bounded fallback

## Linked Issues or Issue Description

**Subsystem affected**

The packages/plugins Daytona provider and the shared duplex execution
path.

**Problem or motivation**

The Daytona callback bridge uses a bespoke line-framed protocol over the
provider duplex channel. This adds protocol work and limits stream
handling.

**Proposed solution**

Carry raw bytes through the cross-layer channel. Add an authenticated
Node HTTP/2 host server and sandbox client gateway. Select http2_v1 for
Daytona and retain queue_v1 as the fallback.

**Alternatives considered**

Keep the current duplex_v1 protocol. This keeps the bespoke framing path
and does not provide one HTTP/2 session for callback streams.

**Roadmap alignment**

ROADMAP.md lists Daytona under cloud and sandbox agents. This change
improves the shipped Daytona provider path.

**Additional context**

The branch adds no dependency. Node 24 provides the http2 module. The
host token check and canonical path parser remain the single dispatch
path.

## What Changed

- Carry raw Uint8Array chunks through the adapter, plugin, worker,
runtime, and Daytona layers.
- Encode bytes as base64 only across the JSON-RPC hop, because JSON has
no binary type.
- Add the bounded host HTTP/2 server and the in-sandbox HTTP/2 client
gateway.
- Authenticate every stream with the per-run bridge token before route
work.
- Parse the path once and reuse the canonical result for route and
forwarding work.
- Select http2_v1 for Daytona and fall back once to queue_v1 when the
client preface is absent.
- Add transport, session, stream, and fallback telemetry.
- Mark HTTP/2 as the preferred transport and queue_v1 as the
soft-deprecated fallback.

## Verification

- `npx vitest run packages/adapter-utils/src` — 990 passed and 4
skipped.
- `npx vitest run
server/src/__tests__/plugin-worker-manager-duplex.test.ts` — 32 passed.
- `npx vitest run --config
packages/plugins/sandbox-providers/daytona/vitest.config.ts` — 220
passed and 6 skipped.
- `npx tsc --noEmit` in `packages/adapter-utils`, `packages/shared`,
`packages/plugins/sdk`, and `server` — clean.
- No `package.json` or `pnpm-lock.yaml` file changed.
- The live Daytona test skips when `DAYTONA_API_KEY` is absent.
- The root `npx tsc --noEmit` command has a pre-existing missing
`packages/adapters/droid-local` reference on this branch and on
`master`.

## Risks

- The transport change affects several duplex layers and could expose
byte-boundary errors.
- A missing HTTP/2 client preface falls back once to queue_v1 and
records `preface_missing`.
- The host token check and canonical path parser must remain on the
shared dispatch path.
- The live Daytona test needs `DAYTONA_API_KEY` and does not run in this
agent sandbox.

## Model Used

OpenAI GPT-5, tool-enabled coding agent with repository inspection,
GitHub CLI, and shell execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-25 07:35:39 -07:00
Nicky Leach c5050396c7
fix(daytona-duplex): chunk host-to-sandbox writes and make a transport close legible (#11986)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip runs agent work through adapters and sandbox providers
> - The Daytona duplex path sends host input through a provider
pseudo-terminal WebSocket
> - Large messages exceed the provider limit, and a transport close can
look like a process exit
> - This pull request chunks UTF-8 input and carries transport-close
state through the duplex path
> - The benefit is reliable large input and accurate loss reporting

## Linked Issues or Issue Description

**What happened?**

The Daytona duplex path sent a full input payload as one WebSocket
message. A payload above the provider limit closed the channel. The wait
path also mapped a non-numeric exit result to a process exit without
exit data.

**Expected behavior**

The provider must receive large input as ordered UTF-8 chunks. A
transport close without exit data must record `transport_closed`, while
a numeric exit must record `provider_exit`.

**Steps to reproduce**

1. Start a Daytona duplex session.
2. Send an input payload larger than 65536 bytes.
3. Observe that one message closes the provider channel.
4. End a session without a numeric exit code.
5. Observe that the loss reason reports a process exit.

**Paperclip version or commit**

Commit `1761e79ec9097c65d94f90a8ba20416f8ab718a6`.

**Deployment mode**

Built from source with the Daytona sandbox provider.

## What Changed

- Add a shared UTF-8 byte chunker with a 32768-byte cap.
- Route both Daytona pseudo-terminal write paths through the chunker.
- Preserve multi-byte UTF-8 sequences across read-side chunks.
- Carry an explicit `transportClosed` state through the worker and host
wait paths.
- Record `transport_closed` for a reason-less transport close and
`provider_exit` for a numeric exit.
- Keep orderly completion suppression for both exit paths.

## Verification

- The Daytona plugin suite passes 194 tests.
- The adapter-utils broker, codec, and telemetry suites pass 73 tests.
- The plugin SDK duplex and worker RPC host suites pass 37 tests.
- The server plugin worker manager duplex suite passes 78 tests.
- The execution target sandbox and ACPX execute suites pass 257 tests.
- TypeScript checks pass for adapter-utils, plugin SDK, server, and the
standalone Daytona plugin.

## Risks

The chunk size adds a loop for large input payloads. The 32768-byte cap
stays below the provider limit. The optional loss field preserves
compatibility for other providers.

## Model Used

OpenAI Codex, GPT-5, extended reasoning, tool use, and code execution.
The runtime does not expose a separate context-window value.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-22 16:35:47 -07:00
Nicky Leach 10d2781a29
feat(sandbox): add the duplex bridge broker, gated transport selection, and fixed observability (#11769)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sandbox adapters provide controlled execution for untrusted provider
environments.
> - The sandbox channel needs one persistent duplex transport with
strict host control.
> - The transport must remain off unless the instance setting and
provider capability both allow it.
> - The host must detect loss, bound resource use, and expose only safe
telemetry.
> - This pull request adds the broker, gated selection, kill-switch
wiring, fixed observability, and real-process proof.
> - The benefit is safer sandbox execution with bounded failure behavior
and inspectable transport results.

## Linked Issues or Issue Description

No public issue exists for this change. The related pull requests are
#11738 and #11750.

**Problem or motivation**

The sandbox duplex channel needs a host-controlled broker, strict
transport gates, bounded provider input, and safe loss telemetry.
Without these controls, a provider can cause replay, resource growth,
unsafe endpoint selection, or data exposure through telemetry.

**Proposed solution**

Add a host broker with nested time limits, request limits, one-shot
loss, and per-id deduplication. Select duplex transport only when the
instance setting and provider capability both equal true. Assign the
endpoint and nonce on the host. Reject invalid readiness data and use
the file bridge on failure. Add fixed redacted telemetry and a
real-process end-to-end test harness.

**Alternatives considered**

Keep the file bridge as the only transport. This avoids new channel
behavior but does not provide persistent duplex operation for supported
sandbox providers.

**Roadmap alignment**

This change supports the Cloud / Sandbox agents section in ROADMAP.md.

## What Changed

- Add the duplex bridge broker with bounded forward, response, and
gateway wait budgets.
- Bound concurrent requests, lifetime requests, and request-id bytes
before retention or forwarding.
- Select duplex transport only when both required gates are true.
- Assign the loopback port and nonce on the host and enforce a
liveness-only READY frame.
- Fall back to the file bridge after invalid readiness, contamination,
bind failure, or timeout.
- Carry the kill switch through the server, acpx engine, and six local
adapters.
- Add fixed, redacted duplex telemetry with a provider allowlist.
- Add a real-process end-to-end harness for readiness, round trips,
loss, and teardown.
- Add regression coverage for limits, loss, UTF-8 splits, concurrency,
and telemetry dimensions.

## Verification

- Adapter-utils, server, and Daytona typechecks pass locally.
- Adapter-utils tests pass, including the codec, broker,
execution-target sandbox, and real-process harness.
- Server kill-switch tests pass.
- Live Daytona tests pass with the required provider key and skip
without that key.
- The root pnpm-lock.yaml file has no diff.
- The branch contains ten commits after origin/master.

## Risks

- Duplex transport remains disabled unless both gates equal true.
- A provider remains an untrusted boundary and needs least-privilege
credentials and quotas.
- The server telemetry recorder stays deferred; the default recorder
does nothing.
- A provider that pre-binds the host port causes a fail-closed fallback
to the file bridge.
- The change adds no database migration and changes no root lockfile.

## Model Used

OpenAI GPT-5, exact model family GPT-5, large context window, reasoning,
and tool use. The model assisted with Git handoff validation and PR
preparation. The implementation commits came from the engineering
worktree.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with Fixes: # / Closes #
/ Refs # OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub #NNN / github.com/paperclipai/paperclip URLs)
- [x] My branch name describes the change (e.g. docs/... or fix/...) and
contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
- [x] I searched the GitHub PR list for similar PRs and confirmed this
is not a duplicate

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-22 09:01:31 -07:00
Nicky Leach faab2620ad
feat(sandbox): add a duplex transport for Daytona behind a default-off kill switch (#11750)
## Thinking Path

> - Paperclip is an open source app that manages AI agents for work
> - Paperclip runs agents in local and remote sandbox environments
> - A sandbox needs a bounded channel for commands and asynchronous
input
> - Daytona needs a real pseudo-terminal transport for this channel
> - The sandbox gateway also needs a mode that handles channel loss
safely
> - This pull request adds the Daytona transport and gateway mode behind
a default-off kill switch
> - The benefit is a tested foundation for later transport selection

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting (multiple of the above): sandbox providers, plugin SDK,
server settings, and shared types.

**Problem or motivation**

The merged sandbox protocol has no runtime transport for Daytona. The
generated sandbox gateway also has no duplex mode. A later
transport-selection change needs both parts and a safe per-run gate.

**Proposed solution**

Add a Daytona `duplexCommandStream` transport over a raw
pseudo-terminal. Add a generated gateway mode named `duplex_v1`. Add the
`enableSandboxDuplexBridge` setting with a default value of `false`.
Keep transport selection disabled until a later pull request.

**Alternatives considered**

Keep the protocol unused until the transport-selection change. This
would delay provider tests and leave the gateway path without direct
coverage.

**Roadmap alignment**

This change supports the completed Roadmap item for cloud and sandbox
agents. It extends the merged sandbox channel foundation in pull request
#11738.

**Additional context**

The Daytona provider remains an untrusted boundary. Deployments must use
least-privilege provider credentials and provider-side quota controls.
Operators must name an owner for duplex telemetry retention before
rollout.

## What Changed

- Add the Daytona `duplexCommandStream` capability over a raw
pseudo-terminal.
- Add a launch wrapper that disables echo and newline translation for
NDJSON frames.
- Close channels on lease release, destroy, resume of a stopped worker,
and worker shutdown.
- Declare the capability in the Daytona manifest and set
`PLUGIN_VERSION` to `0.1.5`.
- Add the worker-to-host notification sink at `ctx.duplexChannel.data`
and `ctx.duplexChannel.exit`.
- Add the generated sandbox gateway mode
`PAPERCLIP_API_BRIDGE_MODE=duplex_v1`.
- Add channel-loss results of `409 outcome_indeterminate` and `503
bridge_unavailable`.
- Add the per-run setting `enableSandboxDuplexBridge`, with a default
value of `false`.
- Add unit tests, generated-source codec tests, lifecycle tests, and a
credential-gated live Daytona test.

## Verification

- Daytona suite: 185 tests pass.
- Adapter utilities: 754 tests pass and 4 tests skip.
- Plugin SDK: 62 tests pass.
- Shared package: 28 tests pass.
- Server duplex tests pass.
- Shared, plugin SDK, server, and Daytona TypeScript checks pass.
- The live Daytona test passes 3 cases when `DAYTONA_API_KEY` is set.
- The live Daytona test skips 3 cases without `DAYTONA_API_KEY`.
- CI must run the full workspace typecheck, test, and build gates after
PR creation.

## Risks

- The Daytona control plane and pseudo-terminal remain untrusted
boundaries.
- The duplex gateway changes behavior only when the mode and per-run
setting enable it.
- A lost channel fails requests without replay, so callers must handle
indeterminate outcomes.
- The transport-selection change must require both `duplexCommandStream
=== true` and `enableSandboxDuplexBridge === true`.
- The provider credential and quota limits need operator control before
rollout.

## Model Used

OpenAI Codex, GPT-5, tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-19 17:14:47 -07:00