feat(runner): add secure remote transport (#12639)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner gives native runs a durable and governed execution path. > - The lower stack PR adds authenticated remote execution targets and provider ingress. > - The Rust daemon currently accepts only loopback plaintext WebSocket connections. > - Remote Codex needs authenticated WSS dialing and provider-ingress listener mode. > - This pull request adds the bounded Rust transport contract. > - The benefit is a secure transport layer for the Codex remote vertical slice. ## Linked Issues or Issue Description Refs #12638. Refs #12616. Refs #12352. **Subsystem affected** Paperclip Runner Rust transport and remote runner networking. **Problem or motivation** The runner daemon cannot connect to a public control plane with TLS. It also cannot accept a provider preview connection on the run-bound ingress path. **Proposed solution** Add WSS with native trust roots and an optional private CA bundle. Add a fixed authenticated listener mode for provider ingress. Advertise the exact transport contract through build metadata. **Alternatives considered** Plaintext public WebSocket connections would weaken the transport boundary. A general listener would expose more network surface than the run-bound provider ingress requires. **Roadmap alignment** This work supports the Cloud and Sandbox agents milestone. It also supports self-healing native runs. ## Stack - Lower merged PR: #12638. - This PR contains only its 13-file delta against `master`. - Later stack PRs add the task workspace and administrator UI. ## What Changed - Added WSS dialing with rustls and native certificate roots. - Added an optional bounded private CA bundle that augments native roots. - Kept plaintext WebSocket dialing restricted to loopback addresses. - Pinned resolved dial addresses for the process lifetime. - Added a fixed `0.0.0.0:43127` listener with an exact run-bound path. - Rejected listener queries, ambiguous paths, and WebSocket extensions. - Kept frame and message size bounds. - Added bounded reconnect grace and exponential jitter. - Retried bootstrap failures only before authentication proof transmission begins. - Kept post-proof failures fail-closed and bounded the welcome exchange at two seconds. - Added runnerd build metadata for the versioned transport contract. - Updated Rust dependencies and `Cargo.lock` only for TLS and certificate handling. - Did not add provider dispatch, Pi, AWS, `pnpm-lock.yaml`, migrations, or workflows. ## Verification - GitHub Actions will run Cargo formatting, Rust tests, repository tests, typecheck, build, security, and policy gates. - Rust tests cover URL validation, listener path validation, build metadata, durable recovery, and the existing Codex provider path. - Local tests were not run. The requested verification policy uses GitHub Actions for this series. - `git diff --check master...HEAD` passes. - The delta contains 13 files. ## Risks - TLS and listener changes affect the runner trust boundary. - Public plaintext transport remains rejected. - The listener uses one fixed port and one exact run-bound path. - PRP authentication remains required after the WebSocket upgrade. - The optional CA file uses the existing private-file checks and a 4 MiB limit. - This PR does not enable another provider or change direct adapters. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex with GPT-5.6. The work used high-reasoning agent mode, repository tools, GitHub tools, and parallel code-audit agents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with Fixes: / Closes / Refs OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta
GitHub
parent
0a422fda52
commit
0bdbf61564
|
|
@ -20,6 +20,8 @@ The PRP identity binds company, issue, agent, run, environment lease, runner ins
|
|||
- Allow the sandbox provider's private preview proxy to reach runnerd on TCP 43127. Do not expose the port with a public sandbox or signed URL.
|
||||
- Allow the sandbox outbound access only to destinations explicitly required by the provider runtime.
|
||||
- Use Daytona's `wss://` preview URL with normal certificate and hostname validation. The preview token is sent only as the `X-Daytona-Preview-Token` header by Paperclip.
|
||||
- Runnerd rejects public plaintext `ws://` destinations. Public dial targets require `wss://` with hostname validation and either the platform trust store or an explicitly staged private CA bundle.
|
||||
- Listener mode binds only `0.0.0.0:43127`, accepts only the run-specific `/api/runner/v1/connect/:runId` path, and rejects WebSocket extension negotiation. PRP authentication and application-layer secure frames remain mandatory above the preview-proxy hop.
|
||||
- Do not put the bootstrap ticket in argv, files, provider environment, logs, or model context. Inject it into runnerd's initial environment/secret channel; runnerd already removes it from its environment immediately.
|
||||
- Runnerd is the only process allowed to reach PRP. The provider communicates with runnerd over inherited pipes.
|
||||
|
||||
|
|
|
|||
|
|
@ -187,8 +187,9 @@ These envelopes are local Local runner implementation contracts.
|
|||
|
||||
## Durable wire rules
|
||||
|
||||
- The runner opens an outbound WebSocket and sends PRP v1 `hello` before any
|
||||
command result or event.
|
||||
- The runner opens loopback `ws://` or hostname-verified `wss://`, or accepts a
|
||||
preview-proxy connection on its fixed listener, and completes the PRP v1
|
||||
authenticated handshake before any command result or event.
|
||||
- A one-use bootstrap bearer capability returns a short-lived connection lease
|
||||
in `welcome`. Later connections use that lease. Neither raw capability is
|
||||
durable state.
|
||||
|
|
@ -205,8 +206,16 @@ These envelopes are local Local runner implementation contracts.
|
|||
- Frames are bounded at 1 MiB and upgrade headers at 16 KiB. Unknown or invalid
|
||||
required protocol data fails closed; malformed JSON is a bounded diagnostic.
|
||||
|
||||
These are package-local Durable recovery rules. Production TLS, control-plane admission,
|
||||
and deployment policy remain separately reviewed work.
|
||||
Runnerd build-metadata contract v2 advertises the exact transport inventory:
|
||||
`dial_ws_loopback`, `dial_wss`, and `listen_ws`. Plaintext dial destinations
|
||||
must resolve entirely to loopback. Public dial targets require TLS trust and
|
||||
hostname validation; a private CA bundle augments the platform roots and must
|
||||
be a bounded, private, regular file. Listener mode is fixed to port 43127 and a
|
||||
single run-bound path. All modes retain the same message/frame bounds and PRP
|
||||
authentication.
|
||||
|
||||
These are package-local Durable recovery and transport rules. Control-plane
|
||||
admission and deployment policy remain separately reviewed work.
|
||||
|
||||
## Change policy
|
||||
|
||||
|
|
|
|||
|
|
@ -126,6 +126,16 @@ version = "1.12.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
|
||||
|
||||
[[package]]
|
||||
name = "cc"
|
||||
version = "1.4.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0ad534f4357a5264cce5019c989cf66a4f0dc4e0d1b1d15f8aacec0ff7360273"
|
||||
dependencies = [
|
||||
"find-msvc-tools",
|
||||
"shlex",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfg-if"
|
||||
version = "1.0.4"
|
||||
|
|
@ -142,6 +152,22 @@ dependencies = [
|
|||
"inout",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "core-foundation"
|
||||
version = "0.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
|
||||
dependencies = [
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "core-foundation-sys"
|
||||
version = "0.8.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.2.17"
|
||||
|
|
@ -225,6 +251,12 @@ dependencies = [
|
|||
"regex-syntax",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d45db016d36b838f563236e9193d0ee6ce38f3f68b6c94e914b4929c96bbb890"
|
||||
|
||||
[[package]]
|
||||
name = "fluent-uri"
|
||||
version = "0.4.1"
|
||||
|
|
@ -649,6 +681,12 @@ version = "0.3.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381"
|
||||
|
||||
[[package]]
|
||||
name = "openssl-probe"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
|
||||
|
||||
[[package]]
|
||||
name = "outref"
|
||||
version = "0.5.2"
|
||||
|
|
@ -665,6 +703,9 @@ dependencies = [
|
|||
"jsonschema",
|
||||
"num-bigint",
|
||||
"num-traits",
|
||||
"rustls",
|
||||
"rustls-native-certs",
|
||||
"rustls-pemfile",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
|
|
@ -867,18 +908,119 @@ version = "0.8.11"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
|
||||
|
||||
[[package]]
|
||||
name = "ring"
|
||||
version = "0.17.14"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"cfg-if",
|
||||
"getrandom 0.2.17",
|
||||
"libc",
|
||||
"untrusted",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls"
|
||||
version = "0.23.43"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06"
|
||||
dependencies = [
|
||||
"once_cell",
|
||||
"ring",
|
||||
"rustls-pki-types",
|
||||
"rustls-webpki",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-native-certs"
|
||||
version = "0.8.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d"
|
||||
dependencies = [
|
||||
"openssl-probe",
|
||||
"rustls-pki-types",
|
||||
"schannel",
|
||||
"security-framework",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-pemfile"
|
||||
version = "2.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50"
|
||||
dependencies = [
|
||||
"rustls-pki-types",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-pki-types"
|
||||
version = "1.15.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
|
||||
dependencies = [
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-webpki"
|
||||
version = "0.103.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2"
|
||||
dependencies = [
|
||||
"ring",
|
||||
"rustls-pki-types",
|
||||
"untrusted",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustversion"
|
||||
version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
|
||||
|
||||
[[package]]
|
||||
name = "schannel"
|
||||
version = "0.1.29"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
|
||||
dependencies = [
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "scopeguard"
|
||||
version = "1.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
|
||||
|
||||
[[package]]
|
||||
name = "security-framework"
|
||||
version = "3.7.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"core-foundation",
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
"security-framework-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "security-framework-sys"
|
||||
version = "2.17.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
|
||||
dependencies = [
|
||||
"core-foundation-sys",
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.229"
|
||||
|
|
@ -944,6 +1086,12 @@ dependencies = [
|
|||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "shlex"
|
||||
version = "2.0.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
|
||||
|
||||
[[package]]
|
||||
name = "smallvec"
|
||||
version = "1.15.2"
|
||||
|
|
@ -1058,6 +1206,9 @@ dependencies = [
|
|||
"httparse",
|
||||
"log",
|
||||
"rand",
|
||||
"rustls",
|
||||
"rustls-native-certs",
|
||||
"rustls-pki-types",
|
||||
"sha1",
|
||||
"thiserror",
|
||||
"utf-8",
|
||||
|
|
@ -1091,6 +1242,12 @@ dependencies = [
|
|||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "untrusted"
|
||||
version = "0.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
|
||||
|
||||
[[package]]
|
||||
name = "utf-8"
|
||||
version = "0.7.6"
|
||||
|
|
@ -1191,6 +1348,88 @@ version = "0.2.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
||||
|
||||
[[package]]
|
||||
name = "windows-sys"
|
||||
version = "0.52.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
|
||||
dependencies = [
|
||||
"windows-targets",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-sys"
|
||||
version = "0.61.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
|
||||
dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-targets"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
|
||||
dependencies = [
|
||||
"windows_aarch64_gnullvm",
|
||||
"windows_aarch64_msvc",
|
||||
"windows_i686_gnu",
|
||||
"windows_i686_gnullvm",
|
||||
"windows_i686_msvc",
|
||||
"windows_x86_64_gnu",
|
||||
"windows_x86_64_gnullvm",
|
||||
"windows_x86_64_msvc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows_aarch64_gnullvm"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
|
||||
|
||||
[[package]]
|
||||
name = "windows_aarch64_msvc"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
|
||||
|
||||
[[package]]
|
||||
name = "windows_i686_gnu"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
|
||||
|
||||
[[package]]
|
||||
name = "windows_i686_gnullvm"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
|
||||
|
||||
[[package]]
|
||||
name = "windows_i686_msvc"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
|
||||
|
||||
[[package]]
|
||||
name = "windows_x86_64_gnu"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
|
||||
|
||||
[[package]]
|
||||
name = "windows_x86_64_gnullvm"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
|
||||
|
||||
[[package]]
|
||||
name = "windows_x86_64_msvc"
|
||||
version = "0.52.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
|
||||
|
||||
[[package]]
|
||||
name = "wit-bindgen"
|
||||
version = "0.57.1"
|
||||
|
|
@ -1267,6 +1506,12 @@ dependencies = [
|
|||
"synstructure",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zeroize"
|
||||
version = "1.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
|
||||
|
||||
[[package]]
|
||||
name = "zerotrie"
|
||||
version = "0.2.5"
|
||||
|
|
|
|||
|
|
@ -18,4 +18,7 @@ num-traits = "0.2"
|
|||
serde = { version = "1.0", features = ["derive"] }
|
||||
serde_json = "1.0"
|
||||
sha2 = "0.10"
|
||||
tungstenite = { version = "0.28", default-features = false, features = ["handshake"] }
|
||||
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12"] }
|
||||
rustls-native-certs = "0.8"
|
||||
rustls-pemfile = "2.2"
|
||||
tungstenite = { version = "0.28", default-features = false, features = ["handshake", "rustls-tls-native-roots"] }
|
||||
|
|
|
|||
|
|
@ -15,6 +15,9 @@ num-traits.workspace = true
|
|||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
sha2.workspace = true
|
||||
rustls.workspace = true
|
||||
rustls-native-certs.workspace = true
|
||||
rustls-pemfile.workspace = true
|
||||
tungstenite.workspace = true
|
||||
|
||||
[[bin]]
|
||||
|
|
|
|||
|
|
@ -7,6 +7,27 @@ use paperclip_runner_core::durable::{
|
|||
};
|
||||
use paperclip_runner_core::local_runner::{run_local_runner, LocalRunnerError, RunnerConfig};
|
||||
use paperclip_runner_core::provider_backend::CodexCommandExecutor;
|
||||
use serde_json::json;
|
||||
|
||||
const RUNNERD_BUILD_METADATA_SCHEMA: &str = "paperclip-runner/runnerd-build-metadata/v1";
|
||||
|
||||
fn build_metadata() -> serde_json::Value {
|
||||
json!({
|
||||
"schema": RUNNERD_BUILD_METADATA_SCHEMA,
|
||||
"binaryName": "paperclip-runnerd",
|
||||
"packageName": "@paperclipai/paperclip-runner",
|
||||
"packageVersion": env!("CARGO_PKG_VERSION"),
|
||||
"binaryContractVersion": 2,
|
||||
"nativeExecutionVersion": 1,
|
||||
"harnessDriverVersion": 1,
|
||||
"prp": {
|
||||
"name": "paperclip.runner",
|
||||
"minimumVersion": 1,
|
||||
"maximumVersion": 1
|
||||
},
|
||||
"prpTransportModes": ["dial_ws_loopback", "dial_wss", "listen_ws"]
|
||||
})
|
||||
}
|
||||
|
||||
fn value(args: &[String], name: &str) -> Result<String, LocalRunnerError> {
|
||||
let index = args
|
||||
|
|
@ -50,12 +71,52 @@ fn run_durable(args: &[String]) -> Result<(), LocalRunnerError> {
|
|||
optional_u64(args, name).map(|value| Duration::from_millis(value.unwrap_or(default)))
|
||||
};
|
||||
let state_dir = PathBuf::from(value(args, "--state-dir")?);
|
||||
let run_id = value(args, "--run-id")?;
|
||||
let has_connect = args.iter().any(|argument| argument == "--connect-url");
|
||||
let has_listener = ["--listen-address", "--listen-port", "--listen-path"]
|
||||
.iter()
|
||||
.any(|name| args.iter().any(|argument| argument == name));
|
||||
let connect_url = match (has_connect, has_listener) {
|
||||
(true, false) => value(args, "--connect-url")?,
|
||||
(false, true) => {
|
||||
let address = value(args, "--listen-address")?;
|
||||
let port = value(args, "--listen-port")?;
|
||||
let path = value(args, "--listen-path")?;
|
||||
if address != "0.0.0.0" || port != "43127" {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"runner listener requires --listen-address 0.0.0.0 and --listen-port 43127",
|
||||
));
|
||||
}
|
||||
if path != format!("/api/runner/v1/connect/{run_id}") {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"runner listener path must exactly match the configured run",
|
||||
));
|
||||
}
|
||||
format!("listen://{address}:{port}{path}")
|
||||
}
|
||||
_ => {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"durable runner requires exactly one connect URL or complete listener group",
|
||||
))
|
||||
}
|
||||
};
|
||||
let ca_bundle_path = args
|
||||
.iter()
|
||||
.any(|argument| argument == "--ca-bundle-path")
|
||||
.then(|| value(args, "--ca-bundle-path").map(PathBuf::from))
|
||||
.transpose()?;
|
||||
if ca_bundle_path.is_some() && !connect_url.starts_with("wss://") {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"--ca-bundle-path is accepted only with wss://",
|
||||
));
|
||||
}
|
||||
let config = DurableRunnerConfig {
|
||||
connect_url: value(args, "--connect-url")?,
|
||||
connect_url,
|
||||
ca_bundle_path,
|
||||
state_dir: state_dir.clone(),
|
||||
runner_instance_id: value(args, "--runner-id")?,
|
||||
environment_lease_id: value(args, "--environment-lease-id")?,
|
||||
run_id: value(args, "--run-id")?,
|
||||
run_id,
|
||||
normalized_session_id: value(args, "--session-id")?,
|
||||
turn_id: value(args, "--turn-id")?,
|
||||
item_id: value(args, "--item-id")?,
|
||||
|
|
@ -65,6 +126,7 @@ fn run_durable(args: &[String]) -> Result<(), LocalRunnerError> {
|
|||
p0_reserve_bytes: usize_value(args, "--p0-reserve-bytes", 1024 * 1024)?,
|
||||
max_frame_bytes: usize_value(args, "--max-frame-bytes", 1024 * 1024)?,
|
||||
reconnect_delay: duration("--reconnect-delay-ms", 250)?,
|
||||
reconnect_grace: optional_u64(args, "--reconnect-grace-ms")?.map(Duration::from_millis),
|
||||
max_runtime: duration("--max-runtime-ms", 60 * 60 * 1000)?,
|
||||
};
|
||||
let executor = CodexCommandExecutor::with_runner_config(state_dir, &config);
|
||||
|
|
@ -74,7 +136,15 @@ fn run_durable(args: &[String]) -> Result<(), LocalRunnerError> {
|
|||
|
||||
fn run() -> Result<(), LocalRunnerError> {
|
||||
let args = std::env::args().skip(1).collect::<Vec<_>>();
|
||||
if args.iter().any(|argument| argument == "--connect-url") {
|
||||
if args.as_slice() == ["--build-metadata"] {
|
||||
println!("{}", build_metadata());
|
||||
return Ok(());
|
||||
}
|
||||
if args.iter().any(|argument| argument == "--connect-url")
|
||||
|| args.iter().any(|argument| argument == "--listen-address")
|
||||
|| args.iter().any(|argument| argument == "--listen-port")
|
||||
|| args.iter().any(|argument| argument == "--listen-path")
|
||||
{
|
||||
return run_durable(&args);
|
||||
}
|
||||
run_local_runner(RunnerConfig {
|
||||
|
|
@ -95,6 +165,22 @@ fn run() -> Result<(), LocalRunnerError> {
|
|||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn build_metadata_advertises_the_remote_transport_contract() {
|
||||
let metadata = build_metadata();
|
||||
assert_eq!(metadata["schema"], RUNNERD_BUILD_METADATA_SCHEMA);
|
||||
assert_eq!(metadata["binaryContractVersion"], 2);
|
||||
assert_eq!(
|
||||
metadata["prpTransportModes"],
|
||||
json!(["dial_ws_loopback", "dial_wss", "listen_ws"])
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn main() -> ExitCode {
|
||||
match run() {
|
||||
Ok(()) => ExitCode::SUCCESS,
|
||||
|
|
|
|||
|
|
@ -98,6 +98,7 @@ pub fn capture_bootstrap_ticket() -> Result<Option<BootstrapTicket>, DurableRunn
|
|||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct DurableRunnerConfig {
|
||||
pub connect_url: String,
|
||||
pub ca_bundle_path: Option<PathBuf>,
|
||||
pub state_dir: PathBuf,
|
||||
pub runner_instance_id: String,
|
||||
pub environment_lease_id: String,
|
||||
|
|
@ -111,6 +112,7 @@ pub struct DurableRunnerConfig {
|
|||
pub p0_reserve_bytes: usize,
|
||||
pub max_frame_bytes: usize,
|
||||
pub reconnect_delay: Duration,
|
||||
pub reconnect_grace: Option<Duration>,
|
||||
pub max_runtime: Duration,
|
||||
}
|
||||
|
||||
|
|
@ -172,6 +174,11 @@ impl DurableRunnerConfig {
|
|||
"reconnect delay must be between one millisecond and 60 seconds",
|
||||
));
|
||||
}
|
||||
if self.reconnect_grace.is_some_and(|grace| grace.is_zero()) {
|
||||
return Err(DurableRunnerError::invalid(
|
||||
"reconnect grace must be non-zero when configured",
|
||||
));
|
||||
}
|
||||
if self.max_runtime > Duration::from_secs(7 * 24 * 60 * 60) {
|
||||
return Err(DurableRunnerError::invalid(
|
||||
"durable runner max runtime must not exceed seven days",
|
||||
|
|
@ -188,6 +195,7 @@ mod tests {
|
|||
fn config() -> DurableRunnerConfig {
|
||||
DurableRunnerConfig {
|
||||
connect_url: "ws://127.0.0.1/runner".to_owned(),
|
||||
ca_bundle_path: None,
|
||||
state_dir: PathBuf::from("state"),
|
||||
runner_instance_id: "runner-1".to_owned(),
|
||||
environment_lease_id: "lease-1".to_owned(),
|
||||
|
|
@ -201,6 +209,7 @@ mod tests {
|
|||
p0_reserve_bytes: 64 * 1024,
|
||||
max_frame_bytes: 64 * 1024,
|
||||
reconnect_delay: Duration::from_millis(1),
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(60),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
use std::thread;
|
||||
use std::time::Instant;
|
||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::{json, Value};
|
||||
|
|
@ -10,7 +10,7 @@ use super::state::{
|
|||
};
|
||||
use super::transport::{
|
||||
current_unix_ms, validate_control_identity, AuthenticatedTransport, ConnectionMetadata,
|
||||
LeaseCredential, ResolvedWsTarget,
|
||||
LeaseCredential, RunnerTransportEndpoint,
|
||||
};
|
||||
use super::{BootstrapTicket, DurableRunnerConfig, DurableRunnerError, PROTOCOL, PROTOCOL_VERSION};
|
||||
|
||||
|
|
@ -45,6 +45,49 @@ enum CommandLifecycle {
|
|||
Shutdown,
|
||||
}
|
||||
|
||||
fn sleep_for_reconnect(base: Duration, max_delay: Duration, attempt: &mut u32) {
|
||||
let multiplier = 1_u128 << (*attempt).min(5);
|
||||
let uncapped = base.as_millis().saturating_mul(multiplier);
|
||||
let capped = uncapped.clamp(1, 5_000) as u64;
|
||||
let nanos = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map_or(0, |duration| u64::from(duration.subsec_nanos()));
|
||||
let jitter_percent = 75 + nanos % 51;
|
||||
*attempt = attempt.saturating_add(1);
|
||||
let delay = Duration::from_millis(capped.saturating_mul(jitter_percent) / 100).min(max_delay);
|
||||
if !delay.is_zero() {
|
||||
thread::sleep(delay);
|
||||
}
|
||||
}
|
||||
|
||||
fn sleep_before_deadline(delay: Duration, deadline: Instant) {
|
||||
let bounded_delay = delay.min(deadline.saturating_duration_since(Instant::now()));
|
||||
if !bounded_delay.is_zero() {
|
||||
thread::sleep(bounded_delay);
|
||||
}
|
||||
}
|
||||
|
||||
fn connection_attempt_deadline(
|
||||
config: &DurableRunnerConfig,
|
||||
started: Instant,
|
||||
disconnected_since: Option<Instant>,
|
||||
) -> Instant {
|
||||
let now = Instant::now();
|
||||
let runtime_remaining = config
|
||||
.max_runtime
|
||||
.saturating_sub(now.saturating_duration_since(started));
|
||||
let remaining = disconnected_since.zip(config.reconnect_grace).map_or(
|
||||
runtime_remaining,
|
||||
|(disconnected_at, grace)| {
|
||||
runtime_remaining
|
||||
.min(grace.saturating_sub(now.saturating_duration_since(disconnected_at)))
|
||||
},
|
||||
);
|
||||
// Validation caps max_runtime at seven days, and reconnect grace can only
|
||||
// shorten this budget, so adding it to a current Instant cannot overflow.
|
||||
now + remaining
|
||||
}
|
||||
|
||||
impl CommandLifecycle {
|
||||
fn for_completed(command: &Command) -> Self {
|
||||
match command.command_type.as_str() {
|
||||
|
|
@ -112,14 +155,36 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
)?;
|
||||
store.save(&state)?;
|
||||
}
|
||||
// Resolve once. Every reconnect uses the same validated concrete addresses,
|
||||
// so DNS cannot redirect a retry after the trust decision.
|
||||
let target = ResolvedWsTarget::resolve(&config.connect_url)?;
|
||||
// Bind listener mode or resolve dial mode before processing commands. Dial
|
||||
// reconnects retain the same validated addresses so DNS cannot redirect a
|
||||
// retry after the trust decision.
|
||||
let endpoint = RunnerTransportEndpoint::new(&config.connect_url, &config.run_id)?;
|
||||
let started = Instant::now();
|
||||
let mut bootstrap_ticket = Some(bootstrap_ticket);
|
||||
let mut lease: Option<LeaseCredential> = None;
|
||||
let mut authenticated_once = false;
|
||||
let mut disconnected_since: Option<Instant> = None;
|
||||
let mut reconnect_attempt = 0_u32;
|
||||
|
||||
loop {
|
||||
if authenticated_once {
|
||||
let disconnected_at = disconnected_since.get_or_insert_with(Instant::now);
|
||||
if config
|
||||
.reconnect_grace
|
||||
.is_some_and(|grace| disconnected_at.elapsed() >= grace)
|
||||
{
|
||||
let _ = executor.shutdown();
|
||||
state.lifecycle = "recoverable_failure".to_owned();
|
||||
state.recoverable_failure = Some("transport_reconnect_grace_exceeded".to_owned());
|
||||
state.record_diagnostic(
|
||||
"transport reconnect grace exceeded; durable state is preserved",
|
||||
);
|
||||
store.save(&state)?;
|
||||
return Err(DurableRunnerError::invalid(
|
||||
"transport reconnect grace exceeded; durable state is preserved",
|
||||
));
|
||||
}
|
||||
}
|
||||
if started.elapsed() >= config.max_runtime {
|
||||
let _ = executor.shutdown();
|
||||
state.lifecycle = "recoverable_failure".to_owned();
|
||||
|
|
@ -146,27 +211,51 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
}
|
||||
|
||||
let using_bootstrap = lease.is_none();
|
||||
let connect_deadline = connection_attempt_deadline(&config, started, disconnected_since);
|
||||
let connection = AuthenticatedTransport::connect(
|
||||
&target,
|
||||
&endpoint,
|
||||
&config,
|
||||
&state,
|
||||
bootstrap_ticket.as_ref(),
|
||||
lease.as_ref(),
|
||||
connect_deadline,
|
||||
);
|
||||
let (mut transport, welcome) = match connection {
|
||||
Ok(connection) => connection,
|
||||
Ok(Some(connection)) => connection,
|
||||
Ok(None) => {
|
||||
sleep_before_deadline(config.reconnect_delay, connect_deadline);
|
||||
continue;
|
||||
}
|
||||
Err(error) => {
|
||||
state.record_diagnostic(format!("transport reconnect scheduled: {error}"));
|
||||
store.save(&state)?;
|
||||
if Instant::now() >= connect_deadline {
|
||||
// Re-enter the lifecycle checks immediately so an auth
|
||||
// timeout cannot be misreported as a reusable-bootstrap
|
||||
// failure or delayed by reconnect backoff.
|
||||
continue;
|
||||
}
|
||||
if using_bootstrap && error.bootstrap_maybe_consumed {
|
||||
return Err(DurableRunnerError::invalid(
|
||||
"bootstrap connection failed closed; provide a fresh one-use ticket",
|
||||
));
|
||||
}
|
||||
thread::sleep(config.reconnect_delay);
|
||||
sleep_for_reconnect(
|
||||
config.reconnect_delay,
|
||||
connect_deadline.saturating_duration_since(Instant::now()),
|
||||
&mut reconnect_attempt,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if Instant::now() >= connect_deadline {
|
||||
// A transport that authenticated after its lifecycle deadline is
|
||||
// never allowed to clear reconnect state or process commands.
|
||||
continue;
|
||||
}
|
||||
authenticated_once = true;
|
||||
disconnected_since = None;
|
||||
reconnect_attempt = 0;
|
||||
if let Some(next_lease) = welcome.lease {
|
||||
lease = Some(next_lease);
|
||||
// A bootstrap capability is one-use. It is destroyed only after a
|
||||
|
|
@ -207,9 +296,12 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
return Ok(());
|
||||
}
|
||||
if disconnected {
|
||||
disconnected_since.get_or_insert_with(Instant::now);
|
||||
state.reconnect_count = state.reconnect_count.saturating_add(1);
|
||||
store.save(&state)?;
|
||||
thread::sleep(config.reconnect_delay);
|
||||
let reconnect_deadline =
|
||||
connection_attempt_deadline(&config, started, disconnected_since);
|
||||
sleep_before_deadline(config.reconnect_delay, reconnect_deadline);
|
||||
continue;
|
||||
}
|
||||
|
||||
|
|
@ -220,6 +312,7 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
}
|
||||
poll_executor_events(&mut state, &store, &config, &mut executor)?;
|
||||
if let Err(error) = send_outbox(&mut transport, &state, &mut sent_source_seq) {
|
||||
disconnected_since.get_or_insert_with(Instant::now);
|
||||
state.record_diagnostic(error.to_string());
|
||||
state.reconnect_count = state.reconnect_count.saturating_add(1);
|
||||
store.save(&state)?;
|
||||
|
|
@ -239,6 +332,7 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
Ok(Some(message)) => message,
|
||||
Ok(None) => continue,
|
||||
Err(error) => {
|
||||
disconnected_since.get_or_insert_with(Instant::now);
|
||||
state.record_diagnostic(error.to_string());
|
||||
state.reconnect_count = state.reconnect_count.saturating_add(1);
|
||||
store.save(&state)?;
|
||||
|
|
@ -246,6 +340,7 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
}
|
||||
};
|
||||
if let Err(error) = validate_control_identity(&message, &state, Some(&connection)) {
|
||||
disconnected_since.get_or_insert_with(Instant::now);
|
||||
state.record_diagnostic(format!(
|
||||
"control identity mismatch closed the connection: {error}"
|
||||
));
|
||||
|
|
@ -276,6 +371,7 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
.send_json(&command_result_envelope(&state, &result))
|
||||
.and_then(|()| send_outbox(&mut transport, &state, &mut sent_source_seq));
|
||||
if let Err(error) = delivery {
|
||||
disconnected_since.get_or_insert_with(Instant::now);
|
||||
state.record_diagnostic(error.to_string());
|
||||
state.reconnect_count = state.reconnect_count.saturating_add(1);
|
||||
store.save(&state)?;
|
||||
|
|
@ -319,6 +415,7 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
))?;
|
||||
}
|
||||
_ => {
|
||||
disconnected_since.get_or_insert_with(Instant::now);
|
||||
state.record_diagnostic(
|
||||
"malformed or unsupported control frame closed the connection",
|
||||
);
|
||||
|
|
@ -328,7 +425,9 @@ pub fn run_durable_runner<E: CommandExecutor>(
|
|||
}
|
||||
}
|
||||
}
|
||||
thread::sleep(config.reconnect_delay);
|
||||
disconnected_since.get_or_insert_with(Instant::now);
|
||||
let reconnect_deadline = connection_attempt_deadline(&config, started, disconnected_since);
|
||||
sleep_before_deadline(config.reconnect_delay, reconnect_deadline);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -507,6 +606,7 @@ mod tests {
|
|||
fn config(directory: PathBuf) -> DurableRunnerConfig {
|
||||
DurableRunnerConfig {
|
||||
connect_url: "ws://127.0.0.1:3000/path".to_owned(),
|
||||
ca_bundle_path: None,
|
||||
state_dir: directory,
|
||||
runner_instance_id: "runner_1".to_owned(),
|
||||
environment_lease_id: "environment_1".to_owned(),
|
||||
|
|
@ -520,6 +620,7 @@ mod tests {
|
|||
p0_reserve_bytes: 4096,
|
||||
max_frame_bytes: 64 * 1024,
|
||||
reconnect_delay: Duration::from_millis(1),
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(1),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1197,6 +1197,7 @@ mod tests {
|
|||
fn config(state_dir: PathBuf) -> DurableRunnerConfig {
|
||||
DurableRunnerConfig {
|
||||
connect_url: "ws://127.0.0.1:3000/api/runner/v1/connect/run_1".to_owned(),
|
||||
ca_bundle_path: None,
|
||||
state_dir,
|
||||
runner_instance_id: "runner_1".to_owned(),
|
||||
environment_lease_id: "environment_1".to_owned(),
|
||||
|
|
@ -1210,6 +1211,7 @@ mod tests {
|
|||
p0_reserve_bytes: 4096,
|
||||
max_frame_bytes: 65_536,
|
||||
reconnect_delay: Duration::from_millis(1),
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(1),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -80,6 +80,7 @@ fn task_context_tool_set() -> AuthorizedToolSet {
|
|||
fn durable_config(directory: &Path) -> DurableRunnerConfig {
|
||||
DurableRunnerConfig {
|
||||
connect_url: "ws://127.0.0.1:3000/runner".to_owned(),
|
||||
ca_bundle_path: None,
|
||||
state_dir: directory.to_path_buf(),
|
||||
runner_instance_id: "runner-1".to_owned(),
|
||||
environment_lease_id: "lease-1".to_owned(),
|
||||
|
|
@ -93,6 +94,7 @@ fn durable_config(directory: &Path) -> DurableRunnerConfig {
|
|||
p0_reserve_bytes: 1024 * 1024,
|
||||
max_frame_bytes: 1024 * 1024,
|
||||
reconnect_delay: std::time::Duration::from_millis(1),
|
||||
reconnect_grace: None,
|
||||
max_runtime: std::time::Duration::from_secs(5),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ static NEXT_TEMPORARY_DIRECTORY: AtomicU64 = AtomicU64::new(0);
|
|||
fn config(state_dir: PathBuf) -> DurableRunnerConfig {
|
||||
DurableRunnerConfig {
|
||||
connect_url: "ws://127.0.0.1:3000/api/runner/v1/connect/run_1".to_owned(),
|
||||
ca_bundle_path: None,
|
||||
state_dir,
|
||||
runner_instance_id: "runner_1".to_owned(),
|
||||
environment_lease_id: "environment_1".to_owned(),
|
||||
|
|
@ -26,6 +27,7 @@ fn config(state_dir: PathBuf) -> DurableRunnerConfig {
|
|||
p0_reserve_bytes: 4096,
|
||||
max_frame_bytes: 65_536,
|
||||
reconnect_delay: Duration::from_millis(1),
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(1),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue