## Thinking Path
> - Paperclip manages AI agents that perform work.
> - The paid runner matrix verifies complete runner behavior with real
providers.
> - Each matrix job currently repeats work on GitHub-hosted runners.
> - Paperclip has an ephemeral AWS runner fleet for trusted workflows.
> - The paid workflow needs a reviewed and fail-closed route to that
fleet.
> - This pull request adds that route and keeps the existing hosted
runner as the disabled-state fallback.
> - The benefit is faster paid campaigns with the same actor,
environment, and secret boundaries.
## Linked Issues or Issue Description
**What happened?**
The Runner Full-Stack E2E workflow always uses `ubuntu-latest-m`. It
limits the matrix to 57 parallel jobs. The repository AWS fleet can run
100 ephemeral jobs, but the paid workflow cannot select it.
**Expected behavior**
An explicit repository flag must select the reviewed AWS fleet label. A
missing or invalid flag must keep the existing hosted runner. The
workflow must authorize the stable actor identity before it routes any
paid job.
**Steps to reproduce**
1. Dispatch the Runner Full-Stack E2E workflow from `master`.
2. Inspect a paid matrix job.
3. Observe that the job requests `ubuntu-latest-m` even when the AWS
fleet should be used.
**Paperclip version or commit**
`da0947d3582ac7779d6bf11851c9938eca6c5c8c`
**Deployment mode**
GitHub Actions paid runner campaign.
## What Changed
- Add a fail-closed `RUNNER_E2E_AWS_ENABLED` switch.
- Select only the reviewed AWS fleet label or the existing hosted label.
- Permit up to 100 parallel jobs in AWS mode.
- Keep the hosted-runner limit at 57.
- Reauthorize paid execution before checkout and provider access.
- Stop paid checkouts from storing GitHub credentials.
- Cancel superseded validation-ref campaigns while preserving `master`
audit runs.
- Add workflow policy checks and operator documentation.
## Verification
- `git diff --check`
- `actionlint -ignore SC2129
.github/workflows/runner-full-stack-e2e.yml`
- The organization runner group permits this workflow only from
`refs/heads/master`.
- The repository AWS switch remains disabled until this pull request is
merged and a one-cell probe succeeds.
## Risks
- A wrong fleet policy can leave jobs queued. The disabled state keeps
the existing hosted runner.
- The AWS fleet uses paid compute. The workflow validates a configured
maximum of 100 jobs.
- The runner group, actor allowlist, and paid environment remain
separate enforcement layers.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex based on GPT-5 with agentic reasoning, repository
inspection, code editing, Git, GitHub API coordination, and static
workflow analysis. The exact deployed model identifier and
context-window size are not exposed to this task.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip uses local adapters to connect agent sessions to the
control plane
> - The Codex adapter emits turn events from response and notification
channels
> - A terminal notification can arrive before the turn/start response
> - This pull request gates the terminal event on turn.accepted
> - The result keeps the event order stable for consumers and tests
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip uses local adapters to connect agent sessions to the
control plane
> - The Codex adapter emits turn events from response and notification
channels
> - A terminal notification can arrive before the turn/start response
> - This pull request gates the terminal event on turn.accepted
> - The result keeps the event order stable for consumers and tests
## Linked Issues or Issue Description
**What happened?**
The Codex harness session emitted `turn.accepted` only after the
`turn/start` response resolved. A terminal notification could arrive
before that response and reach consumers first.
**Expected behavior**
The Codex driver must emit `turn.accepted` before any terminal event for
the same turn.
**Steps to reproduce**
1. Start a Codex harness session.
2. Keep the `turn/start` response pending.
3. Send `turn/started` and `turn/completed` notifications.
4. Observe the event order.
**Paperclip version or commit**
`afbcd28dae9e51108738c4258929b95ca359186c`
**Deployment mode**
Built from source with the Codex driver test harness.
**Agent adapter(s) involved**
Codex.
## What Changed
- Add session state that tracks a pending `turn/start` operation.
- Resolve the state when `turn/start` succeeds or fails.
- Wait for that state before the terminal notification handler emits its
event.
- Add a regression test that delivers a terminal notification while
`turn/start` remains pending.
## Verification
- The regression test failed 5 of 5 times before this change and passed
5 of 5 times after it.
- The Codex driver suite passed 189 of 189 tests.
- The affected live transport test file passed 46 of 46 tests on 10
consecutive runs.
- The TypeScript check exited with status 0.
- Continuous integration must pass before merge.
## Risks
The change affects only Codex turn event ordering. It adds no sleep,
retry, or timeout. The main risk is a provider path that does not settle
`turn/start`; existing provider response handling still controls
completion.
## Model Used
OpenAI GPT-5. The exact deployment identifier is not exposed in this
environment. Tool use and code execution assisted this change.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip runs agent heartbeats and stores their run state in a
database
> - The direct-adapter native-isolation tests start heartbeat runs and
then clear database state
> - A terminal run status does not prove that its background database
work has stopped
> - The teardown can then deadlock with a live run during PostgreSQL
`TRUNCATE`
> - This pull request drains active runs before teardown and adds a
guard for queued or running runs
> - The benefit is stable test teardown without a production code change
## Linked Issues or Issue Description
This change fixes an intermittent test deadlock in the direct-adapter
native-isolation suite.
**What happened?**
The test teardown could run PostgreSQL `TRUNCATE` while a heartbeat
execution still held a write transaction. PostgreSQL then returned error
`40P01` during some test runs.
**Expected behavior**
The test teardown must wait until all heartbeat executions finish before
it clears the test database.
**Steps to reproduce**
1. Run
`server/src/__tests__/heartbeat-direct-adapter-native-isolation.test.ts`
repeatedly.
2. Run the suite against PostgreSQL-backed native isolation.
3. Observe intermittent deadlock error `40P01` during teardown.
**Paperclip version or commit**
Commit `57515726d3ef45a07df9b5ee2dfaf7d108556478`.
**Deployment mode**
Built from source with the native-isolation test suite.
**Agent adapter(s) involved**
Not adapter-specific. The test covers the direct adapter path.
**Database mode**
External PostgreSQL used by the native-isolation test suite.
**Additional context**
Related prior attempt:
[#12715](https://github.com/paperclipai/paperclip/pull/12715). This pull
request starts from current `master` and does not depend on that pull
request.
## What Changed
- Drain active heartbeat run executions before `afterEach` runs
`TRUNCATE`.
- Assert that no heartbeat run remains `queued` or `running` before
teardown.
- Drain active executions before `afterAll` removes the temporary
database.
- Create one shared `heartbeatService` instance in `beforeAll` so the
drain tracks the test runs.
## Verification
- Run
`server/src/__tests__/heartbeat-direct-adapter-native-isolation.test.ts`
20 times. All 20 runs pass.
- Run the target suite with
`server/src/__tests__/native-run-finalizer.test.ts`. Both files pass
with 19 tests.
- Run `tsc --noEmit`. The branch adds no new error compared with
`master`.
- Run the pull request checks after GitHub starts them.
## Risks
Low risk. The change affects one test file and no production code. The
added drain can expose an incomplete test run before teardown, which is
the intended guard.
## Model Used
OpenAI GPT-5. Exact runtime model ID: GPT-5. The context window is not
exposed to this agent. The model used tool calls and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
Dotta
dependabot[bot]
Dotta
Nicky Leach