ci(runner): route paid matrix to AWS fleet (#12765)
## Thinking Path > - Paperclip manages AI agents that perform work. > - The paid runner matrix verifies complete runner behavior with real providers. > - Each matrix job currently repeats work on GitHub-hosted runners. > - Paperclip has an ephemeral AWS runner fleet for trusted workflows. > - The paid workflow needs a reviewed and fail-closed route to that fleet. > - This pull request adds that route and keeps the existing hosted runner as the disabled-state fallback. > - The benefit is faster paid campaigns with the same actor, environment, and secret boundaries. ## Linked Issues or Issue Description **What happened?** The Runner Full-Stack E2E workflow always uses `ubuntu-latest-m`. It limits the matrix to 57 parallel jobs. The repository AWS fleet can run 100 ephemeral jobs, but the paid workflow cannot select it. **Expected behavior** An explicit repository flag must select the reviewed AWS fleet label. A missing or invalid flag must keep the existing hosted runner. The workflow must authorize the stable actor identity before it routes any paid job. **Steps to reproduce** 1. Dispatch the Runner Full-Stack E2E workflow from `master`. 2. Inspect a paid matrix job. 3. Observe that the job requests `ubuntu-latest-m` even when the AWS fleet should be used. **Paperclip version or commit** `da0947d3582ac7779d6bf11851c9938eca6c5c8c` **Deployment mode** GitHub Actions paid runner campaign. ## What Changed - Add a fail-closed `RUNNER_E2E_AWS_ENABLED` switch. - Select only the reviewed AWS fleet label or the existing hosted label. - Permit up to 100 parallel jobs in AWS mode. - Keep the hosted-runner limit at 57. - Reauthorize paid execution before checkout and provider access. - Stop paid checkouts from storing GitHub credentials. - Cancel superseded validation-ref campaigns while preserving `master` audit runs. - Add workflow policy checks and operator documentation. ## Verification - `git diff --check` - `actionlint -ignore SC2129 .github/workflows/runner-full-stack-e2e.yml` - The organization runner group permits this workflow only from `refs/heads/master`. - The repository AWS switch remains disabled until this pull request is merged and a one-cell probe succeeds. ## Risks - A wrong fleet policy can leave jobs queued. The disabled state keeps the existing hosted runner. - The AWS fleet uses paid compute. The workflow validates a configured maximum of 100 jobs. - The runner group, actor allowlist, and paid environment remain separate enforcement layers. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex based on GPT-5 with agentic reasoning, repository inspection, code editing, Git, GitHub API coordination, and static workflow analysis. The exact deployed model identifier and context-window size are not exposed to this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta
GitHub
parent
e4afd163bf
commit
1b74561fea
|
|
@ -39,7 +39,9 @@ permissions:
|
|||
|
||||
concurrency:
|
||||
group: runner-full-stack-e2e-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
# Development-only validation refs supersede older runs on the same ref.
|
||||
# Preserve every protected default-branch campaign for its paid audit trail.
|
||||
cancel-in-progress: ${{ github.ref != format('refs/heads/{0}', github.event.repository.default_branch) }}
|
||||
|
||||
jobs:
|
||||
authorize:
|
||||
|
|
@ -49,6 +51,10 @@ jobs:
|
|||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
test_runner: ${{ steps.runner.outputs.runner }}
|
||||
max_parallel_default: ${{ steps.runner.outputs.max_parallel_default }}
|
||||
max_parallel_limit: ${{ steps.runner.outputs.max_parallel_limit }}
|
||||
steps:
|
||||
- name: Require default branch and allowlisted numeric actor IDs
|
||||
env:
|
||||
|
|
@ -83,6 +89,31 @@ jobs:
|
|||
fi
|
||||
done
|
||||
|
||||
- name: Select paid test runner
|
||||
id: runner
|
||||
env:
|
||||
AWS_PAID_RUNNER_ENABLED: ${{ vars.RUNNER_E2E_AWS_ENABLED }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
github_runner='ubuntu-latest-m'
|
||||
aws_runner='runs-on/fleet=paperclip-public-pr-x64/env=public-ci'
|
||||
|
||||
if [ "$AWS_PAID_RUNNER_ENABLED" = true ]; then
|
||||
{
|
||||
echo "runner=$aws_runner"
|
||||
echo "max_parallel_default=100"
|
||||
echo "max_parallel_limit=100"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
echo '::notice title=Paid runner routing::Using an ephemeral RunsOn Fleet runner'
|
||||
else
|
||||
{
|
||||
echo "runner=$github_runner"
|
||||
echo "max_parallel_default=32"
|
||||
echo "max_parallel_limit=57"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
echo '::notice title=Paid runner routing::RUNNER_E2E_AWS_ENABLED is not true; using the existing paid runner'
|
||||
fi
|
||||
|
||||
catalog:
|
||||
name: Validate catalog and select cells
|
||||
needs: authorize
|
||||
|
|
@ -128,7 +159,8 @@ jobs:
|
|||
SELECT_ENVIRONMENT: ${{ inputs.environment }}
|
||||
SELECT_CASE: ${{ inputs.case }}
|
||||
SELECT_ID: ${{ inputs.id }}
|
||||
MAX_PARALLEL: ${{ vars.RUNNER_E2E_MAX_PARALLEL || '32' }}
|
||||
MAX_PARALLEL: ${{ vars.RUNNER_E2E_MAX_PARALLEL || needs.authorize.outputs.max_parallel_default }}
|
||||
MAX_PARALLEL_LIMIT: ${{ needs.authorize.outputs.max_parallel_limit }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
args=(--matrix-json)
|
||||
|
|
@ -176,8 +208,12 @@ jobs:
|
|||
echo "matrix=$(jq -c '{include: .include}' <<< "$catalog_json")" >> "$GITHUB_OUTPUT"
|
||||
echo "needs_daytona=$(jq -r '.needsDaytona' <<< "$catalog_json")" >> "$GITHUB_OUTPUT"
|
||||
echo "execution_ids=$(jq -c '.executionIds' <<< "$catalog_json")" >> "$GITHUB_OUTPUT"
|
||||
if ! [[ "$MAX_PARALLEL" =~ ^[1-9][0-9]*$ ]] || [ "$MAX_PARALLEL" -gt 57 ]; then
|
||||
echo "RUNNER_E2E_MAX_PARALLEL must be an integer from 1 through 57." >&2
|
||||
if ! [[ "$MAX_PARALLEL_LIMIT" =~ ^[1-9][0-9]*$ ]] || [ "$MAX_PARALLEL_LIMIT" -gt 100 ]; then
|
||||
echo "Runner selection emitted an invalid max-parallel limit." >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! [[ "$MAX_PARALLEL" =~ ^[1-9][0-9]*$ ]] || [ "$MAX_PARALLEL" -gt "$MAX_PARALLEL_LIMIT" ]; then
|
||||
echo "RUNNER_E2E_MAX_PARALLEL must be an integer from 1 through $MAX_PARALLEL_LIMIT for the selected runner." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "max_parallel=$MAX_PARALLEL" >> "$GITHUB_OUTPUT"
|
||||
|
|
@ -286,8 +322,10 @@ jobs:
|
|||
|
||||
test:
|
||||
name: ${{ matrix.executionId }}
|
||||
needs: [catalog, daytona_image]
|
||||
runs-on: ubuntu-latest-m
|
||||
needs: [authorize, catalog, daytona_image]
|
||||
# The authorize job selects only one of two literal, reviewed runner labels;
|
||||
# no dispatch input or repository variable can inject an arbitrary label.
|
||||
runs-on: ${{ needs.authorize.outputs.test_runner }}
|
||||
timeout-minutes: ${{ matrix.timeoutMinutes }}
|
||||
permissions:
|
||||
contents: read
|
||||
|
|
@ -315,6 +353,8 @@ jobs:
|
|||
jq -e --argjson candidate "$ACTOR_ID" 'index($candidate) != null' <<< "$ALLOWED_ACTOR_IDS" >/dev/null
|
||||
|
||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
|
||||
with:
|
||||
|
|
|
|||
|
|
@ -286,17 +286,27 @@ accept comma-separated values for repeatable dimensions.
|
|||
The nightly cron is `08:47 UTC`, but scheduled execution is intentionally gated
|
||||
by the repository variable `RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED=true`. Set it
|
||||
only after the live acceptance ladder in the architecture plan is green.
|
||||
Set `RUNNER_E2E_MAX_PARALLEL` to an integer from 1–57 (default 32). Paid cells
|
||||
run on `ubuntu-latest-m`; multi-turn steps are sequential inside their cell
|
||||
while independent cells overlap. Artifacts and merged HTML/JUnit/normalized
|
||||
reports are retained for 30 days.
|
||||
Set `RUNNER_E2E_AWS_ENABLED=true` to route paid cells to the repository-scoped
|
||||
ephemeral AWS RunsOn fleet selected by
|
||||
`runs-on/fleet=paperclip-public-pr-x64/env=public-ci`. Any other value retains
|
||||
the existing `ubuntu-latest-m` target. Set `RUNNER_E2E_MAX_PARALLEL` to an
|
||||
integer from 1–100 on AWS (default 100); use at least 71 to run the current
|
||||
complete catalog in one wave. The fallback runner retains its 1–57 limit and
|
||||
default of 32. Multi-turn steps are sequential inside their cell while
|
||||
independent cells overlap. Artifacts and merged HTML/JUnit/normalized reports
|
||||
are retained for 30 days.
|
||||
|
||||
Restrict the `ubuntu-latest-m` runner group to this workflow and the selected
|
||||
repository. Do not let pull-request or fork-triggered workflows target that
|
||||
group, do not mix it with untrusted workloads, and use ephemeral/reimaged
|
||||
runners so one paid cell cannot leave state for the next. These runner-group
|
||||
controls are external GitHub settings and are as important as the workflow
|
||||
checks in a public repository.
|
||||
Restrict the RunsOn fleet to this repository and independently trusted
|
||||
workflows. Do not let untrusted pull-request or fork-triggered workflows target
|
||||
it, and require a fresh ephemeral instance for each job so one paid cell cannot
|
||||
leave state for the next. Provider secrets remain protected by the stable-ID
|
||||
authorization checks and the default-branch-only `runner-e2e-paid` environment;
|
||||
the fleet itself is not an authorization boundary. These external fleet controls
|
||||
are as important as the workflow checks in a public repository.
|
||||
|
||||
Non-default validation runs share a concurrency key per ref and cancel an older
|
||||
run when a replacement is dispatched. Protected default-branch paid campaigns
|
||||
are retained and are never auto-cancelled, preserving their audit trail.
|
||||
|
||||
GitHub Actions artifacts are access-controlled 30-day operational copies, not
|
||||
the permanent public history. They retain packaged PNG/WebM and generated
|
||||
|
|
|
|||
|
|
@ -62,15 +62,29 @@ job. It contains no long-lived AWS key. Required reviewers may be added when a
|
|||
human approval on every nightly publication is acceptable; otherwise rely on
|
||||
the actor gate, environment branch restriction, and protected default branch.
|
||||
|
||||
## Runner group isolation
|
||||
## Runner fleet isolation
|
||||
|
||||
Restrict the `ubuntu-latest-m` runner group to `paperclipai/paperclip` and, when
|
||||
the GitHub plan supports selected-workflow restrictions, to
|
||||
`.github/workflows/runner-full-stack-e2e.yml` on the default branch. Never let
|
||||
fork or pull-request workflows target the group. Use ephemeral runners, or
|
||||
guaranteed reimaging between jobs, and do not share this group with untrusted
|
||||
workloads. Disable interactive SSH/debug access for paid jobs unless a separate
|
||||
incident procedure explicitly authorizes it.
|
||||
When `RUNNER_E2E_AWS_ENABLED=true`, paid matrix cells use the exact RunsOn fleet
|
||||
selector `runs-on/fleet=paperclip-public-pr-x64/env=public-ci`, matching the AWS
|
||||
fleet selected by `pr-trusted.yml` only after its stable numeric-ID trust gate.
|
||||
Any other or missing toggle value falls back to the existing `ubuntu-latest-m`
|
||||
paid runner and its lower concurrency ceiling. The workflow chooses between
|
||||
those two reviewed literal labels; it never evaluates a configured runner label.
|
||||
|
||||
Keep both runner targets restricted to `paperclipai/paperclip` and workflows
|
||||
that independently authorize trusted source revisions. Never let a fork or
|
||||
untrusted pull-request workflow target them. The RunsOn fleet must launch a
|
||||
fresh ephemeral instance for every job, prohibit persistent runner reuse, and
|
||||
disable interactive SSH/debug access unless a separate incident procedure
|
||||
explicitly authorizes it.
|
||||
|
||||
Changing the runner does not widen secret access. The paid workflow still has
|
||||
only schedule and manual triggers, requires the protected default branch and
|
||||
allowlisted stable actor IDs before checkout, repeats that authorization as the
|
||||
first matrix step, and receives provider credentials only from the protected
|
||||
`runner-e2e-paid` environment. The fleet selector is an exact workflow literal;
|
||||
the only repository-controlled input is its boolean rollout switch, so
|
||||
configuration cannot redirect a secret-bearing job to an arbitrary runner.
|
||||
|
||||
## AWS OIDC and S3
|
||||
|
||||
|
|
|
|||
|
|
@ -25,9 +25,10 @@ describe("public repository paid workflow security", () => {
|
|||
const providerAccess = contents.search(
|
||||
/(?:OPENAI|ANTHROPIC|OPENROUTER|DAYTONA)_API_KEY:\s*\$\{\{\s*[^}]*secrets\./,
|
||||
);
|
||||
expect(authorize, `${name} must have an authorization job`).toBeGreaterThan(
|
||||
0,
|
||||
);
|
||||
expect(
|
||||
authorize,
|
||||
`${name} must have an authorization job`,
|
||||
).toBeGreaterThan(0);
|
||||
expect(
|
||||
reauthorize,
|
||||
`${name} must reauthorize partial job reruns`,
|
||||
|
|
@ -43,7 +44,7 @@ describe("public repository paid workflow security", () => {
|
|||
expect(contents).toContain("RUNNER_E2E_ALLOWED_ACTOR_IDS");
|
||||
expect(contents).toContain("github.actor_id");
|
||||
expect(contents).toContain("github.triggering_actor");
|
||||
expect(contents).toContain('refs/heads/$DEFAULT_BRANCH');
|
||||
expect(contents).toContain("refs/heads/$DEFAULT_BRANCH");
|
||||
expect(contents).toContain("needs: authorize");
|
||||
expect(contents).toContain("name: runner-e2e-paid");
|
||||
expect(contents).not.toMatch(
|
||||
|
|
@ -59,6 +60,37 @@ describe("public repository paid workflow security", () => {
|
|||
}
|
||||
|
||||
const fullStack = workflows[0]!.contents;
|
||||
const paidJob = fullStack.slice(
|
||||
fullStack.indexOf(" test:"),
|
||||
fullStack.indexOf(" report:"),
|
||||
);
|
||||
const authorizeJob = fullStack.slice(
|
||||
fullStack.indexOf(" authorize:"),
|
||||
fullStack.indexOf(" catalog:"),
|
||||
);
|
||||
expect(authorizeJob).toContain(
|
||||
"aws_runner='runs-on/fleet=paperclip-public-pr-x64/env=public-ci'",
|
||||
);
|
||||
expect(authorizeJob).toContain("github_runner='ubuntu-latest-m'");
|
||||
expect(authorizeJob).toContain(
|
||||
"AWS_PAID_RUNNER_ENABLED: ${{ vars.RUNNER_E2E_AWS_ENABLED }}",
|
||||
);
|
||||
expect(paidJob).toContain(
|
||||
"runs-on: ${{ needs.authorize.outputs.test_runner }}",
|
||||
);
|
||||
expect(paidJob).toContain("needs: [authorize, catalog, daytona_image]");
|
||||
expect(paidJob).toContain("name: runner-e2e-paid");
|
||||
expect(paidJob).toMatch(
|
||||
/Reauthorize paid execution before provider access[\s\S]*actions\/checkout@[0-9a-f]{40}[\s\S]*persist-credentials: false/,
|
||||
);
|
||||
expect(authorizeJob).toContain('echo "max_parallel_limit=100"');
|
||||
expect(fullStack).toContain('[ "$MAX_PARALLEL_LIMIT" -gt 100 ]');
|
||||
expect(fullStack).toContain(
|
||||
'[ "$MAX_PARALLEL" -gt "$MAX_PARALLEL_LIMIT" ]',
|
||||
);
|
||||
expect(fullStack).toContain(
|
||||
"cancel-in-progress: ${{ github.ref != format('refs/heads/{0}', github.event.repository.default_branch) }}",
|
||||
);
|
||||
for (const [secret, condition] of Object.entries({
|
||||
OPENAI_API_KEY: "matrix.credentialName == 'OPENAI_API_KEY'",
|
||||
ANTHROPIC_API_KEY: "matrix.credentialName == 'ANTHROPIC_API_KEY'",
|
||||
|
|
@ -83,7 +115,10 @@ describe("public repository paid workflow security", () => {
|
|||
);
|
||||
|
||||
for (const name of names) {
|
||||
const contents = await readFile(path.join(workflowDirectory, name), "utf8");
|
||||
const contents = await readFile(
|
||||
path.join(workflowDirectory, name),
|
||||
"utf8",
|
||||
);
|
||||
const providerSecretReferences = [
|
||||
...contents.matchAll(
|
||||
/secrets(?:\.(?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|DAYTONA_API_KEY)\b|\[['"](?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|DAYTONA_API_KEY)['"]\])/g,
|
||||
|
|
@ -101,10 +136,7 @@ describe("public repository paid workflow security", () => {
|
|||
it("runs paid scheduled campaigns only on Sundays", async () => {
|
||||
const workflows = await Promise.all(
|
||||
["runner-full-stack-e2e.yml", "runner-live-evals.yml"].map((name) =>
|
||||
readFile(
|
||||
path.join(repositoryRoot, ".github/workflows", name),
|
||||
"utf8",
|
||||
),
|
||||
readFile(path.join(repositoryRoot, ".github/workflows", name), "utf8"),
|
||||
),
|
||||
);
|
||||
for (const workflow of workflows) {
|
||||
|
|
|
|||
Loading…
Reference in New Issue