166 lines
7.4 KiB
Markdown
166 lines
7.4 KiB
Markdown
# Runner E2E security for a public repository
|
|
|
|
This suite can spend provider money, expose four API credentials to isolated
|
|
test processes, publish a container, retain private visual evidence, and write
|
|
public structured evidence. Treat changes to the workflow, harness, fixture
|
|
prompts, evidence packager, and publisher as security-sensitive production
|
|
changes.
|
|
|
|
## GitHub authorization
|
|
|
|
Set `RUNNER_E2E_ALLOWED_ACTOR_IDS` to a non-empty JSON array of numeric GitHub
|
|
user IDs, for example `[123456,789012]`. Resolve each ID from the authenticated
|
|
CLI and verify the login before adding it:
|
|
|
|
```bash
|
|
gh api users/LOGIN --jq '{login,id}'
|
|
```
|
|
|
|
The paid workflows reject manual dispatches outside the default branch before
|
|
checkout. They verify both the original actor and triggering actor for every
|
|
scheduled or manual attempt, including human reruns. Every
|
|
secret-bearing job repeats this check as its first step so GitHub's partial-job
|
|
rerun feature cannot bypass a successful predecessor authorization job. The
|
|
legacy manually dispatched E2E workflow uses the same gate. Numeric IDs are
|
|
stable across username changes and prevent lookalike-name authorization.
|
|
|
|
The full-stack and live campaigns have one Sunday UTC schedule each and also
|
|
support explicit manual dispatch. Their legacy-named nightly repository
|
|
variables remain independent kill switches. Neither paid workflow accepts
|
|
pull-request, push, workflow-run, or reusable-workflow triggers.
|
|
|
|
Protect the default branch, require review for workflow/harness paths, restrict
|
|
workflow dispatch permission, and restrict repository variable/environment
|
|
administration to the same trusted maintainers. Configure the organization to
|
|
allow only approved GitHub Actions. A malicious change merged into the default
|
|
branch executes with the same authority as the suite.
|
|
|
|
Every external action in the paid workflow is pinned to a full commit SHA. Keep
|
|
the adjacent major-version comment for update tooling, and resolve and review a
|
|
new immutable SHA before upgrading an action. The credential-free security test
|
|
rejects mutable tag or branch references.
|
|
|
|
## Secrets and protected environments
|
|
|
|
Create `runner-e2e-paid`, restrict deployments to the default branch, and put
|
|
only `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, and
|
|
`DAYTONA_API_KEY` in it. Do not duplicate these credentials as repository- or
|
|
organization-level Actions secrets: environment scoping is the boundary that
|
|
prevents branch or pull-request jobs from requesting them. Require approval
|
|
from an account in `RUNNER_E2E_ALLOWED_ACTOR_IDS` for this environment and
|
|
disable administrator bypass. The authorize,
|
|
catalog, image, report, history, and Pages jobs receive none of these secrets.
|
|
Each full-stack matrix cell receives only its selected profile credential, plus
|
|
Daytona only for Daytona cells. Secret-bearing and OIDC jobs use frozen installs
|
|
without a shared dependency cache.
|
|
The Paperclip server process also receives none; the browser posts each value
|
|
once to the encrypted company secret API and agents/environments retain only
|
|
secret references.
|
|
|
|
Create `runner-e2e-history`, also default-branch-only, for the OIDC publishing
|
|
job. It contains no long-lived AWS key. Required reviewers may be added when a
|
|
human approval on every nightly publication is acceptable; otherwise rely on
|
|
the actor gate, environment branch restriction, and protected default branch.
|
|
|
|
## Runner group isolation
|
|
|
|
Restrict the `ubuntu-latest-m` runner group to `paperclipai/paperclip` and, when
|
|
the GitHub plan supports selected-workflow restrictions, to
|
|
`.github/workflows/runner-full-stack-e2e.yml` on the default branch. Never let
|
|
fork or pull-request workflows target the group. Use ephemeral runners, or
|
|
guaranteed reimaging between jobs, and do not share this group with untrusted
|
|
workloads. Disable interactive SSH/debug access for paid jobs unless a separate
|
|
incident procedure explicitly authorizes it.
|
|
|
|
## AWS OIDC and S3
|
|
|
|
The AWS role trust policy should accept only GitHub's OIDC audience and the
|
|
publishing environment subject:
|
|
|
|
```json
|
|
{
|
|
"Version": "2012-10-17",
|
|
"Statement": [
|
|
{
|
|
"Effect": "Allow",
|
|
"Principal": {
|
|
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
|
|
},
|
|
"Action": "sts:AssumeRoleWithWebIdentity",
|
|
"Condition": {
|
|
"StringEquals": {
|
|
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
|
|
"token.actions.githubusercontent.com:sub": "repo:paperclipai/paperclip:environment:runner-e2e-history"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Grant only List on the bucket prefix and Get/Put on its objects. Do not grant
|
|
Delete, ACL, bucket-policy, or wildcard-resource permissions:
|
|
|
|
```json
|
|
{
|
|
"Version": "2012-10-17",
|
|
"Statement": [
|
|
{
|
|
"Effect": "Allow",
|
|
"Action": "s3:ListBucket",
|
|
"Resource": "arn:aws:s3:::BUCKET",
|
|
"Condition": {
|
|
"StringLike": { "s3:prefix": ["runner-e2e", "runner-e2e/*"] }
|
|
}
|
|
},
|
|
{
|
|
"Effect": "Allow",
|
|
"Action": ["s3:GetObject", "s3:PutObject"],
|
|
"Resource": "arn:aws:s3:::BUCKET/runner-e2e/*"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Enable S3 versioning, default encryption, and Block Public Access. Disable
|
|
object ACLs. CloudFront receives read-only access through Origin Access Control;
|
|
the bucket itself stays private. Log S3 data writes and alert on attempts to
|
|
write outside the prefix or assume the role with a different subject.
|
|
|
|
Campaign prefixes are content-digested and immutable. The publisher refuses a
|
|
different digest at an existing campaign key. Only the compact history and
|
|
latest pointers are mutable, and S3 versioning makes those updates recoverable.
|
|
|
|
## Public evidence boundary
|
|
|
|
CloudFront and GitHub Pages are public. Fixture identifiers, timing, token
|
|
usage, costs, normalized results, and allowlisted inert structured per-attempt
|
|
evidence are expected public data. Screenshots, video, archives, generated
|
|
Playwright/blob/HTML report trees, credentials, Paperclip homes, databases,
|
|
workspaces, master keys, raw/unredacted logs, and unallowlisted files are not.
|
|
Only allowlisted `.log` copies that passed exact-value/key-shape scanning and
|
|
redaction may cross the public boundary.
|
|
|
|
The packaged evidence uploaded as a 30-day GitHub Actions artifact has a
|
|
different, access-controlled boundary. Text is exact-value and key-shape
|
|
scanned and redacted. PNG and WebM are raw-byte scanned but cannot be inspected
|
|
for credentials rendered as pixels, so they remain only in local evidence and
|
|
the access-controlled artifact. SVG is rejected during packaging because it is
|
|
active content.
|
|
|
|
Before permanent publication, the campaign publisher prunes raster/video
|
|
files, archives, and generated report trees. It then regenerates the dashboard
|
|
from the remaining allowlisted `.json`, `.log`, `.md`, and `.txt` evidence and
|
|
accepts only that dashboard, normalized JSON/JUnit/summary, fixed
|
|
branding assets, and the inert structured evidence paths. Per-attempt XML is
|
|
excluded because browsers can process XML/XSLT; the only public XML is the
|
|
root `junit.xml`, which the report aggregator constructs from fixed markup and
|
|
XML-escaped fields. The same pruned tree feeds both S3/CloudFront history and
|
|
the optional GitHub Pages artifact. A leak fails the cell and withholds the
|
|
unsafe file.
|
|
|
|
Rotate the affected credential immediately if a secret-scanning failure or
|
|
unexpected public object is observed. Preserve the access-controlled Actions
|
|
artifact and S3 object versions for incident analysis; do not weaken scanning
|
|
to make a campaign publish.
|