feat(server): authenticate server-side git clone and fetch with a company-secret GitHub token (#10720)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Repo-only project workspaces are materialized by a server-side `git clone`, and isolated `git_worktree` runs refresh their base ref with server-side `git fetch` > - Both operations run outside the agent process with no credentials, so private GitHub repositories can never be cloned or refreshed — agent-scoped credential env bindings do not reach them > - The company secret store already has a well-known GitHub token convention (`GITHUB_TOKEN` / `GH_TOKEN` / `PAPERCLIP_GITHUB_TOKEN`, consumed by the external-object provider for API reads), but nothing server-side consults it for git > - This pull request resolves that token per run and authenticates the managed clone and every base-ref refresh with it through an ephemeral credential helper > - The benefit is that isolated workspaces work on private repositories with one company secret, while public repositories and self-hosted ambient git configuration keep working unchanged ## Linked Issues or Issue Description **Subsystem affected** Server workspace materialization (`server/src/services/heartbeat.ts`) and execution-workspace realization (`server/src/services/workspace-runtime.ts`). **Problem or motivation** A project workspace configured with only a private GitHub `repoUrl` cannot be used for isolated `git_worktree` runs: the managed `git clone` runs with a sanitized, credential-less environment, and plain git cannot consume a bare token env variable without a credential helper. There is no way to give the server a git credential — storing a `GH_TOKEN` company secret has no effect on server-side git, and a credential-less private clone hangs on a terminal prompt until the ten-minute clone timeout. Base-ref refreshes (`git fetch`) during worktree realization have the same gap. **Proposed solution** A `git-credentials` module resolves a token per run — company secret by well-known name (`GITHUB_TOKEN`, `GH_TOKEN`, `PAPERCLIP_GITHUB_TOKEN`), then `GITHUB_TOKEN`/`GH_TOKEN` in the server process environment for self-hosted deployments, then none — and builds a git invocation that authenticates via an inline credential helper. The token travels in an env variable; it never appears in argv, URLs, or on disk. Only `https://github.com` remotes are authenticated; everything else keeps ambient behavior. The provider is a single factory seam so a future brokered credential source can replace it without touching call sites. **Alternatives considered** - A GitHub OAuth "connect your account" flow: heavier product surface, needs app registration and callback custody; out of scope for a server credential and better served by a dedicated connector later. The provider seam keeps that path open. - `gh auth setup-git`: writes helper configuration to disk and requires a global token env; rejected in favor of per-invocation config with no persistent state. - Embedding the token in the clone URL: leaks into argv, error messages, and `.git/config`; rejected. ## What Changed - New `server/src/services/git-credentials.ts`: `createGitRemoteAuthProvider` (memoized per run, one secret resolution and one audit event), `buildGitAuthInvocation` (helper-reset + inline helper, `x-access-token` username, `GIT_TERMINAL_PROMPT=0`), `isGitHubHttpsRemoteUrl` host gating (rejects ssh/GHES/http/other hosts/userinfo URLs), `describeGitAuthFailure`, and the canonical `scrubGitCredentialText`. Secret resolutions pass a `system` consumer access context so they are recorded as secret access events. - `ensureManagedProjectWorkspace` (now exported) accepts an optional auth provider; the clone env spreads the token after `sanitizeRuntimeServiceBaseEnv` (which strips `PAPERCLIP_*`), always sets `GIT_TERMINAL_PROMPT=0`, distinguishes "credential rejected" from "no credential configured — add a GITHUB_TOKEN or GH_TOKEN company secret" in the error, and removes the partially created directory on clone failure so a timeout-killed clone cannot be adopted as a broken checkout by the next run. - `refreshRemoteTrackingBaseRef` (now exported) captures the remote URL it already looked up, asks the provider for an invocation, and attributes failed authenticated fetches to the credential in a scrubbed warning. The optional provider threads through `detectDefaultBranch`, `resolveAuthoritativeBaseRef`, `inspectExecutionWorkspaceBaseDrift`, `realizeExecutionWorkspace`, and `ensurePersistedExecutionWorkspaceAvailable`; heartbeat builds one provider per run for both the anchor-resolution clone path and workspace realization/restore. - `github-external-object-provider.ts` imports the shared secret-name list; `isGitHubDotCom` is exported from `github-fetch.ts`. - Docs: "Private repositories and repo-only project workspaces" section in the execution-workspaces guide, cross-linked from the secrets deploy doc. ## Verification - `cd server && npx vitest run src/__tests__/git-credentials.test.ts` — resolution chain order and precedence, env fallback, memoization, audited access context, host-gating matrix, invocation shape (token absent from argv), scrubber, failure descriptions, and a real-git `git credential fill` round trip that proves the helper executes and answers with the env-carried token (no network). - `cd server && npx vitest run src/__tests__/heartbeat-managed-clone-credentials.test.ts` — clones behave byte-identically with no provider or a null-returning provider (local repos, no network), authenticated-failure errors name the credential, non-auth failures do not mention credentials, partial clone directories are removed, pre-existing non-git directories keep the "Using it as-is" path, and the sanitizer spread order keeps the token env alive. - `cd server && npx vitest run src/__tests__/workspace-runtime.test.ts` — new `refreshRemoteTrackingBaseRef` cases: provider offered the remote URL and null keeps behavior identical; failed authenticated fetch warning names the credential; unauthenticated failure warning stays credential-free. - `pnpm --filter @paperclipai/server typecheck` is clean. - Manual (optional, networked): store a `GH_TOKEN` company secret, configure a repo-only project workspace pointing at a private GitHub repository, run an isolated-workspace issue — the managed clone succeeds and the worktree run proceeds. ## Risks - Every new parameter is optional; with no provider the git invocations are byte-identical to before. Public repos and ambient credential helpers keep working whenever no token resolves. - Precedence change when a token exists: a stored company secret now wins over ambient helpers for `https://github.com` remotes (the helper list is reset for that invocation). The rejected-credential error names the secret so an operator can fix or remove it. - `GIT_TERMINAL_PROMPT=0` on the managed clone is the one always-on change: a credential-less private clone now fails fast with a clear message instead of hanging until the ten-minute timeout (it could only ever "succeed" interactively on a TTY dev server). - The token is scoped to the git process env for one invocation; it is never written to agent env, run context, disk, or logs, and error text is scrubbed of URL userinfo. - No migrations, no image changes (git ships in the image). ## Model Used Claude Fable 5 (`claude-fable-5`, extended thinking, agentic tool use via Claude Code CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Devin Foley
GitHub
parent
97590ff8c4
commit
e0c2448267
|
|
@ -51,6 +51,13 @@ Project env applies to every issue run in that project. When a project env key
|
|||
matches an agent env key, the project value wins before Paperclip injects its
|
||||
own `PAPERCLIP_*` runtime variables.
|
||||
|
||||
Separately from env bindings, the **server itself** consumes a company secret
|
||||
named `GITHUB_TOKEN`, `GH_TOKEN`, or `PAPERCLIP_GITHUB_TOKEN` (by name, no
|
||||
binding needed) to authenticate server-side git operations — cloning private
|
||||
GitHub repos for repo-only project workspaces and refreshing worktree base
|
||||
refs. See
|
||||
[Execution workspaces](../guides/board-operator/execution-workspaces-and-runtime-services.md#private-repositories-and-repo-only-project-workspaces).
|
||||
|
||||
## User-Specific Secrets
|
||||
|
||||
User-specific secrets let a shared agent or project declare a slot such as
|
||||
|
|
|
|||
|
|
@ -77,6 +77,32 @@ Some workspaces need heavy one-time setup — seeding a database, warming caches
|
|||
- **Provisioning failed** — the command failed; the workspace detail links to the runtime logs for the failing operation.
|
||||
- While the command runs, the runtime service shows a **Provisioning…** state before it transitions to starting/running.
|
||||
|
||||
## Private repositories and repo-only project workspaces
|
||||
|
||||
A project workspace can be **repo-only**: a `Repo URL` with no local path. The server then
|
||||
materializes a managed checkout on demand (`git clone` into a managed directory) and, for
|
||||
isolated `git_worktree` runs, refreshes the base ref (`git fetch`) before preparing each
|
||||
worktree. Both operations run on the server, outside any agent process — so agent-scoped
|
||||
credential env bindings do not apply to them.
|
||||
|
||||
For **private GitHub repositories**, store a token as a **company secret** named one of
|
||||
`GITHUB_TOKEN`, `GH_TOKEN`, or `PAPERCLIP_GITHUB_TOKEN` (checked in that order; Settings →
|
||||
Secrets). The server resolves it per run and authenticates managed clones and base-ref
|
||||
fetches with it. Details and caveats:
|
||||
|
||||
- Scope: only `https://github.com/...` repo URLs are authenticated this way. SSH URLs, GitHub
|
||||
Enterprise hosts, and other providers keep ambient behavior (system git config/credential
|
||||
helpers on the server host). URLs that embed their own credentials are never overridden.
|
||||
- Fallback: with no matching company secret, the server falls back to a `GITHUB_TOKEN` or
|
||||
`GH_TOKEN` variable in the **server process environment** (useful for self-hosted single-tenant
|
||||
deployments), then to unauthenticated access — public repos keep working with no setup.
|
||||
- The token never appears in command lines, URLs, or on disk; it is passed to git through an
|
||||
ephemeral credential helper. Each resolution is recorded as a secret access event.
|
||||
- This is separate from the **agent push credential**: agents pushing branches/PRs still need
|
||||
`GH_TOKEN`/`GITHUB_TOKEN` bound at agent or project scope (see
|
||||
[deploy/secrets](../../deploy/secrets.md)) so the token reaches the agent process env. The
|
||||
same company secret can back both uses via a binding.
|
||||
|
||||
## Cross-run persistence (no-remote-git contract)
|
||||
|
||||
Code state moves between runs through the local execution-workspace cwd alone — not through a git remote.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,285 @@
|
|||
import { spawn } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
DEFAULT_GITHUB_TOKEN_SECRET_NAMES,
|
||||
GIT_CREDENTIAL_TOKEN_ENV_KEY,
|
||||
buildGitAuthInvocation,
|
||||
createGitRemoteAuthProvider,
|
||||
describeGitAuthFailure,
|
||||
isGitHubHttpsRemoteUrl,
|
||||
scrubGitCredentialText,
|
||||
} from "../services/git-credentials.ts";
|
||||
|
||||
const fakeDb = null as unknown as Db;
|
||||
|
||||
function buildSecretsFake(byName: Record<string, string | Error>) {
|
||||
const getByName = vi.fn(async (_companyId: string, name: string) => {
|
||||
if (!(name in byName)) return null;
|
||||
return { id: `secret-${name}` };
|
||||
});
|
||||
const resolveSecretValue = vi.fn(async (_companyId: string, secretId: string) => {
|
||||
const name = secretId.replace(/^secret-/, "");
|
||||
const value = byName[name];
|
||||
if (value instanceof Error) throw value;
|
||||
return value ?? "";
|
||||
});
|
||||
return { getByName, resolveSecretValue };
|
||||
}
|
||||
|
||||
describe("isGitHubHttpsRemoteUrl", () => {
|
||||
it("accepts https github.com and www.github.com URLs", () => {
|
||||
expect(isGitHubHttpsRemoteUrl("https://github.com/example/repo.git")).toBe(true);
|
||||
expect(isGitHubHttpsRemoteUrl("https://www.github.com/example/repo.git")).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects ssh, http, enterprise hosts, other providers, userinfo URLs, and non-URLs", () => {
|
||||
expect(isGitHubHttpsRemoteUrl("git@github.com:example/repo.git")).toBe(false);
|
||||
expect(isGitHubHttpsRemoteUrl("ssh://git@github.com/example/repo.git")).toBe(false);
|
||||
expect(isGitHubHttpsRemoteUrl("http://github.com/example/repo.git")).toBe(false);
|
||||
expect(isGitHubHttpsRemoteUrl("https://github.enterprise.example/org/repo.git")).toBe(false);
|
||||
expect(isGitHubHttpsRemoteUrl("https://gitlab.com/example/repo.git")).toBe(false);
|
||||
expect(isGitHubHttpsRemoteUrl("https://alice:token@github.com/example/repo.git")).toBe(false);
|
||||
expect(isGitHubHttpsRemoteUrl("/local/path/repo.git")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("createGitRemoteAuthProvider", () => {
|
||||
const githubUrl = "https://github.com/example/repo.git";
|
||||
|
||||
it("prefers company secrets in declared order", async () => {
|
||||
const secrets = buildSecretsFake({ GH_TOKEN: "gh-token", PAPERCLIP_GITHUB_TOKEN: "pc-token" });
|
||||
const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, {
|
||||
secrets,
|
||||
env: { GITHUB_TOKEN: "env-token" },
|
||||
});
|
||||
const invocation = await provider(githubUrl);
|
||||
expect(invocation?.env[GIT_CREDENTIAL_TOKEN_ENV_KEY]).toBe("gh-token");
|
||||
expect(invocation?.source).toBe("company_secret");
|
||||
expect(invocation?.secretName).toBe("GH_TOKEN");
|
||||
// GITHUB_TOKEN is probed first even though only GH_TOKEN exists.
|
||||
expect(secrets.getByName.mock.calls.map((call) => call[1])).toEqual(["GITHUB_TOKEN", "GH_TOKEN"]);
|
||||
});
|
||||
|
||||
it("falls back to the server env, GITHUB_TOKEN before GH_TOKEN", async () => {
|
||||
const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, {
|
||||
secrets: buildSecretsFake({}),
|
||||
env: { GITHUB_TOKEN: "env-github", GH_TOKEN: "env-gh" },
|
||||
});
|
||||
const invocation = await provider(githubUrl);
|
||||
expect(invocation?.env[GIT_CREDENTIAL_TOKEN_ENV_KEY]).toBe("env-github");
|
||||
expect(invocation?.source).toBe("server_env");
|
||||
expect(invocation?.secretName).toBeNull();
|
||||
});
|
||||
|
||||
it("returns null when no token is available anywhere", async () => {
|
||||
const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, {
|
||||
secrets: buildSecretsFake({}),
|
||||
env: {},
|
||||
});
|
||||
await expect(provider(githubUrl)).resolves.toBeNull();
|
||||
});
|
||||
|
||||
it("returns null for out-of-scope URLs without touching the secret store", async () => {
|
||||
const secrets = buildSecretsFake({ GITHUB_TOKEN: "token" });
|
||||
const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, {
|
||||
secrets,
|
||||
env: {},
|
||||
});
|
||||
await expect(provider("git@github.com:example/repo.git")).resolves.toBeNull();
|
||||
await expect(provider("https://gitlab.com/example/repo.git")).resolves.toBeNull();
|
||||
expect(secrets.getByName).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("memoizes the credential lookup across calls", async () => {
|
||||
const secrets = buildSecretsFake({ GITHUB_TOKEN: "token" });
|
||||
const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, {
|
||||
secrets,
|
||||
env: {},
|
||||
});
|
||||
await provider(githubUrl);
|
||||
await provider(githubUrl);
|
||||
await provider("https://github.com/example/another.git");
|
||||
expect(secrets.getByName).toHaveBeenCalledTimes(1);
|
||||
expect(secrets.resolveSecretValue).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("passes a system access context so resolution is audited", async () => {
|
||||
const secrets = buildSecretsFake({ GITHUB_TOKEN: "token" });
|
||||
const provider = createGitRemoteAuthProvider(
|
||||
fakeDb,
|
||||
"company-1",
|
||||
{ issueId: "issue-1", heartbeatRunId: "run-1" },
|
||||
{ secrets, env: {} },
|
||||
);
|
||||
await provider(githubUrl);
|
||||
expect(secrets.resolveSecretValue).toHaveBeenCalledWith("company-1", "secret-GITHUB_TOKEN", "latest", {
|
||||
accessContext: expect.objectContaining({
|
||||
consumerType: "system",
|
||||
consumerId: "workspace-git-credential",
|
||||
actorType: "system",
|
||||
issueId: "issue-1",
|
||||
heartbeatRunId: "run-1",
|
||||
}),
|
||||
});
|
||||
});
|
||||
|
||||
it("continues down the chain when one secret fails to resolve", async () => {
|
||||
const secrets = buildSecretsFake({
|
||||
GITHUB_TOKEN: new Error("provider outage"),
|
||||
GH_TOKEN: "gh-token",
|
||||
});
|
||||
const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, {
|
||||
secrets,
|
||||
env: {},
|
||||
});
|
||||
const invocation = await provider(githubUrl);
|
||||
expect(invocation?.secretName).toBe("GH_TOKEN");
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildGitAuthInvocation", () => {
|
||||
it("keeps the token out of argv and installs the helper URL-scoped to github.com", () => {
|
||||
const invocation = buildGitAuthInvocation({
|
||||
token: "super-secret-token",
|
||||
source: "company_secret",
|
||||
secretName: "GITHUB_TOKEN",
|
||||
});
|
||||
expect(invocation.configArgs.join(" ")).not.toContain("super-secret-token");
|
||||
expect(invocation.configArgs[0]).toBe("-c");
|
||||
expect(invocation.configArgs[1]).toBe("credential.helper=");
|
||||
expect(invocation.configArgs[3]).toContain("credential.https://github.com.helper=");
|
||||
expect(invocation.configArgs[3]).toContain("x-access-token");
|
||||
expect(invocation.configArgs[5]).toContain("credential.https://www.github.com.helper=");
|
||||
expect(invocation.env[GIT_CREDENTIAL_TOKEN_ENV_KEY]).toBe("super-secret-token");
|
||||
expect(invocation.env.GIT_TERMINAL_PROMPT).toBe("0");
|
||||
});
|
||||
});
|
||||
|
||||
describe("credential helper execution (real git, no network)", () => {
|
||||
async function runCredentialFill(description: string) {
|
||||
const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-cred-fill-"));
|
||||
try {
|
||||
const invocation = buildGitAuthInvocation({
|
||||
token: "abc123",
|
||||
source: "company_secret",
|
||||
secretName: "GITHUB_TOKEN",
|
||||
});
|
||||
return await new Promise<{ code: number | null; stdout: string; stderr: string }>(
|
||||
(resolve, reject) => {
|
||||
const child = spawn("git", [...invocation.configArgs, "credential", "fill"], {
|
||||
cwd,
|
||||
env: { ...process.env, ...invocation.env },
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
});
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout.on("data", (chunk) => { stdout += String(chunk); });
|
||||
child.stderr.on("data", (chunk) => { stderr += String(chunk); });
|
||||
child.on("error", reject);
|
||||
child.on("close", (code) => resolve({ code, stdout, stderr }));
|
||||
child.stdin.write(description);
|
||||
child.stdin.end();
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
await fs.rm(cwd, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
it("answers a github.com https request with the env-carried token", async () => {
|
||||
const result = await runCredentialFill("protocol=https\nhost=github.com\n\n");
|
||||
expect(result.code).toBe(0);
|
||||
expect(result.stdout).toContain("username=x-access-token");
|
||||
expect(result.stdout).toContain("password=abc123");
|
||||
});
|
||||
|
||||
it("never hands the token to another host, even if git asks", async () => {
|
||||
// Simulates a request whose effective host changed after our pre-invocation URL check
|
||||
// (for example a repository-local url.<base>.insteadOf rewrite): the URL-scoped helper
|
||||
// config keeps git from consulting the helper, prompts are disabled, so the fill fails
|
||||
// and the token is never emitted.
|
||||
const result = await runCredentialFill("protocol=https\nhost=evil.example\n\n");
|
||||
expect(result.code).not.toBe(0);
|
||||
expect(result.stdout).not.toContain("abc123");
|
||||
});
|
||||
|
||||
it("never answers plain-http requests for github.com", async () => {
|
||||
const result = await runCredentialFill("protocol=http\nhost=github.com\n\n");
|
||||
expect(result.code).not.toBe(0);
|
||||
expect(result.stdout).not.toContain("abc123");
|
||||
});
|
||||
});
|
||||
|
||||
describe("scrubGitCredentialText", () => {
|
||||
it("masks URL userinfo", () => {
|
||||
expect(scrubGitCredentialText("https://x-access-token:ghp_secret@github.com/a/b.git")).toBe(
|
||||
"https://***@github.com/a/b.git",
|
||||
);
|
||||
});
|
||||
|
||||
it("masks userinfo on non-HTTP schemes, leaving scp-style remotes alone", () => {
|
||||
expect(scrubGitCredentialText("ssh://deploy:hunter2@internal.example/repo.git")).toBe(
|
||||
"ssh://***@internal.example/repo.git",
|
||||
);
|
||||
expect(scrubGitCredentialText("git@github.com:example/repo.git")).toBe(
|
||||
"git@github.com:example/repo.git",
|
||||
);
|
||||
});
|
||||
|
||||
it("masks entire URL query strings regardless of parameter names", () => {
|
||||
expect(scrubGitCredentialText("https://github.com/a/b.git?access_token=ghs_secret&ref=main")).toBe(
|
||||
"https://github.com/a/b.git?***",
|
||||
);
|
||||
expect(scrubGitCredentialText("https://host.example/r.git?obscure_cred_name=secret")).toBe(
|
||||
"https://host.example/r.git?***",
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves credential-free text unchanged", () => {
|
||||
expect(scrubGitCredentialText("fatal: repository not found")).toBe("fatal: repository not found");
|
||||
});
|
||||
});
|
||||
|
||||
describe("describeGitAuthFailure", () => {
|
||||
it("names the company secret when a stored credential was used", () => {
|
||||
expect(describeGitAuthFailure({
|
||||
error: "fatal: Authentication failed",
|
||||
used: { source: "company_secret", secretName: "GH_TOKEN" },
|
||||
})).toContain("the GH_TOKEN company-secret GitHub credential");
|
||||
});
|
||||
|
||||
it("names the server environment when an env credential was used", () => {
|
||||
expect(describeGitAuthFailure({
|
||||
error: "fatal: Authentication failed",
|
||||
used: { source: "server_env", secretName: null },
|
||||
})).toContain("server-environment GitHub credential");
|
||||
});
|
||||
|
||||
it("points at Settings → Secrets for auth-looking failures without a credential", () => {
|
||||
expect(describeGitAuthFailure({
|
||||
error: "fatal: could not read Username for 'https://github.com': terminal prompts disabled",
|
||||
used: null,
|
||||
})).toContain("add a GITHUB_TOKEN or GH_TOKEN company secret");
|
||||
});
|
||||
|
||||
it("stays silent for non-auth failures without a credential", () => {
|
||||
expect(describeGitAuthFailure({
|
||||
error: "fatal: unable to resolve host example.invalid",
|
||||
used: null,
|
||||
})).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("DEFAULT_GITHUB_TOKEN_SECRET_NAMES", () => {
|
||||
it("keeps the shared name order stable", () => {
|
||||
expect([...DEFAULT_GITHUB_TOKEN_SECRET_NAMES]).toEqual([
|
||||
"GITHUB_TOKEN",
|
||||
"GH_TOKEN",
|
||||
"PAPERCLIP_GITHUB_TOKEN",
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
|
@ -0,0 +1,161 @@
|
|||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { ensureManagedProjectWorkspace } from "../services/heartbeat.ts";
|
||||
import { buildGitAuthInvocation, GIT_CREDENTIAL_TOKEN_ENV_KEY } from "../services/git-credentials.ts";
|
||||
import { sanitizeRuntimeServiceBaseEnv } from "../services/workspace-runtime.ts";
|
||||
import { resolveManagedProjectWorkspaceDir } from "../home-paths.ts";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
|
||||
let tempHome: string;
|
||||
let originalHome: string | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
originalHome = process.env.PAPERCLIP_HOME;
|
||||
tempHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-managed-clone-"));
|
||||
process.env.PAPERCLIP_HOME = tempHome;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (originalHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = originalHome;
|
||||
await fs.rm(tempHome, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function createLocalSourceRepo() {
|
||||
const sourceRepo = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-clone-source-"));
|
||||
await execFile("git", ["init"], { cwd: sourceRepo });
|
||||
await execFile("git", ["config", "user.email", "paperclip@example.com"], { cwd: sourceRepo });
|
||||
await execFile("git", ["config", "user.name", "Paperclip Test"], { cwd: sourceRepo });
|
||||
await fs.writeFile(path.join(sourceRepo, "README.md"), "hello\n", "utf8");
|
||||
await execFile("git", ["add", "README.md"], { cwd: sourceRepo });
|
||||
await execFile("git", ["commit", "-m", "init"], { cwd: sourceRepo });
|
||||
return sourceRepo;
|
||||
}
|
||||
|
||||
describe("ensureManagedProjectWorkspace clone credentials", () => {
|
||||
it("clones exactly as before when no auth provider is configured", async () => {
|
||||
const sourceRepo = await createLocalSourceRepo();
|
||||
try {
|
||||
const result = await ensureManagedProjectWorkspace({
|
||||
companyId: "company-noauth",
|
||||
projectId: "project-1",
|
||||
repoUrl: sourceRepo,
|
||||
});
|
||||
expect(result.warning).toBeNull();
|
||||
const gitDir = await fs.stat(path.join(result.cwd, ".git"));
|
||||
expect(gitDir.isDirectory()).toBe(true);
|
||||
} finally {
|
||||
await fs.rm(sourceRepo, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("consults the provider with the repo URL and clones normally when it returns null", async () => {
|
||||
const sourceRepo = await createLocalSourceRepo();
|
||||
const resolveGitAuth = vi.fn(async () => null);
|
||||
try {
|
||||
const result = await ensureManagedProjectWorkspace({
|
||||
companyId: "company-nullauth",
|
||||
projectId: "project-1",
|
||||
repoUrl: sourceRepo,
|
||||
resolveGitAuth,
|
||||
});
|
||||
expect(resolveGitAuth).toHaveBeenCalledWith(sourceRepo);
|
||||
const gitDir = await fs.stat(path.join(result.cwd, ".git"));
|
||||
expect(gitDir.isDirectory()).toBe(true);
|
||||
} finally {
|
||||
await fs.rm(sourceRepo, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("names the company-secret credential when an authenticated clone fails", async () => {
|
||||
const missingRepo = path.join(os.tmpdir(), "paperclip-definitely-missing", "repo.git");
|
||||
const resolveGitAuth = vi.fn(async () => ({
|
||||
// Empty configArgs keep this offline: the failure comes from the missing local path,
|
||||
// the message must still attribute the credential that was in play.
|
||||
configArgs: [],
|
||||
env: { [GIT_CREDENTIAL_TOKEN_ENV_KEY]: "token", GIT_TERMINAL_PROMPT: "0" },
|
||||
source: "company_secret" as const,
|
||||
secretName: "GH_TOKEN",
|
||||
}));
|
||||
await expect(ensureManagedProjectWorkspace({
|
||||
companyId: "company-authfail",
|
||||
projectId: "project-1",
|
||||
repoUrl: missingRepo,
|
||||
resolveGitAuth,
|
||||
})).rejects.toThrow(/the GH_TOKEN company-secret GitHub credential/);
|
||||
});
|
||||
|
||||
it("does not mention credentials when an unauthenticated clone fails for non-auth reasons", async () => {
|
||||
// The Settings → Secrets hint is reserved for auth-shaped failures (covered in
|
||||
// git-credentials.test.ts); a plain missing-repo failure must not suggest credentials.
|
||||
const missingRepo = path.join(os.tmpdir(), "paperclip-definitely-missing", "repo.git");
|
||||
const error = await ensureManagedProjectWorkspace({
|
||||
companyId: "company-noauthfail",
|
||||
projectId: "project-1",
|
||||
repoUrl: missingRepo,
|
||||
}).then(
|
||||
() => { throw new Error("expected the clone to fail"); },
|
||||
(err: unknown) => err as Error,
|
||||
);
|
||||
expect(error.message).toContain("Failed to prepare managed checkout");
|
||||
expect(error.message).not.toContain("company secret");
|
||||
});
|
||||
|
||||
it("removes the partially created directory when the clone fails", async () => {
|
||||
const missingRepo = path.join(os.tmpdir(), "paperclip-definitely-missing", "repo.git");
|
||||
const companyId = "company-cleanup";
|
||||
const projectId = "project-1";
|
||||
await expect(ensureManagedProjectWorkspace({
|
||||
companyId,
|
||||
projectId,
|
||||
repoUrl: missingRepo,
|
||||
})).rejects.toThrow();
|
||||
// Filesystem-path repo "URLs" derive no repo name, so the managed dir is the _default slot.
|
||||
const cwd = resolveManagedProjectWorkspaceDir({ companyId, projectId });
|
||||
await expect(fs.stat(cwd)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
|
||||
it("keeps using a pre-existing non-git directory as-is without attempting a clone", async () => {
|
||||
const companyId = "company-existing";
|
||||
const projectId = "project-1";
|
||||
const sourceRepo = await createLocalSourceRepo();
|
||||
try {
|
||||
const cwd = resolveManagedProjectWorkspaceDir({ companyId, projectId });
|
||||
await fs.mkdir(cwd, { recursive: true });
|
||||
await fs.writeFile(path.join(cwd, "keep.txt"), "operator data\n", "utf8");
|
||||
const result = await ensureManagedProjectWorkspace({
|
||||
companyId,
|
||||
projectId,
|
||||
repoUrl: sourceRepo,
|
||||
});
|
||||
expect(result.cwd).toBe(cwd);
|
||||
expect(result.warning).toContain("Using it as-is");
|
||||
await expect(fs.readFile(path.join(cwd, "keep.txt"), "utf8")).resolves.toBe("operator data\n");
|
||||
} finally {
|
||||
await fs.rm(sourceRepo, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps the credential env alive through the sanitizer spread order", () => {
|
||||
// The clone env is `{ ...sanitize(process.env), GIT_TERMINAL_PROMPT, ...auth.env }`. The
|
||||
// sanitizer strips every PAPERCLIP_* key, so the token env must be spread after it.
|
||||
const invocation = buildGitAuthInvocation({
|
||||
token: "tok",
|
||||
source: "company_secret",
|
||||
secretName: "GITHUB_TOKEN",
|
||||
});
|
||||
const cloneEnv = {
|
||||
...sanitizeRuntimeServiceBaseEnv({ ...process.env, [GIT_CREDENTIAL_TOKEN_ENV_KEY]: "stale" }),
|
||||
GIT_TERMINAL_PROMPT: "0",
|
||||
...invocation.env,
|
||||
};
|
||||
expect(cloneEnv[GIT_CREDENTIAL_TOKEN_ENV_KEY]).toBe("tok");
|
||||
expect(sanitizeRuntimeServiceBaseEnv({ [GIT_CREDENTIAL_TOKEN_ENV_KEY]: "stale" })[GIT_CREDENTIAL_TOKEN_ENV_KEY])
|
||||
.toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
|
@ -34,6 +34,7 @@ import {
|
|||
normalizeAdapterManagedRuntimeServices,
|
||||
reconcilePersistedRuntimeServicesOnStartup,
|
||||
realizeExecutionWorkspace,
|
||||
refreshRemoteTrackingBaseRef,
|
||||
releaseRuntimeServicesForRun,
|
||||
resetRuntimeServicesForTests,
|
||||
resolveWorkspaceRuntimeReadinessTimeoutSec,
|
||||
|
|
@ -379,6 +380,42 @@ describe("sanitizeRuntimeServiceBaseEnv", () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe("refreshRemoteTrackingBaseRef git auth", () => {
|
||||
it("offers the remote URL to the provider and keeps ambient behavior when it returns null", async () => {
|
||||
const { remotePath, repoRoot } = await createClonedRepoWithRemote();
|
||||
const offeredUrls: string[] = [];
|
||||
const warnings = await refreshRemoteTrackingBaseRef(repoRoot, "origin/master", async (remoteUrl) => {
|
||||
offeredUrls.push(remoteUrl);
|
||||
return null;
|
||||
});
|
||||
expect(warnings).toEqual([]);
|
||||
expect(offeredUrls).toEqual([remotePath]);
|
||||
});
|
||||
|
||||
it("attributes a failed authenticated fetch to the credential that was used", async () => {
|
||||
const { repoRoot } = await createClonedRepoWithRemote();
|
||||
await runGit(repoRoot, ["remote", "set-url", "origin", path.join(os.tmpdir(), "paperclip-missing-remote", "repo.git")]);
|
||||
const warnings = await refreshRemoteTrackingBaseRef(repoRoot, "origin/master", async () => ({
|
||||
configArgs: [],
|
||||
env: { GIT_TERMINAL_PROMPT: "0" },
|
||||
source: "company_secret",
|
||||
secretName: "GH_TOKEN",
|
||||
}));
|
||||
expect(warnings).toHaveLength(1);
|
||||
expect(warnings[0]).toContain("Could not refresh base ref origin/master");
|
||||
expect(warnings[0]).toContain("the GH_TOKEN company-secret GitHub credential");
|
||||
});
|
||||
|
||||
it("keeps the unauthenticated failure warning credential-free without a provider", async () => {
|
||||
const { repoRoot } = await createClonedRepoWithRemote();
|
||||
await runGit(repoRoot, ["remote", "set-url", "origin", path.join(os.tmpdir(), "paperclip-missing-remote", "repo.git")]);
|
||||
const warnings = await refreshRemoteTrackingBaseRef(repoRoot, "origin/master");
|
||||
expect(warnings).toHaveLength(1);
|
||||
expect(warnings[0]).toContain("Could not refresh base ref origin/master");
|
||||
expect(warnings[0]).not.toContain("GitHub credential");
|
||||
});
|
||||
});
|
||||
|
||||
describe("ensureServerWorkspaceLinksCurrent", () => {
|
||||
it("relinks stale server workspace dependencies inside the current repo root", async () => {
|
||||
const repoRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-links-"));
|
||||
|
|
|
|||
|
|
@ -0,0 +1,202 @@
|
|||
import type { Db } from "@paperclipai/db";
|
||||
import { isGitHubDotCom } from "./github-fetch.js";
|
||||
import { secretService } from "./secrets.js";
|
||||
|
||||
/**
|
||||
* Server-side git credentials for managed project checkouts and execution-workspace base
|
||||
* refreshes. Operators store a GitHub token as a company secret under one of the well-known
|
||||
* names below (the same convention the GitHub external-object provider reads); this module
|
||||
* resolves it and turns it into a git invocation that authenticates clone/fetch against
|
||||
* github.com over HTTPS without ever placing the token in argv, URLs, or on disk.
|
||||
*
|
||||
* The provider factory is deliberately the single seam for future credential sources (for
|
||||
* example a brokered GitHub connection): swap the factory, keep every call site unchanged.
|
||||
*/
|
||||
|
||||
/** Company-secret names probed for a GitHub token, in priority order. */
|
||||
export const DEFAULT_GITHUB_TOKEN_SECRET_NAMES = ["GITHUB_TOKEN", "GH_TOKEN", "PAPERCLIP_GITHUB_TOKEN"] as const;
|
||||
|
||||
/** Env var the credential helper reads the token from; never appears in argv. */
|
||||
export const GIT_CREDENTIAL_TOKEN_ENV_KEY = "PAPERCLIP_GIT_TOKEN";
|
||||
|
||||
// `!`-prefixed helpers run via `sh -c` with the credential action appended as "$1". Only the
|
||||
// `get` action answers; store/erase drain stdin and exit 0 silently. `x-access-token`
|
||||
// authenticates classic PATs, fine-grained PATs, and GitHub App installation tokens alike.
|
||||
//
|
||||
// The helper re-validates the credential request from its stdin description and answers only
|
||||
// for `protocol=https` + `host=github.com`/`www.github.com`. The pre-invocation URL check
|
||||
// runs before git applies configuration like repository-local `url.<base>.insteadOf`
|
||||
// rewrites, so a rewritten remote could otherwise request the token for an arbitrary host.
|
||||
// The helper is additionally installed URL-scoped (`credential.https://github.com.helper`)
|
||||
// so git does not consult it for other hosts in the first place — two independent gates.
|
||||
const GIT_CREDENTIAL_HELPER =
|
||||
`!f() { ok=; proto=; while IFS= read -r l && [ -n "$l" ]; do case "$l" in host=github.com|host=www.github.com) ok=1;; protocol=https) proto=1;; esac; done; if [ "$1" = get ] && [ -n "$ok" ] && [ -n "$proto" ]; then printf 'username=x-access-token\\npassword=%s\\n' "$PAPERCLIP_GIT_TOKEN"; fi; }; f`;
|
||||
|
||||
export type GitCredential = {
|
||||
token: string;
|
||||
source: "company_secret" | "server_env";
|
||||
/** The company-secret name the token came from; null for a server-environment token. */
|
||||
secretName: string | null;
|
||||
};
|
||||
|
||||
/** A prepared, credential-bearing git invocation: config args plus the env that carries the token. */
|
||||
export type GitAuthInvocation = {
|
||||
configArgs: string[];
|
||||
env: Record<string, string>;
|
||||
source: GitCredential["source"];
|
||||
secretName: string | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Resolve auth for one remote URL. Returns null when the URL is out of scope (non-GitHub,
|
||||
* ssh, or already credentialed) or when no token is available — callers then run git with
|
||||
* ambient behavior, exactly as before this module existed.
|
||||
*/
|
||||
export type GitRemoteAuthProvider = (remoteUrl: string) => Promise<GitAuthInvocation | null>;
|
||||
|
||||
/**
|
||||
* True only for `https://github.com/...` (or `www.`) URLs without inline userinfo. GHES and
|
||||
* other hosts are out of scope for now — sending a github.com token to an arbitrary host
|
||||
* would leak it, and an operator's inline URL credential must never be overridden.
|
||||
*/
|
||||
export function isGitHubHttpsRemoteUrl(remoteUrl: string): boolean {
|
||||
let parsed: URL;
|
||||
try {
|
||||
parsed = new URL(remoteUrl);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
if (parsed.protocol !== "https:") return false;
|
||||
if (parsed.username || parsed.password) return false;
|
||||
return isGitHubDotCom(parsed.hostname);
|
||||
}
|
||||
|
||||
/**
|
||||
* Mask credential material embedded in URLs so it never reaches warnings, run errors, or
|
||||
* persisted payloads: userinfo on any scheme (`https://user:token@host`,
|
||||
* `ssh://user:pass@host`) and the entire query string of any URL (`?access_token=…` and
|
||||
* every other parameter — masked wholesale rather than by an inevitably incomplete
|
||||
* parameter-name list). Scp-style remotes (`git@host:path`) carry no password and are left
|
||||
* alone.
|
||||
*/
|
||||
export function scrubGitCredentialText(text: string): string {
|
||||
return text
|
||||
.replace(/([a-z][a-z0-9+.-]*:\/\/)[^/@\s]+@/gi, "$1***@")
|
||||
.replace(/([a-z][a-z0-9+.-]*:\/\/[^\s"'?]*)\?[^\s"']*/gi, "$1?***");
|
||||
}
|
||||
|
||||
export function buildGitAuthInvocation(credential: GitCredential): GitAuthInvocation {
|
||||
return {
|
||||
// The leading empty helper clears ambient helpers (gh, osxkeychain, credential-store) so
|
||||
// they neither outrank the resolved token nor receive store/erase callbacks for it. The
|
||||
// token helper is installed URL-scoped: git consults it only for credential requests
|
||||
// whose context matches github.com over https, so an `insteadOf`-rewritten remote never
|
||||
// reaches it (and the helper itself re-checks the request host — see above).
|
||||
configArgs: [
|
||||
"-c", "credential.helper=",
|
||||
"-c", `credential.https://github.com.helper=${GIT_CREDENTIAL_HELPER}`,
|
||||
"-c", `credential.https://www.github.com.helper=${GIT_CREDENTIAL_HELPER}`,
|
||||
],
|
||||
env: {
|
||||
[GIT_CREDENTIAL_TOKEN_ENV_KEY]: credential.token,
|
||||
GIT_TERMINAL_PROMPT: "0",
|
||||
},
|
||||
source: credential.source,
|
||||
secretName: credential.secretName,
|
||||
};
|
||||
}
|
||||
|
||||
const GIT_AUTH_FAILURE_PATTERN =
|
||||
/authentication failed|could not read username|could not read password|invalid username or password|terminal prompts disabled|repository not found|not accessible|permission denied|HTTP 40[13]|The requested URL returned error: 40[13]/i;
|
||||
|
||||
/**
|
||||
* Turn a failed authenticated (or unauthenticated) git network operation into an actionable
|
||||
* suffix for the error message. Returns null when the failure does not look auth-related and
|
||||
* no credential was in play.
|
||||
*/
|
||||
export function describeGitAuthFailure(input: {
|
||||
error: string;
|
||||
used: { source: GitCredential["source"]; secretName: string | null } | null;
|
||||
}): string | null {
|
||||
if (input.used) {
|
||||
const label = input.used.secretName
|
||||
? `the ${input.used.secretName} company-secret GitHub credential`
|
||||
: "the server-environment GitHub credential";
|
||||
return `The operation authenticated with ${label}, which was rejected or lacks access to this repository.`;
|
||||
}
|
||||
if (GIT_AUTH_FAILURE_PATTERN.test(input.error)) {
|
||||
return "No GitHub credential is configured — add a GITHUB_TOKEN or GH_TOKEN company secret in Settings → Secrets, or configure a local checkout cwd for this project workspace.";
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
type SecretServiceLike = ReturnType<typeof secretService>;
|
||||
|
||||
type GitCredentialSecretsDeps = {
|
||||
getByName: (
|
||||
companyId: string,
|
||||
name: string,
|
||||
) => Promise<{ id: string } | null | undefined> | ReturnType<SecretServiceLike["getByName"]>;
|
||||
resolveSecretValue: SecretServiceLike["resolveSecretValue"];
|
||||
};
|
||||
|
||||
/**
|
||||
* Build the credential provider for one run. Resolution order: company secret by well-known
|
||||
* name, then the server process env (`GITHUB_TOKEN`/`GH_TOKEN`) for self-hosted operators,
|
||||
* then null. The lookup is memoized per provider instance so one run performs at most one
|
||||
* secret resolution (and writes at most one audit event) no matter how many git operations
|
||||
* it authenticates.
|
||||
*/
|
||||
export function createGitRemoteAuthProvider(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
context?: {
|
||||
issueId?: string | null;
|
||||
heartbeatRunId?: string | null;
|
||||
responsibleUserId?: string | null;
|
||||
},
|
||||
deps?: {
|
||||
secrets?: GitCredentialSecretsDeps;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
secretNames?: readonly string[];
|
||||
},
|
||||
): GitRemoteAuthProvider {
|
||||
const secrets: GitCredentialSecretsDeps = deps?.secrets ?? secretService(db);
|
||||
const env = deps?.env ?? process.env;
|
||||
const secretNames = deps?.secretNames ?? DEFAULT_GITHUB_TOKEN_SECRET_NAMES;
|
||||
let credentialPromise: Promise<GitCredential | null> | null = null;
|
||||
|
||||
const resolveCredential = async (): Promise<GitCredential | null> => {
|
||||
for (const secretName of secretNames) {
|
||||
const secret = await Promise.resolve(secrets.getByName(companyId, secretName)).catch(() => null);
|
||||
if (!secret) continue;
|
||||
// A resolution failure (inactive secret, provider outage) records its own failure audit
|
||||
// event; fall through to the next source instead of failing the whole git operation here.
|
||||
const token = await secrets
|
||||
.resolveSecretValue(companyId, secret.id, "latest", {
|
||||
accessContext: {
|
||||
consumerType: "system",
|
||||
consumerId: "workspace-git-credential",
|
||||
actorType: "system",
|
||||
issueId: context?.issueId ?? null,
|
||||
heartbeatRunId: context?.heartbeatRunId ?? null,
|
||||
responsibleUserId: context?.responsibleUserId ?? null,
|
||||
},
|
||||
})
|
||||
.then((value) => value.trim())
|
||||
.catch(() => "");
|
||||
if (token) return { token, source: "company_secret", secretName };
|
||||
}
|
||||
const envToken = env.GITHUB_TOKEN?.trim() || env.GH_TOKEN?.trim() || "";
|
||||
if (envToken) return { token: envToken, source: "server_env", secretName: null };
|
||||
return null;
|
||||
};
|
||||
|
||||
return async (remoteUrl: string) => {
|
||||
if (!isGitHubHttpsRemoteUrl(remoteUrl)) return null;
|
||||
credentialPromise ??= resolveCredential();
|
||||
const credential = await credentialPromise;
|
||||
if (!credential) return null;
|
||||
return buildGitAuthInvocation(credential);
|
||||
};
|
||||
}
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
import type { Db } from "@paperclipai/db";
|
||||
import type { ExternalObjectCanonicalUrl } from "@paperclipai/shared";
|
||||
import { DEFAULT_GITHUB_TOKEN_SECRET_NAMES } from "./git-credentials.js";
|
||||
import { ghFetch, gitHubApiBase } from "./github-fetch.js";
|
||||
import { secretService } from "./secrets.js";
|
||||
import type {
|
||||
|
|
@ -27,7 +28,6 @@ interface GitHubObjectIdentity {
|
|||
pathKind: "pull" | "issues";
|
||||
}
|
||||
|
||||
const DEFAULT_GITHUB_TOKEN_SECRET_NAMES = ["GITHUB_TOKEN", "GH_TOKEN", "PAPERCLIP_GITHUB_TOKEN"] as const;
|
||||
const GITHUB_OBJECT_TTL_SECONDS = 300;
|
||||
|
||||
function isGitHubHost(host: string) {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import { unprocessable } from "../errors.js";
|
||||
|
||||
function isGitHubDotCom(hostname: string) {
|
||||
export function isGitHubDotCom(hostname: string) {
|
||||
const h = hostname.toLowerCase();
|
||||
return h === "github.com" || h === "www.github.com";
|
||||
}
|
||||
|
|
|
|||
|
|
@ -70,6 +70,12 @@ import {
|
|||
import { conflict, HttpError, notFound } from "../errors.js";
|
||||
import { getStartupTraceContext } from "../instrumentation.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import {
|
||||
createGitRemoteAuthProvider,
|
||||
describeGitAuthFailure,
|
||||
scrubGitCredentialText,
|
||||
type GitRemoteAuthProvider,
|
||||
} from "./git-credentials.js";
|
||||
import { publishLiveEvent } from "./live-events.js";
|
||||
import { normalizeResponsibleUserDenialCode } from "./responsible-user-denial-run-outcomes.js";
|
||||
import { getRunLogStore, type RunLogHandle } from "./run-log-store.js";
|
||||
|
|
@ -1489,10 +1495,12 @@ function deriveRepoNameFromRepoUrl(repoUrl: string | null): string | null {
|
|||
}
|
||||
}
|
||||
|
||||
async function ensureManagedProjectWorkspace(input: {
|
||||
export async function ensureManagedProjectWorkspace(input: {
|
||||
companyId: string;
|
||||
projectId: string;
|
||||
repoUrl: string | null;
|
||||
/** Optional git credential source for cloning private repos; null/absent preserves ambient behavior. */
|
||||
resolveGitAuth?: GitRemoteAuthProvider | null;
|
||||
}): Promise<{ cwd: string; warning: string | null }> {
|
||||
const cwd = resolveManagedProjectWorkspaceDir({
|
||||
companyId: input.companyId,
|
||||
|
|
@ -1528,15 +1536,34 @@ async function ensureManagedProjectWorkspace(input: {
|
|||
await fs.rm(cwd, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
const auth = input.resolveGitAuth ? await input.resolveGitAuth(input.repoUrl) : null;
|
||||
try {
|
||||
await execFile("git", ["clone", input.repoUrl, cwd], {
|
||||
env: sanitizeRuntimeServiceBaseEnv(process.env),
|
||||
await execFile("git", [...(auth?.configArgs ?? []), "clone", input.repoUrl, cwd], {
|
||||
env: {
|
||||
// Spread order matters: the sanitizer strips PAPERCLIP_*, which would remove the
|
||||
// credential-helper token env if it came first. GIT_TERMINAL_PROMPT=0 fails a
|
||||
// credential-less private clone immediately instead of hanging on a prompt until
|
||||
// the clone timeout.
|
||||
...sanitizeRuntimeServiceBaseEnv(process.env),
|
||||
GIT_TERMINAL_PROMPT: "0",
|
||||
...(auth?.env ?? {}),
|
||||
},
|
||||
timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS,
|
||||
});
|
||||
return { cwd, warning: null };
|
||||
} catch (error) {
|
||||
// The clone only ever starts from a missing or just-emptied directory, so removing it on
|
||||
// failure cannot destroy operator data — but leaving it would poison the next run: a
|
||||
// timeout-killed clone leaves a partial .git that the gitDirExists probe above adopts.
|
||||
await fs.rm(cwd, { recursive: true, force: true }).catch(() => undefined);
|
||||
const reason = error instanceof Error ? error.message : String(error);
|
||||
throw new Error(`Failed to prepare managed checkout for "${input.repoUrl}" at "${cwd}": ${reason}`);
|
||||
const authNote = describeGitAuthFailure({
|
||||
error: reason,
|
||||
used: auth ? { source: auth.source, secretName: auth.secretName } : null,
|
||||
});
|
||||
throw new Error(scrubGitCredentialText(
|
||||
`Failed to prepare managed checkout for "${input.repoUrl}" at "${cwd}": ${reason}${authNote ? ` ${authNote}` : ""}`,
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1551,6 +1578,7 @@ async function resolveConfiguredOrManagedProjectCwd(input: {
|
|||
projectId: string;
|
||||
cwd: string | null;
|
||||
repoUrl: string | null;
|
||||
resolveGitAuth?: GitRemoteAuthProvider | null;
|
||||
}): Promise<{ cwd: string; warning: string | null }> {
|
||||
const configuredCwd = readNonEmptyString(input.cwd);
|
||||
if (configuredCwd && configuredCwd !== REPO_ONLY_CWD_SENTINEL) {
|
||||
|
|
@ -1560,6 +1588,7 @@ async function resolveConfiguredOrManagedProjectCwd(input: {
|
|||
companyId: input.companyId,
|
||||
projectId: input.projectId,
|
||||
repoUrl: readNonEmptyString(input.repoUrl),
|
||||
resolveGitAuth: input.resolveGitAuth ?? null,
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -1587,8 +1616,16 @@ function defaultAdditionalProjectWorkspaceDeps(db: Db): ResolveAdditionalProject
|
|||
.from(projectWorkspaces)
|
||||
.where(and(eq(projectWorkspaces.companyId, companyId), eq(projectWorkspaces.projectId, projectId)))
|
||||
.orderBy(asc(projectWorkspaces.createdAt), asc(projectWorkspaces.id)),
|
||||
resolveConfiguredOrManagedProjectCwd,
|
||||
ensureManagedProjectWorkspace,
|
||||
resolveConfiguredOrManagedProjectCwd: (input) =>
|
||||
resolveConfiguredOrManagedProjectCwd({
|
||||
...input,
|
||||
resolveGitAuth: input.resolveGitAuth ?? createGitRemoteAuthProvider(db, input.companyId),
|
||||
}),
|
||||
ensureManagedProjectWorkspace: (input) =>
|
||||
ensureManagedProjectWorkspace({
|
||||
...input,
|
||||
resolveGitAuth: input.resolveGitAuth ?? createGitRemoteAuthProvider(db, input.companyId),
|
||||
}),
|
||||
// A realized workspace must hold real content. An empty directory gives the agent an empty
|
||||
// referenced workspace, so treat an empty directory the same as a missing one.
|
||||
directoryHasContents: async (cwd) => {
|
||||
|
|
@ -8419,6 +8456,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
preferredWorkspaceWarning =
|
||||
`Selected project workspace "${preferredProjectWorkspaceId}" is not available on this project.`;
|
||||
}
|
||||
const resolveGitAuth = createGitRemoteAuthProvider(db, agent.companyId, { issueId });
|
||||
for (const workspace of projectWorkspaceRows) {
|
||||
let projectCwd: string;
|
||||
let managedWorkspaceWarning: string | null = null;
|
||||
|
|
@ -8428,6 +8466,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
projectId: workspaceProjectId ?? resolvedProjectId ?? workspace.projectId,
|
||||
cwd: workspace.cwd,
|
||||
repoUrl: workspace.repoUrl,
|
||||
resolveGitAuth,
|
||||
});
|
||||
projectCwd = resolvedCwd.cwd;
|
||||
managedWorkspaceWarning = resolvedCwd.warning;
|
||||
|
|
@ -14050,6 +14089,13 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
: null,
|
||||
issueId,
|
||||
});
|
||||
// One credential provider per run: base-ref refreshes during workspace realization and
|
||||
// restore authenticate against private GitHub remotes with the same company-secret token
|
||||
// the managed clone uses.
|
||||
const workspaceGitAuthProvider = createGitRemoteAuthProvider(db, agent.companyId, {
|
||||
issueId,
|
||||
heartbeatRunId: run.id,
|
||||
});
|
||||
const { executionWorkspace, reusedExecutionWorkspace, policy: resolvedWorkspaceReusePolicy } =
|
||||
await provisionExecutionWorkspaceForFreshnessDecision<RealizedExecutionWorkspace>({
|
||||
requestedShouldReuseExisting,
|
||||
|
|
@ -14098,6 +14144,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
enableWorkspaceDirtyQuarantineRepair:
|
||||
resolvedInstanceSettings.experimental.enableWorkspaceDirtyQuarantineRepair,
|
||||
recorder: workspaceOperationRecorder,
|
||||
resolveGitAuth: workspaceGitAuthProvider,
|
||||
})
|
||||
: null,
|
||||
realizeWorkspace: () => realizeExecutionWorkspace({
|
||||
|
|
@ -14116,6 +14163,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||
enableWorkspaceDirtyQuarantineRepair:
|
||||
resolvedInstanceSettings.experimental.enableWorkspaceDirtyQuarantineRepair,
|
||||
recorder: workspaceOperationRecorder,
|
||||
resolveGitAuth: workspaceGitAuthProvider,
|
||||
}),
|
||||
});
|
||||
const resolvedProjectId = executionWorkspace.projectId ?? issueRef?.projectId ?? executionProjectId ?? null;
|
||||
|
|
|
|||
|
|
@ -77,6 +77,21 @@ export interface ExecutionWorkspaceInput {
|
|||
additionalWorkspaces?: ExecutionWorkspaceAdditionalInput[];
|
||||
}
|
||||
|
||||
/**
|
||||
* A prepared credential-bearing git invocation for one remote URL, or null to keep ambient
|
||||
* behavior. Structurally compatible with the provider built by `git-credentials.ts` — this
|
||||
* module deliberately takes prepared invocations rather than tokens, so it never imports the
|
||||
* secrets layer and test fakes stay trivial.
|
||||
*/
|
||||
export type GitRemoteAuthInvocation = {
|
||||
configArgs: string[];
|
||||
env: Record<string, string>;
|
||||
source?: string;
|
||||
secretName?: string | null;
|
||||
};
|
||||
|
||||
export type GitRemoteAuthProvider = (remoteUrl: string) => Promise<GitRemoteAuthInvocation | null>;
|
||||
|
||||
export interface ExecutionWorkspaceIssueRef {
|
||||
id: string;
|
||||
identifier: string | null;
|
||||
|
|
@ -562,11 +577,12 @@ async function executeProcess(input: {
|
|||
};
|
||||
}
|
||||
|
||||
async function runGit(args: string[], cwd: string): Promise<string> {
|
||||
async function runGit(args: string[], cwd: string, opts?: { env?: NodeJS.ProcessEnv }): Promise<string> {
|
||||
const proc = await executeProcess({
|
||||
command: "git",
|
||||
args,
|
||||
cwd,
|
||||
env: opts?.env,
|
||||
});
|
||||
if (proc.code !== 0) {
|
||||
throw new Error(proc.stderr.trim() || proc.stdout.trim() || `git ${args.join(" ")} failed`);
|
||||
|
|
@ -597,26 +613,40 @@ function parseRemoteTrackingRef(ref: string): { remote: string; branch: string }
|
|||
return { remote, branch };
|
||||
}
|
||||
|
||||
async function refreshRemoteTrackingBaseRef(repoRoot: string, baseRef: string): Promise<string[]> {
|
||||
export async function refreshRemoteTrackingBaseRef(
|
||||
repoRoot: string,
|
||||
baseRef: string,
|
||||
resolveGitAuth?: GitRemoteAuthProvider | null,
|
||||
): Promise<string[]> {
|
||||
const remoteTracking = parseRemoteTrackingRef(baseRef);
|
||||
if (!remoteTracking) return [];
|
||||
|
||||
const remoteExists = await runGit(["remote", "get-url", remoteTracking.remote], repoRoot)
|
||||
.then(() => true)
|
||||
.catch(() => false);
|
||||
if (!remoteExists) return [];
|
||||
const remoteUrl = await runGit(["remote", "get-url", remoteTracking.remote], repoRoot)
|
||||
.then((value) => value.trim() || null)
|
||||
.catch(() => null);
|
||||
if (!remoteUrl) return [];
|
||||
|
||||
const auth = resolveGitAuth ? await resolveGitAuth(remoteUrl).catch(() => null) : null;
|
||||
try {
|
||||
await runGit([
|
||||
...(auth?.configArgs ?? []),
|
||||
"fetch",
|
||||
"--prune",
|
||||
remoteTracking.remote,
|
||||
`+refs/heads/${remoteTracking.branch}:refs/remotes/${remoteTracking.remote}/${remoteTracking.branch}`,
|
||||
], repoRoot);
|
||||
], repoRoot, auth ? { env: { ...process.env, ...auth.env } } : undefined);
|
||||
return [];
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
return [`Could not refresh base ref ${baseRef} before preparing the execution workspace: ${message}`];
|
||||
const rawMessage = error instanceof Error ? error.message : String(error);
|
||||
// Mask URL userinfo (any scheme) and whole URL query strings before the message rides
|
||||
// warnings that reach run logs.
|
||||
const message = rawMessage
|
||||
.replace(/([a-z][a-z0-9+.-]*:\/\/)[^/@\s]+@/gi, "$1***@")
|
||||
.replace(/([a-z][a-z0-9+.-]*:\/\/[^\s"'?]*)\?[^\s"']*/gi, "$1?***");
|
||||
const authNote = auth
|
||||
? ` The fetch authenticated with ${auth.secretName ? `the ${auth.secretName} company-secret GitHub credential` : "the server-environment GitHub credential"}, which may have been rejected.`
|
||||
: "";
|
||||
return [`Could not refresh base ref ${baseRef} before preparing the execution workspace: ${message}${authNote}`];
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -637,6 +667,7 @@ export async function inspectExecutionWorkspaceBaseDrift(input: {
|
|||
baseRef: string | null;
|
||||
recordedBaseRefSha?: string | null;
|
||||
skipRefresh?: boolean;
|
||||
resolveGitAuth?: GitRemoteAuthProvider | null;
|
||||
}): Promise<{
|
||||
warnings: string[];
|
||||
currentBaseRefSha: string | null;
|
||||
|
|
@ -647,7 +678,9 @@ export async function inspectExecutionWorkspaceBaseDrift(input: {
|
|||
return { warnings: [], currentBaseRefSha: null, branchBaseRefSha: null };
|
||||
}
|
||||
|
||||
const warnings = input.skipRefresh ? [] : await refreshRemoteTrackingBaseRef(input.repoRoot, baseRef);
|
||||
const warnings = input.skipRefresh
|
||||
? []
|
||||
: await refreshRemoteTrackingBaseRef(input.repoRoot, baseRef, input.resolveGitAuth);
|
||||
const currentBaseRefSha = await resolveBaseRefSha(input.repoRoot, baseRef);
|
||||
if (!currentBaseRefSha) {
|
||||
warnings.push(`Could not resolve base ref ${baseRef} while checking execution workspace freshness.`);
|
||||
|
|
@ -1991,9 +2024,10 @@ export async function ensureGitWorktreeBranchCoherent(input: {
|
|||
async function resolveAuthoritativeBaseRef(
|
||||
repoRoot: string,
|
||||
configuredBaseRef: string | null,
|
||||
resolveGitAuth?: GitRemoteAuthProvider | null,
|
||||
): Promise<{ baseRef: string; warnings: string[]; refreshed: boolean }> {
|
||||
const warnings: string[] = [];
|
||||
const detectOrHead = async () => (await detectDefaultBranch(repoRoot)) ?? "HEAD";
|
||||
const detectOrHead = async () => (await detectDefaultBranch(repoRoot, resolveGitAuth)) ?? "HEAD";
|
||||
|
||||
const configured = configuredBaseRef?.trim();
|
||||
if (!configured || configured === "HEAD") {
|
||||
|
|
@ -2008,7 +2042,7 @@ async function resolveAuthoritativeBaseRef(
|
|||
const remoteCandidate = `origin/${configured}`;
|
||||
// Refresh here and keep the warnings; the caller skips its own refresh of
|
||||
// the returned ref (see `refreshed`) so we never fetch the same ref twice.
|
||||
warnings.push(...await refreshRemoteTrackingBaseRef(repoRoot, remoteCandidate));
|
||||
warnings.push(...await refreshRemoteTrackingBaseRef(repoRoot, remoteCandidate, resolveGitAuth));
|
||||
if (await resolveBaseRefSha(repoRoot, remoteCandidate)) {
|
||||
return { baseRef: remoteCandidate, warnings, refreshed: true };
|
||||
}
|
||||
|
|
@ -2179,9 +2213,12 @@ async function isGitCheckout(cwd: string): Promise<boolean> {
|
|||
return Boolean(await runGit(["rev-parse", "--git-dir"], cwd).catch(() => null));
|
||||
}
|
||||
|
||||
async function detectDefaultBranch(repoRoot: string): Promise<string | null> {
|
||||
async function detectDefaultBranch(
|
||||
repoRoot: string,
|
||||
resolveGitAuth?: GitRemoteAuthProvider | null,
|
||||
): Promise<string | null> {
|
||||
const originMasterRef = "origin/master";
|
||||
await refreshRemoteTrackingBaseRef(repoRoot, originMasterRef);
|
||||
await refreshRemoteTrackingBaseRef(repoRoot, originMasterRef, resolveGitAuth);
|
||||
if (await resolveBaseRefSha(repoRoot, originMasterRef)) {
|
||||
return originMasterRef;
|
||||
}
|
||||
|
|
@ -2193,7 +2230,7 @@ async function detectDefaultBranch(repoRoot: string): Promise<string | null> {
|
|||
repoRoot,
|
||||
);
|
||||
if (remoteHead) {
|
||||
await refreshRemoteTrackingBaseRef(repoRoot, remoteHead);
|
||||
await refreshRemoteTrackingBaseRef(repoRoot, remoteHead, resolveGitAuth);
|
||||
if (await resolveBaseRefSha(repoRoot, remoteHead)) return remoteHead;
|
||||
}
|
||||
} catch {
|
||||
|
|
@ -2203,7 +2240,7 @@ async function detectDefaultBranch(repoRoot: string): Promise<string | null> {
|
|||
// Fallback: check for common default branch names on the remote
|
||||
for (const candidate of ["origin/master", "origin/main", "main", "master"]) {
|
||||
try {
|
||||
await refreshRemoteTrackingBaseRef(repoRoot, candidate);
|
||||
await refreshRemoteTrackingBaseRef(repoRoot, candidate, resolveGitAuth);
|
||||
await runGit(["rev-parse", "--verify", `${candidate}^{commit}`], repoRoot);
|
||||
return candidate;
|
||||
} catch {
|
||||
|
|
@ -2689,6 +2726,7 @@ export async function realizeExecutionWorkspace(input: {
|
|||
enableWorkspaceBranchReconcileForward?: boolean;
|
||||
enableWorkspaceDirtyQuarantineRepair?: boolean;
|
||||
recorder?: WorkspaceOperationRecorder | null;
|
||||
resolveGitAuth?: GitRemoteAuthProvider | null;
|
||||
}): Promise<RealizedExecutionWorkspace> {
|
||||
const rawStrategy = parseObject(input.config.workspaceStrategy);
|
||||
const strategyType = asString(rawStrategy.type, "project_primary");
|
||||
|
|
@ -2727,10 +2765,10 @@ export async function realizeExecutionWorkspace(input: {
|
|||
baseRef,
|
||||
warnings: baseRefResolutionWarnings,
|
||||
refreshed: baseRefAlreadyRefreshed,
|
||||
} = await resolveAuthoritativeBaseRef(repoRoot, configuredBaseRef);
|
||||
} = await resolveAuthoritativeBaseRef(repoRoot, configuredBaseRef, input.resolveGitAuth);
|
||||
const baseRefreshWarnings = [
|
||||
...baseRefResolutionWarnings,
|
||||
...(baseRefAlreadyRefreshed ? [] : await refreshRemoteTrackingBaseRef(repoRoot, baseRef)),
|
||||
...(baseRefAlreadyRefreshed ? [] : await refreshRemoteTrackingBaseRef(repoRoot, baseRef, input.resolveGitAuth)),
|
||||
];
|
||||
const currentBaseRefSha = await resolveBaseRefSha(repoRoot, baseRef);
|
||||
|
||||
|
|
@ -2962,6 +3000,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
|||
enableWorkspaceBranchReconcileForward?: boolean;
|
||||
enableWorkspaceDirtyQuarantineRepair?: boolean;
|
||||
recorder?: WorkspaceOperationRecorder | null;
|
||||
resolveGitAuth?: GitRemoteAuthProvider | null;
|
||||
}): Promise<RealizedExecutionWorkspace | null> {
|
||||
const cwd = asString(input.workspace.cwd ?? input.workspace.providerRef, "").trim();
|
||||
if (!cwd) return null;
|
||||
|
|
@ -3039,7 +3078,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
|||
);
|
||||
}
|
||||
const baseRefreshWarnings = reuseBaseRef
|
||||
? await refreshRemoteTrackingBaseRef(repoRoot, reuseBaseRef)
|
||||
? await refreshRemoteTrackingBaseRef(repoRoot, reuseBaseRef, input.resolveGitAuth)
|
||||
: [];
|
||||
const currentBaseRefSha = reuseBaseRef ? await resolveBaseRefSha(repoRoot, reuseBaseRef) : null;
|
||||
const refresh = reuseBaseRef && currentBaseRefSha
|
||||
|
|
@ -3090,7 +3129,9 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
|||
await fs.mkdir(path.dirname(worktreePath), { recursive: true });
|
||||
await runGit(["worktree", "prune"], repoRoot).catch(() => {});
|
||||
const restoreBaseRef = input.workspace.baseRef ?? input.base.repoRef ?? null;
|
||||
const restoreRefreshWarnings = restoreBaseRef ? await refreshRemoteTrackingBaseRef(repoRoot, restoreBaseRef) : [];
|
||||
const restoreRefreshWarnings = restoreBaseRef
|
||||
? await refreshRemoteTrackingBaseRef(repoRoot, restoreBaseRef, input.resolveGitAuth)
|
||||
: [];
|
||||
const restoreCurrentBaseRefSha = restoreBaseRef ? await resolveBaseRefSha(repoRoot, restoreBaseRef) : null;
|
||||
|
||||
let created = false;
|
||||
|
|
|
|||
Loading…
Reference in New Issue