From e0c2448267721f672bc2cb6cd16f6da99fdcf89e Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sun, 2 Aug 2026 20:27:09 -0700 Subject: [PATCH] feat(server): authenticate server-side git clone and fetch with a company-secret GitHub token (#10720) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Repo-only project workspaces are materialized by a server-side `git clone`, and isolated `git_worktree` runs refresh their base ref with server-side `git fetch` > - Both operations run outside the agent process with no credentials, so private GitHub repositories can never be cloned or refreshed — agent-scoped credential env bindings do not reach them > - The company secret store already has a well-known GitHub token convention (`GITHUB_TOKEN` / `GH_TOKEN` / `PAPERCLIP_GITHUB_TOKEN`, consumed by the external-object provider for API reads), but nothing server-side consults it for git > - This pull request resolves that token per run and authenticates the managed clone and every base-ref refresh with it through an ephemeral credential helper > - The benefit is that isolated workspaces work on private repositories with one company secret, while public repositories and self-hosted ambient git configuration keep working unchanged ## Linked Issues or Issue Description **Subsystem affected** Server workspace materialization (`server/src/services/heartbeat.ts`) and execution-workspace realization (`server/src/services/workspace-runtime.ts`). **Problem or motivation** A project workspace configured with only a private GitHub `repoUrl` cannot be used for isolated `git_worktree` runs: the managed `git clone` runs with a sanitized, credential-less environment, and plain git cannot consume a bare token env variable without a credential helper. There is no way to give the server a git credential — storing a `GH_TOKEN` company secret has no effect on server-side git, and a credential-less private clone hangs on a terminal prompt until the ten-minute clone timeout. Base-ref refreshes (`git fetch`) during worktree realization have the same gap. **Proposed solution** A `git-credentials` module resolves a token per run — company secret by well-known name (`GITHUB_TOKEN`, `GH_TOKEN`, `PAPERCLIP_GITHUB_TOKEN`), then `GITHUB_TOKEN`/`GH_TOKEN` in the server process environment for self-hosted deployments, then none — and builds a git invocation that authenticates via an inline credential helper. The token travels in an env variable; it never appears in argv, URLs, or on disk. Only `https://github.com` remotes are authenticated; everything else keeps ambient behavior. The provider is a single factory seam so a future brokered credential source can replace it without touching call sites. **Alternatives considered** - A GitHub OAuth "connect your account" flow: heavier product surface, needs app registration and callback custody; out of scope for a server credential and better served by a dedicated connector later. The provider seam keeps that path open. - `gh auth setup-git`: writes helper configuration to disk and requires a global token env; rejected in favor of per-invocation config with no persistent state. - Embedding the token in the clone URL: leaks into argv, error messages, and `.git/config`; rejected. ## What Changed - New `server/src/services/git-credentials.ts`: `createGitRemoteAuthProvider` (memoized per run, one secret resolution and one audit event), `buildGitAuthInvocation` (helper-reset + inline helper, `x-access-token` username, `GIT_TERMINAL_PROMPT=0`), `isGitHubHttpsRemoteUrl` host gating (rejects ssh/GHES/http/other hosts/userinfo URLs), `describeGitAuthFailure`, and the canonical `scrubGitCredentialText`. Secret resolutions pass a `system` consumer access context so they are recorded as secret access events. - `ensureManagedProjectWorkspace` (now exported) accepts an optional auth provider; the clone env spreads the token after `sanitizeRuntimeServiceBaseEnv` (which strips `PAPERCLIP_*`), always sets `GIT_TERMINAL_PROMPT=0`, distinguishes "credential rejected" from "no credential configured — add a GITHUB_TOKEN or GH_TOKEN company secret" in the error, and removes the partially created directory on clone failure so a timeout-killed clone cannot be adopted as a broken checkout by the next run. - `refreshRemoteTrackingBaseRef` (now exported) captures the remote URL it already looked up, asks the provider for an invocation, and attributes failed authenticated fetches to the credential in a scrubbed warning. The optional provider threads through `detectDefaultBranch`, `resolveAuthoritativeBaseRef`, `inspectExecutionWorkspaceBaseDrift`, `realizeExecutionWorkspace`, and `ensurePersistedExecutionWorkspaceAvailable`; heartbeat builds one provider per run for both the anchor-resolution clone path and workspace realization/restore. - `github-external-object-provider.ts` imports the shared secret-name list; `isGitHubDotCom` is exported from `github-fetch.ts`. - Docs: "Private repositories and repo-only project workspaces" section in the execution-workspaces guide, cross-linked from the secrets deploy doc. ## Verification - `cd server && npx vitest run src/__tests__/git-credentials.test.ts` — resolution chain order and precedence, env fallback, memoization, audited access context, host-gating matrix, invocation shape (token absent from argv), scrubber, failure descriptions, and a real-git `git credential fill` round trip that proves the helper executes and answers with the env-carried token (no network). - `cd server && npx vitest run src/__tests__/heartbeat-managed-clone-credentials.test.ts` — clones behave byte-identically with no provider or a null-returning provider (local repos, no network), authenticated-failure errors name the credential, non-auth failures do not mention credentials, partial clone directories are removed, pre-existing non-git directories keep the "Using it as-is" path, and the sanitizer spread order keeps the token env alive. - `cd server && npx vitest run src/__tests__/workspace-runtime.test.ts` — new `refreshRemoteTrackingBaseRef` cases: provider offered the remote URL and null keeps behavior identical; failed authenticated fetch warning names the credential; unauthenticated failure warning stays credential-free. - `pnpm --filter @paperclipai/server typecheck` is clean. - Manual (optional, networked): store a `GH_TOKEN` company secret, configure a repo-only project workspace pointing at a private GitHub repository, run an isolated-workspace issue — the managed clone succeeds and the worktree run proceeds. ## Risks - Every new parameter is optional; with no provider the git invocations are byte-identical to before. Public repos and ambient credential helpers keep working whenever no token resolves. - Precedence change when a token exists: a stored company secret now wins over ambient helpers for `https://github.com` remotes (the helper list is reset for that invocation). The rejected-credential error names the secret so an operator can fix or remove it. - `GIT_TERMINAL_PROMPT=0` on the managed clone is the one always-on change: a credential-less private clone now fails fast with a clear message instead of hanging until the ten-minute timeout (it could only ever "succeed" interactively on a TTY dev server). - The token is scoped to the git process env for one invocation; it is never written to agent env, run context, disk, or logs, and error text is scrubbed of URL userinfo. - No migrations, no image changes (git ships in the image). ## Model Used Claude Fable 5 (`claude-fable-5`, extended thinking, agentic tool use via Claude Code CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- docs/deploy/secrets.md | 7 + ...ecution-workspaces-and-runtime-services.md | 26 ++ server/src/__tests__/git-credentials.test.ts | 285 ++++++++++++++++++ ...eartbeat-managed-clone-credentials.test.ts | 161 ++++++++++ .../src/__tests__/workspace-runtime.test.ts | 37 +++ server/src/services/git-credentials.ts | 202 +++++++++++++ .../github-external-object-provider.ts | 2 +- server/src/services/github-fetch.ts | 2 +- server/src/services/heartbeat.ts | 60 +++- server/src/services/workspace-runtime.ts | 81 +++-- 10 files changed, 835 insertions(+), 28 deletions(-) create mode 100644 server/src/__tests__/git-credentials.test.ts create mode 100644 server/src/__tests__/heartbeat-managed-clone-credentials.test.ts create mode 100644 server/src/services/git-credentials.ts diff --git a/docs/deploy/secrets.md b/docs/deploy/secrets.md index 0d6591574a..e557086ebf 100644 --- a/docs/deploy/secrets.md +++ b/docs/deploy/secrets.md @@ -51,6 +51,13 @@ Project env applies to every issue run in that project. When a project env key matches an agent env key, the project value wins before Paperclip injects its own `PAPERCLIP_*` runtime variables. +Separately from env bindings, the **server itself** consumes a company secret +named `GITHUB_TOKEN`, `GH_TOKEN`, or `PAPERCLIP_GITHUB_TOKEN` (by name, no +binding needed) to authenticate server-side git operations — cloning private +GitHub repos for repo-only project workspaces and refreshing worktree base +refs. See +[Execution workspaces](../guides/board-operator/execution-workspaces-and-runtime-services.md#private-repositories-and-repo-only-project-workspaces). + ## User-Specific Secrets User-specific secrets let a shared agent or project declare a slot such as diff --git a/docs/guides/board-operator/execution-workspaces-and-runtime-services.md b/docs/guides/board-operator/execution-workspaces-and-runtime-services.md index 1b0394d52f..6d4e927400 100644 --- a/docs/guides/board-operator/execution-workspaces-and-runtime-services.md +++ b/docs/guides/board-operator/execution-workspaces-and-runtime-services.md @@ -77,6 +77,32 @@ Some workspaces need heavy one-time setup — seeding a database, warming caches - **Provisioning failed** — the command failed; the workspace detail links to the runtime logs for the failing operation. - While the command runs, the runtime service shows a **Provisioning…** state before it transitions to starting/running. +## Private repositories and repo-only project workspaces + +A project workspace can be **repo-only**: a `Repo URL` with no local path. The server then +materializes a managed checkout on demand (`git clone` into a managed directory) and, for +isolated `git_worktree` runs, refreshes the base ref (`git fetch`) before preparing each +worktree. Both operations run on the server, outside any agent process — so agent-scoped +credential env bindings do not apply to them. + +For **private GitHub repositories**, store a token as a **company secret** named one of +`GITHUB_TOKEN`, `GH_TOKEN`, or `PAPERCLIP_GITHUB_TOKEN` (checked in that order; Settings → +Secrets). The server resolves it per run and authenticates managed clones and base-ref +fetches with it. Details and caveats: + +- Scope: only `https://github.com/...` repo URLs are authenticated this way. SSH URLs, GitHub + Enterprise hosts, and other providers keep ambient behavior (system git config/credential + helpers on the server host). URLs that embed their own credentials are never overridden. +- Fallback: with no matching company secret, the server falls back to a `GITHUB_TOKEN` or + `GH_TOKEN` variable in the **server process environment** (useful for self-hosted single-tenant + deployments), then to unauthenticated access — public repos keep working with no setup. +- The token never appears in command lines, URLs, or on disk; it is passed to git through an + ephemeral credential helper. Each resolution is recorded as a secret access event. +- This is separate from the **agent push credential**: agents pushing branches/PRs still need + `GH_TOKEN`/`GITHUB_TOKEN` bound at agent or project scope (see + [deploy/secrets](../../deploy/secrets.md)) so the token reaches the agent process env. The + same company secret can back both uses via a binding. + ## Cross-run persistence (no-remote-git contract) Code state moves between runs through the local execution-workspace cwd alone — not through a git remote. diff --git a/server/src/__tests__/git-credentials.test.ts b/server/src/__tests__/git-credentials.test.ts new file mode 100644 index 0000000000..17fed11736 --- /dev/null +++ b/server/src/__tests__/git-credentials.test.ts @@ -0,0 +1,285 @@ +import { spawn } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { + DEFAULT_GITHUB_TOKEN_SECRET_NAMES, + GIT_CREDENTIAL_TOKEN_ENV_KEY, + buildGitAuthInvocation, + createGitRemoteAuthProvider, + describeGitAuthFailure, + isGitHubHttpsRemoteUrl, + scrubGitCredentialText, +} from "../services/git-credentials.ts"; + +const fakeDb = null as unknown as Db; + +function buildSecretsFake(byName: Record) { + const getByName = vi.fn(async (_companyId: string, name: string) => { + if (!(name in byName)) return null; + return { id: `secret-${name}` }; + }); + const resolveSecretValue = vi.fn(async (_companyId: string, secretId: string) => { + const name = secretId.replace(/^secret-/, ""); + const value = byName[name]; + if (value instanceof Error) throw value; + return value ?? ""; + }); + return { getByName, resolveSecretValue }; +} + +describe("isGitHubHttpsRemoteUrl", () => { + it("accepts https github.com and www.github.com URLs", () => { + expect(isGitHubHttpsRemoteUrl("https://github.com/example/repo.git")).toBe(true); + expect(isGitHubHttpsRemoteUrl("https://www.github.com/example/repo.git")).toBe(true); + }); + + it("rejects ssh, http, enterprise hosts, other providers, userinfo URLs, and non-URLs", () => { + expect(isGitHubHttpsRemoteUrl("git@github.com:example/repo.git")).toBe(false); + expect(isGitHubHttpsRemoteUrl("ssh://git@github.com/example/repo.git")).toBe(false); + expect(isGitHubHttpsRemoteUrl("http://github.com/example/repo.git")).toBe(false); + expect(isGitHubHttpsRemoteUrl("https://github.enterprise.example/org/repo.git")).toBe(false); + expect(isGitHubHttpsRemoteUrl("https://gitlab.com/example/repo.git")).toBe(false); + expect(isGitHubHttpsRemoteUrl("https://alice:token@github.com/example/repo.git")).toBe(false); + expect(isGitHubHttpsRemoteUrl("/local/path/repo.git")).toBe(false); + }); +}); + +describe("createGitRemoteAuthProvider", () => { + const githubUrl = "https://github.com/example/repo.git"; + + it("prefers company secrets in declared order", async () => { + const secrets = buildSecretsFake({ GH_TOKEN: "gh-token", PAPERCLIP_GITHUB_TOKEN: "pc-token" }); + const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, { + secrets, + env: { GITHUB_TOKEN: "env-token" }, + }); + const invocation = await provider(githubUrl); + expect(invocation?.env[GIT_CREDENTIAL_TOKEN_ENV_KEY]).toBe("gh-token"); + expect(invocation?.source).toBe("company_secret"); + expect(invocation?.secretName).toBe("GH_TOKEN"); + // GITHUB_TOKEN is probed first even though only GH_TOKEN exists. + expect(secrets.getByName.mock.calls.map((call) => call[1])).toEqual(["GITHUB_TOKEN", "GH_TOKEN"]); + }); + + it("falls back to the server env, GITHUB_TOKEN before GH_TOKEN", async () => { + const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, { + secrets: buildSecretsFake({}), + env: { GITHUB_TOKEN: "env-github", GH_TOKEN: "env-gh" }, + }); + const invocation = await provider(githubUrl); + expect(invocation?.env[GIT_CREDENTIAL_TOKEN_ENV_KEY]).toBe("env-github"); + expect(invocation?.source).toBe("server_env"); + expect(invocation?.secretName).toBeNull(); + }); + + it("returns null when no token is available anywhere", async () => { + const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, { + secrets: buildSecretsFake({}), + env: {}, + }); + await expect(provider(githubUrl)).resolves.toBeNull(); + }); + + it("returns null for out-of-scope URLs without touching the secret store", async () => { + const secrets = buildSecretsFake({ GITHUB_TOKEN: "token" }); + const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, { + secrets, + env: {}, + }); + await expect(provider("git@github.com:example/repo.git")).resolves.toBeNull(); + await expect(provider("https://gitlab.com/example/repo.git")).resolves.toBeNull(); + expect(secrets.getByName).not.toHaveBeenCalled(); + }); + + it("memoizes the credential lookup across calls", async () => { + const secrets = buildSecretsFake({ GITHUB_TOKEN: "token" }); + const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, { + secrets, + env: {}, + }); + await provider(githubUrl); + await provider(githubUrl); + await provider("https://github.com/example/another.git"); + expect(secrets.getByName).toHaveBeenCalledTimes(1); + expect(secrets.resolveSecretValue).toHaveBeenCalledTimes(1); + }); + + it("passes a system access context so resolution is audited", async () => { + const secrets = buildSecretsFake({ GITHUB_TOKEN: "token" }); + const provider = createGitRemoteAuthProvider( + fakeDb, + "company-1", + { issueId: "issue-1", heartbeatRunId: "run-1" }, + { secrets, env: {} }, + ); + await provider(githubUrl); + expect(secrets.resolveSecretValue).toHaveBeenCalledWith("company-1", "secret-GITHUB_TOKEN", "latest", { + accessContext: expect.objectContaining({ + consumerType: "system", + consumerId: "workspace-git-credential", + actorType: "system", + issueId: "issue-1", + heartbeatRunId: "run-1", + }), + }); + }); + + it("continues down the chain when one secret fails to resolve", async () => { + const secrets = buildSecretsFake({ + GITHUB_TOKEN: new Error("provider outage"), + GH_TOKEN: "gh-token", + }); + const provider = createGitRemoteAuthProvider(fakeDb, "company-1", undefined, { + secrets, + env: {}, + }); + const invocation = await provider(githubUrl); + expect(invocation?.secretName).toBe("GH_TOKEN"); + }); +}); + +describe("buildGitAuthInvocation", () => { + it("keeps the token out of argv and installs the helper URL-scoped to github.com", () => { + const invocation = buildGitAuthInvocation({ + token: "super-secret-token", + source: "company_secret", + secretName: "GITHUB_TOKEN", + }); + expect(invocation.configArgs.join(" ")).not.toContain("super-secret-token"); + expect(invocation.configArgs[0]).toBe("-c"); + expect(invocation.configArgs[1]).toBe("credential.helper="); + expect(invocation.configArgs[3]).toContain("credential.https://github.com.helper="); + expect(invocation.configArgs[3]).toContain("x-access-token"); + expect(invocation.configArgs[5]).toContain("credential.https://www.github.com.helper="); + expect(invocation.env[GIT_CREDENTIAL_TOKEN_ENV_KEY]).toBe("super-secret-token"); + expect(invocation.env.GIT_TERMINAL_PROMPT).toBe("0"); + }); +}); + +describe("credential helper execution (real git, no network)", () => { + async function runCredentialFill(description: string) { + const cwd = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-cred-fill-")); + try { + const invocation = buildGitAuthInvocation({ + token: "abc123", + source: "company_secret", + secretName: "GITHUB_TOKEN", + }); + return await new Promise<{ code: number | null; stdout: string; stderr: string }>( + (resolve, reject) => { + const child = spawn("git", [...invocation.configArgs, "credential", "fill"], { + cwd, + env: { ...process.env, ...invocation.env }, + stdio: ["pipe", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + child.stdout.on("data", (chunk) => { stdout += String(chunk); }); + child.stderr.on("data", (chunk) => { stderr += String(chunk); }); + child.on("error", reject); + child.on("close", (code) => resolve({ code, stdout, stderr })); + child.stdin.write(description); + child.stdin.end(); + }, + ); + } finally { + await fs.rm(cwd, { recursive: true, force: true }); + } + } + + it("answers a github.com https request with the env-carried token", async () => { + const result = await runCredentialFill("protocol=https\nhost=github.com\n\n"); + expect(result.code).toBe(0); + expect(result.stdout).toContain("username=x-access-token"); + expect(result.stdout).toContain("password=abc123"); + }); + + it("never hands the token to another host, even if git asks", async () => { + // Simulates a request whose effective host changed after our pre-invocation URL check + // (for example a repository-local url..insteadOf rewrite): the URL-scoped helper + // config keeps git from consulting the helper, prompts are disabled, so the fill fails + // and the token is never emitted. + const result = await runCredentialFill("protocol=https\nhost=evil.example\n\n"); + expect(result.code).not.toBe(0); + expect(result.stdout).not.toContain("abc123"); + }); + + it("never answers plain-http requests for github.com", async () => { + const result = await runCredentialFill("protocol=http\nhost=github.com\n\n"); + expect(result.code).not.toBe(0); + expect(result.stdout).not.toContain("abc123"); + }); +}); + +describe("scrubGitCredentialText", () => { + it("masks URL userinfo", () => { + expect(scrubGitCredentialText("https://x-access-token:ghp_secret@github.com/a/b.git")).toBe( + "https://***@github.com/a/b.git", + ); + }); + + it("masks userinfo on non-HTTP schemes, leaving scp-style remotes alone", () => { + expect(scrubGitCredentialText("ssh://deploy:hunter2@internal.example/repo.git")).toBe( + "ssh://***@internal.example/repo.git", + ); + expect(scrubGitCredentialText("git@github.com:example/repo.git")).toBe( + "git@github.com:example/repo.git", + ); + }); + + it("masks entire URL query strings regardless of parameter names", () => { + expect(scrubGitCredentialText("https://github.com/a/b.git?access_token=ghs_secret&ref=main")).toBe( + "https://github.com/a/b.git?***", + ); + expect(scrubGitCredentialText("https://host.example/r.git?obscure_cred_name=secret")).toBe( + "https://host.example/r.git?***", + ); + }); + + it("leaves credential-free text unchanged", () => { + expect(scrubGitCredentialText("fatal: repository not found")).toBe("fatal: repository not found"); + }); +}); + +describe("describeGitAuthFailure", () => { + it("names the company secret when a stored credential was used", () => { + expect(describeGitAuthFailure({ + error: "fatal: Authentication failed", + used: { source: "company_secret", secretName: "GH_TOKEN" }, + })).toContain("the GH_TOKEN company-secret GitHub credential"); + }); + + it("names the server environment when an env credential was used", () => { + expect(describeGitAuthFailure({ + error: "fatal: Authentication failed", + used: { source: "server_env", secretName: null }, + })).toContain("server-environment GitHub credential"); + }); + + it("points at Settings → Secrets for auth-looking failures without a credential", () => { + expect(describeGitAuthFailure({ + error: "fatal: could not read Username for 'https://github.com': terminal prompts disabled", + used: null, + })).toContain("add a GITHUB_TOKEN or GH_TOKEN company secret"); + }); + + it("stays silent for non-auth failures without a credential", () => { + expect(describeGitAuthFailure({ + error: "fatal: unable to resolve host example.invalid", + used: null, + })).toBeNull(); + }); +}); + +describe("DEFAULT_GITHUB_TOKEN_SECRET_NAMES", () => { + it("keeps the shared name order stable", () => { + expect([...DEFAULT_GITHUB_TOKEN_SECRET_NAMES]).toEqual([ + "GITHUB_TOKEN", + "GH_TOKEN", + "PAPERCLIP_GITHUB_TOKEN", + ]); + }); +}); diff --git a/server/src/__tests__/heartbeat-managed-clone-credentials.test.ts b/server/src/__tests__/heartbeat-managed-clone-credentials.test.ts new file mode 100644 index 0000000000..146a11df50 --- /dev/null +++ b/server/src/__tests__/heartbeat-managed-clone-credentials.test.ts @@ -0,0 +1,161 @@ +import { execFile as execFileCallback } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { ensureManagedProjectWorkspace } from "../services/heartbeat.ts"; +import { buildGitAuthInvocation, GIT_CREDENTIAL_TOKEN_ENV_KEY } from "../services/git-credentials.ts"; +import { sanitizeRuntimeServiceBaseEnv } from "../services/workspace-runtime.ts"; +import { resolveManagedProjectWorkspaceDir } from "../home-paths.ts"; + +const execFile = promisify(execFileCallback); + +let tempHome: string; +let originalHome: string | undefined; + +beforeAll(async () => { + originalHome = process.env.PAPERCLIP_HOME; + tempHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-managed-clone-")); + process.env.PAPERCLIP_HOME = tempHome; +}); + +afterAll(async () => { + if (originalHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = originalHome; + await fs.rm(tempHome, { recursive: true, force: true }); +}); + +async function createLocalSourceRepo() { + const sourceRepo = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-clone-source-")); + await execFile("git", ["init"], { cwd: sourceRepo }); + await execFile("git", ["config", "user.email", "paperclip@example.com"], { cwd: sourceRepo }); + await execFile("git", ["config", "user.name", "Paperclip Test"], { cwd: sourceRepo }); + await fs.writeFile(path.join(sourceRepo, "README.md"), "hello\n", "utf8"); + await execFile("git", ["add", "README.md"], { cwd: sourceRepo }); + await execFile("git", ["commit", "-m", "init"], { cwd: sourceRepo }); + return sourceRepo; +} + +describe("ensureManagedProjectWorkspace clone credentials", () => { + it("clones exactly as before when no auth provider is configured", async () => { + const sourceRepo = await createLocalSourceRepo(); + try { + const result = await ensureManagedProjectWorkspace({ + companyId: "company-noauth", + projectId: "project-1", + repoUrl: sourceRepo, + }); + expect(result.warning).toBeNull(); + const gitDir = await fs.stat(path.join(result.cwd, ".git")); + expect(gitDir.isDirectory()).toBe(true); + } finally { + await fs.rm(sourceRepo, { recursive: true, force: true }); + } + }); + + it("consults the provider with the repo URL and clones normally when it returns null", async () => { + const sourceRepo = await createLocalSourceRepo(); + const resolveGitAuth = vi.fn(async () => null); + try { + const result = await ensureManagedProjectWorkspace({ + companyId: "company-nullauth", + projectId: "project-1", + repoUrl: sourceRepo, + resolveGitAuth, + }); + expect(resolveGitAuth).toHaveBeenCalledWith(sourceRepo); + const gitDir = await fs.stat(path.join(result.cwd, ".git")); + expect(gitDir.isDirectory()).toBe(true); + } finally { + await fs.rm(sourceRepo, { recursive: true, force: true }); + } + }); + + it("names the company-secret credential when an authenticated clone fails", async () => { + const missingRepo = path.join(os.tmpdir(), "paperclip-definitely-missing", "repo.git"); + const resolveGitAuth = vi.fn(async () => ({ + // Empty configArgs keep this offline: the failure comes from the missing local path, + // the message must still attribute the credential that was in play. + configArgs: [], + env: { [GIT_CREDENTIAL_TOKEN_ENV_KEY]: "token", GIT_TERMINAL_PROMPT: "0" }, + source: "company_secret" as const, + secretName: "GH_TOKEN", + })); + await expect(ensureManagedProjectWorkspace({ + companyId: "company-authfail", + projectId: "project-1", + repoUrl: missingRepo, + resolveGitAuth, + })).rejects.toThrow(/the GH_TOKEN company-secret GitHub credential/); + }); + + it("does not mention credentials when an unauthenticated clone fails for non-auth reasons", async () => { + // The Settings → Secrets hint is reserved for auth-shaped failures (covered in + // git-credentials.test.ts); a plain missing-repo failure must not suggest credentials. + const missingRepo = path.join(os.tmpdir(), "paperclip-definitely-missing", "repo.git"); + const error = await ensureManagedProjectWorkspace({ + companyId: "company-noauthfail", + projectId: "project-1", + repoUrl: missingRepo, + }).then( + () => { throw new Error("expected the clone to fail"); }, + (err: unknown) => err as Error, + ); + expect(error.message).toContain("Failed to prepare managed checkout"); + expect(error.message).not.toContain("company secret"); + }); + + it("removes the partially created directory when the clone fails", async () => { + const missingRepo = path.join(os.tmpdir(), "paperclip-definitely-missing", "repo.git"); + const companyId = "company-cleanup"; + const projectId = "project-1"; + await expect(ensureManagedProjectWorkspace({ + companyId, + projectId, + repoUrl: missingRepo, + })).rejects.toThrow(); + // Filesystem-path repo "URLs" derive no repo name, so the managed dir is the _default slot. + const cwd = resolveManagedProjectWorkspaceDir({ companyId, projectId }); + await expect(fs.stat(cwd)).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("keeps using a pre-existing non-git directory as-is without attempting a clone", async () => { + const companyId = "company-existing"; + const projectId = "project-1"; + const sourceRepo = await createLocalSourceRepo(); + try { + const cwd = resolveManagedProjectWorkspaceDir({ companyId, projectId }); + await fs.mkdir(cwd, { recursive: true }); + await fs.writeFile(path.join(cwd, "keep.txt"), "operator data\n", "utf8"); + const result = await ensureManagedProjectWorkspace({ + companyId, + projectId, + repoUrl: sourceRepo, + }); + expect(result.cwd).toBe(cwd); + expect(result.warning).toContain("Using it as-is"); + await expect(fs.readFile(path.join(cwd, "keep.txt"), "utf8")).resolves.toBe("operator data\n"); + } finally { + await fs.rm(sourceRepo, { recursive: true, force: true }); + } + }); + + it("keeps the credential env alive through the sanitizer spread order", () => { + // The clone env is `{ ...sanitize(process.env), GIT_TERMINAL_PROMPT, ...auth.env }`. The + // sanitizer strips every PAPERCLIP_* key, so the token env must be spread after it. + const invocation = buildGitAuthInvocation({ + token: "tok", + source: "company_secret", + secretName: "GITHUB_TOKEN", + }); + const cloneEnv = { + ...sanitizeRuntimeServiceBaseEnv({ ...process.env, [GIT_CREDENTIAL_TOKEN_ENV_KEY]: "stale" }), + GIT_TERMINAL_PROMPT: "0", + ...invocation.env, + }; + expect(cloneEnv[GIT_CREDENTIAL_TOKEN_ENV_KEY]).toBe("tok"); + expect(sanitizeRuntimeServiceBaseEnv({ [GIT_CREDENTIAL_TOKEN_ENV_KEY]: "stale" })[GIT_CREDENTIAL_TOKEN_ENV_KEY]) + .toBeUndefined(); + }); +}); diff --git a/server/src/__tests__/workspace-runtime.test.ts b/server/src/__tests__/workspace-runtime.test.ts index 5543fc1526..447583fff7 100644 --- a/server/src/__tests__/workspace-runtime.test.ts +++ b/server/src/__tests__/workspace-runtime.test.ts @@ -34,6 +34,7 @@ import { normalizeAdapterManagedRuntimeServices, reconcilePersistedRuntimeServicesOnStartup, realizeExecutionWorkspace, + refreshRemoteTrackingBaseRef, releaseRuntimeServicesForRun, resetRuntimeServicesForTests, resolveWorkspaceRuntimeReadinessTimeoutSec, @@ -379,6 +380,42 @@ describe("sanitizeRuntimeServiceBaseEnv", () => { }); }); +describe("refreshRemoteTrackingBaseRef git auth", () => { + it("offers the remote URL to the provider and keeps ambient behavior when it returns null", async () => { + const { remotePath, repoRoot } = await createClonedRepoWithRemote(); + const offeredUrls: string[] = []; + const warnings = await refreshRemoteTrackingBaseRef(repoRoot, "origin/master", async (remoteUrl) => { + offeredUrls.push(remoteUrl); + return null; + }); + expect(warnings).toEqual([]); + expect(offeredUrls).toEqual([remotePath]); + }); + + it("attributes a failed authenticated fetch to the credential that was used", async () => { + const { repoRoot } = await createClonedRepoWithRemote(); + await runGit(repoRoot, ["remote", "set-url", "origin", path.join(os.tmpdir(), "paperclip-missing-remote", "repo.git")]); + const warnings = await refreshRemoteTrackingBaseRef(repoRoot, "origin/master", async () => ({ + configArgs: [], + env: { GIT_TERMINAL_PROMPT: "0" }, + source: "company_secret", + secretName: "GH_TOKEN", + })); + expect(warnings).toHaveLength(1); + expect(warnings[0]).toContain("Could not refresh base ref origin/master"); + expect(warnings[0]).toContain("the GH_TOKEN company-secret GitHub credential"); + }); + + it("keeps the unauthenticated failure warning credential-free without a provider", async () => { + const { repoRoot } = await createClonedRepoWithRemote(); + await runGit(repoRoot, ["remote", "set-url", "origin", path.join(os.tmpdir(), "paperclip-missing-remote", "repo.git")]); + const warnings = await refreshRemoteTrackingBaseRef(repoRoot, "origin/master"); + expect(warnings).toHaveLength(1); + expect(warnings[0]).toContain("Could not refresh base ref origin/master"); + expect(warnings[0]).not.toContain("GitHub credential"); + }); +}); + describe("ensureServerWorkspaceLinksCurrent", () => { it("relinks stale server workspace dependencies inside the current repo root", async () => { const repoRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-links-")); diff --git a/server/src/services/git-credentials.ts b/server/src/services/git-credentials.ts new file mode 100644 index 0000000000..4675103fd0 --- /dev/null +++ b/server/src/services/git-credentials.ts @@ -0,0 +1,202 @@ +import type { Db } from "@paperclipai/db"; +import { isGitHubDotCom } from "./github-fetch.js"; +import { secretService } from "./secrets.js"; + +/** + * Server-side git credentials for managed project checkouts and execution-workspace base + * refreshes. Operators store a GitHub token as a company secret under one of the well-known + * names below (the same convention the GitHub external-object provider reads); this module + * resolves it and turns it into a git invocation that authenticates clone/fetch against + * github.com over HTTPS without ever placing the token in argv, URLs, or on disk. + * + * The provider factory is deliberately the single seam for future credential sources (for + * example a brokered GitHub connection): swap the factory, keep every call site unchanged. + */ + +/** Company-secret names probed for a GitHub token, in priority order. */ +export const DEFAULT_GITHUB_TOKEN_SECRET_NAMES = ["GITHUB_TOKEN", "GH_TOKEN", "PAPERCLIP_GITHUB_TOKEN"] as const; + +/** Env var the credential helper reads the token from; never appears in argv. */ +export const GIT_CREDENTIAL_TOKEN_ENV_KEY = "PAPERCLIP_GIT_TOKEN"; + +// `!`-prefixed helpers run via `sh -c` with the credential action appended as "$1". Only the +// `get` action answers; store/erase drain stdin and exit 0 silently. `x-access-token` +// authenticates classic PATs, fine-grained PATs, and GitHub App installation tokens alike. +// +// The helper re-validates the credential request from its stdin description and answers only +// for `protocol=https` + `host=github.com`/`www.github.com`. The pre-invocation URL check +// runs before git applies configuration like repository-local `url..insteadOf` +// rewrites, so a rewritten remote could otherwise request the token for an arbitrary host. +// The helper is additionally installed URL-scoped (`credential.https://github.com.helper`) +// so git does not consult it for other hosts in the first place — two independent gates. +const GIT_CREDENTIAL_HELPER = + `!f() { ok=; proto=; while IFS= read -r l && [ -n "$l" ]; do case "$l" in host=github.com|host=www.github.com) ok=1;; protocol=https) proto=1;; esac; done; if [ "$1" = get ] && [ -n "$ok" ] && [ -n "$proto" ]; then printf 'username=x-access-token\\npassword=%s\\n' "$PAPERCLIP_GIT_TOKEN"; fi; }; f`; + +export type GitCredential = { + token: string; + source: "company_secret" | "server_env"; + /** The company-secret name the token came from; null for a server-environment token. */ + secretName: string | null; +}; + +/** A prepared, credential-bearing git invocation: config args plus the env that carries the token. */ +export type GitAuthInvocation = { + configArgs: string[]; + env: Record; + source: GitCredential["source"]; + secretName: string | null; +}; + +/** + * Resolve auth for one remote URL. Returns null when the URL is out of scope (non-GitHub, + * ssh, or already credentialed) or when no token is available — callers then run git with + * ambient behavior, exactly as before this module existed. + */ +export type GitRemoteAuthProvider = (remoteUrl: string) => Promise; + +/** + * True only for `https://github.com/...` (or `www.`) URLs without inline userinfo. GHES and + * other hosts are out of scope for now — sending a github.com token to an arbitrary host + * would leak it, and an operator's inline URL credential must never be overridden. + */ +export function isGitHubHttpsRemoteUrl(remoteUrl: string): boolean { + let parsed: URL; + try { + parsed = new URL(remoteUrl); + } catch { + return false; + } + if (parsed.protocol !== "https:") return false; + if (parsed.username || parsed.password) return false; + return isGitHubDotCom(parsed.hostname); +} + +/** + * Mask credential material embedded in URLs so it never reaches warnings, run errors, or + * persisted payloads: userinfo on any scheme (`https://user:token@host`, + * `ssh://user:pass@host`) and the entire query string of any URL (`?access_token=…` and + * every other parameter — masked wholesale rather than by an inevitably incomplete + * parameter-name list). Scp-style remotes (`git@host:path`) carry no password and are left + * alone. + */ +export function scrubGitCredentialText(text: string): string { + return text + .replace(/([a-z][a-z0-9+.-]*:\/\/)[^/@\s]+@/gi, "$1***@") + .replace(/([a-z][a-z0-9+.-]*:\/\/[^\s"'?]*)\?[^\s"']*/gi, "$1?***"); +} + +export function buildGitAuthInvocation(credential: GitCredential): GitAuthInvocation { + return { + // The leading empty helper clears ambient helpers (gh, osxkeychain, credential-store) so + // they neither outrank the resolved token nor receive store/erase callbacks for it. The + // token helper is installed URL-scoped: git consults it only for credential requests + // whose context matches github.com over https, so an `insteadOf`-rewritten remote never + // reaches it (and the helper itself re-checks the request host — see above). + configArgs: [ + "-c", "credential.helper=", + "-c", `credential.https://github.com.helper=${GIT_CREDENTIAL_HELPER}`, + "-c", `credential.https://www.github.com.helper=${GIT_CREDENTIAL_HELPER}`, + ], + env: { + [GIT_CREDENTIAL_TOKEN_ENV_KEY]: credential.token, + GIT_TERMINAL_PROMPT: "0", + }, + source: credential.source, + secretName: credential.secretName, + }; +} + +const GIT_AUTH_FAILURE_PATTERN = + /authentication failed|could not read username|could not read password|invalid username or password|terminal prompts disabled|repository not found|not accessible|permission denied|HTTP 40[13]|The requested URL returned error: 40[13]/i; + +/** + * Turn a failed authenticated (or unauthenticated) git network operation into an actionable + * suffix for the error message. Returns null when the failure does not look auth-related and + * no credential was in play. + */ +export function describeGitAuthFailure(input: { + error: string; + used: { source: GitCredential["source"]; secretName: string | null } | null; +}): string | null { + if (input.used) { + const label = input.used.secretName + ? `the ${input.used.secretName} company-secret GitHub credential` + : "the server-environment GitHub credential"; + return `The operation authenticated with ${label}, which was rejected or lacks access to this repository.`; + } + if (GIT_AUTH_FAILURE_PATTERN.test(input.error)) { + return "No GitHub credential is configured — add a GITHUB_TOKEN or GH_TOKEN company secret in Settings → Secrets, or configure a local checkout cwd for this project workspace."; + } + return null; +} + +type SecretServiceLike = ReturnType; + +type GitCredentialSecretsDeps = { + getByName: ( + companyId: string, + name: string, + ) => Promise<{ id: string } | null | undefined> | ReturnType; + resolveSecretValue: SecretServiceLike["resolveSecretValue"]; +}; + +/** + * Build the credential provider for one run. Resolution order: company secret by well-known + * name, then the server process env (`GITHUB_TOKEN`/`GH_TOKEN`) for self-hosted operators, + * then null. The lookup is memoized per provider instance so one run performs at most one + * secret resolution (and writes at most one audit event) no matter how many git operations + * it authenticates. + */ +export function createGitRemoteAuthProvider( + db: Db, + companyId: string, + context?: { + issueId?: string | null; + heartbeatRunId?: string | null; + responsibleUserId?: string | null; + }, + deps?: { + secrets?: GitCredentialSecretsDeps; + env?: NodeJS.ProcessEnv; + secretNames?: readonly string[]; + }, +): GitRemoteAuthProvider { + const secrets: GitCredentialSecretsDeps = deps?.secrets ?? secretService(db); + const env = deps?.env ?? process.env; + const secretNames = deps?.secretNames ?? DEFAULT_GITHUB_TOKEN_SECRET_NAMES; + let credentialPromise: Promise | null = null; + + const resolveCredential = async (): Promise => { + for (const secretName of secretNames) { + const secret = await Promise.resolve(secrets.getByName(companyId, secretName)).catch(() => null); + if (!secret) continue; + // A resolution failure (inactive secret, provider outage) records its own failure audit + // event; fall through to the next source instead of failing the whole git operation here. + const token = await secrets + .resolveSecretValue(companyId, secret.id, "latest", { + accessContext: { + consumerType: "system", + consumerId: "workspace-git-credential", + actorType: "system", + issueId: context?.issueId ?? null, + heartbeatRunId: context?.heartbeatRunId ?? null, + responsibleUserId: context?.responsibleUserId ?? null, + }, + }) + .then((value) => value.trim()) + .catch(() => ""); + if (token) return { token, source: "company_secret", secretName }; + } + const envToken = env.GITHUB_TOKEN?.trim() || env.GH_TOKEN?.trim() || ""; + if (envToken) return { token: envToken, source: "server_env", secretName: null }; + return null; + }; + + return async (remoteUrl: string) => { + if (!isGitHubHttpsRemoteUrl(remoteUrl)) return null; + credentialPromise ??= resolveCredential(); + const credential = await credentialPromise; + if (!credential) return null; + return buildGitAuthInvocation(credential); + }; +} diff --git a/server/src/services/github-external-object-provider.ts b/server/src/services/github-external-object-provider.ts index f434b24b41..6664b6c7cf 100644 --- a/server/src/services/github-external-object-provider.ts +++ b/server/src/services/github-external-object-provider.ts @@ -1,5 +1,6 @@ import type { Db } from "@paperclipai/db"; import type { ExternalObjectCanonicalUrl } from "@paperclipai/shared"; +import { DEFAULT_GITHUB_TOKEN_SECRET_NAMES } from "./git-credentials.js"; import { ghFetch, gitHubApiBase } from "./github-fetch.js"; import { secretService } from "./secrets.js"; import type { @@ -27,7 +28,6 @@ interface GitHubObjectIdentity { pathKind: "pull" | "issues"; } -const DEFAULT_GITHUB_TOKEN_SECRET_NAMES = ["GITHUB_TOKEN", "GH_TOKEN", "PAPERCLIP_GITHUB_TOKEN"] as const; const GITHUB_OBJECT_TTL_SECONDS = 300; function isGitHubHost(host: string) { diff --git a/server/src/services/github-fetch.ts b/server/src/services/github-fetch.ts index 787ae0ef5d..af9fd955fd 100644 --- a/server/src/services/github-fetch.ts +++ b/server/src/services/github-fetch.ts @@ -1,6 +1,6 @@ import { unprocessable } from "../errors.js"; -function isGitHubDotCom(hostname: string) { +export function isGitHubDotCom(hostname: string) { const h = hostname.toLowerCase(); return h === "github.com" || h === "www.github.com"; } diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index ab27ce02b2..38f1d66375 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -70,6 +70,12 @@ import { import { conflict, HttpError, notFound } from "../errors.js"; import { getStartupTraceContext } from "../instrumentation.js"; import { logger } from "../middleware/logger.js"; +import { + createGitRemoteAuthProvider, + describeGitAuthFailure, + scrubGitCredentialText, + type GitRemoteAuthProvider, +} from "./git-credentials.js"; import { publishLiveEvent } from "./live-events.js"; import { normalizeResponsibleUserDenialCode } from "./responsible-user-denial-run-outcomes.js"; import { getRunLogStore, type RunLogHandle } from "./run-log-store.js"; @@ -1489,10 +1495,12 @@ function deriveRepoNameFromRepoUrl(repoUrl: string | null): string | null { } } -async function ensureManagedProjectWorkspace(input: { +export async function ensureManagedProjectWorkspace(input: { companyId: string; projectId: string; repoUrl: string | null; + /** Optional git credential source for cloning private repos; null/absent preserves ambient behavior. */ + resolveGitAuth?: GitRemoteAuthProvider | null; }): Promise<{ cwd: string; warning: string | null }> { const cwd = resolveManagedProjectWorkspaceDir({ companyId: input.companyId, @@ -1528,15 +1536,34 @@ async function ensureManagedProjectWorkspace(input: { await fs.rm(cwd, { recursive: true, force: true }); } + const auth = input.resolveGitAuth ? await input.resolveGitAuth(input.repoUrl) : null; try { - await execFile("git", ["clone", input.repoUrl, cwd], { - env: sanitizeRuntimeServiceBaseEnv(process.env), + await execFile("git", [...(auth?.configArgs ?? []), "clone", input.repoUrl, cwd], { + env: { + // Spread order matters: the sanitizer strips PAPERCLIP_*, which would remove the + // credential-helper token env if it came first. GIT_TERMINAL_PROMPT=0 fails a + // credential-less private clone immediately instead of hanging on a prompt until + // the clone timeout. + ...sanitizeRuntimeServiceBaseEnv(process.env), + GIT_TERMINAL_PROMPT: "0", + ...(auth?.env ?? {}), + }, timeout: MANAGED_WORKSPACE_GIT_CLONE_TIMEOUT_MS, }); return { cwd, warning: null }; } catch (error) { + // The clone only ever starts from a missing or just-emptied directory, so removing it on + // failure cannot destroy operator data — but leaving it would poison the next run: a + // timeout-killed clone leaves a partial .git that the gitDirExists probe above adopts. + await fs.rm(cwd, { recursive: true, force: true }).catch(() => undefined); const reason = error instanceof Error ? error.message : String(error); - throw new Error(`Failed to prepare managed checkout for "${input.repoUrl}" at "${cwd}": ${reason}`); + const authNote = describeGitAuthFailure({ + error: reason, + used: auth ? { source: auth.source, secretName: auth.secretName } : null, + }); + throw new Error(scrubGitCredentialText( + `Failed to prepare managed checkout for "${input.repoUrl}" at "${cwd}": ${reason}${authNote ? ` ${authNote}` : ""}`, + )); } } @@ -1551,6 +1578,7 @@ async function resolveConfiguredOrManagedProjectCwd(input: { projectId: string; cwd: string | null; repoUrl: string | null; + resolveGitAuth?: GitRemoteAuthProvider | null; }): Promise<{ cwd: string; warning: string | null }> { const configuredCwd = readNonEmptyString(input.cwd); if (configuredCwd && configuredCwd !== REPO_ONLY_CWD_SENTINEL) { @@ -1560,6 +1588,7 @@ async function resolveConfiguredOrManagedProjectCwd(input: { companyId: input.companyId, projectId: input.projectId, repoUrl: readNonEmptyString(input.repoUrl), + resolveGitAuth: input.resolveGitAuth ?? null, }); } @@ -1587,8 +1616,16 @@ function defaultAdditionalProjectWorkspaceDeps(db: Db): ResolveAdditionalProject .from(projectWorkspaces) .where(and(eq(projectWorkspaces.companyId, companyId), eq(projectWorkspaces.projectId, projectId))) .orderBy(asc(projectWorkspaces.createdAt), asc(projectWorkspaces.id)), - resolveConfiguredOrManagedProjectCwd, - ensureManagedProjectWorkspace, + resolveConfiguredOrManagedProjectCwd: (input) => + resolveConfiguredOrManagedProjectCwd({ + ...input, + resolveGitAuth: input.resolveGitAuth ?? createGitRemoteAuthProvider(db, input.companyId), + }), + ensureManagedProjectWorkspace: (input) => + ensureManagedProjectWorkspace({ + ...input, + resolveGitAuth: input.resolveGitAuth ?? createGitRemoteAuthProvider(db, input.companyId), + }), // A realized workspace must hold real content. An empty directory gives the agent an empty // referenced workspace, so treat an empty directory the same as a missing one. directoryHasContents: async (cwd) => { @@ -8419,6 +8456,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) preferredWorkspaceWarning = `Selected project workspace "${preferredProjectWorkspaceId}" is not available on this project.`; } + const resolveGitAuth = createGitRemoteAuthProvider(db, agent.companyId, { issueId }); for (const workspace of projectWorkspaceRows) { let projectCwd: string; let managedWorkspaceWarning: string | null = null; @@ -8428,6 +8466,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) projectId: workspaceProjectId ?? resolvedProjectId ?? workspace.projectId, cwd: workspace.cwd, repoUrl: workspace.repoUrl, + resolveGitAuth, }); projectCwd = resolvedCwd.cwd; managedWorkspaceWarning = resolvedCwd.warning; @@ -14050,6 +14089,13 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) : null, issueId, }); + // One credential provider per run: base-ref refreshes during workspace realization and + // restore authenticate against private GitHub remotes with the same company-secret token + // the managed clone uses. + const workspaceGitAuthProvider = createGitRemoteAuthProvider(db, agent.companyId, { + issueId, + heartbeatRunId: run.id, + }); const { executionWorkspace, reusedExecutionWorkspace, policy: resolvedWorkspaceReusePolicy } = await provisionExecutionWorkspaceForFreshnessDecision({ requestedShouldReuseExisting, @@ -14098,6 +14144,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) enableWorkspaceDirtyQuarantineRepair: resolvedInstanceSettings.experimental.enableWorkspaceDirtyQuarantineRepair, recorder: workspaceOperationRecorder, + resolveGitAuth: workspaceGitAuthProvider, }) : null, realizeWorkspace: () => realizeExecutionWorkspace({ @@ -14116,6 +14163,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) enableWorkspaceDirtyQuarantineRepair: resolvedInstanceSettings.experimental.enableWorkspaceDirtyQuarantineRepair, recorder: workspaceOperationRecorder, + resolveGitAuth: workspaceGitAuthProvider, }), }); const resolvedProjectId = executionWorkspace.projectId ?? issueRef?.projectId ?? executionProjectId ?? null; diff --git a/server/src/services/workspace-runtime.ts b/server/src/services/workspace-runtime.ts index 288a7d8773..b7e8ea5388 100644 --- a/server/src/services/workspace-runtime.ts +++ b/server/src/services/workspace-runtime.ts @@ -77,6 +77,21 @@ export interface ExecutionWorkspaceInput { additionalWorkspaces?: ExecutionWorkspaceAdditionalInput[]; } +/** + * A prepared credential-bearing git invocation for one remote URL, or null to keep ambient + * behavior. Structurally compatible with the provider built by `git-credentials.ts` — this + * module deliberately takes prepared invocations rather than tokens, so it never imports the + * secrets layer and test fakes stay trivial. + */ +export type GitRemoteAuthInvocation = { + configArgs: string[]; + env: Record; + source?: string; + secretName?: string | null; +}; + +export type GitRemoteAuthProvider = (remoteUrl: string) => Promise; + export interface ExecutionWorkspaceIssueRef { id: string; identifier: string | null; @@ -562,11 +577,12 @@ async function executeProcess(input: { }; } -async function runGit(args: string[], cwd: string): Promise { +async function runGit(args: string[], cwd: string, opts?: { env?: NodeJS.ProcessEnv }): Promise { const proc = await executeProcess({ command: "git", args, cwd, + env: opts?.env, }); if (proc.code !== 0) { throw new Error(proc.stderr.trim() || proc.stdout.trim() || `git ${args.join(" ")} failed`); @@ -597,26 +613,40 @@ function parseRemoteTrackingRef(ref: string): { remote: string; branch: string } return { remote, branch }; } -async function refreshRemoteTrackingBaseRef(repoRoot: string, baseRef: string): Promise { +export async function refreshRemoteTrackingBaseRef( + repoRoot: string, + baseRef: string, + resolveGitAuth?: GitRemoteAuthProvider | null, +): Promise { const remoteTracking = parseRemoteTrackingRef(baseRef); if (!remoteTracking) return []; - const remoteExists = await runGit(["remote", "get-url", remoteTracking.remote], repoRoot) - .then(() => true) - .catch(() => false); - if (!remoteExists) return []; + const remoteUrl = await runGit(["remote", "get-url", remoteTracking.remote], repoRoot) + .then((value) => value.trim() || null) + .catch(() => null); + if (!remoteUrl) return []; + const auth = resolveGitAuth ? await resolveGitAuth(remoteUrl).catch(() => null) : null; try { await runGit([ + ...(auth?.configArgs ?? []), "fetch", "--prune", remoteTracking.remote, `+refs/heads/${remoteTracking.branch}:refs/remotes/${remoteTracking.remote}/${remoteTracking.branch}`, - ], repoRoot); + ], repoRoot, auth ? { env: { ...process.env, ...auth.env } } : undefined); return []; } catch (error) { - const message = error instanceof Error ? error.message : String(error); - return [`Could not refresh base ref ${baseRef} before preparing the execution workspace: ${message}`]; + const rawMessage = error instanceof Error ? error.message : String(error); + // Mask URL userinfo (any scheme) and whole URL query strings before the message rides + // warnings that reach run logs. + const message = rawMessage + .replace(/([a-z][a-z0-9+.-]*:\/\/)[^/@\s]+@/gi, "$1***@") + .replace(/([a-z][a-z0-9+.-]*:\/\/[^\s"'?]*)\?[^\s"']*/gi, "$1?***"); + const authNote = auth + ? ` The fetch authenticated with ${auth.secretName ? `the ${auth.secretName} company-secret GitHub credential` : "the server-environment GitHub credential"}, which may have been rejected.` + : ""; + return [`Could not refresh base ref ${baseRef} before preparing the execution workspace: ${message}${authNote}`]; } } @@ -637,6 +667,7 @@ export async function inspectExecutionWorkspaceBaseDrift(input: { baseRef: string | null; recordedBaseRefSha?: string | null; skipRefresh?: boolean; + resolveGitAuth?: GitRemoteAuthProvider | null; }): Promise<{ warnings: string[]; currentBaseRefSha: string | null; @@ -647,7 +678,9 @@ export async function inspectExecutionWorkspaceBaseDrift(input: { return { warnings: [], currentBaseRefSha: null, branchBaseRefSha: null }; } - const warnings = input.skipRefresh ? [] : await refreshRemoteTrackingBaseRef(input.repoRoot, baseRef); + const warnings = input.skipRefresh + ? [] + : await refreshRemoteTrackingBaseRef(input.repoRoot, baseRef, input.resolveGitAuth); const currentBaseRefSha = await resolveBaseRefSha(input.repoRoot, baseRef); if (!currentBaseRefSha) { warnings.push(`Could not resolve base ref ${baseRef} while checking execution workspace freshness.`); @@ -1991,9 +2024,10 @@ export async function ensureGitWorktreeBranchCoherent(input: { async function resolveAuthoritativeBaseRef( repoRoot: string, configuredBaseRef: string | null, + resolveGitAuth?: GitRemoteAuthProvider | null, ): Promise<{ baseRef: string; warnings: string[]; refreshed: boolean }> { const warnings: string[] = []; - const detectOrHead = async () => (await detectDefaultBranch(repoRoot)) ?? "HEAD"; + const detectOrHead = async () => (await detectDefaultBranch(repoRoot, resolveGitAuth)) ?? "HEAD"; const configured = configuredBaseRef?.trim(); if (!configured || configured === "HEAD") { @@ -2008,7 +2042,7 @@ async function resolveAuthoritativeBaseRef( const remoteCandidate = `origin/${configured}`; // Refresh here and keep the warnings; the caller skips its own refresh of // the returned ref (see `refreshed`) so we never fetch the same ref twice. - warnings.push(...await refreshRemoteTrackingBaseRef(repoRoot, remoteCandidate)); + warnings.push(...await refreshRemoteTrackingBaseRef(repoRoot, remoteCandidate, resolveGitAuth)); if (await resolveBaseRefSha(repoRoot, remoteCandidate)) { return { baseRef: remoteCandidate, warnings, refreshed: true }; } @@ -2179,9 +2213,12 @@ async function isGitCheckout(cwd: string): Promise { return Boolean(await runGit(["rev-parse", "--git-dir"], cwd).catch(() => null)); } -async function detectDefaultBranch(repoRoot: string): Promise { +async function detectDefaultBranch( + repoRoot: string, + resolveGitAuth?: GitRemoteAuthProvider | null, +): Promise { const originMasterRef = "origin/master"; - await refreshRemoteTrackingBaseRef(repoRoot, originMasterRef); + await refreshRemoteTrackingBaseRef(repoRoot, originMasterRef, resolveGitAuth); if (await resolveBaseRefSha(repoRoot, originMasterRef)) { return originMasterRef; } @@ -2193,7 +2230,7 @@ async function detectDefaultBranch(repoRoot: string): Promise { repoRoot, ); if (remoteHead) { - await refreshRemoteTrackingBaseRef(repoRoot, remoteHead); + await refreshRemoteTrackingBaseRef(repoRoot, remoteHead, resolveGitAuth); if (await resolveBaseRefSha(repoRoot, remoteHead)) return remoteHead; } } catch { @@ -2203,7 +2240,7 @@ async function detectDefaultBranch(repoRoot: string): Promise { // Fallback: check for common default branch names on the remote for (const candidate of ["origin/master", "origin/main", "main", "master"]) { try { - await refreshRemoteTrackingBaseRef(repoRoot, candidate); + await refreshRemoteTrackingBaseRef(repoRoot, candidate, resolveGitAuth); await runGit(["rev-parse", "--verify", `${candidate}^{commit}`], repoRoot); return candidate; } catch { @@ -2689,6 +2726,7 @@ export async function realizeExecutionWorkspace(input: { enableWorkspaceBranchReconcileForward?: boolean; enableWorkspaceDirtyQuarantineRepair?: boolean; recorder?: WorkspaceOperationRecorder | null; + resolveGitAuth?: GitRemoteAuthProvider | null; }): Promise { const rawStrategy = parseObject(input.config.workspaceStrategy); const strategyType = asString(rawStrategy.type, "project_primary"); @@ -2727,10 +2765,10 @@ export async function realizeExecutionWorkspace(input: { baseRef, warnings: baseRefResolutionWarnings, refreshed: baseRefAlreadyRefreshed, - } = await resolveAuthoritativeBaseRef(repoRoot, configuredBaseRef); + } = await resolveAuthoritativeBaseRef(repoRoot, configuredBaseRef, input.resolveGitAuth); const baseRefreshWarnings = [ ...baseRefResolutionWarnings, - ...(baseRefAlreadyRefreshed ? [] : await refreshRemoteTrackingBaseRef(repoRoot, baseRef)), + ...(baseRefAlreadyRefreshed ? [] : await refreshRemoteTrackingBaseRef(repoRoot, baseRef, input.resolveGitAuth)), ]; const currentBaseRefSha = await resolveBaseRefSha(repoRoot, baseRef); @@ -2962,6 +3000,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { enableWorkspaceBranchReconcileForward?: boolean; enableWorkspaceDirtyQuarantineRepair?: boolean; recorder?: WorkspaceOperationRecorder | null; + resolveGitAuth?: GitRemoteAuthProvider | null; }): Promise { const cwd = asString(input.workspace.cwd ?? input.workspace.providerRef, "").trim(); if (!cwd) return null; @@ -3039,7 +3078,7 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { ); } const baseRefreshWarnings = reuseBaseRef - ? await refreshRemoteTrackingBaseRef(repoRoot, reuseBaseRef) + ? await refreshRemoteTrackingBaseRef(repoRoot, reuseBaseRef, input.resolveGitAuth) : []; const currentBaseRefSha = reuseBaseRef ? await resolveBaseRefSha(repoRoot, reuseBaseRef) : null; const refresh = reuseBaseRef && currentBaseRefSha @@ -3090,7 +3129,9 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: { await fs.mkdir(path.dirname(worktreePath), { recursive: true }); await runGit(["worktree", "prune"], repoRoot).catch(() => {}); const restoreBaseRef = input.workspace.baseRef ?? input.base.repoRef ?? null; - const restoreRefreshWarnings = restoreBaseRef ? await refreshRemoteTrackingBaseRef(repoRoot, restoreBaseRef) : []; + const restoreRefreshWarnings = restoreBaseRef + ? await refreshRemoteTrackingBaseRef(repoRoot, restoreBaseRef, input.resolveGitAuth) + : []; const restoreCurrentBaseRefSha = restoreBaseRef ? await resolveBaseRefSha(repoRoot, restoreBaseRef) : null; let created = false;