From d54ff52fc33dd58fc723222a4ef141397f202876 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 22 Jul 2026 11:53:29 -0500 Subject: [PATCH] test(heartbeat): await execution drain before cleanup (#10023) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source control plane people use to manage AI agents and their work > - Heartbeat scheduling tests protect the orchestration rules that serialize an agent's runs > - The dependency scheduling suite waits for run rows to become terminal before deleting shared database fixtures > - A terminal row is persisted before asynchronous execution finalization and successful-run handoff work fully drain > - The test then clears process tracking and deletes heartbeat events while finalization can still append another event > - This pull request waits for each tracked run's execution promise to drain before resetting mocks or deleting fixtures > - The benefit is deterministic cleanup that preserves the production lifecycle ordering and prevents release CI flakes ## Linked Issues or Issue Description ### What happened? Release run `29936031931` failed in `heartbeat-dependency-scheduling.test.ts` while deleting `heartbeat_runs`. Asynchronous heartbeat finalization inserted a new `heartbeat_run_events` row after the test had already deleted existing events, causing the run-row delete to violate the event foreign key. ### Expected behavior The serialized heartbeat test suite should finish all asynchronous run execution work before destructive fixture cleanup. ### Steps to reproduce 1. Check out commit `2aef4641b48e88f5ce7e75ce69fbe3bf6bbfc60d`. 2. Run `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/heartbeat-dependency-scheduling.test.ts --pool=forks --isolate` repeatedly with PostgreSQL test support enabled. 3. Observe that teardown can delete heartbeat events while execution finalization is still able to append another event, causing a foreign-key failure when heartbeat runs are deleted. ### Paperclip version or commit `2aef4641b48e88f5ce7e75ce69fbe3bf6bbfc60d` ### Deployment mode Other — GitHub Actions release verification. ### Installation method Built from source with pnpm. ### Agent adapter(s) involved Not adapter-specific (core heartbeat test lifecycle). ### Database mode External PostgreSQL test database. ### Relevant logs or output `delete from "heartbeat_runs"` failed because the run remained referenced by `heartbeat_run_events_run_id_heartbeat_runs_id_fk`. ## What Changed - Collect heartbeat run IDs after queued/running rows settle and await `heartbeat.waitForRunExecutionDrain()` for each run. - Reset the adapter mock and clear process tracking only after asynchronous heartbeat finalization has completed. ## Verification - Ran `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/heartbeat-dependency-scheduling.test.ts --pool=forks --isolate` 10 consecutive times; all 10 runs passed with 6/6 tests. ## Risks - Low risk: test-only cleanup ordering change using an existing heartbeat service drain API. Production behavior is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with exact model IDs `gpt-5.5` for this heartbeat and `gpt-5.6-sol` for the recovered initial implementation run; tool-enabled code inspection, GitHub diagnostics, and shell test execution. Runtime context-window sizes were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../heartbeat-dependency-scheduling.test.ts | 28 +++++++++++-------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/server/src/__tests__/heartbeat-dependency-scheduling.test.ts b/server/src/__tests__/heartbeat-dependency-scheduling.test.ts index 454fa19244..bb875a3f21 100644 --- a/server/src/__tests__/heartbeat-dependency-scheduling.test.ts +++ b/server/src/__tests__/heartbeat-dependency-scheduling.test.ts @@ -100,17 +100,6 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () = }, 20_000); afterEach(async () => { - mockAdapterExecute.mockReset(); - mockAdapterExecute.mockImplementation(async () => ({ - exitCode: 0, - signal: null, - timedOut: false, - errorMessage: null, - summary: "Dependency-aware heartbeat test run.", - provider: "test", - model: "test-model", - })); - runningProcesses.clear(); let idlePolls = 0; for (let attempt = 0; attempt < 100; attempt += 1) { const runs = await db @@ -125,7 +114,22 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () = } await new Promise((resolve) => setTimeout(resolve, 50)); } - await new Promise((resolve) => setTimeout(resolve, 50)); + const runIds = await db + .select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .then((runs) => runs.map((run) => run.id)); + await Promise.all(runIds.map((runId) => heartbeat.waitForRunExecutionDrain(runId))); + mockAdapterExecute.mockReset(); + mockAdapterExecute.mockImplementation(async () => ({ + exitCode: 0, + signal: null, + timedOut: false, + errorMessage: null, + summary: "Dependency-aware heartbeat test run.", + provider: "test", + model: "test-model", + })); + runningProcesses.clear(); await db.delete(environmentLeases); await db.delete(activityLog); await db.delete(companySkills);