the sparklines were stretched across the whole gantt bar, but the profiler
wraps a SINGLE step (.github/actions/profile runs it between start/stop), so
on a job dominated by checkout + uv sync + post-job cleanup the samples only
describe a slice in the middle. reproduced: a 30s profile inside a 100s job
whose profiled step ran t=60..90 drew at left 0% width 100% instead of
left 60% width 30% — putting a cpu spike visually under a step that never ran.
the profile json had no wall-clock anchor to place it with, only duration_s,
so emit started_at/completed_at as iso-8601 utc in the same shape as github's
job timestamps. monotonic() still drives the sampling loop (immune to clock
steps); the timestamps are purely for placement.
_profile_window_pct() converts that window into bar-relative percentages and
both overlay states now use it — the expanded holder directly, the collapsed
strip via a .res-clip wrapper so its 100%-width is relative to the window
rather than the bar. the two states are asserted to agree on the x-axis.
falls back to the full bar, i.e. exactly today's behaviour, when the profile
predates these fields, when the timestamps don't parse, or when the window
doesn't overlap the job at all (clock skew between the runner writing the
profile and github's timestamps). a profiler that outran the job's
completed_at is clamped to the bar, and a sub-percent window keeps a 0.5%
hairline so it can't collapse to invisible.
7 new tests. verified they discriminate: forcing the old always-stretch
behaviour fails 4 of them, while the 3 fallback tests keep passing since
full-bar is what they want. tests/ci 145/145.
ops/s alone can't tell you whether a device is saturated — it reads low on a
few large IOs that pin the disk at 100% busy, and high on many small cached
ones. sample io_ticks (diskstats field 13) instead: its delta over the
interval is device busy time, i.e. iostat's %util. take the busiest single
device rather than the sum, since summing across devices exceeds 100% on a
multi-disk node and means nothing as a saturation percentage.
divide by the real elapsed gap rather than the nominal interval — a loaded or
throttled runner drifts well past 1.0s and would otherwise report >100%.
emit a downsampled series (cpu/mem/disk, 0-100 ints) alongside the existing
summary, mean-bucketed to 180 points so a 40min job costs the same few KB as
a 40s one. the report inlines every profile into one self-contained html
file, so an unbounded 1Hz series would dominate its size. mean, not every Nth
sample: a spike that survives decimation by luck is misleading.
the report renders the series as svg sparklines over each gantt bar, in two
states off the same markup (3px strip when collapsed, full height when
expanded) via preserveAspectRatio=none. profiles predating the series field
degrade to table-only, no overlay.
tests/ci 62/62.
Run a duplicate of CI on the new ARC (Actions Runner Controller) runners in
GKE, beside the existing CI. The duplicate does not change production CI.
Every workflow in .github/workflows/ that does not start with newci- is
byte-identical to main. Watch the shadow runs for a few days, then migrate.
The shadow set is 16 files: newci-ci.yml plus the 15 reusable workflows that
ci.yml calls on a pull request. Only pull-request workflows are copied.
js-autofix, deploy-site, and skills-index run on push or on a schedule. A
copy of those would push branches and deploy the site a second time.
Safety properties of the shadow:
- Concurrency groups are newci-prefixed. This is the important one. The
production groups use cancel-in-progress, so a shared group would let a
shadow run cancel the production run.
- Cache keys are newci-prefixed. The shadow cannot poison or evict a
production cache entry.
- Reusable-workflow calls point only at other newci-* files. No shadow job
calls a production workflow.
- The PR review comment runs with --dry-run. It prints the comment body to
the job log. Two pollers cannot fight over the hermes-ci-review-bot
comment.
- The gate job is renamed to "[newci] All checks pass (informational)". The
production check "All required checks pass" stays the only merge gate.
- The shadow runs on pull_request only. The push trigger is removed.
- docker publish and merge jobs are unreachable. Their conditions require a
push to main or a release.
Runner infrastructure, in the shadow copies only:
- Jobs go to three scale sets: arc-runner-small for short gate jobs,
arc-runner-set for general work, arc-runner-docker and arc-runner-arm64
for image builds. dind is only on the docker sets, so the other jobs stop
paying for a privileged sidecar.
- The runner image supplies node 26, npm 12, uv, Python, and ripgrep. The
setup-node, setup-uv, and per-job install steps are gone.
- Checkout uses a node-local git mirror, seeded from the runner pod env.
- buildx layer cache moved to Artifact Registry in us-central1, the same
region as the runners. Reads are keyless through GKE Workload Identity.
Writes use GitHub OIDC and happen only on main pushes and releases, so
pull-request code cannot write a layer that the publish job reads.
Merge-base work, in the shadow copies only:
- A new composite action, .github/actions/merge-base, deepens a shallow
clone until the two histories connect. fetch-depth: 0 fetches all ~1400
refs and measured 76-81s, against 3-6s for a shallow checkout.
- The action fails by default when no merge base exists. A three-dot diff
over a missing merge base scans nothing and reports clean, so the
supply-chain audit must stop. history-check sets fail-on-missing to false,
because absence is the result it measures.
- lint diffs against the base commit directly. The job checks out the PR
merge ref, so base.sha is already the correct comparison point.
- contributor-check uses origin/main..HEAD. The result equals the merge-base
form, and the extra git call also expanded a SHA without quotes.
Other changes:
- .github/actionlint.yaml declares the four ARC labels. actionlint knows
only GitHub-hosted labels, so every runs-on in the repo was reported as an
unknown label: 40 warnings that hid real findings.
- scripts/ci/resource_profile.py records CPU and memory for a job step. The
timing report shows the data per step.
- run_tests_parallel.py can list test files from the git index. The slice
generator then needs no blobs.
- Docker test files are split so boot-heavy tests run in parallel.
- Container-environment parity fixes in doctor, gateway, and skill_utils,
with tests.
To retire the shadow: delete .github/workflows/newci-*.yml.
ethernet