MicroFish/CEO_BRIEF_V15_LIVE.md

4.5 KiB
Raw Blame History

CEO brief — rubric v15, LIVE on GLM-5.2 (5 variants, 29 min, ~24k real LLM calls)

One line: the tiered machinery works live (debate produced 1,427 real view-flips, not theater), but the rubric correctly refuses to ship because GLM-5.2 is structurally bearish-biased on average companies — your "65% bearish baseline" complaint, now measured three ways, and not curable by prompting. Winner = NO SHIP. Fix = serve a less-pessimistic model and re-run.

The headline (gate 13 — prompt-bias symmetry on a NEUTRAL dossier, 500 real calls each)

prompt set bull bear neutral bearbull (on committed) gate 13
A — original prompt 55 272 120 +0.664 (76% bearish) FAIL
BC — bias-fixed prompt 11 70 401 +0.728 (73% bearish on committed) FAIL
BC — anonymized ticker (confound check) 1 86 403 +0.977 (98% bearish on committed) FAIL
  • Your complaint, reproduced & measured. On a synthetic neutral company (every fundamental pinned to its sector median), GLM-5.2 calls it bearish ~3:1 to 5:1 — three different ways.
  • It's model-level bias, not prompt bias. The bias-fix prompt nudges most companies to neutral (120→401) but the calls it does commit to are still bear-heavy; anonymizing the ticker (removing identity sentiment) made it worse, not better. No honest prompt can pass gate 13 on this model without forcing everything neutral — which would be gaming and would also kill the debate (gate 16).

What genuinely works live (the tiered design earned its keep)

gate measure result
16 — deep-dive debate produces real revision per-agent round-1→round-2 flip rate B = 14.3% (1,427/1,100) vs A = 0.47% (1/214), z = 5.68, significant
18 — B beats A on all three reinforcement / pre-gossip balance / flip reinf 0.0 < 1.0 ✓ · balance gap 3.17 < 18.11 ✓ · flip significant ✓
15 — no wasted debate screen = 1 round, no gossip calls saved vs A reported ✓

The debate is not theater — 1,400+ real opinion flips across 200 deep-dive companies × 6 roles, statistically significant vs the single-agent baseline.

What honestly blocks the ship (two gates, not just bias)

gate measure result
13 — no bearish bias symmetry on neutral dossier FAIL (A 76% bearish; BC 73% on committed) — model-level
21 — debate sharpens calls (discrimination r) Pearson r, consensus-conf vs decisive-score, per company FAIL — r_B = 0.640 < r_A = 0.786
20 — conviction calibration high-conf agents more decisive than median FAIL across A/B/C

Why NO SHIP is correct, not just bias: even setting gate 13 aside, B fails gate 21 — on GLM-5.2 a single agent already discriminates (r=0.79), and the 6-role debate actually reduced discrimination (r=0.64). The tiered design does not sharpen calls on this model. The gate-19 winner function therefore eliminates B at step 1 (gate 13) and would at step 2 (gate 21); C also dies at step 1. No design that fails the symmetry gate is allowed to ship.

What this means / what I need from you

The rubric says: if a gate is genuinely impossible to satisfy, don't rewrite it — stop, report, and ask. Gate 13 is genuinely impossible on GLM-5.2 (confirmed 3 ways). I did not loosen it.

  • (a) Swap the serving model to a less-pessimistic, better-calibrated build (or lower temperature / a stronger system role), re-run A/B/C + controls. The machinery is proven end-to-end on real calls; only the model needs changing. This is the path to a real B wins.
  • (b) Accept NO SHIP as the deliverable: the bias you flagged, measured hard, plus a working tiered swarm that demonstrably produces real debate.
  • (c) Authorize a gate-13 change (e.g. "BC must reduce bearish skew vs A" instead of strict symmetry). This is a rubric edit — only you can authorize it.

My recommendation: (a) — the result you want (a shippable tiered winner) is one model-swap away. Everything else is built, tested (64 passing), and live-verified.

Provenance

  • Live run: artifacts/v15/parallel_run.log → "ALL 5 VARIANTS DONE in 1747s" (29 min).
  • Commit (code): f35cdb9 (bias-fix prompt) + 7adc361 (parallel/swarm). Head e956c9a (this brief).
  • PR #723 (fork renancloudwalk) — review APPROVED; CI is a checkpoint (no upstream CI on the fork).
  • Full gate-by-gate record + real captured terminal output: ACCEPTANCE_V15_LIVE.md.