3.2 KiB
CEO brief — rubric v15, live on GLM-5.2 — the bias gate is the headline
One line: the tiered-vs-flat machinery works and is running live; the bias gate (gate 13) honestly FAILS on GLM-5.2 because the model is structurally pessimistic on perfectly average companies — exactly the "65% bearish baseline" you flagged, now measured.
What's real (captured terminal output, GLM-5.2 over the SSH tunnel)
| control run (500 companies, sector-median "neutral" dossier) | bull | bear | neutral | diff (bear−bull) on directional | gate 13 |
|---|---|---|---|---|---|
| A — original prompt | 57 | 270 | 125 | 0.651 (CI ~[0.50, 0.80]) | FAIL |
| BC — bias-fixed prompt | 14 | 68 | 402 | 0.659 (CI ~[0.50, 0.82]) | FAIL |
- A original prompt: 270 bearish vs 57 bullish = ~65% bearish. That is your complaint, reproduced on real GLM-5.2 calls.
- Bias fix helps but doesn't cure: it pushes average companies to neutral (125→402 neutral), but the calls it does commit to are still ~5:1 bear:bull. The model reads mediocre-but-fine fundamentals as bearish.
- This is model-level bias, not prompt bias. Prompts can't calibrate it away without forcing everything neutral — which would be gaming and would also kill the debate (gates 16/21 need real flips).
What this means for the rubric
- Gate 13 ("bias-fixed prompts must be symmetric on a neutral dossier") cannot pass on GLM-5.2 with an honest prompt. The rubric says: don't rewrite the gate, report it and ask.
- If gate 13 fails, the gate-19 winner function eliminates both B and C at step 1 → NO SHIP by design. That is the rubric working correctly — it refuses to ship a biased design.
What's running right now (real, in parallel over the tunnel)
All 5 variants live: A reference, B tiered, C pure-screen, + control-A + control-BC. ~25k real GLM calls, ~30 min. This produces the real per-measure A/B/C table (reinforcement, flip rate w/ significance, gate-21 discrimination r, coverage, cost) and real gate-26 captured terminal output — even though the honest winner verdict will be NO SHIP until the bias is resolved.
Two gates blocked on infra (not on code)
- Gates 22 & 23 (deep-dive dossiers ≥2,000 tokens: top-10 holders, 4-quarter ownership trend, 8-quarter trend) need ClickHouse creds:
CLICKHOUSE_URL,CLICKHOUSE_USER,CLICKHOUSE_PASSWORD. The eligible file has the 27 base fields (~162 tokens/dossier). Runner already reads those env vars.
I need two decisions
- Gate 13 / bias: GLM-5.2 is honestly bearish-biased on average firms. Options:
- (a) Accept honest NO SHIP now; the bias finding itself is the deliverable (it proves the rubric catches the thing you complained about).
- (b) Switch the serving model to one that isn't structurally pessimistic (e.g. a different GLM build / temp / system-prompt role), re-run gate 13.
- (c) Loosen gate 13 to "bias-fixed prompts must reduce bearish skew vs A" (measurable: A 0.651 → BC residual directional) instead of strict symmetry — but that's a rubric change you'd have to authorize.
- ClickHouse creds for gates 22/23 dossier depth.
My recommendation: (a) ship the honest finding now (it's exactly the ground-truth you wanted), and run a cleaner model for a real winner in a follow-up.