ced8e30217
Ships the hard A/B evaluation used for the August 2026 core-toolset performance batch (#77056) as a reusable harness: 9 error-inducing trap tasks derived from measured production waste classes, two-arm PYTHONPATH-only comparison, ATOF-trace-based scoring, resume-safe batteries. Hardened from the original one-off: paths de-hardcoded (ABEVAL_ROOT / ABEVAL_HOME), encoding= on all file IO, startup crashes retry on resume instead of polluting cells, post-hoc grading fix for err_inline_script baked in. Live-smoked end to end (baseline arm, qwen3-coder-30b, err_multi_dir: exit 0, correct on-disk verification, resume record written). |
||
|---|---|---|
| .. | ||
| ci | ||
| lib | ||
| observability | ||
| sandbox | ||
| tests | ||
| toolperf_abeval | ||
| whatsapp-bridge | ||
| LIVETEST_README.md | ||
| add_contributor.py | ||
| analyze_livetest.py | ||
| audit_pr_attribution.py | ||
| benchmark_browser_eval.py | ||
| build_model_catalog.py | ||
| build_skills_index.py | ||
| capture-cage-terminal.sh | ||
| check-windows-footguns.py | ||
| check_subprocess_stdin.py | ||
| contributor_audit.py | ||
| dev-sandbox.sh | ||
| discord-voice-doctor.py | ||
| docker_config_migrate.py | ||
| docker_rebootstrap_nous_session.py | ||
| generate_conformance_vectors.py | ||
| hermes-gateway | ||
| install.cmd | ||
| install.ps1 | ||
| install.sh | ||
| install_psutil_android.py | ||
| iso-certify.py | ||
| keystroke_diagnostic.py | ||
| kill_modal.sh | ||
| lint_diff.py | ||
| micro_compaction_report.py | ||
| profile-tui.py | ||
| release.py | ||
| run_tests.sh | ||
| run_tests_parallel.py | ||
| sample_and_compress.py | ||
| smoke_nemo_relay_shared_metrics.py | ||
| tool_search_livetest.py | ||
| tool_search_livetest2.py | ||
| tool_search_livetest_ue.py | ||
| tool_search_livetest_ue_disc.py | ||
| tool_search_livetest_ue_hard.py | ||