309 lines
12 KiB
Plaintext
309 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Autoresearch Memory Smoke Test\n",
|
|
"\n",
|
|
"This notebook validates the first synthesized memory layer for `autoresearch-macos`.\n",
|
|
"\n",
|
|
"It checks:\n",
|
|
"\n",
|
|
"1. the `memory/` directory exists\n",
|
|
"2. `MEMORY.md` exists and links the expected topic files\n",
|
|
"3. each topic file has frontmatter with `name`, `description`, and `type`\n",
|
|
"4. operational memory files still exist:\n",
|
|
" - `results.tsv`\n",
|
|
" - `experiment_memory.jsonl`\n",
|
|
"5. the synthesized memory layer complements, rather than replaces, the exact run logs\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'claw_root': '/Users/wongdowling/Documents/autoresearch_harness/harness/claw-code', 'autoresearch_root': '/Users/wongdowling/Documents/autoresearch_harness/nodes/autoresearch-macos', 'memory_root': '/Users/wongdowling/Documents/autoresearch_harness/nodes/autoresearch-macos/memory'}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from __future__ import annotations\n",
|
|
"\n",
|
|
"from pathlib import Path\n",
|
|
"import json\n",
|
|
"import re\n",
|
|
"import sys\n",
|
|
"\n",
|
|
"def is_claw_root(candidate: Path) -> bool:\n",
|
|
" return (candidate / 'src' / 'main.py').exists() and (candidate / 'tests' / 'test_porting_workspace.py').exists()\n",
|
|
"\n",
|
|
"def find_claw_root(start: Path | None = None) -> Path:\n",
|
|
" current = (start or Path.cwd()).resolve()\n",
|
|
" for candidate in (current, *current.parents):\n",
|
|
" if is_claw_root(candidate):\n",
|
|
" return candidate\n",
|
|
" for nested in (candidate / 'claw-code', candidate / 'harness' / 'claw-code'):\n",
|
|
" if is_claw_root(nested):\n",
|
|
" return nested\n",
|
|
" raise RuntimeError(f'Could not locate claw-code root from {current}')\n",
|
|
"\n",
|
|
"CLAW_ROOT = find_claw_root()\n",
|
|
"WORKSPACE_ROOT = CLAW_ROOT.parent.parent\n",
|
|
"AUTORESEARCH_ROOT = WORKSPACE_ROOT / 'nodes' / 'autoresearch-macos'\n",
|
|
"MEMORY_ROOT = AUTORESEARCH_ROOT / 'memory'\n",
|
|
"\n",
|
|
"print({\n",
|
|
" 'claw_root': str(CLAW_ROOT),\n",
|
|
" 'autoresearch_root': str(AUTORESEARCH_ROOT),\n",
|
|
" 'memory_root': str(MEMORY_ROOT),\n",
|
|
"})"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"['MEMORY.md',\n",
|
|
" 'project_current_strategy.md',\n",
|
|
" 'project_failed_ideas.md',\n",
|
|
" 'project_promising_ideas.md',\n",
|
|
" 'reference_external_sources.md']"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"expected_files = [\n",
|
|
" 'MEMORY.md',\n",
|
|
" 'project_current_strategy.md',\n",
|
|
" 'project_failed_ideas.md',\n",
|
|
" 'project_promising_ideas.md',\n",
|
|
" 'reference_external_sources.md',\n",
|
|
"]\n",
|
|
"missing = [name for name in expected_files if not (MEMORY_ROOT / name).exists()]\n",
|
|
"if missing:\n",
|
|
" raise RuntimeError(f'Missing memory files: {missing}')\n",
|
|
"sorted(path.name for path in MEMORY_ROOT.iterdir())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"# Memory Index\n",
|
|
"\n",
|
|
"This directory is the synthesized project memory layer for `autoresearch-macos`.\n",
|
|
"\n",
|
|
"It sits between:\n",
|
|
"\n",
|
|
"- raw operational records such as [`results.tsv`](/Users/wongdowling/Documents/autoresearch_harness/nodes/autoresearch-macos/results.tsv) and [`experiment_memory.jsonl`](/Users/wongdowling/Documents/autoresearch_harness/nodes/autoresearch-macos/experiment_memory.jsonl)\n",
|
|
"- future higher-level retrieval or wiki tooling\n",
|
|
"\n",
|
|
"The goal is not to replace exact run logs. The goal is to keep a compact, maintained set of topic pages that accumulate what the manager and worker learn over time.\n",
|
|
"\n",
|
|
"## Topics\n",
|
|
"\n",
|
|
"- [Current strategy](./project_current_strategy.md) - Current operating model, control-plane shape, and near-term execution priorities.\n",
|
|
"- [Failed ideas](./project_failed_ideas.md) - Approaches or patterns that did not work, or that should be treated cautiously.\n",
|
|
"- [Promising ideas](./project_promising_ideas.md) - Directions worth revisiting or expanding in future experiment cycles.\n",
|
|
"- [External references](./reference_external_sources.md) - Pointers to papers, repos, dashboards, or notes that are useful but live outside the codebase.\n",
|
|
"\n",
|
|
"## Memory policy\n",
|
|
"\n",
|
|
"- Update existing topic files before creating new ones.\n",
|
|
"- Prefer synthesized conclusions over dumping raw logs.\n",
|
|
"- Treat concrete file/function/flag references as historical claims and re-verify before acting on them.\n",
|
|
"- Keep the exact experiment ledger in `results.tsv`; keep the append-only event log in `experiment_memory.jsonl`.\n",
|
|
"- Use this directory for cross-run synthesis, strategy, and retrieval-friendly summaries.\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"index_text = (MEMORY_ROOT / 'MEMORY.md').read_text(encoding='utf-8')\n",
|
|
"for required_link in [\n",
|
|
" './project_current_strategy.md',\n",
|
|
" './project_failed_ideas.md',\n",
|
|
" './project_promising_ideas.md',\n",
|
|
" './reference_external_sources.md',\n",
|
|
"]:\n",
|
|
" if required_link not in index_text:\n",
|
|
" raise RuntimeError(f'MEMORY.md is missing link: {required_link}')\n",
|
|
"print(index_text[:2000])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'project_current_strategy.md': {'name': 'Current strategy',\n",
|
|
" 'description': 'Current operating strategy for the autoresearch manager/worker system and how to use the control plane safely.',\n",
|
|
" 'type': 'project'},\n",
|
|
" 'project_failed_ideas.md': {'name': 'Failed ideas',\n",
|
|
" 'description': 'Failed or cautionary patterns for the autoresearch system, especially around control-plane validation and worker behavior.',\n",
|
|
" 'type': 'project'},\n",
|
|
" 'project_promising_ideas.md': {'name': 'Promising ideas',\n",
|
|
" 'description': 'Promising next steps for memory, orchestration, and experiment quality in autoresearch.',\n",
|
|
" 'type': 'project'},\n",
|
|
" 'reference_external_sources.md': {'name': 'External references',\n",
|
|
" 'description': 'External sources, tools, or future systems that are relevant to the autoresearch memory and orchestration roadmap.',\n",
|
|
" 'type': 'reference'}}"
|
|
]
|
|
},
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"def parse_frontmatter(path: Path) -> dict[str, str]:\n",
|
|
" text = path.read_text(encoding='utf-8')\n",
|
|
" match = re.match(r'^---\\n(.*?)\\n---\\n', text, re.DOTALL)\n",
|
|
" if not match:\n",
|
|
" raise RuntimeError(f'Missing frontmatter in {path.name}')\n",
|
|
" result = {}\n",
|
|
" for line in match.group(1).splitlines():\n",
|
|
" if ':' not in line:\n",
|
|
" continue\n",
|
|
" key, value = line.split(':', 1)\n",
|
|
" result[key.strip()] = value.strip()\n",
|
|
" for field in ('name', 'description', 'type'):\n",
|
|
" if field not in result or not result[field]:\n",
|
|
" raise RuntimeError(f'Missing {field} in {path.name}')\n",
|
|
" return result\n",
|
|
"\n",
|
|
"metadata = {\n",
|
|
" name: parse_frontmatter(MEMORY_ROOT / name)\n",
|
|
" for name in expected_files\n",
|
|
" if name != 'MEMORY.md'\n",
|
|
"}\n",
|
|
"metadata"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'results_tsv': '/Users/wongdowling/Documents/autoresearch_harness/nodes/autoresearch-macos/results.tsv', 'experiment_memory_jsonl': '/Users/wongdowling/Documents/autoresearch_harness/nodes/autoresearch-macos/experiment_memory.jsonl', 'event_count': 30}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"results_path = AUTORESEARCH_ROOT / 'results.tsv'\n",
|
|
"event_log_path = AUTORESEARCH_ROOT / 'experiment_memory.jsonl'\n",
|
|
"if not results_path.exists():\n",
|
|
" raise RuntimeError('results.tsv is missing')\n",
|
|
"if not event_log_path.exists():\n",
|
|
" raise RuntimeError('experiment_memory.jsonl is missing')\n",
|
|
"\n",
|
|
"event_count = sum(1 for line in event_log_path.read_text(encoding='utf-8').splitlines() if line.strip())\n",
|
|
"print({\n",
|
|
" 'results_tsv': str(results_path),\n",
|
|
" 'experiment_memory_jsonl': str(event_log_path),\n",
|
|
" 'event_count': event_count,\n",
|
|
"})"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'memory_dir_exists': True,\n",
|
|
" 'topic_file_count': 4,\n",
|
|
" 'has_exact_ledger': True,\n",
|
|
" 'has_append_only_event_log': True,\n",
|
|
" 'project_topic_types': ['project', 'reference']}"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"summary = {\n",
|
|
" 'memory_dir_exists': MEMORY_ROOT.exists(),\n",
|
|
" 'topic_file_count': len(expected_files) - 1,\n",
|
|
" 'has_exact_ledger': results_path.exists(),\n",
|
|
" 'has_append_only_event_log': event_log_path.exists(),\n",
|
|
" 'project_topic_types': sorted({item['type'] for item in metadata.values()}),\n",
|
|
"}\n",
|
|
"summary"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Success Criteria\n",
|
|
"\n",
|
|
"This notebook is a successful smoke test when:\n",
|
|
"\n",
|
|
"- the `memory/` directory exists\n",
|
|
"- `MEMORY.md` exists and links the expected topic files\n",
|
|
"- each topic file contains valid frontmatter with `name`, `description`, and `type`\n",
|
|
"- `results.tsv` still exists as the exact experiment ledger\n",
|
|
"- `experiment_memory.jsonl` still exists as the append-only event log\n",
|
|
"\n",
|
|
"That confirms the project now has the intended hybrid short-term memory layout:\n",
|
|
"\n",
|
|
"- exact operational records\n",
|
|
"- append-only event history\n",
|
|
"- small synthesized project memory\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "base",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.20"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|