diff --git a/.kiro/steering/api-standards.md b/.kiro/steering/api-standards.md new file mode 100644 index 00000000..298d4a41 --- /dev/null +++ b/.kiro/steering/api-standards.md @@ -0,0 +1,143 @@ +# API Standards + +These are the conventions for the **Flask backend** consumed by the +Vue frontend. Generic REST guidance is secondary to the patterns +already established in `backend/app/api/`. + +## Philosophy + +- The frontend is the only consumer; we optimize for *that* contract, + not for a hypothetical public API. +- Long-running work returns immediately with a `task_id`; clients + poll. There are no streaming responses or websockets. +- The backend is stateless across restarts of `Project`/`Task` data + (in-memory), with deterministic recovery on boot. + +## URL Pattern + +Routes live under `/api//` where domain matches the +Flask blueprint: + +- `/api/graph/...` — `graph_bp` (`api/graph.py`) +- `/api/simulation/...` — `simulation_bp` (`api/simulation.py`) +- `/api/report/...` — `report_bp` (`api/report.py`) + +Within a blueprint, resource sub-paths are accepted but **action-style +endpoints are equally common** (`/ontology/generate`, +`/project//reset`). Don't force REST verbs onto operations that +aren't naturally CRUD — match the surrounding file. + +The Vite dev server proxies `/api/*` from `:3000` to `:5001`. Don't +hard-code the backend host in frontend code. + +## Response Envelope + +Every response uses this shape — do **not** invent a new one: + +```json +// Success +{ "success": true, "data": { ... } } + +// Failure +{ "success": false, "error": "Human-readable message" } +``` + +- `success` is always present and boolean. +- Successful responses put the payload under `data`. List endpoints may + also include sibling fields (`count`, etc.) — see + `/project/list` for the precedent. +- Error responses use `error: `. There is no error-code enum. + Messages may be in English or Chinese to match the rest of the + module — keep both styles working. +- HTTP status follows the outcome: `200` on success, `400` for client + validation, `404` for missing entities, `500` for unhandled + exceptions. + +## Long-Running Operations: The Task Polling Contract + +This is the defining backend pattern. Anything that takes more than a +few seconds (ontology generation, graph build, profile generation, +simulation, report generation) **must** use it. + +### Submit endpoint + +- Validates input synchronously. +- Creates a `Task` via `TaskManager().create_task(task_type, metadata)`. +- Spawns a background `threading.Thread` that runs the work and + updates the task as it progresses. +- Returns immediately: + +```json +{ "success": true, "data": { "task_id": "...", "project_id": "..." } } +``` + +### Background worker + +- Calls `TaskManager().update_task(task_id, progress=…, message=…, progress_detail=…)` + at meaningful checkpoints (not every loop iteration). +- On success: `complete_task(task_id, result_dict)`. +- On failure: `fail_task(task_id, error_string)` — never let the + exception escape the thread; tasks must always reach a terminal + state. + +### Status endpoint + +- A polling endpoint (typically `/api//task/` or + similar) returns the current `Task.to_dict()`. +- The frontend service layer (`frontend/src/api/*.js`) handles + exponential backoff + a 5-min timeout; new endpoints don't need + custom retry logic on the client. + +### Task lifecycle + +`PENDING → PROCESSING → COMPLETED | FAILED`. Other status fields +(`progress` 0–100, `progress_detail` dict) are advisory — the frontend +decides how to render them. Don't add new statuses without a +frontend-side change. + +## Where Logic Belongs + +- **`api/` (handlers)**: validate input, look up `Project`/`Task`, + dispatch to a service, format the envelope. No graph access, no LLM + calls, no `subprocess`. +- **`services/`**: all business logic, including spawning the + background thread for long-running work. +- **`models/`**: state shape only. + +If a handler is doing more than a few lines of orchestration, the work +belongs in a service. + +## Authentication + +There is no user-level authentication today. Endpoints assume a +trusted operator on the same network (dev, Docker, internal +deployment). **Do not add ad-hoc auth checks scattered through +handlers** — if/when auth is needed, it goes through Flask middleware +and is documented in a new steering file. Until then, treat all +endpoints as authenticated by deployment. + +## Versioning + +No version prefix in URLs. The frontend ships with the backend in a +single repo, so backwards compatibility for the API is not a concern. +If that ever changes (public API, multiple frontend versions), version +the affected blueprint, not the whole API. + +## Pagination + +- `/project/list` accepts `?limit=` (default 50). Match this + pattern for new list endpoints. +- Graph queries use `utils/zep_paging.py` for cursor-style paging + (legacy name; still the canonical helper). + +## What Not to Do + +- Don't return raw exceptions or stack traces in `error`. +- Don't bypass `TaskManager` for long-running work (e.g. with a custom + status field on `Project`). +- Don't add new response envelope shapes — extend `data`. +- Don't introduce streaming (SSE, websockets) without a steering-level + discussion; the polling model is intentional. + +--- +_Focus on patterns and decisions, not endpoint catalogs._ diff --git a/.kiro/steering/database.md b/.kiro/steering/database.md new file mode 100644 index 00000000..a34dc840 --- /dev/null +++ b/.kiro/steering/database.md @@ -0,0 +1,123 @@ +# Database / Knowledge Graph Standards + +The "database" in MiroFish is **Neo4j accessed via Graphiti**, not a +relational store. There is no SQL, no migrations file, no ORM. Generic +relational guidance does not apply — these are the project-specific +patterns. + +## Architecture + +- **Engine**: Neo4j 5.x Community over `bolt://`. +- **Graph layer**: `graphiti-core` ≥ 0.3 — handles node/edge writes, + embeddings, hybrid search, reranking. +- **Adapter**: `backend/app/services/graphiti_adapter.py` is the **only** + module that imports `graphiti_core` directly. Every other module talks + to the graph through this adapter. + +The adapter exposes a Zep-Cloud-shaped namespace +(`client.graph.add_episode(...)`, `client.graph.search(...)`, etc.) so +legacy `zep_*` services kept their existing call sites after the +migration. New code should use the same surface — do not introduce a +parallel API. + +## Core Rule: `group_id` Isolation + +**Every read or write to the graph must be scoped by the project's +`group_id`.** The graph is multi-tenant by construction; cross-project +access is not permitted and is grounds for rejecting a change in review. + +- A project's `group_id` lives on its `Project` model and never changes + after creation. +- When constructing search filters, episode adds, or node/edge fetches, + always pass `group_id=project.group_id` (or the equivalent + `group_ids=[...]`). +- If you need data spanning projects (e.g. an admin view), aggregate + per-project at the API layer; do not query the graph without a + `group_id` filter. + +## Adapter Patterns That Must Stay Intact + +These are non-obvious and break subtly when violated: + +- **Single Graphiti singleton.** `_get_graphiti()` lazily constructs one + `Graphiti` instance for the whole process. Do not instantiate + `Graphiti` in services or tests. +- **Persistent event loop in a dedicated thread.** All async graph calls + are dispatched through `_run(coro)` onto a single background event + loop (see `graphiti-event-loop` thread). The Neo4j async driver is + bound to whichever loop opened it; crossing loops corrupts the driver + state. Never call `asyncio.run(...)` on a Graphiti coroutine, and + never schedule one on a request thread's loop. +- **Indices and constraints on first init.** `build_indices_and_constraints()` + runs once when the singleton is created. New required indexes go + through Graphiti's mechanisms, not raw Cypher in services. + +## What Belongs in the Graph + +- **Entities** — Domain objects extracted by the ontology generator + (people, organizations, concepts, events, etc.). +- **Edges** — Relationships between entities, typed per the project's + generated ontology. +- **Episodes** — The raw text/units the entities were derived from; + Graphiti owns chunking and embedding. + +What does **not** belong in the graph: + +- Project / task metadata (lives in in-memory `ProjectManager` and + `TaskManager`). +- Simulation state (owned by OASIS subprocesses). +- User-uploaded files (filesystem only — paths, not contents, are + passed through the API). + +## Schema & Ontology + +- Ontology (entity types + edge types) is **generated per project** by + the LLM in step 1, stored on the `Project` model, and used to + constrain extraction during graph build. +- There is no global, hand-maintained schema file. Don't add one — the + ontology is intentionally per-project. +- Reasoning-model outputs from ontology generation are stripped of + `` blocks and code fences before JSON parsing (see + `tech.md`'s "reasoning-model output stripping" decision). + +## Embeddings + +- `EMBEDDING_MODEL` is configurable per provider: + - OpenAI default: `text-embedding-3-small` + - Gemini: `text-embedding-004` / `gemini-embedding-001` +- Embedding model selection lives in `config.py`. Don't hard-code it in + services. +- Switching embedding model **invalidates existing project graphs** — + document this if you add an option that changes the default. + +## Query Patterns + +- Read via the adapter's search methods (hybrid RRF recipes are wired + in `graphiti_adapter.py`); avoid raw Cypher in feature code. +- If a feature genuinely requires raw Cypher, add it as a method on the + adapter, scoped by `group_id`, with a comment explaining why + Graphiti's API is insufficient. +- Pagination over Graphiti results uses `utils/zep_paging.py` (legacy + name, still applicable). + +## Startup Recovery + +`_recover_stuck_projects` runs on app boot and promotes any project +left in `GRAPH_BUILDING` to `GRAPH_COMPLETED` if the graph already has +that project's nodes — handling the case where the original task was +killed by a restart. **Any new long-running graph operation must +either:** + +1. Be safe to re-run from the start, OR +2. Add an analogous recovery path so a restart mid-task doesn't strand + the project. + +## Backups + +Graph data is treated as **regenerable from seed material**, not as +durable user data — there is no project-managed backup/restore. If a +deployment requires durability, that's an operator concern (Neo4j +backups), not a feature-code one. + +--- +_Focus on patterns and decisions. No environment-specific settings._ diff --git a/.kiro/steering/error-handling.md b/.kiro/steering/error-handling.md new file mode 100644 index 00000000..40dd50ee --- /dev/null +++ b/.kiro/steering/error-handling.md @@ -0,0 +1,140 @@ +# Error Handling Standards + +Most errors in MiroFish originate from **LLM calls**, **graph +operations**, **subprocess simulation**, or **user-uploaded files** — +not classical 4xx/5xx web flows. These standards target those failure +modes specifically. + +## Philosophy + +- Fail fast in services; convert to a stable response envelope at the + API layer. +- Long-running tasks must always reach a terminal state + (`COMPLETED` or `FAILED`) — a stuck `PROCESSING` task is a bug. +- LLM responses are untrusted by default: validate, strip, parse, then + use. +- Background-thread errors are silent unless explicitly captured — + always wrap the work in `try/except`. + +## Error Surfaces (where they appear, where they're handled) + +| Surface | Handle in | Convert to | +| -------------------- | ------------------------------------------ | --------------------------------- | +| HTTP request errors | `api/` handler `try/except` + envelope | `{"success": false, "error": …}` | +| Background task | Worker thread `try/except` → `fail_task()` | `Task.status = FAILED` + `error` | +| LLM call failures | `retry_with_backoff` decorator | Exception bubbles after retries | +| Graph adapter errors | Caller catches & maps | Service-specific error or `Task.fail` | +| Simulation IPC | `simulation_ipc.py` catches & logs | Task fail or simulation cleanup | +| File parsing | `utils/file_parser.py` | Raised as `ValueError` to caller | + +A handler should never let an exception reach Flask's default 500 +formatter — wrap and return the canonical envelope instead. + +## LLM-Specific Failure Modes + +These are recurring and worth handling explicitly: + +### 1. Reasoning-model output contamination + +Some providers (MiniMax, GLM, certain Qwen variants) emit `… +` blocks and/or markdown code fences (```` ```json ... ``` ````) +around JSON output. + +**Rule:** Strip both before `json.loads(...)`. The fix lives in commit +`985f89f` for context. Any new LLM-output JSON parser must do the same +— do not call `json.loads` on raw model output. + +### 2. Transient API errors + +Network blips, rate limits, intermittent 5xx from the provider. + +**Rule:** Use `utils/retry.py`: + +```python +from app.utils.retry import retry_with_backoff + +@retry_with_backoff(max_retries=3, exceptions=(SomeAPIError,)) +def call_llm(...): ... +``` + +- Sync version: `retry_with_backoff` +- Async version: `retry_with_backoff_async` +- For batch processing where partial failure is acceptable, use + `RetryableAPIClient.call_batch_with_retry(items, fn, + continue_on_failure=True)`. + +Don't write a hand-rolled retry loop — it'll drift from the project's +backoff/jitter conventions. + +### 3. Schema mismatch in structured output + +LLM returns valid JSON but missing/extra fields. + +**Rule:** Validate with Pydantic v2 models where the call expects +structure. Fail loudly (raise) rather than silently coercing — better +to retry the LLM call than to feed bad data downstream. + +## Background Task Errors + +Inside a worker thread spawned from an API handler: + +```python +def _worker(task_id, project_id, ...): + try: + # work + TaskManager().update_task(task_id, progress=50, message=...) + result = do_real_work(...) + TaskManager().complete_task(task_id, result) + except Exception as e: + logger.exception(f"task {task_id} failed") + TaskManager().fail_task(task_id, str(e)) +``` + +Rules: + +- The outer `except` must be broad (`Exception`) — the goal is "task + always terminates," not "narrow down failures here." +- Log the full traceback (`logger.exception`), then store a concise + `str(e)` on the task for the frontend to display. +- Never re-raise from the worker; the thread has no caller. +- Update related `Project` state (e.g. revert `GRAPH_BUILDING` → + previous status) **inside** the except, before `fail_task`. + +## Graph & Subprocess Errors + +- **Graphiti / Neo4j errors:** caller decides — usually fail the task + with a user-friendly message; for non-fatal search failures, log and + return empty results. +- **OASIS subprocess crashes:** `simulation_ipc.py` is the single + surface. It owns lifecycle, logging, and signaling task failure. + Don't catch subprocess errors elsewhere. +- **Startup recovery:** `_recover_stuck_projects` re-classifies + projects left `GRAPH_BUILDING` after a restart — see `database.md`. + +## Logging + +- Use `utils/logger.get_logger('mirofish.')` — never + `print` or `logging.getLogger` directly. +- Levels: + - `ERROR` — task failure, unrecoverable exception + - `WARNING` — retry triggered, transient failure, recovered state + - `INFO` — task lifecycle (created, completed), pipeline milestones + - `DEBUG` — payload shapes, intermediate counts, off by default +- User-visible log messages should go through `utils/locale.t(...)` so + they translate; internal diagnostic logs stay in the file's existing + language (English or Chinese — match the surrounding code). +- **Never log:** API keys, full LLM prompts containing user-uploaded + text (truncate or hash), Neo4j credentials, full `.env` contents. + +## What Not to Do + +- Don't catch `Exception` inside an API handler just to log and + continue — fail the request and return the envelope. +- Don't retry non-idempotent work (e.g. graph writes that may have + partially completed). +- Don't translate exceptions into `success: true` responses with an + embedded error message; use `success: false`. +- Don't surface raw stack traces or LLM internals to the frontend. + +--- +_Focus on patterns and decisions. No implementation details or exhaustive lists._ diff --git a/.kiro/steering/product.md b/.kiro/steering/product.md new file mode 100644 index 00000000..4dd55fbb --- /dev/null +++ b/.kiro/steering/product.md @@ -0,0 +1,57 @@ +# Product Overview + +MiroFish is a multi-agent **swarm intelligence prediction engine**. Given seed +material (news, policy drafts, financial signals, novel chapters, etc.) and a +natural-language prediction question, it builds a knowledge graph, populates a +parallel "digital sandbox" with thousands of personality-driven AI agents, +runs a social simulation, and returns an analytical report plus an explorable +simulated world. + +The user-facing experience is a guided **5-step workflow**: Graph Build → +Environment Setup → Simulation → Report → Interaction. Long-running steps +(LLM ontology extraction, graph build, profile generation, simulation, report) +execute as background tasks the UI polls for progress. + +## Core Capabilities + +- **Knowledge graph construction** — Files (PDF, text) are parsed, an LLM + extracts ontology, and Graphiti writes nodes/edges into Neo4j scoped per + project (`group_id`). +- **Persona-driven agent generation** — Entities pulled from the graph become + OASIS agent profiles with traits, memory, and behavior priors. +- **Dual-platform social simulation** — CAMEL-OASIS runs Twitter and Reddit + agents in parallel rounds with a configurable action set. +- **ReACT-loop report agent** — A reasoning agent answers the prediction + question using graph tools (`SearchResult`, `InsightForge`, `Panorama`, + `Interview`). +- **Post-simulation interaction** — Users can chat with any simulated agent + or the report agent to probe results. + +## Target Use Cases + +- **Macro decision rehearsal** — Stress-test policies, PR strategies, or + market moves against a synthetic public before committing. +- **Public-opinion / political forecasting** — Project how an event or + narrative may diffuse across social platforms. +- **Narrative and creative simulation** — Explore alternate endings, + what-if scenarios, or fiction continuations (e.g. *Dream of the Red + Chamber* lost-ending demo). +- **Operator-led research** — Internal analysts upload reports and inspect + the resulting graph + simulation rather than running ad-hoc surveys. + +## Value Proposition + +MiroFish converts a static document into a **dynamic, interrogable digital +society**. Where traditional forecasting summarizes data, MiroFish lets +decision-makers *watch the future play out* — observing emergent collective +behavior, intervening from a "god view," and reading both an analytical +report and the underlying agent interactions that produced it. + +The pipeline is deliberately **provider-agnostic** at the LLM layer (any +OpenAI-SDK-compatible endpoint works) and **self-hosted** at the graph layer +(Neo4j + Graphiti, no third-party graph service required), so the same +system can run from a developer laptop to a managed deployment without +vendor lock-in. + +--- +_Focus on patterns and purpose, not exhaustive feature lists_ diff --git a/.kiro/steering/structure.md b/.kiro/steering/structure.md new file mode 100644 index 00000000..7189acdc --- /dev/null +++ b/.kiro/steering/structure.md @@ -0,0 +1,166 @@ +# Project Structure + +## Organization Philosophy + +A **monorepo split by runtime** (`backend/` Python, `frontend/` Vue), +with each side organized **by layer**: + +- Backend: `api/` (HTTP) → `services/` (logic) → `models/` (state) → + `utils/` (cross-cutting helpers), with `config.py` as the single source + of configuration. +- Frontend: `views/` (route-level pages) → `components/` (reusable UI) → + `api/` (HTTP services) → `store/`, `router/`, `i18n/`, `assets/`. + +The core workflow is **pipeline-shaped** (5 sequential steps), and the +codebase mirrors that: each step has a backend service or service group, +a Flask blueprint endpoint, a frontend Step component, and (where useful) +a route-level view. + +## Directory Patterns + +### Backend HTTP Layer +**Location**: `backend/app/api/` +**Purpose**: One Flask blueprint per pipeline domain — thin handlers +that validate input, dispatch to services, and return JSON. No business +logic. +**Files**: `graph.py` (`graph_bp`), `simulation.py` (`simulation_bp`), +`report.py` (`report_bp`). + +### Backend Services +**Location**: `backend/app/services/` +**Purpose**: All business logic. Each file owns one responsibility +(graph build, profile generation, simulation runner, report agent, etc.). +**Pattern**: Long-running operations expose an async/background entrypoint +that returns a `Task` and runs work off the request thread. Direct calls +to Neo4j, OASIS subprocesses, or LLM streaming live here, not in `api/`. +**Naming note**: Files prefixed `zep_*` are legacy from the Zep→Graphiti +migration. Don't rename casually (imports across the project), and don't +add new `zep_*` files. + +### Backend State Models +**Location**: `backend/app/models/` +**Purpose**: In-memory, JSON-serializable state objects. `Project` +tracks per-project pipeline state and `group_id`; `Task` tracks +background-job status, progress, and result. These are the polling +contract with the frontend — change their shape with care. + +### Backend Utilities +**Location**: `backend/app/utils/` +**Purpose**: Cross-cutting helpers usable from any service — +LLM client wrapper (`llm_client.py`), file parsing (`file_parser.py`), +retry (`retry.py`), logging (`logger.py`), locale (`locale.py`), +pagination helpers (`zep_paging.py`). +**Rule**: Utils never import from `services/` or `api/`. + +### Backend Config +**Location**: `backend/app/config.py` +**Purpose**: Single file for LLM, Neo4j, embedding, chunking, OASIS, +and ReportAgent parameters. Read env vars here; consume the resulting +constants elsewhere. Avoid `os.getenv` calls scattered through services. + +### Frontend Views (Routes) +**Location**: `frontend/src/views/` +**Purpose**: Page-level components mapped to routes (`Home.vue`, +`MainView.vue`, `Process.vue`, `SimulationView.vue`, +`SimulationRunView.vue`, `InteractionView.vue`, `ReportView.vue`). +**Pattern**: `Process.vue` is the workflow orchestrator (~50KB); it +composes the Step components and owns step transitions. + +### Frontend Components +**Location**: `frontend/src/components/` +**Purpose**: Reusable UI. Step components (`Step1GraphBuild.vue`, +`Step2EnvSetup.vue`, `Step3Simulation.vue`, `Step4Report.vue`, +`Step5Interaction.vue`) implement one pipeline stage each; +`GraphPanel.vue` renders the D3 knowledge graph; +`HistoryDatabase.vue`, `LanguageSwitcher.vue` are general-purpose. + +### Frontend API Services +**Location**: `frontend/src/api/` +**Purpose**: Axios services that wrap the backend blueprints — +`graph.js`, `simulation.js`, `report.js`, with `index.js` as the shared +client (5-min timeout, exponential retry). +**Rule**: Components and views call these services; they do not import +`axios` directly. New endpoints add a method on the matching service. + +### Locales (shared) +**Location**: `/locales/` at repo root +**Purpose**: i18n source for both frontend (`vue-i18n`) and backend +(logger). Vite aliases `@locales` to this folder. Files: `en.json`, +`zh.json`, `languages.json`. + +### Static Assets +**Location**: `static/` at repo root +**Purpose**: Images and demo assets referenced from READMEs (logos, +screenshots, video covers). Not bundled by Vite. + +## Naming Conventions + +- **Python files / modules / functions / vars**: `snake_case` +- **Python classes**: `PascalCase` +- **Python constants**: `UPPER_SNAKE_CASE` +- **Vue Single-File Components**: `PascalCase.vue` +- **Vue route views**: `View.vue` or domain noun + (`Home.vue`, `Process.vue`, `MainView.vue`) +- **Vue step components**: `Step.vue` (matches the pipeline stage) +- **Frontend non-component JS**: `camelCase.js` (e.g. + `pendingUpload.js`) +- **Locale files**: lowercase ISO code (`en.json`, `zh.json`) + + `languages.json` for the language list +- **Booleans (Python and JS)**: prefix with `is_` / `has_` / `should_` + where it improves clarity, but match local style first + +## Import Organization + +### Frontend (`frontend/src/`) +```js +// Vendor +import { ref, computed } from 'vue' +import axios from 'axios' + +// Absolute (via @ alias) +import GraphPanel from '@/components/GraphPanel.vue' +import { fetchGraph } from '@/api/graph' + +// Locales (shared with backend) +import en from '@locales/en.json' + +// Relative (same feature only) +import { useStep } from './useStep' +``` + +**Path aliases** (`vite.config.js`): +- `@/` → `frontend/src/` +- `@locales` → repo-root `/locales/` + +### Backend (`backend/app/`) +- Use absolute package imports (`from app.services.graph_builder import ...`). +- Layer dependency rule: `api → services → models / utils`. Services + may import from `models` and `utils`; `models` and `utils` never + import from `services` or `api`. +- All Neo4j/Graphiti access goes through `services/graphiti_adapter.py`. + +## Code Organization Principles + +- **Pipeline-aligned modules.** When adding a new pipeline-touching + feature, place code in the same backend service group and the same + frontend Step component as the stage it belongs to. Don't split a + stage across multiple services unless responsibilities genuinely + diverge. +- **Background tasks are uniform.** Any operation taking more than a + few seconds returns a `Task` and is polled. Don't introduce ad-hoc + status fields on `Project`; extend `Task`. +- **Per-project isolation.** Every graph operation must filter by + `group_id`. Cross-project reads are out of scope and should be + flagged in review. +- **IPC has one door.** Subprocess communication for the simulator goes + through `services/simulation_ipc.py`. Do not call `subprocess` / + pipe primitives elsewhere. +- **Configuration is centralized.** New tunables go in + `backend/app/config.py` (and an `.env.example` line if env-driven), + not as constants scattered through services. +- **Legacy filenames stay.** `zep_*` files predate the Graphiti + migration; leave the names alone to avoid touching every importer, + but don't add new `zep_*` files. + +--- +_Document patterns, not file trees. New files following patterns shouldn't require updates_ diff --git a/.kiro/steering/tech.md b/.kiro/steering/tech.md new file mode 100644 index 00000000..de60d0a6 --- /dev/null +++ b/.kiro/steering/tech.md @@ -0,0 +1,157 @@ +# Technology Stack + +## Architecture + +A two-tier web app with a long-running **background-task** core: + +- **Frontend** (Vue 3 + Vite) — Single-page UI orchestrating the 5-step + workflow. Polls the backend for task progress; renders the knowledge + graph with D3. +- **Backend** (Flask + `uv`) — Stateless HTTP API on top of in-memory + `Project` and `Task` models. Heavy work (ontology extraction, graph + build, profile generation, simulation, report) runs as background + tasks tracked through `Task` and exposed via polling endpoints. +- **Knowledge graph** — Neo4j is the durable store; Graphiti is the + write/read layer. All queries are scoped by per-project `group_id`. +- **Simulation** — CAMEL-OASIS executes in subprocesses; the Flask app + communicates with them only through `services/simulation_ipc.py`. + +The system favors **process isolation** for the simulator and **in-memory +state with restart recovery** for project/task tracking, rather than a +classic job queue + persistent DB. + +## Core Technologies + +- **Backend language**: Python ≥3.11, ≤3.12 +- **Backend framework**: Flask 3.0 + flask-cors +- **Backend tooling**: `uv` for dependency management +- **Frontend framework**: Vue 3.5 + Vue Router 4 + `vue-i18n` 11 +- **Frontend tooling**: Vite 7 +- **Graph DB**: Neo4j 5.x (Community) via `bolt://` +- **Graph layer**: `graphiti-core` ≥ 0.3 +- **Simulation**: `camel-oasis` 0.2.5 + `camel-ai` 0.2.78 +- **LLM access**: OpenAI SDK against any OpenAI-compatible endpoint + +## Key Libraries + +Only the libraries that shape how new code is written: + +- **`openai`** — Sole LLM client; new providers are integrated by changing + `LLM_BASE_URL`/`LLM_MODEL_NAME`, **not** by adding a second SDK. +- **`graphiti-core`** — All graph reads/writes go through the + `graphiti_adapter`; do not call Neo4j drivers directly from feature + code. +- **`camel-oasis` / `camel-ai`** — Pinned versions; upgrading either + requires re-validating the simulation pipeline end-to-end. +- **`PyMuPDF`, `charset-normalizer`, `chardet`** — File ingestion; + encoding detection is mandatory because seed material is frequently + non-UTF-8 (notably mixed Chinese/English). +- **`pydantic` v2** — Used for structured LLM output / validation. +- **`axios`** (frontend) — All API calls go through `src/api/*.js` + services with a 5-min timeout and exponential retry; components must + not call `fetch`/`axios` directly. +- **`d3` v7** — Knowledge-graph visualization in `GraphPanel.vue`. + +## Development Standards + +### Type Safety +- Python: type hints where the surrounding file uses them. Don't retrofit + hints into untyped modules just for consistency. +- Frontend: plain JavaScript, not TypeScript. Use JSDoc only when it + improves clarity. + +### Code Quality +- **No enforced linter or formatter** in this repo by design. Match the + surrounding file's style. Discuss with the user before introducing + ESLint/Prettier/Ruff/Black. +- 4-space indentation everywhere. +- Python: `snake_case`. Existing files mix English and Chinese in + comments/docstrings — preserve both; do not translate one into the + other unless asked. + +### Testing +- pytest is wired (`backend/scripts/test_profile_format.py`) but coverage + is intentionally minimal. Don't add a heavy test harness without + discussing scope. +- For UI changes, run `npm run dev` and exercise the feature in a + browser; type-check/test passes do not prove feature correctness here. + +### Internationalization +- User-visible strings live in repo-root `/locales/*.json` (`en.json`, + `zh.json`, `languages.json`). The `frontend/vite.config.js` aliases + `@locales` to that root folder so the backend logger and frontend share + the same keys. +- Backend logger messages are part of the i18n surface — translate keys, + not raw log lines, when adding new logs that surface to users. + +## Development Environment + +### Required Tools + +| Tool | Version | +| --------- | ------------- | +| Node.js | ≥18 | +| Python | ≥3.11, ≤3.12 | +| `uv` | latest | +| Neo4j | 5.x Community | +| Docker | optional | + +### Common Commands + +```bash +# Setup (one-shot) +npm run setup:all + +# Dev (backend on :5001, frontend on :3000 with /api proxy) +npm run dev + +# Run individually +npm run backend +npm run frontend + +# Build frontend +npm run build + +# Backend tests +cd backend && uv run python -m pytest + +# Full stack (incl. Neo4j) +docker compose up +``` + +## Key Technical Decisions + +- **Neo4j + Graphiti replaces Zep Cloud.** Several services still carry + the legacy `zep_*` filename prefix (`zep_tools.py`, + `zep_entity_reader.py`, `zep_graph_memory_updater.py`). New code must + not depend on Zep Cloud. The `ZEP_API_KEY` env var is kept (empty + string is fine) only for backwards compatibility. +- **Per-project graph isolation via `group_id`.** Every Graphiti read or + write must filter by the project's `group_id`. There is no + cross-project graph access. +- **Reasoning-model output stripping.** Models like MiniMax and GLM emit + `` blocks and markdown fences; outputs are stripped before JSON + parsing (see commit `985f89f`). New LLM-output parsers must do the + same. +- **Background tasks via `Task` model, not a queue.** Anything taking + more than a few seconds returns immediately and tracks progress on a + `Task` object the frontend polls. There is no Celery/RQ/etc. +- **Startup recovery for stuck projects.** On boot, + `_recover_stuck_projects` promotes projects in `GRAPH_BUILDING` to + `GRAPH_COMPLETED` if Neo4j already has their nodes. New long-running + task types should follow the same recovery pattern. +- **Subprocess cleanup is centralized.** `SimulationRunner.register_cleanup()` + registers a shutdown hook so simulation subprocesses die with the app. + Don't spawn subprocesses outside this path. +- **Configuration is a single Python file.** `backend/app/config.py` + holds LLM, Neo4j, embedding, chunking, OASIS, and ReportAgent + settings. Prefer extending it over scattering env-var reads through + the codebase. +- **Default simulation parameters.** Max 10 rounds. Twitter actions: + `CREATE_POST`, `LIKE_POST`, `REPOST`, `FOLLOW`, `QUOTE_POST`, + `DO_NOTHING`. Reddit additionally: `CREATE_COMMENT`, `LIKE_COMMENT`, + `DISLIKE_*`, `SEARCH_*`, `TREND`, `REFRESH`, `MUTE`. Changes go in + `config.py`, not per-call. + +--- +_Document standards and patterns, not every dependency_