chore(claude): bootstrap kiro steering files (Step 5)
Add .kiro/steering/ as persistent project memory for CC-SDD / Kiro
workflows. Three core files (product, tech, structure) capture
purpose, stack, and organization patterns; three custom files
(database, api-standards, error-handling) pin the load-bearing
project-specific conventions:
- group_id isolation and the Graphiti adapter / event-loop singleton
- {success, data|error} envelope and the Task polling contract
- reasoning-model output stripping and the retry_with_backoff helper
Files focus on patterns and decisions, not catalogs, per the
steering-principles "golden rule".
This commit is contained in:
Dominik Seemann
parent
8144290fc6
commit
d4f1a9aee0
|
|
@ -0,0 +1,143 @@
|
|||
# API Standards
|
||||
|
||||
These are the conventions for the **Flask backend** consumed by the
|
||||
Vue frontend. Generic REST guidance is secondary to the patterns
|
||||
already established in `backend/app/api/`.
|
||||
|
||||
## Philosophy
|
||||
|
||||
- The frontend is the only consumer; we optimize for *that* contract,
|
||||
not for a hypothetical public API.
|
||||
- Long-running work returns immediately with a `task_id`; clients
|
||||
poll. There are no streaming responses or websockets.
|
||||
- The backend is stateless across restarts of `Project`/`Task` data
|
||||
(in-memory), with deterministic recovery on boot.
|
||||
|
||||
## URL Pattern
|
||||
|
||||
Routes live under `/api/<domain>/<action>` where domain matches the
|
||||
Flask blueprint:
|
||||
|
||||
- `/api/graph/...` — `graph_bp` (`api/graph.py`)
|
||||
- `/api/simulation/...` — `simulation_bp` (`api/simulation.py`)
|
||||
- `/api/report/...` — `report_bp` (`api/report.py`)
|
||||
|
||||
Within a blueprint, resource sub-paths are accepted but **action-style
|
||||
endpoints are equally common** (`/ontology/generate`,
|
||||
`/project/<id>/reset`). Don't force REST verbs onto operations that
|
||||
aren't naturally CRUD — match the surrounding file.
|
||||
|
||||
The Vite dev server proxies `/api/*` from `:3000` to `:5001`. Don't
|
||||
hard-code the backend host in frontend code.
|
||||
|
||||
## Response Envelope
|
||||
|
||||
Every response uses this shape — do **not** invent a new one:
|
||||
|
||||
```json
|
||||
// Success
|
||||
{ "success": true, "data": { ... } }
|
||||
|
||||
// Failure
|
||||
{ "success": false, "error": "Human-readable message" }
|
||||
```
|
||||
|
||||
- `success` is always present and boolean.
|
||||
- Successful responses put the payload under `data`. List endpoints may
|
||||
also include sibling fields (`count`, etc.) — see
|
||||
`/project/list` for the precedent.
|
||||
- Error responses use `error: <string>`. There is no error-code enum.
|
||||
Messages may be in English or Chinese to match the rest of the
|
||||
module — keep both styles working.
|
||||
- HTTP status follows the outcome: `200` on success, `400` for client
|
||||
validation, `404` for missing entities, `500` for unhandled
|
||||
exceptions.
|
||||
|
||||
## Long-Running Operations: The Task Polling Contract
|
||||
|
||||
This is the defining backend pattern. Anything that takes more than a
|
||||
few seconds (ontology generation, graph build, profile generation,
|
||||
simulation, report generation) **must** use it.
|
||||
|
||||
### Submit endpoint
|
||||
|
||||
- Validates input synchronously.
|
||||
- Creates a `Task` via `TaskManager().create_task(task_type, metadata)`.
|
||||
- Spawns a background `threading.Thread` that runs the work and
|
||||
updates the task as it progresses.
|
||||
- Returns immediately:
|
||||
|
||||
```json
|
||||
{ "success": true, "data": { "task_id": "...", "project_id": "..." } }
|
||||
```
|
||||
|
||||
### Background worker
|
||||
|
||||
- Calls `TaskManager().update_task(task_id, progress=…, message=…, progress_detail=…)`
|
||||
at meaningful checkpoints (not every loop iteration).
|
||||
- On success: `complete_task(task_id, result_dict)`.
|
||||
- On failure: `fail_task(task_id, error_string)` — never let the
|
||||
exception escape the thread; tasks must always reach a terminal
|
||||
state.
|
||||
|
||||
### Status endpoint
|
||||
|
||||
- A polling endpoint (typically `/api/<domain>/task/<task_id>` or
|
||||
similar) returns the current `Task.to_dict()`.
|
||||
- The frontend service layer (`frontend/src/api/*.js`) handles
|
||||
exponential backoff + a 5-min timeout; new endpoints don't need
|
||||
custom retry logic on the client.
|
||||
|
||||
### Task lifecycle
|
||||
|
||||
`PENDING → PROCESSING → COMPLETED | FAILED`. Other status fields
|
||||
(`progress` 0–100, `progress_detail` dict) are advisory — the frontend
|
||||
decides how to render them. Don't add new statuses without a
|
||||
frontend-side change.
|
||||
|
||||
## Where Logic Belongs
|
||||
|
||||
- **`api/` (handlers)**: validate input, look up `Project`/`Task`,
|
||||
dispatch to a service, format the envelope. No graph access, no LLM
|
||||
calls, no `subprocess`.
|
||||
- **`services/`**: all business logic, including spawning the
|
||||
background thread for long-running work.
|
||||
- **`models/`**: state shape only.
|
||||
|
||||
If a handler is doing more than a few lines of orchestration, the work
|
||||
belongs in a service.
|
||||
|
||||
## Authentication
|
||||
|
||||
There is no user-level authentication today. Endpoints assume a
|
||||
trusted operator on the same network (dev, Docker, internal
|
||||
deployment). **Do not add ad-hoc auth checks scattered through
|
||||
handlers** — if/when auth is needed, it goes through Flask middleware
|
||||
and is documented in a new steering file. Until then, treat all
|
||||
endpoints as authenticated by deployment.
|
||||
|
||||
## Versioning
|
||||
|
||||
No version prefix in URLs. The frontend ships with the backend in a
|
||||
single repo, so backwards compatibility for the API is not a concern.
|
||||
If that ever changes (public API, multiple frontend versions), version
|
||||
the affected blueprint, not the whole API.
|
||||
|
||||
## Pagination
|
||||
|
||||
- `/project/list` accepts `?limit=<n>` (default 50). Match this
|
||||
pattern for new list endpoints.
|
||||
- Graph queries use `utils/zep_paging.py` for cursor-style paging
|
||||
(legacy name; still the canonical helper).
|
||||
|
||||
## What Not to Do
|
||||
|
||||
- Don't return raw exceptions or stack traces in `error`.
|
||||
- Don't bypass `TaskManager` for long-running work (e.g. with a custom
|
||||
status field on `Project`).
|
||||
- Don't add new response envelope shapes — extend `data`.
|
||||
- Don't introduce streaming (SSE, websockets) without a steering-level
|
||||
discussion; the polling model is intentional.
|
||||
|
||||
---
|
||||
_Focus on patterns and decisions, not endpoint catalogs._
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
# Database / Knowledge Graph Standards
|
||||
|
||||
The "database" in MiroFish is **Neo4j accessed via Graphiti**, not a
|
||||
relational store. There is no SQL, no migrations file, no ORM. Generic
|
||||
relational guidance does not apply — these are the project-specific
|
||||
patterns.
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Engine**: Neo4j 5.x Community over `bolt://`.
|
||||
- **Graph layer**: `graphiti-core` ≥ 0.3 — handles node/edge writes,
|
||||
embeddings, hybrid search, reranking.
|
||||
- **Adapter**: `backend/app/services/graphiti_adapter.py` is the **only**
|
||||
module that imports `graphiti_core` directly. Every other module talks
|
||||
to the graph through this adapter.
|
||||
|
||||
The adapter exposes a Zep-Cloud-shaped namespace
|
||||
(`client.graph.add_episode(...)`, `client.graph.search(...)`, etc.) so
|
||||
legacy `zep_*` services kept their existing call sites after the
|
||||
migration. New code should use the same surface — do not introduce a
|
||||
parallel API.
|
||||
|
||||
## Core Rule: `group_id` Isolation
|
||||
|
||||
**Every read or write to the graph must be scoped by the project's
|
||||
`group_id`.** The graph is multi-tenant by construction; cross-project
|
||||
access is not permitted and is grounds for rejecting a change in review.
|
||||
|
||||
- A project's `group_id` lives on its `Project` model and never changes
|
||||
after creation.
|
||||
- When constructing search filters, episode adds, or node/edge fetches,
|
||||
always pass `group_id=project.group_id` (or the equivalent
|
||||
`group_ids=[...]`).
|
||||
- If you need data spanning projects (e.g. an admin view), aggregate
|
||||
per-project at the API layer; do not query the graph without a
|
||||
`group_id` filter.
|
||||
|
||||
## Adapter Patterns That Must Stay Intact
|
||||
|
||||
These are non-obvious and break subtly when violated:
|
||||
|
||||
- **Single Graphiti singleton.** `_get_graphiti()` lazily constructs one
|
||||
`Graphiti` instance for the whole process. Do not instantiate
|
||||
`Graphiti` in services or tests.
|
||||
- **Persistent event loop in a dedicated thread.** All async graph calls
|
||||
are dispatched through `_run(coro)` onto a single background event
|
||||
loop (see `graphiti-event-loop` thread). The Neo4j async driver is
|
||||
bound to whichever loop opened it; crossing loops corrupts the driver
|
||||
state. Never call `asyncio.run(...)` on a Graphiti coroutine, and
|
||||
never schedule one on a request thread's loop.
|
||||
- **Indices and constraints on first init.** `build_indices_and_constraints()`
|
||||
runs once when the singleton is created. New required indexes go
|
||||
through Graphiti's mechanisms, not raw Cypher in services.
|
||||
|
||||
## What Belongs in the Graph
|
||||
|
||||
- **Entities** — Domain objects extracted by the ontology generator
|
||||
(people, organizations, concepts, events, etc.).
|
||||
- **Edges** — Relationships between entities, typed per the project's
|
||||
generated ontology.
|
||||
- **Episodes** — The raw text/units the entities were derived from;
|
||||
Graphiti owns chunking and embedding.
|
||||
|
||||
What does **not** belong in the graph:
|
||||
|
||||
- Project / task metadata (lives in in-memory `ProjectManager` and
|
||||
`TaskManager`).
|
||||
- Simulation state (owned by OASIS subprocesses).
|
||||
- User-uploaded files (filesystem only — paths, not contents, are
|
||||
passed through the API).
|
||||
|
||||
## Schema & Ontology
|
||||
|
||||
- Ontology (entity types + edge types) is **generated per project** by
|
||||
the LLM in step 1, stored on the `Project` model, and used to
|
||||
constrain extraction during graph build.
|
||||
- There is no global, hand-maintained schema file. Don't add one — the
|
||||
ontology is intentionally per-project.
|
||||
- Reasoning-model outputs from ontology generation are stripped of
|
||||
`<think>` blocks and code fences before JSON parsing (see
|
||||
`tech.md`'s "reasoning-model output stripping" decision).
|
||||
|
||||
## Embeddings
|
||||
|
||||
- `EMBEDDING_MODEL` is configurable per provider:
|
||||
- OpenAI default: `text-embedding-3-small`
|
||||
- Gemini: `text-embedding-004` / `gemini-embedding-001`
|
||||
- Embedding model selection lives in `config.py`. Don't hard-code it in
|
||||
services.
|
||||
- Switching embedding model **invalidates existing project graphs** —
|
||||
document this if you add an option that changes the default.
|
||||
|
||||
## Query Patterns
|
||||
|
||||
- Read via the adapter's search methods (hybrid RRF recipes are wired
|
||||
in `graphiti_adapter.py`); avoid raw Cypher in feature code.
|
||||
- If a feature genuinely requires raw Cypher, add it as a method on the
|
||||
adapter, scoped by `group_id`, with a comment explaining why
|
||||
Graphiti's API is insufficient.
|
||||
- Pagination over Graphiti results uses `utils/zep_paging.py` (legacy
|
||||
name, still applicable).
|
||||
|
||||
## Startup Recovery
|
||||
|
||||
`_recover_stuck_projects` runs on app boot and promotes any project
|
||||
left in `GRAPH_BUILDING` to `GRAPH_COMPLETED` if the graph already has
|
||||
that project's nodes — handling the case where the original task was
|
||||
killed by a restart. **Any new long-running graph operation must
|
||||
either:**
|
||||
|
||||
1. Be safe to re-run from the start, OR
|
||||
2. Add an analogous recovery path so a restart mid-task doesn't strand
|
||||
the project.
|
||||
|
||||
## Backups
|
||||
|
||||
Graph data is treated as **regenerable from seed material**, not as
|
||||
durable user data — there is no project-managed backup/restore. If a
|
||||
deployment requires durability, that's an operator concern (Neo4j
|
||||
backups), not a feature-code one.
|
||||
|
||||
---
|
||||
_Focus on patterns and decisions. No environment-specific settings._
|
||||
|
|
@ -0,0 +1,140 @@
|
|||
# Error Handling Standards
|
||||
|
||||
Most errors in MiroFish originate from **LLM calls**, **graph
|
||||
operations**, **subprocess simulation**, or **user-uploaded files** —
|
||||
not classical 4xx/5xx web flows. These standards target those failure
|
||||
modes specifically.
|
||||
|
||||
## Philosophy
|
||||
|
||||
- Fail fast in services; convert to a stable response envelope at the
|
||||
API layer.
|
||||
- Long-running tasks must always reach a terminal state
|
||||
(`COMPLETED` or `FAILED`) — a stuck `PROCESSING` task is a bug.
|
||||
- LLM responses are untrusted by default: validate, strip, parse, then
|
||||
use.
|
||||
- Background-thread errors are silent unless explicitly captured —
|
||||
always wrap the work in `try/except`.
|
||||
|
||||
## Error Surfaces (where they appear, where they're handled)
|
||||
|
||||
| Surface | Handle in | Convert to |
|
||||
| -------------------- | ------------------------------------------ | --------------------------------- |
|
||||
| HTTP request errors | `api/` handler `try/except` + envelope | `{"success": false, "error": …}` |
|
||||
| Background task | Worker thread `try/except` → `fail_task()` | `Task.status = FAILED` + `error` |
|
||||
| LLM call failures | `retry_with_backoff` decorator | Exception bubbles after retries |
|
||||
| Graph adapter errors | Caller catches & maps | Service-specific error or `Task.fail` |
|
||||
| Simulation IPC | `simulation_ipc.py` catches & logs | Task fail or simulation cleanup |
|
||||
| File parsing | `utils/file_parser.py` | Raised as `ValueError` to caller |
|
||||
|
||||
A handler should never let an exception reach Flask's default 500
|
||||
formatter — wrap and return the canonical envelope instead.
|
||||
|
||||
## LLM-Specific Failure Modes
|
||||
|
||||
These are recurring and worth handling explicitly:
|
||||
|
||||
### 1. Reasoning-model output contamination
|
||||
|
||||
Some providers (MiniMax, GLM, certain Qwen variants) emit `<think>…
|
||||
</think>` blocks and/or markdown code fences (```` ```json ... ``` ````)
|
||||
around JSON output.
|
||||
|
||||
**Rule:** Strip both before `json.loads(...)`. The fix lives in commit
|
||||
`985f89f` for context. Any new LLM-output JSON parser must do the same
|
||||
— do not call `json.loads` on raw model output.
|
||||
|
||||
### 2. Transient API errors
|
||||
|
||||
Network blips, rate limits, intermittent 5xx from the provider.
|
||||
|
||||
**Rule:** Use `utils/retry.py`:
|
||||
|
||||
```python
|
||||
from app.utils.retry import retry_with_backoff
|
||||
|
||||
@retry_with_backoff(max_retries=3, exceptions=(SomeAPIError,))
|
||||
def call_llm(...): ...
|
||||
```
|
||||
|
||||
- Sync version: `retry_with_backoff`
|
||||
- Async version: `retry_with_backoff_async`
|
||||
- For batch processing where partial failure is acceptable, use
|
||||
`RetryableAPIClient.call_batch_with_retry(items, fn,
|
||||
continue_on_failure=True)`.
|
||||
|
||||
Don't write a hand-rolled retry loop — it'll drift from the project's
|
||||
backoff/jitter conventions.
|
||||
|
||||
### 3. Schema mismatch in structured output
|
||||
|
||||
LLM returns valid JSON but missing/extra fields.
|
||||
|
||||
**Rule:** Validate with Pydantic v2 models where the call expects
|
||||
structure. Fail loudly (raise) rather than silently coercing — better
|
||||
to retry the LLM call than to feed bad data downstream.
|
||||
|
||||
## Background Task Errors
|
||||
|
||||
Inside a worker thread spawned from an API handler:
|
||||
|
||||
```python
|
||||
def _worker(task_id, project_id, ...):
|
||||
try:
|
||||
# work
|
||||
TaskManager().update_task(task_id, progress=50, message=...)
|
||||
result = do_real_work(...)
|
||||
TaskManager().complete_task(task_id, result)
|
||||
except Exception as e:
|
||||
logger.exception(f"task {task_id} failed")
|
||||
TaskManager().fail_task(task_id, str(e))
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- The outer `except` must be broad (`Exception`) — the goal is "task
|
||||
always terminates," not "narrow down failures here."
|
||||
- Log the full traceback (`logger.exception`), then store a concise
|
||||
`str(e)` on the task for the frontend to display.
|
||||
- Never re-raise from the worker; the thread has no caller.
|
||||
- Update related `Project` state (e.g. revert `GRAPH_BUILDING` →
|
||||
previous status) **inside** the except, before `fail_task`.
|
||||
|
||||
## Graph & Subprocess Errors
|
||||
|
||||
- **Graphiti / Neo4j errors:** caller decides — usually fail the task
|
||||
with a user-friendly message; for non-fatal search failures, log and
|
||||
return empty results.
|
||||
- **OASIS subprocess crashes:** `simulation_ipc.py` is the single
|
||||
surface. It owns lifecycle, logging, and signaling task failure.
|
||||
Don't catch subprocess errors elsewhere.
|
||||
- **Startup recovery:** `_recover_stuck_projects` re-classifies
|
||||
projects left `GRAPH_BUILDING` after a restart — see `database.md`.
|
||||
|
||||
## Logging
|
||||
|
||||
- Use `utils/logger.get_logger('mirofish.<module>')` — never
|
||||
`print` or `logging.getLogger` directly.
|
||||
- Levels:
|
||||
- `ERROR` — task failure, unrecoverable exception
|
||||
- `WARNING` — retry triggered, transient failure, recovered state
|
||||
- `INFO` — task lifecycle (created, completed), pipeline milestones
|
||||
- `DEBUG` — payload shapes, intermediate counts, off by default
|
||||
- User-visible log messages should go through `utils/locale.t(...)` so
|
||||
they translate; internal diagnostic logs stay in the file's existing
|
||||
language (English or Chinese — match the surrounding code).
|
||||
- **Never log:** API keys, full LLM prompts containing user-uploaded
|
||||
text (truncate or hash), Neo4j credentials, full `.env` contents.
|
||||
|
||||
## What Not to Do
|
||||
|
||||
- Don't catch `Exception` inside an API handler just to log and
|
||||
continue — fail the request and return the envelope.
|
||||
- Don't retry non-idempotent work (e.g. graph writes that may have
|
||||
partially completed).
|
||||
- Don't translate exceptions into `success: true` responses with an
|
||||
embedded error message; use `success: false`.
|
||||
- Don't surface raw stack traces or LLM internals to the frontend.
|
||||
|
||||
---
|
||||
_Focus on patterns and decisions. No implementation details or exhaustive lists._
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
# Product Overview
|
||||
|
||||
MiroFish is a multi-agent **swarm intelligence prediction engine**. Given seed
|
||||
material (news, policy drafts, financial signals, novel chapters, etc.) and a
|
||||
natural-language prediction question, it builds a knowledge graph, populates a
|
||||
parallel "digital sandbox" with thousands of personality-driven AI agents,
|
||||
runs a social simulation, and returns an analytical report plus an explorable
|
||||
simulated world.
|
||||
|
||||
The user-facing experience is a guided **5-step workflow**: Graph Build →
|
||||
Environment Setup → Simulation → Report → Interaction. Long-running steps
|
||||
(LLM ontology extraction, graph build, profile generation, simulation, report)
|
||||
execute as background tasks the UI polls for progress.
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
- **Knowledge graph construction** — Files (PDF, text) are parsed, an LLM
|
||||
extracts ontology, and Graphiti writes nodes/edges into Neo4j scoped per
|
||||
project (`group_id`).
|
||||
- **Persona-driven agent generation** — Entities pulled from the graph become
|
||||
OASIS agent profiles with traits, memory, and behavior priors.
|
||||
- **Dual-platform social simulation** — CAMEL-OASIS runs Twitter and Reddit
|
||||
agents in parallel rounds with a configurable action set.
|
||||
- **ReACT-loop report agent** — A reasoning agent answers the prediction
|
||||
question using graph tools (`SearchResult`, `InsightForge`, `Panorama`,
|
||||
`Interview`).
|
||||
- **Post-simulation interaction** — Users can chat with any simulated agent
|
||||
or the report agent to probe results.
|
||||
|
||||
## Target Use Cases
|
||||
|
||||
- **Macro decision rehearsal** — Stress-test policies, PR strategies, or
|
||||
market moves against a synthetic public before committing.
|
||||
- **Public-opinion / political forecasting** — Project how an event or
|
||||
narrative may diffuse across social platforms.
|
||||
- **Narrative and creative simulation** — Explore alternate endings,
|
||||
what-if scenarios, or fiction continuations (e.g. *Dream of the Red
|
||||
Chamber* lost-ending demo).
|
||||
- **Operator-led research** — Internal analysts upload reports and inspect
|
||||
the resulting graph + simulation rather than running ad-hoc surveys.
|
||||
|
||||
## Value Proposition
|
||||
|
||||
MiroFish converts a static document into a **dynamic, interrogable digital
|
||||
society**. Where traditional forecasting summarizes data, MiroFish lets
|
||||
decision-makers *watch the future play out* — observing emergent collective
|
||||
behavior, intervening from a "god view," and reading both an analytical
|
||||
report and the underlying agent interactions that produced it.
|
||||
|
||||
The pipeline is deliberately **provider-agnostic** at the LLM layer (any
|
||||
OpenAI-SDK-compatible endpoint works) and **self-hosted** at the graph layer
|
||||
(Neo4j + Graphiti, no third-party graph service required), so the same
|
||||
system can run from a developer laptop to a managed deployment without
|
||||
vendor lock-in.
|
||||
|
||||
---
|
||||
_Focus on patterns and purpose, not exhaustive feature lists_
|
||||
|
|
@ -0,0 +1,166 @@
|
|||
# Project Structure
|
||||
|
||||
## Organization Philosophy
|
||||
|
||||
A **monorepo split by runtime** (`backend/` Python, `frontend/` Vue),
|
||||
with each side organized **by layer**:
|
||||
|
||||
- Backend: `api/` (HTTP) → `services/` (logic) → `models/` (state) →
|
||||
`utils/` (cross-cutting helpers), with `config.py` as the single source
|
||||
of configuration.
|
||||
- Frontend: `views/` (route-level pages) → `components/` (reusable UI) →
|
||||
`api/` (HTTP services) → `store/`, `router/`, `i18n/`, `assets/`.
|
||||
|
||||
The core workflow is **pipeline-shaped** (5 sequential steps), and the
|
||||
codebase mirrors that: each step has a backend service or service group,
|
||||
a Flask blueprint endpoint, a frontend Step component, and (where useful)
|
||||
a route-level view.
|
||||
|
||||
## Directory Patterns
|
||||
|
||||
### Backend HTTP Layer
|
||||
**Location**: `backend/app/api/`
|
||||
**Purpose**: One Flask blueprint per pipeline domain — thin handlers
|
||||
that validate input, dispatch to services, and return JSON. No business
|
||||
logic.
|
||||
**Files**: `graph.py` (`graph_bp`), `simulation.py` (`simulation_bp`),
|
||||
`report.py` (`report_bp`).
|
||||
|
||||
### Backend Services
|
||||
**Location**: `backend/app/services/`
|
||||
**Purpose**: All business logic. Each file owns one responsibility
|
||||
(graph build, profile generation, simulation runner, report agent, etc.).
|
||||
**Pattern**: Long-running operations expose an async/background entrypoint
|
||||
that returns a `Task` and runs work off the request thread. Direct calls
|
||||
to Neo4j, OASIS subprocesses, or LLM streaming live here, not in `api/`.
|
||||
**Naming note**: Files prefixed `zep_*` are legacy from the Zep→Graphiti
|
||||
migration. Don't rename casually (imports across the project), and don't
|
||||
add new `zep_*` files.
|
||||
|
||||
### Backend State Models
|
||||
**Location**: `backend/app/models/`
|
||||
**Purpose**: In-memory, JSON-serializable state objects. `Project`
|
||||
tracks per-project pipeline state and `group_id`; `Task` tracks
|
||||
background-job status, progress, and result. These are the polling
|
||||
contract with the frontend — change their shape with care.
|
||||
|
||||
### Backend Utilities
|
||||
**Location**: `backend/app/utils/`
|
||||
**Purpose**: Cross-cutting helpers usable from any service —
|
||||
LLM client wrapper (`llm_client.py`), file parsing (`file_parser.py`),
|
||||
retry (`retry.py`), logging (`logger.py`), locale (`locale.py`),
|
||||
pagination helpers (`zep_paging.py`).
|
||||
**Rule**: Utils never import from `services/` or `api/`.
|
||||
|
||||
### Backend Config
|
||||
**Location**: `backend/app/config.py`
|
||||
**Purpose**: Single file for LLM, Neo4j, embedding, chunking, OASIS,
|
||||
and ReportAgent parameters. Read env vars here; consume the resulting
|
||||
constants elsewhere. Avoid `os.getenv` calls scattered through services.
|
||||
|
||||
### Frontend Views (Routes)
|
||||
**Location**: `frontend/src/views/`
|
||||
**Purpose**: Page-level components mapped to routes (`Home.vue`,
|
||||
`MainView.vue`, `Process.vue`, `SimulationView.vue`,
|
||||
`SimulationRunView.vue`, `InteractionView.vue`, `ReportView.vue`).
|
||||
**Pattern**: `Process.vue` is the workflow orchestrator (~50KB); it
|
||||
composes the Step components and owns step transitions.
|
||||
|
||||
### Frontend Components
|
||||
**Location**: `frontend/src/components/`
|
||||
**Purpose**: Reusable UI. Step components (`Step1GraphBuild.vue`,
|
||||
`Step2EnvSetup.vue`, `Step3Simulation.vue`, `Step4Report.vue`,
|
||||
`Step5Interaction.vue`) implement one pipeline stage each;
|
||||
`GraphPanel.vue` renders the D3 knowledge graph;
|
||||
`HistoryDatabase.vue`, `LanguageSwitcher.vue` are general-purpose.
|
||||
|
||||
### Frontend API Services
|
||||
**Location**: `frontend/src/api/`
|
||||
**Purpose**: Axios services that wrap the backend blueprints —
|
||||
`graph.js`, `simulation.js`, `report.js`, with `index.js` as the shared
|
||||
client (5-min timeout, exponential retry).
|
||||
**Rule**: Components and views call these services; they do not import
|
||||
`axios` directly. New endpoints add a method on the matching service.
|
||||
|
||||
### Locales (shared)
|
||||
**Location**: `/locales/` at repo root
|
||||
**Purpose**: i18n source for both frontend (`vue-i18n`) and backend
|
||||
(logger). Vite aliases `@locales` to this folder. Files: `en.json`,
|
||||
`zh.json`, `languages.json`.
|
||||
|
||||
### Static Assets
|
||||
**Location**: `static/` at repo root
|
||||
**Purpose**: Images and demo assets referenced from READMEs (logos,
|
||||
screenshots, video covers). Not bundled by Vite.
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
- **Python files / modules / functions / vars**: `snake_case`
|
||||
- **Python classes**: `PascalCase`
|
||||
- **Python constants**: `UPPER_SNAKE_CASE`
|
||||
- **Vue Single-File Components**: `PascalCase.vue`
|
||||
- **Vue route views**: `<Name>View.vue` or domain noun
|
||||
(`Home.vue`, `Process.vue`, `MainView.vue`)
|
||||
- **Vue step components**: `Step<N><Name>.vue` (matches the pipeline stage)
|
||||
- **Frontend non-component JS**: `camelCase.js` (e.g.
|
||||
`pendingUpload.js`)
|
||||
- **Locale files**: lowercase ISO code (`en.json`, `zh.json`) +
|
||||
`languages.json` for the language list
|
||||
- **Booleans (Python and JS)**: prefix with `is_` / `has_` / `should_`
|
||||
where it improves clarity, but match local style first
|
||||
|
||||
## Import Organization
|
||||
|
||||
### Frontend (`frontend/src/`)
|
||||
```js
|
||||
// Vendor
|
||||
import { ref, computed } from 'vue'
|
||||
import axios from 'axios'
|
||||
|
||||
// Absolute (via @ alias)
|
||||
import GraphPanel from '@/components/GraphPanel.vue'
|
||||
import { fetchGraph } from '@/api/graph'
|
||||
|
||||
// Locales (shared with backend)
|
||||
import en from '@locales/en.json'
|
||||
|
||||
// Relative (same feature only)
|
||||
import { useStep } from './useStep'
|
||||
```
|
||||
|
||||
**Path aliases** (`vite.config.js`):
|
||||
- `@/` → `frontend/src/`
|
||||
- `@locales` → repo-root `/locales/`
|
||||
|
||||
### Backend (`backend/app/`)
|
||||
- Use absolute package imports (`from app.services.graph_builder import ...`).
|
||||
- Layer dependency rule: `api → services → models / utils`. Services
|
||||
may import from `models` and `utils`; `models` and `utils` never
|
||||
import from `services` or `api`.
|
||||
- All Neo4j/Graphiti access goes through `services/graphiti_adapter.py`.
|
||||
|
||||
## Code Organization Principles
|
||||
|
||||
- **Pipeline-aligned modules.** When adding a new pipeline-touching
|
||||
feature, place code in the same backend service group and the same
|
||||
frontend Step component as the stage it belongs to. Don't split a
|
||||
stage across multiple services unless responsibilities genuinely
|
||||
diverge.
|
||||
- **Background tasks are uniform.** Any operation taking more than a
|
||||
few seconds returns a `Task` and is polled. Don't introduce ad-hoc
|
||||
status fields on `Project`; extend `Task`.
|
||||
- **Per-project isolation.** Every graph operation must filter by
|
||||
`group_id`. Cross-project reads are out of scope and should be
|
||||
flagged in review.
|
||||
- **IPC has one door.** Subprocess communication for the simulator goes
|
||||
through `services/simulation_ipc.py`. Do not call `subprocess` /
|
||||
pipe primitives elsewhere.
|
||||
- **Configuration is centralized.** New tunables go in
|
||||
`backend/app/config.py` (and an `.env.example` line if env-driven),
|
||||
not as constants scattered through services.
|
||||
- **Legacy filenames stay.** `zep_*` files predate the Graphiti
|
||||
migration; leave the names alone to avoid touching every importer,
|
||||
but don't add new `zep_*` files.
|
||||
|
||||
---
|
||||
_Document patterns, not file trees. New files following patterns shouldn't require updates_
|
||||
|
|
@ -0,0 +1,157 @@
|
|||
# Technology Stack
|
||||
|
||||
## Architecture
|
||||
|
||||
A two-tier web app with a long-running **background-task** core:
|
||||
|
||||
- **Frontend** (Vue 3 + Vite) — Single-page UI orchestrating the 5-step
|
||||
workflow. Polls the backend for task progress; renders the knowledge
|
||||
graph with D3.
|
||||
- **Backend** (Flask + `uv`) — Stateless HTTP API on top of in-memory
|
||||
`Project` and `Task` models. Heavy work (ontology extraction, graph
|
||||
build, profile generation, simulation, report) runs as background
|
||||
tasks tracked through `Task` and exposed via polling endpoints.
|
||||
- **Knowledge graph** — Neo4j is the durable store; Graphiti is the
|
||||
write/read layer. All queries are scoped by per-project `group_id`.
|
||||
- **Simulation** — CAMEL-OASIS executes in subprocesses; the Flask app
|
||||
communicates with them only through `services/simulation_ipc.py`.
|
||||
|
||||
The system favors **process isolation** for the simulator and **in-memory
|
||||
state with restart recovery** for project/task tracking, rather than a
|
||||
classic job queue + persistent DB.
|
||||
|
||||
## Core Technologies
|
||||
|
||||
- **Backend language**: Python ≥3.11, ≤3.12
|
||||
- **Backend framework**: Flask 3.0 + flask-cors
|
||||
- **Backend tooling**: `uv` for dependency management
|
||||
- **Frontend framework**: Vue 3.5 + Vue Router 4 + `vue-i18n` 11
|
||||
- **Frontend tooling**: Vite 7
|
||||
- **Graph DB**: Neo4j 5.x (Community) via `bolt://`
|
||||
- **Graph layer**: `graphiti-core` ≥ 0.3
|
||||
- **Simulation**: `camel-oasis` 0.2.5 + `camel-ai` 0.2.78
|
||||
- **LLM access**: OpenAI SDK against any OpenAI-compatible endpoint
|
||||
|
||||
## Key Libraries
|
||||
|
||||
Only the libraries that shape how new code is written:
|
||||
|
||||
- **`openai`** — Sole LLM client; new providers are integrated by changing
|
||||
`LLM_BASE_URL`/`LLM_MODEL_NAME`, **not** by adding a second SDK.
|
||||
- **`graphiti-core`** — All graph reads/writes go through the
|
||||
`graphiti_adapter`; do not call Neo4j drivers directly from feature
|
||||
code.
|
||||
- **`camel-oasis` / `camel-ai`** — Pinned versions; upgrading either
|
||||
requires re-validating the simulation pipeline end-to-end.
|
||||
- **`PyMuPDF`, `charset-normalizer`, `chardet`** — File ingestion;
|
||||
encoding detection is mandatory because seed material is frequently
|
||||
non-UTF-8 (notably mixed Chinese/English).
|
||||
- **`pydantic` v2** — Used for structured LLM output / validation.
|
||||
- **`axios`** (frontend) — All API calls go through `src/api/*.js`
|
||||
services with a 5-min timeout and exponential retry; components must
|
||||
not call `fetch`/`axios` directly.
|
||||
- **`d3` v7** — Knowledge-graph visualization in `GraphPanel.vue`.
|
||||
|
||||
## Development Standards
|
||||
|
||||
### Type Safety
|
||||
- Python: type hints where the surrounding file uses them. Don't retrofit
|
||||
hints into untyped modules just for consistency.
|
||||
- Frontend: plain JavaScript, not TypeScript. Use JSDoc only when it
|
||||
improves clarity.
|
||||
|
||||
### Code Quality
|
||||
- **No enforced linter or formatter** in this repo by design. Match the
|
||||
surrounding file's style. Discuss with the user before introducing
|
||||
ESLint/Prettier/Ruff/Black.
|
||||
- 4-space indentation everywhere.
|
||||
- Python: `snake_case`. Existing files mix English and Chinese in
|
||||
comments/docstrings — preserve both; do not translate one into the
|
||||
other unless asked.
|
||||
|
||||
### Testing
|
||||
- pytest is wired (`backend/scripts/test_profile_format.py`) but coverage
|
||||
is intentionally minimal. Don't add a heavy test harness without
|
||||
discussing scope.
|
||||
- For UI changes, run `npm run dev` and exercise the feature in a
|
||||
browser; type-check/test passes do not prove feature correctness here.
|
||||
|
||||
### Internationalization
|
||||
- User-visible strings live in repo-root `/locales/*.json` (`en.json`,
|
||||
`zh.json`, `languages.json`). The `frontend/vite.config.js` aliases
|
||||
`@locales` to that root folder so the backend logger and frontend share
|
||||
the same keys.
|
||||
- Backend logger messages are part of the i18n surface — translate keys,
|
||||
not raw log lines, when adding new logs that surface to users.
|
||||
|
||||
## Development Environment
|
||||
|
||||
### Required Tools
|
||||
|
||||
| Tool | Version |
|
||||
| --------- | ------------- |
|
||||
| Node.js | ≥18 |
|
||||
| Python | ≥3.11, ≤3.12 |
|
||||
| `uv` | latest |
|
||||
| Neo4j | 5.x Community |
|
||||
| Docker | optional |
|
||||
|
||||
### Common Commands
|
||||
|
||||
```bash
|
||||
# Setup (one-shot)
|
||||
npm run setup:all
|
||||
|
||||
# Dev (backend on :5001, frontend on :3000 with /api proxy)
|
||||
npm run dev
|
||||
|
||||
# Run individually
|
||||
npm run backend
|
||||
npm run frontend
|
||||
|
||||
# Build frontend
|
||||
npm run build
|
||||
|
||||
# Backend tests
|
||||
cd backend && uv run python -m pytest
|
||||
|
||||
# Full stack (incl. Neo4j)
|
||||
docker compose up
|
||||
```
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
- **Neo4j + Graphiti replaces Zep Cloud.** Several services still carry
|
||||
the legacy `zep_*` filename prefix (`zep_tools.py`,
|
||||
`zep_entity_reader.py`, `zep_graph_memory_updater.py`). New code must
|
||||
not depend on Zep Cloud. The `ZEP_API_KEY` env var is kept (empty
|
||||
string is fine) only for backwards compatibility.
|
||||
- **Per-project graph isolation via `group_id`.** Every Graphiti read or
|
||||
write must filter by the project's `group_id`. There is no
|
||||
cross-project graph access.
|
||||
- **Reasoning-model output stripping.** Models like MiniMax and GLM emit
|
||||
`<think>` blocks and markdown fences; outputs are stripped before JSON
|
||||
parsing (see commit `985f89f`). New LLM-output parsers must do the
|
||||
same.
|
||||
- **Background tasks via `Task` model, not a queue.** Anything taking
|
||||
more than a few seconds returns immediately and tracks progress on a
|
||||
`Task` object the frontend polls. There is no Celery/RQ/etc.
|
||||
- **Startup recovery for stuck projects.** On boot,
|
||||
`_recover_stuck_projects` promotes projects in `GRAPH_BUILDING` to
|
||||
`GRAPH_COMPLETED` if Neo4j already has their nodes. New long-running
|
||||
task types should follow the same recovery pattern.
|
||||
- **Subprocess cleanup is centralized.** `SimulationRunner.register_cleanup()`
|
||||
registers a shutdown hook so simulation subprocesses die with the app.
|
||||
Don't spawn subprocesses outside this path.
|
||||
- **Configuration is a single Python file.** `backend/app/config.py`
|
||||
holds LLM, Neo4j, embedding, chunking, OASIS, and ReportAgent
|
||||
settings. Prefer extending it over scattering env-var reads through
|
||||
the codebase.
|
||||
- **Default simulation parameters.** Max 10 rounds. Twitter actions:
|
||||
`CREATE_POST`, `LIKE_POST`, `REPOST`, `FOLLOW`, `QUOTE_POST`,
|
||||
`DO_NOTHING`. Reddit additionally: `CREATE_COMMENT`, `LIKE_COMMENT`,
|
||||
`DISLIKE_*`, `SEARCH_*`, `TREND`, `REFRESH`, `MUTE`. Changes go in
|
||||
`config.py`, not per-call.
|
||||
|
||||
---
|
||||
_Document standards and patterns, not every dependency_
|
||||
Loading…
Reference in New Issue