RomanNum3ral
/
MicroFish
mirror of https://github.com/666ghj/MiroFish.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into feat/25-i18n-oasis-profile-generator-prompts 

# Conflicts:
#	.kiro/specs/i18n-oasis-profile-generator-prompts/design.md
#	.kiro/specs/i18n-oasis-profile-generator-prompts/requirements.md
#	.kiro/specs/i18n-oasis-profile-generator-prompts/spec.json
#	.kiro/specs/i18n-oasis-profile-generator-prompts/tasks.md
#	backend/app/services/oasis_profile_generator.py
This commit is contained in:
Dominik Seemann 2026-05-11 11:19:46 +02:00
parent 7f74e0a3f8 056f3664be
commit 1f256f3683
145 changed files with 23411 additions and 5332 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.env.example
View File

@ -18,6 +18,14 @@ LLM_MODEL_NAME=qwen-plus
# EMBEDDING_BASE_URL=
EMBEDDING_MODEL=text-embedding-3-small
# Local embeddings via Ollama (run: ollama pull mxbai-embed-large).
# mxbai-embed-large is 1024-dim, matching Graphiti's default EMBEDDING_DIM.
# 768-dim models (e.g. nomic-embed-text) are NOT supported until EMBEDDING_DIM
# becomes configurable. Use host.docker.internal in Docker, localhost in host mode.
# EMBEDDING_BASE_URL=http://host.docker.internal:11434/v1
# EMBEDDING_API_KEY=ollama
# EMBEDDING_MODEL=mxbai-embed-large
# Knowledge graph — Neo4j (default works for both Docker and host modes).
# Docker compose overrides NEO4J_URI to bolt://neo4j:7687 inside the stack.
NEO4J_URI=bolt://localhost:7687

26
.github/workflows/i18n-cjk-guard.yml vendored Normal file
View File

@ -0,0 +1,26 @@
name: i18n CJK Guard
on:
pull_request:
branches: [main]
permissions:
contents: read
jobs:
guard:
runs-on: ubuntu-latest
timeout-minutes: 1
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 1
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Run i18n CJK guard
run: python scripts/ci/i18n_cjk_guard.py

296
.kiro/specs/graphiti-ollama-embedder/design.md Normal file
View File

@ -0,0 +1,296 @@
# Design Document — graphiti-ollama-embedder
## Overview
**Purpose**: Add first-class documentation for using a local Ollama embedder (`mxbai-embed-large`) with the Graphiti adapter, and remove the silent placeholder-UUID fallback in `_GraphNamespace.add_batch` so embedding failures terminate the surrounding graph-build `Task` with the underlying error visible.
**Users**: Self-hosting MiroFish operators who run the LLM/embedder stack locally on Ollama, and any operator hitting a misconfigured embedder (which currently produces an empty graph that *looks* successfully built).
**Impact**: The graph-build pipeline becomes correctly observable: invalid `EMBEDDING_*` configuration produces a `Task.status = FAILED` with the underlying error, instead of `COMPLETED` with no nodes. The change is invisible on the OpenAI/Gemini happy path.
### Goals
- R1 — `.env.example`, `CLAUDE.md`, `README.md`, `docker-compose.yml` document Ollama as a supported embedder configuration with `mxbai-embed-large` and a `curl` smoke test.
- R2 — embedding failures in `_GraphNamespace.add_batch` propagate to the calling background task, which terminates with `status=FAILED` and a non-empty `error`. ERROR-level logging instead of `WARNING`.
- R3 — OpenAI- and Gemini-based deployments are unchanged; no new env var; the 1024-dim constraint is documented.
### Non-Goals
- Adding a startup-time embedder health probe.
- Making `EMBEDDING_DIM` env-configurable to support 768-dim models.
- Adding an `ollama` provider literal in `_build_llm_and_embedder` (Ollama uses the existing `openai` branch with a different `EMBEDDING_BASE_URL`).
- Generic retry/backoff for transient embedder errors. Tracked as an explicit follow-up.
## Boundary Commitments
### This Spec Owns
- The documentation surface for Ollama embedder configuration in `.env.example`, `CLAUDE.md`, `README.md`, and `docker-compose.yml` comments.
- The error-propagation contract of `_GraphNamespace.add_batch` in `backend/app/services/graphiti_adapter.py`.
- Adapter-level ERROR-log emission for failed `add_episode` calls.
### Out of Boundary
- Behavior of `_GraphNamespace.add(...)` (single-episode path; already correct).
- Behavior of `_GraphNamespace.search(...)` (still allowed to log-and-return-empty per steering).
- The `_build_graph_worker` outer `try/except` and `fail_task` plumbing — already implements the contract this spec depends on.
- Any change to `_build_llm_and_embedder` (no provider literal added; existing `openai` branch is sufficient).
- Generic retry policy.
### Allowed Dependencies
- `app.utils.logger.get_logger(...)` for ERROR-level emission.
- The existing `_run` helper that drives async Graphiti calls on the persistent loop.
- The existing `Task` lifecycle methods (`fail_task`) called from `_build_graph_worker` — relied on, not modified.
- `graphiti_core.embedder.openai.OpenAIEmbedder` configured with arbitrary `base_url`.
### Revalidation Triggers
- Any future provider literal added to `_build_llm_and_embedder` (would change which env vars feed which embedder).
- Any change to the contract that `_GraphNamespace.add_batch` returns one `_EpisodeResult` per input episode in input order.
- Any change to how `_build_graph_worker` translates exceptions into `Task` failures (would invalidate the assumption that propagating from the adapter is sufficient).
## Architecture
### Existing Architecture Analysis
- The Graphiti adapter (`backend/app/services/graphiti_adapter.py`) is the **single** read/write surface for Neo4j (`tech.md`: "All graph reads/writes go through the `graphiti_adapter`").
- Graph build runs as a background `Task` (`models/task.py`), tracked through the `Task` model with `status`, `progress`, `error`, polled by the frontend.
- `error-handling.md` mandates that long-running tasks always reach `COMPLETED` or `FAILED`. The current silent-swallow path violates this by producing `COMPLETED` with no nodes.
- The `OpenAIEmbedder` from `graphiti_core` accepts an arbitrary `base_url` / `api_key` / `embedding_model`. Ollama's `/v1/embeddings` is OpenAI-compatible. No new client class is needed.
### Architecture Pattern & Boundary Map
```mermaid
flowchart TD
UI[Frontend Step 1<br/>Graph Build] -->|POST /api/graph/build| API[graph_bp handler]
API --> SVC[GraphBuilderService.build_graph_async]
SVC -->|spawn thread| W[_build_graph_worker]
W --> ADD[GraphBuilderService.add_text_batches]
ADD --> NS["_GraphNamespace.add_batch<br/>(this spec)"]
NS -->|_run| GR[graphiti_core.add_episode]
GR -->|/v1/embeddings| EMB[OpenAI-SDK embedder<br/>OpenAI / Gemini / Ollama]
NS -. raise on failure .-> ADD
ADD -. raise .-> W
W -. fail_task(error) .-> TM[TaskManager]
TM -. status=FAILED .-> UI
classDef changed fill:#fef3c7,stroke:#92400e,stroke-width:2px;
class NS changed;
```
**Architecture Integration**:
- **Selected pattern**: minimal extension of the existing adapter pattern — fix one method's failure semantics, add no new layer.
- **Domain/feature boundaries**: error propagation stays at the adapter; task-state translation stays in the worker; UI rendering of failed tasks is unchanged.
- **Existing patterns preserved**: single-surface graph adapter; background-task `Task` lifecycle; `_run` async-loop helper; `OpenAIEmbedder` reuse for any OpenAI-SDK target.
- **New components rationale**: none — no new module is introduced.
- **Steering compliance**:
- `error-handling.md` § Background Task Errors — failure now terminates the task with a real error.
- `error-handling.md` § Logging — ERROR level for unrecoverable; WARNING reserved for retry/recovered.
- `tech.md` § Key Libraries — adapter remains the single graph read/write surface.
### Technology Stack & Alignment
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Frontend / CLI | Vue 3.5 (unchanged) | Polls `Task` status; renders failure | No code change. |
| Backend / Services | Python ≥3.11, Flask 3.0, `graphiti-core ≥ 0.3` | `_GraphNamespace.add_batch` failure propagation | One method edited. |
| Data / Storage | Neo4j 5.x via `bolt://` (unchanged) | Same writes attempted; failed writes never partially commit because the adapter is the only path. | — |
| Messaging / Events | None | — | — |
| Infrastructure / Runtime | Optional Ollama daemon at `http://host.docker.internal:11434/v1` | Source of `mxbai-embed-large` embeddings (1024-dim). | Documented, not enforced. |
## File Structure Plan
### Modified Files
- `backend/app/services/graphiti_adapter.py` — replace the broad `except Exception` in `_GraphNamespace.add_batch` (lines ~471–473) with `logger.exception(...)` + `raise`. Remove the placeholder-UUID fallback. ~5 LOC delta.
- `.env.example` — add a commented Ollama embedder block (3 commented env-var lines + a 1-line comment about `ollama pull`).
- `CLAUDE.md` — extend the "Required Environment Variables" section to list three supported embedder providers (OpenAI, Gemini, Ollama) and the 1024-dim constraint.
- `README.md` — replace the single Gemini hint comment in the Required Environment Variables block with a short three-option block (OpenAI, Gemini, Ollama) and append a one-line `curl` smoke-test snippet inside the same setup section.
- `docker-compose.yml` — one comment line above the `mirofish` service noting that Ollama on the host is reached via `host.docker.internal:11434`.
### New Files
- None.
> No code is moved or split. All edits are local and additive except the 5-line deletion in `_GraphNamespace.add_batch`.
## System Flows
### Failure flow (the change)
```mermaid
sequenceDiagram
autonumber
participant W as _build_graph_worker
participant A as add_text_batches
participant NS as _GraphNamespace.add_batch
participant G as graphiti_core.add_episode
participant E as Embedder (Ollama / OpenAI)
participant TM as TaskManager
W->>A: chunks, batch_size
loop per batch
A->>NS: add_batch(group_id, episodes)
loop per episode
NS->>G: _run(add_episode(...))
G->>E: POST /v1/embeddings
alt embedder OK
E-->>G: 200, vector(1024)
G-->>NS: EpisodeResult
else embedder error (404 / 401 / connection)
E-->>G: 4xx/5xx
G-->>NS: raise exception
Note right of NS: logger.exception(...); raise
end
end
end
Note over A: try/except wraps add_batch and re-raises
NS-->>A: raise
A-->>W: raise
W->>TM: fail_task(task_id, str(e) + traceback)
TM-->>W: Task.status = FAILED
```
Decisions reflected in the diagram:
- The adapter raises immediately on any exception from `_g.add_episode`.
- The single-episode `add()` path (not shown) is unchanged because it already raises naturally.
- `add_text_batches` already re-raises after a localized progress message — no edit needed there.
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | `.env.example` Ollama block | `.env.example` (modified file) | n/a | n/a |
| 1.2 | `CLAUDE.md` lists three providers + 1024-dim constraint | `CLAUDE.md` (modified file) | n/a | n/a |
| 1.3 | docker-compose / README note about `host.docker.internal:11434` | `docker-compose.yml`, `README.md` (modified files) | n/a | n/a |
| 1.4 | `curl` smoke-test snippet | `README.md` (modified file) | n/a | n/a |
| 1.5 | End-to-end build with `mxbai-embed-large` | `graphiti_adapter._build_llm_and_embedder` (unchanged) | `OpenAIEmbedderConfig` | Failure flow (happy path is identical to today) |
| 2.1 | No placeholder UUID on failure | `_GraphNamespace.add_batch` | `_EpisodeResult` (only emitted on success) | Failure flow |
| 2.2 | Propagate exception | `_GraphNamespace.add_batch` | n/a | Failure flow |
| 2.3 | `Task.FAILED` with non-empty error | `_build_graph_worker` (unchanged) | `TaskManager.fail_task` | Failure flow |
| 2.4 | Log at ERROR level | `_GraphNamespace.add_batch` | `logger.exception(...)` | Failure flow |
| 2.5 | UI shows error, no fake-success placeholder | Frontend Step 1 (unchanged) | Task polling | Failure flow |
| 2.6 | Preserve happy-path UUID contract | `_GraphNamespace.add_batch` | `_EpisodeResult.uuid_` | n/a |
| 3.1 | OpenAI/Gemini behavior unchanged | `_build_llm_and_embedder` (unchanged) | n/a | n/a |
| 3.2 | No new env var | scope rule | n/a | n/a |
| 3.3 | 1024-dim constraint documented | `CLAUDE.md` (modified file) | n/a | n/a |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|--------------|--------------------------|-----------|
| `_GraphNamespace.add_batch` | services / graph-adapter | Ingest a batch of text episodes; raise on first failure; preserve UUIDs on success | 2.1, 2.2, 2.4, 2.6 | `graphiti_core.add_episode` (P0), `app.utils.logger` (P0) | Service |
| Documentation set (`.env.example`, `CLAUDE.md`, `README.md`, `docker-compose.yml`) | docs | Describe Ollama embedder configuration and constraints | 1.1, 1.2, 1.3, 1.4, 3.3 | none | Doc |
### graph-adapter / `_GraphNamespace.add_batch`
| Field | Detail |
|-------|--------|
| Intent | Ingest each episode through `graphiti_core.add_episode`; propagate the first failure to the caller; never substitute a placeholder UUID. |
| Requirements | 2.1, 2.2, 2.4, 2.6 |
**Responsibilities & Constraints**
- Iterate `episodes` in input order.
- For each episode, call `_run(self._g.add_episode(...))` and append a `_EpisodeResult` whose `uuid_` matches the Graphiti-assigned episode UUID.
- On any exception from `_run(...)`, emit `logger.exception(...)` (ERROR level with traceback) including the `graph_id` and the index of the failing episode for diagnosability, then `raise`.
- Do **not** swallow the exception. Do **not** return a `_EpisodeResult` for the failed episode. Do **not** continue the loop after a failure.
- Domain boundary: the method speaks Graphiti and Python exceptions; it does not know about `Task` lifecycles.
- Data ownership: emits `_EpisodeResult` instances only for successfully ingested episodes.
**Dependencies**
- Inbound: `GraphBuilderService.add_text_batches` (P0, sole production caller for this method).
- Outbound: `graphiti_core.add_episode` via `_run(...)` (P0).
- External: `app.utils.logger.get_logger("mirofish.graph_builder")` (P0).
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
##### Service Interface
```python
class _GraphNamespace:
def add_batch(self, graph_id: str, episodes: List[Any]) -> List[_EpisodeResult]:
"""Add a batch of episodes.
Returns a list of _EpisodeResult, one per successfully ingested
episode, in input order. Raises the underlying exception on the
first failure; partial results are not returned.
Preconditions:
- graph_id is a non-empty per-project group_id.
- Each item in `episodes` exposes a `data` attribute (str)
or stringifies to a meaningful body.
Postconditions:
- On success: len(returned list) == len(episodes), each
`_EpisodeResult.uuid_` is the Graphiti-assigned UUID.
- On failure: an exception is raised; no `_EpisodeResult`
is returned for the failing episode and no further episodes
are attempted; partial successes prior to the failure are
committed in Neo4j (this matches today's behavior because
`add_episode` is invoked synchronously per episode).
Invariants:
- Never returns a `_EpisodeResult` whose UUID was generated
locally as a placeholder.
"""
```
- Preconditions: as above.
- Postconditions: as above.
- Invariants: never emit a placeholder UUID.
**Implementation Notes**
- Integration: the method is called from `GraphBuilderService.add_text_batches` (graph_builder.py:289–308), which already wraps the call in `try/except Exception: progress_callback(...); raise`. No caller-side change.
- Validation: input shape unchanged.
- Risks: an environment that was producing "successful" empty graphs because of the silent fallback will now produce a failed `Task`. This is the intended correction; PR description must call it out.
### Documentation set
**Edits (verbatim intent)**:
- `.env.example` — add an opt-in commented block, e.g.:
```env
# Local embeddings via Ollama (run: ollama pull mxbai-embed-large).
# mxbai-embed-large is 1024-dim, matching Graphiti's default EMBEDDING_DIM.
# EMBEDDING_BASE_URL=http://host.docker.internal:11434/v1
# EMBEDDING_API_KEY=ollama
# EMBEDDING_MODEL=mxbai-embed-large
```
- `CLAUDE.md` — extend the embedder note to enumerate OpenAI / Gemini / Ollama and call out the 1024-dim constraint.
- `README.md` — keep the existing Gemini comment, add the Ollama three-line example, append the `curl` smoke-test below the env block.
- `docker-compose.yml` — one comment above the `mirofish` service: `# Note: Ollama on the host is reachable from this container via host.docker.internal:11434`.
These edits are doc-only; they do not affect the runtime contract.
## Data Models
No new data models. The `_EpisodeResult` dataclass shape is unchanged. The `Task` model is unchanged. The `Project.status` lifecycle is unchanged.
## Error Handling
### Error Strategy
- The adapter raises on first failure; the worker catches and routes to `Task.fail_task`. This is the existing project pattern (`error-handling.md` § Background Task Errors), and this spec aligns the adapter with it.
- No retries inside `add_batch`. Transient resilience, if added later, belongs at a layer that owns idempotency considerations (out of scope).
### Error Categories and Responses
- **Embedder configuration errors** (404 unknown model, 401 unauthorized, connection refused) → adapter raises → worker fails the task with the exception's `str()` plus traceback → frontend renders `Task.error`. Operator action: fix `EMBEDDING_*` env vars per the new docs and re-run the build.
- **Embedder transient errors** (timeouts, intermittent 5xx) → today, treated identically to configuration errors (task fails). Future follow-up may narrow this with `retry_with_backoff`.
- **Graphiti-internal errors** unrelated to embeddings (e.g., Neo4j unavailable) → already raised by `_run(...)` and currently swallowed; this fix surfaces them too. Treated as a positive side effect.
### Monitoring
- `logger.exception(...)` in `_GraphNamespace.add_batch` adds a full traceback at ERROR level, enabling existing log-aggregation setups to alert on adapter-level errors.
- `_build_graph_worker` already calls `logger.exception(f"task {task_id} failed")`; the two log lines are complementary (adapter-context vs. task-context).
## Testing Strategy
This is an extension feature — the project's testing stance is intentionally minimal (`tech.md`: "pytest is wired ... but coverage is intentionally minimal. Don't add a heavy test harness without discussing scope.").
### Unit Tests (lightweight, optional)
- If we add a test, the right scope is a single pytest case for `_GraphNamespace.add_batch` that monkeypatches `self._g.add_episode` to raise, calls `add_batch`, and asserts the exception propagates and no `_EpisodeResult` is returned. Do not add a heavier harness.
### Manual / End-to-End
1. **Happy path (OpenAI)**: existing setup — verify graph build still completes with real nodes/edges (no behavior change expected).
2. **Happy path (Ollama)**: `ollama pull mxbai-embed-large`; set the three `EMBEDDING_*` env vars per `.env.example`; run the smoke-test `curl` to confirm 1024-dim response; run a graph build through the UI; verify Neo4j has nodes/edges.
3. **Failure path (typo'd model)**: set `EMBEDDING_MODEL=text-embedding-3-small-typo` against an Ollama base URL; trigger a graph build; verify the task transitions to `FAILED` with the underlying 404 message visible in `Task.error` and the UI; verify backend logs include the ERROR-level traceback.
### Performance / Load
- Not applicable. No throughput change expected on the happy path. Failure path returns earlier than today (bonus).
## Security Considerations
- No new secrets introduced. `EMBEDDING_API_KEY=ollama` is documented as a placeholder string ignored by Ollama; this is consistent with the project's existing handling of `ZEP_API_KEY` (empty string acceptable).
- `error-handling.md` § Logging forbids logging API keys / full prompts. `logger.exception(...)` includes the exception message and traceback — Graphiti's exceptions do not echo API keys, but the ERROR log line should not include the request body. Implementation note: log only `graph_id` and episode index alongside the exception.
## Migration Strategy
- None. The fix is purely additive on documentation and a strictly-more-correct behavior change in `add_batch`. Operators do not need to take action unless their graphs were silently empty, in which case this surfacing IS the migration trigger.

99
.kiro/specs/graphiti-ollama-embedder/gap-analysis.md Normal file
View File

@ -0,0 +1,99 @@
# Gap Analysis — graphiti-ollama-embedder
## 1. Current State Investigation
### Domain assets touched by this feature
- `backend/app/services/graphiti_adapter.py`
- Lines 92–139 — `_build_llm_and_embedder(provider)`. Builds an `OpenAIEmbedder` (when `provider == "openai"`) using `EMBEDDING_API_KEY or LLM_API_KEY`, `EMBEDDING_BASE_URL or LLM_BASE_URL`, and `EMBEDDING_MODEL`. Already supports pointing the embedder at any OpenAI-compatible endpoint — no code change is needed for Ollama support. This is a **documentation gap, not a code gap**.
- Lines 455–475 — `_GraphNamespace.add_batch`. Iterates episodes, calls `add_episode`, and on `except Exception as e` logs a one-line `WARNING` and substitutes a fresh placeholder UUID. This is the silent-swallow path.
- Line 441–453 — `_GraphNamespace.add(...)`. Single-episode path. **Already raises naturally** because there is no `try/except`.
- Lines 504–506 — `_GraphNamespace.search(...)`. Has its own `except Exception` that logs and returns empty results. Per `error-handling.md` ("for non-fatal search failures, log and return empty results") this is the documented contract; out of scope.
- `backend/app/services/graph_builder.py`
- Lines 256–310 — `add_text_batches(...)`. Already wraps `client.graph.add_batch(...)` in `try/except Exception` and **re-raises** after a progress message. So if `_GraphNamespace.add_batch` raises, the exception propagates correctly.
- Lines 143–234 — `_build_graph_worker`. Outer `try/except Exception` calls `self.task_manager.fail_task(task_id, error_msg)` with `f"{str(e)}\n{traceback.format_exc()}"`. This already implements the "task always terminates" rule from `error-handling.md`.
- `backend/app/config.py`
- Lines 40, 50–51 — defines `EMBEDDING_MODEL`, `EMBEDDING_API_KEY`, `EMBEDDING_BASE_URL`. No change required.
- `.env.example` (project root) — currently only documents the OpenAI/Gemini path with commented-out `EMBEDDING_API_KEY` / `EMBEDDING_BASE_URL` lines.
- `CLAUDE.md` lines 60–82 — "Required Environment Variables" section lists `EMBEDDING_MODEL` with a note about Gemini overrides only.
- `README.md` lines 148–165 — "Required Environment Variables" section, mentions "uncomment if using a non-OpenAI provider, e.g. Gemini" but no Ollama example.
- `docker-compose.yml` lines 21–37 — `mirofish` service uses `env_file: .env` and overrides `NEO4J_URI`. No Ollama hint, but the standard `host.docker.internal` route works.
### Conventions extracted from steering
- `tech.md`: "All graph reads/writes go through the `graphiti_adapter`; do not call Neo4j drivers directly from feature code." — adapter is the right place for the fix.
- `error-handling.md`: "Long-running tasks must always reach a terminal state (`COMPLETED` or `FAILED`)" — silent placeholder UUID violates this.
- `error-handling.md`: "Don't catch `Exception` inside an API handler just to log and continue" — same anti-pattern in the adapter today.
- `error-handling.md` § Logging: `WARNING` is for "retry triggered, transient failure, recovered state"; `ERROR` is for "task failure, unrecoverable exception". The current `WARNING` mislabels what is actually an unrecoverable failure for the task.
- `tech.md`: Ollama is **not currently** an officially listed provider. CLAUDE.md only enumerates OpenAI and Gemini.
- `commits.md` / `dev-guidelines.md`: 4-space indent, max 120 chars/line, double-quoted Python strings, snake_case, conventional commits.
### Integration surfaces
- The `OpenAIEmbedder` from `graphiti_core.embedder.openai` already accepts an arbitrary `base_url`. Ollama exposes `/v1/embeddings` at `http://localhost:11434/v1`. No new client class is required.
- Background-task lifecycle: API handler → `GraphBuilderService.build_graph_async()` → background thread → `_build_graph_worker` → `fail_task(task_id, msg)`. Already in place; this feature just needs to stop short-circuiting it.
## 2. Requirements Feasibility Analysis
| Req | Need | Maps to | Gap |
| --- | --- | --- | --- |
| R1.1 | `.env.example` Ollama block | `.env.example` | **Missing** (docs) |
| R1.2 | `CLAUDE.md` lists OpenAI/Gemini/Ollama, dim constraint | `CLAUDE.md` | **Missing** (docs) |
| R1.3 | Docker-compose / README note about `host.docker.internal:11434` | `docker-compose.yml` comments / `README.md` | **Missing** (docs) |
| R1.4 | `curl` smoke-test snippet | `README.md` | **Missing** (docs) |
| R1.5 | Pipeline works end-to-end with mxbai-embed-large | adapter is already provider-agnostic via OpenAI-SDK | **No code gap** — already supported, just undocumented |
| R2.1 | Drop placeholder-UUID fallback | `graphiti_adapter.py:471–473` | **Constraint** — narrow change only |
| R2.2 | Propagate ingest exception | `graphiti_adapter.py:471–473` + caller | **Missing** — adapter swallows; caller re-raises if it sees an exception |
| R2.3 | `Task` transitions to `FAILED` with non-empty `error` | `graph_builder.py:231–234` | **Already implemented** — relies on R2.2 |
| R2.4 | Log at `ERROR` level | `graphiti_adapter.py:472` | **Missing** — currently `WARNING` |
| R2.5 | UI shows error, no fake-success placeholder | downstream of R2.3 | **Already implemented** via task polling |
| R2.6 | Preserve happy-path UUID contract | `graphiti_adapter.py:455–474` | **Constraint** — keep return shape on success |
| R3.1 | OpenAI/Gemini behavior unchanged | `_build_llm_and_embedder` | **No change needed** — branch untouched |
| R3.2 | No new env var | scope rule | **Constraint** |
| R3.3 | Document 1024-dim constraint | `CLAUDE.md` | **Missing** (docs) |
### Research needed
- None for this feature — `OpenAIEmbedder` already supports custom `base_url`, and Ollama's `/v1/embeddings` is OpenAI-compatible (well-known and used in many projects). The 1024-dim constraint comes from `graphiti_core/embedder/client.py:22` (`EMBEDDING_DIM = 1024`) and is documented in the ticket itself.
- One mild unknown: whether to narrow the `except` to a transient subset (e.g., `httpx.TimeoutException`, `httpx.NetworkError`) and retry, or simply drop the catch entirely. Decided in design phase, not blocking.
### Complexity signal
- Mostly documentation. The code change is **5 lines** in one method.
## 3. Implementation Approach Options
### Option A — Pure narrow fix in `_GraphNamespace.add_batch` + docs only (RECOMMENDED)
- **What**: delete the `except Exception` block in `add_batch` (or replace with `logger.exception(...)` + `raise`); update `.env.example`, `CLAUDE.md`, `README.md`, `docker-compose.yml` comments.
- **Files**: `backend/app/services/graphiti_adapter.py`, `.env.example`, `CLAUDE.md`, `README.md`, `docker-compose.yml`.
- **Trade-offs**:
- ✅ Minimal blast radius — adapter behavior outside `add_batch` is untouched.
- ✅ Existing background-task contract carries the failure to the UI for free.
- ✅ Honors steering rules: don't catch `Exception` to log-and-continue; tasks must terminate; ERROR-level logging for unrecoverable failures.
- ❌ Loses the (currently broken) "best effort, keep going on a partial failure" intent. In practice that intent never produced a usable graph anyway, so the loss is theoretical.
### Option B — Narrow the catch to transient errors and retry, fail loud on the rest
- **What**: keep a `try/except`, but only catch a small set of transient classes (`httpx.TimeoutException`, `httpx.NetworkError`, `openai.APIConnectionError`), wrap the whole `add_episode` call in `retry_with_backoff` from `app/utils/retry.py`, and re-raise everything else immediately.
- **Trade-offs**:
- ✅ Adds small resilience for genuinely transient blips.
- ✅ Aligns with the existing `retry_with_backoff` pattern.
- ❌ More moving parts; broader change for a bug fix.
- ❌ Single-episode `add()` would also need the same treatment to avoid two divergent retry semantics.
- ❌ Out-of-scope creep: ticket is focused on stopping the silent swallow + documenting Ollama.
### Option C — Per-provider embedder factory + Option A
- **What**: extend `_build_llm_and_embedder` with a third provider literal (`"ollama"`) that uses `OpenAIEmbedder` under the hood with hardcoded sensible defaults.
- **Trade-offs**:
- ✅ Symmetric with `openai`/`gemini`.
- ❌ The ticket explicitly lists "per-provider embedder factory" as out of scope.
- ❌ Duplicate code path — Ollama is just OpenAI-SDK with a different base URL.
## 4. Effort & Risk
- **Effort**: **S** (≤1 day). One file, one method, ~5 LOC delta plus 4 doc edits.
- **Risk**: **Low**. The change makes a previously-silent failure loud; it cannot break the happy path because the happy-path branch is the same return statement. Documentation changes are not load-bearing.
One non-zero risk: if there are real-world users today whose graph builds succeed only by accident (i.e., the fallback hides intermittent embedding failures), they will start seeing failed tasks instead of (broken) successful ones. This is the intended correction — but worth noting in the PR description so the operator can re-check their embedder credentials.
## 5. Recommendations for design phase
- **Preferred approach**: **Option A**. Smallest correct fix; documentation reflects the already-supported configuration; follows steering's error-handling philosophy literally.
- **Key decisions to lock in design**:
1. Drop the `except` entirely, or narrow it? Default: drop. Rationale: the only retry path that matters is transient network blips, and those would also kill the surrounding `_run` loop today; addressing them would be a follow-up using the project's `retry_with_backoff` decorator on the underlying graph driver call, not a band-aid in `add_batch`.
2. Which docs files mention Ollama? Default: `.env.example`, `CLAUDE.md`, `README.md`, `docker-compose.yml` comment. Two-file or three-file split?
- **Carry-forward research**: none.

128
.kiro/specs/graphiti-ollama-embedder/requirements.md Normal file
View File

@ -0,0 +1,128 @@
# Requirements Document
## Project Description (Input)
Fix Graphiti embedding integration with Ollama (mxbai-embed-large) and stop silently swallowing embedding failures. Two bugs: (1) No first-class support for local Ollama embedders — `EMBEDDING_MODEL` defaults to OpenAI's `text-embedding-3-small` and the embedder reuses `LLM_BASE_URL` when `EMBEDDING_BASE_URL` is unset, so Ollama users get 404s; `.env.example` and `CLAUDE.md` don't document Ollama. (2) `backend/app/services/graphiti_adapter.py:471-473` catches every exception during episode ingestion, logs a truncated `WARNING`, and substitutes a placeholder UUID, so a graph build appears to succeed but writes nothing. Tracked as GitHub issue #18.
## Introduction
This feature adds first-class documentation for using a local Ollama embedder
(`mxbai-embed-large`, 1024-dim) with the Graphiti adapter and removes the
silent placeholder-UUID fallback in `_GraphNamespace.add_batch` so that
embedding failures terminate the surrounding background `Task` with the
underlying error visible in the UI and logs.
The work spans two narrowly scoped changes:
1. **Documentation update** — `.env.example`, `CLAUDE.md`, and the README /
docker-compose comments gain a short Ollama section that explains how to
point the embedder at a local Ollama instance, why `mxbai-embed-large` is
the recommended model (1024-dim, matches Graphiti's default
`EMBEDDING_DIM`), and how to smoke-test connectivity with one `curl`
command before kicking off a graph build.
2. **Loud failure** — the broad `except Exception` in
`_GraphNamespace.add_batch` is removed (or narrowed to a small set of
transient network errors). Episode ingestion failures now propagate to
the calling background task, which marks itself `FAILED` with the
underlying error message attached, rather than logging a `WARNING` and
returning a fake UUID.
No new dependency, environment variable, or config flag is introduced.
All existing OpenAI/Gemini configurations continue to work unchanged.
## Boundary Context
- **In scope**: documenting Ollama as a third supported embedder provider
in `.env.example`, `CLAUDE.md`, and the docker-compose / README comments;
removing the silent placeholder-UUID fallback in
`_GraphNamespace.add_batch`; surfacing the underlying ingestion error to
the background `Task` so it terminates with `status=FAILED`; documenting
a one-line `curl` smoke test for embedder connectivity.
- **Out of scope**: a startup-time embedder health probe that refuses to
boot on dim/model mismatch; making `EMBEDDING_DIM` env-configurable so
768-dim or 1536-dim embedders can be used; adding a per-provider
embedder factory (today the adapter only branches on `openai` and
`gemini`); generic retry/backoff policy changes elsewhere in the
pipeline.
- **Adjacent expectations**: the existing background-task error-handling
contract from `.kiro/steering/error-handling.md` already specifies that
worker exceptions must call `fail_task(...)`. This feature relies on
that contract — it does not introduce a new one. The single-episode
`_GraphNamespace.add(...)` path is left untouched because it already
re-raises naturally.
## Requirements
### Requirement 1: Ollama Embedder Documentation
**Objective:** As a self-hosting MiroFish operator, I want first-class
documentation for using a local Ollama embedder, so that I can run the
Graphiti pipeline without needing an OpenAI- or Gemini-compatible
embeddings endpoint.
#### Acceptance Criteria
1. The `.env.example` file shall contain a commented Ollama embedder block
showing `EMBEDDING_BASE_URL`, `EMBEDDING_API_KEY`, and `EMBEDDING_MODEL`
set to `http://host.docker.internal:11434/v1`, a non-empty placeholder
string, and `mxbai-embed-large` respectively, with a comment noting the
`ollama pull mxbai-embed-large` prerequisite.
2. The `CLAUDE.md` file shall list the three supported embedder providers
(OpenAI, Gemini, Ollama) and shall state the 1024-dim constraint that
forces `mxbai-embed-large` over `nomic-embed-text` (768-dim).
3. Where the user runs MiroFish in Docker, the docker-compose comments or
README shall note that Ollama on the host is reached from the
`mirofish` container via `host.docker.internal:11434`.
4. The documentation shall include a one-line `curl` example that calls
`$EMBEDDING_BASE_URL/embeddings` with the configured model and confirms
the response embedding length is 1024.
5. When the operator follows the documented Ollama configuration with
`mxbai-embed-large` pulled in Ollama, the existing graph-build pipeline
shall complete end-to-end and write real nodes and edges to Neo4j with
no code changes beyond the env-var configuration.
### Requirement 2: Loud Embedding Failure
**Objective:** As a MiroFish operator, I want embedding failures during
graph build to surface as a visible task failure with the underlying
error, so that I can fix my embedder configuration instead of seeing an
"empty graph" with no diagnostic.
#### Acceptance Criteria
1. The `_GraphNamespace.add_batch` method shall not return a placeholder
`_EpisodeResult` UUID when the underlying `add_episode` call raises an
exception.
2. If `add_episode` raises any exception other than a narrowly defined set
of transient network errors, then `_GraphNamespace.add_batch` shall
propagate the exception to its caller.
3. When `_GraphNamespace.add_batch` propagates an exception, the
surrounding graph-build background `Task` shall transition to
`FAILED` with `Task.error` containing a non-empty message derived from
the underlying exception (per the existing
`.kiro/steering/error-handling.md` contract).
4. While a graph-build task is failing because of a misconfigured
`EMBEDDING_MODEL`, `EMBEDDING_BASE_URL`, or `EMBEDDING_API_KEY`, the
adapter shall log the underlying `add_episode` error at `ERROR` level
(not `WARNING`) before raising, so the root cause is visible in
server logs.
5. Where the configured `EMBEDDING_MODEL` is invalid (e.g. a typo, or a
model not pulled in Ollama), the user-facing project state shall move
out of `GRAPH_BUILDING` and the task shall surface the underlying
embedder error to the frontend without producing a placeholder-UUID
"successful" episode.
6. The `_GraphNamespace.add_batch` method shall preserve its current
contract for successful episodes: each successfully ingested episode
shall still produce one `_EpisodeResult` whose `uuid_` matches the
Graphiti-assigned episode UUID, in input order.
### Requirement 3: Backwards Compatibility
**Objective:** As an existing MiroFish operator already running with an
OpenAI- or Gemini-compatible embedder, I want this change to be invisible
on the happy path, so that no upgrade action is required.
#### Acceptance Criteria
1. Where `EMBEDDING_BASE_URL`, `EMBEDDING_API_KEY`, and `EMBEDDING_MODEL`
are unset or set to OpenAI/Gemini-compatible values, the embedder
construction in `_build_llm_and_embedder` shall behave identically to
the current implementation.
2. The graph-build pipeline shall not require any new environment
variable to function; Ollama support shall be enabled purely by
setting the three existing `EMBEDDING_*` variables.
3. While Graphiti's default `EMBEDDING_DIM` is 1024, the documentation
shall explicitly note that any embedder model with a different output
dimension is unsupported by this change and is an explicit follow-up
item.

103
.kiro/specs/graphiti-ollama-embedder/research.md Normal file
View File

@ -0,0 +1,103 @@
# Research & Design Decisions — graphiti-ollama-embedder
## Summary
- **Feature**: `graphiti-ollama-embedder`
- **Discovery Scope**: Extension (small, narrowly scoped change to an existing adapter + supporting docs)
- **Key Findings**:
- The Graphiti `OpenAIEmbedder` already accepts an arbitrary `base_url` and `api_key`. Pointing it at Ollama's OpenAI-compatible `/v1/embeddings` endpoint requires **no code change** — only documentation.
- The silent placeholder-UUID fallback in `_GraphNamespace.add_batch` violates the project's existing background-task error-handling contract (`error-handling.md`: "Long-running tasks must always reach a terminal state"). The plumbing to surface a failure already exists in `_build_graph_worker`.
- `mxbai-embed-large` is the only widely-available local embedder that matches Graphiti's hard-coded `EMBEDDING_DIM = 1024`. Smaller models (`nomic-embed-text` at 768) would silently mis-fit Neo4j vector indexes and are out of scope.
## Research Log
### Ollama's OpenAI-compatible embeddings API
- **Context**: Verify that no Ollama-specific Graphiti embedder class is required.
- **Sources Consulted**: Existing code at `backend/app/services/graphiti_adapter.py:92–115` (`OpenAIEmbedderConfig` accepts arbitrary `base_url`); ticket #18 description; Graphiti `embedder/client.py:22` (`EMBEDDING_DIM = 1024`).
- **Findings**:
- Ollama exposes `POST /v1/embeddings` mirroring the OpenAI shape.
- The current `_build_llm_and_embedder("openai")` branch already uses `EMBEDDING_API_KEY or LLM_API_KEY` and `EMBEDDING_BASE_URL or LLM_BASE_URL`, so any OpenAI-compatible endpoint just works.
- Ollama ignores the auth header but `OpenAIEmbedderConfig` requires a non-empty `api_key`; the literal string `"ollama"` is the de-facto convention.
- **Implications**: This is a documentation-only ask for R1. No new provider literal, no new factory branch.
### Failure-propagation contract
- **Context**: Confirm that removing the broad `except` in `_GraphNamespace.add_batch` will result in `Task.status = FAILED` in the UI.
- **Sources Consulted**:
- `.kiro/steering/error-handling.md` § Background Task Errors — outer `except Exception` in worker calls `fail_task(task_id, str(e))`.
- `backend/app/services/graph_builder.py:289–308` — `add_text_batches` already wraps `client.graph.add_batch` in `try/except` and re-raises after a localized progress message.
- `backend/app/services/graph_builder.py:231–234` — `_build_graph_worker` catches every exception and calls `self.task_manager.fail_task(task_id, error_msg)` with a full traceback.
- **Findings**: The chain `add_episode → _GraphNamespace.add_batch → add_text_batches → _build_graph_worker → fail_task` is intact except for the swallow at the adapter layer. Removing the swallow is sufficient; no caller-side change is required.
- **Implications**: R2.3 / R2.5 are realized for free as soon as R2.2 is implemented.
### Single vs. batch ingestion path
- **Context**: Determine whether the single-episode `_GraphNamespace.add(...)` (line 441) needs a parallel fix.
- **Sources Consulted**: `graphiti_adapter.py:441–453`. No `try/except`; exceptions bubble naturally.
- **Findings**: Only the batch path swallows. The single path already complies.
- **Implications**: Fix is local to `add_batch`. Do not introduce symmetric handling in `add(...)`.
### Logging level
- **Context**: Decide between `WARNING` and `ERROR` for the failure log line.
- **Sources Consulted**: `.kiro/steering/error-handling.md` § Logging:
- `ERROR` — task failure, unrecoverable exception
- `WARNING` — retry triggered, transient failure, recovered state
- **Findings**: A failure that terminates the surrounding task is unrecoverable from the task's perspective, so `ERROR` is correct. The current `WARNING` is mislabelled.
- **Implications**: R2.4 — change to `logger.exception(...)` (which logs at ERROR with traceback).
### Documentation surfaces
- **Context**: Decide which files need updating to satisfy R1.
- **Sources Consulted**: `.env.example` (canonical config), `CLAUDE.md` lines 60–82, `README.md` lines 148–165, `docker-compose.yml` lines 21–37.
- **Findings**: All four are appropriate. `README.md` already has a placeholder for "non-OpenAI provider" and is the natural home for the `curl` smoke test snippet. `docker-compose.yml` benefits from one additional comment about `host.docker.internal`.
- **Implications**: Update all four; keep edits minimal and additive.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| A. Drop swallow + docs | Remove `except` block in `add_batch`; update four docs files | Smallest surface; honors steering rules; symmetric with `add()` | Loses (broken) "best effort" intent | Recommended |
| B. Narrow + retry | Catch only transient classes (`httpx.TimeoutException`, `openai.APIConnectionError`); use `retry_with_backoff` from `app/utils/retry.py`; raise everything else | Adds resilience to genuine network blips | More moving parts; would also need to update `add()` for symmetry | Defer to follow-up |
| C. New `ollama` provider literal | Extend `_build_llm_and_embedder` with a third branch | Symmetric with `openai`/`gemini` | Explicitly out of scope per ticket; duplicate code path (Ollama is OpenAI-SDK with custom `base_url`) | Rejected |
## Design Decisions
### Decision: Adopt Option A (drop the placeholder fallback entirely; documentation only for Ollama support)
- **Context**: R2 mandates that embedding failures during graph build surface as visible task failures. R1 mandates documentation for an Ollama embedder. The adapter already supports any OpenAI-compatible base URL.
- **Alternatives Considered**:
1. **Option B (narrow + retry)** — keep a small `except` clause for transient errors and use the project's `retry_with_backoff`.
2. **Option C (new provider literal)** — add an `ollama` branch in `_build_llm_and_embedder`.
- **Selected Approach**:
- In `_GraphNamespace.add_batch`, replace the `try/except Exception` block with a straightforward call. Failures from `_run(self._g.add_episode(...))` propagate to the caller.
- Use `logger.exception(...)` immediately before re-raise is unnecessary — `_build_graph_worker` already calls `logger.exception(f"task {task_id} failed")` per the error-handling steering. To honor R2.4 explicitly without double-logging, wrap the call in a narrow `try/except: logger.exception(...); raise` so the adapter-level context (`group_id`, episode index) is captured before bubbling.
- Update `.env.example`, `CLAUDE.md`, `README.md`, and `docker-compose.yml` to document Ollama configuration (R1).
- **Rationale**:
- The ticket explicitly lists transient-retry behavior and per-provider factory as out of scope.
- Steering's error-handling chapter forbids catch-and-continue in service code.
- Smaller surface = lower regression risk.
- **Trade-offs**:
- +Visibility: real config errors now surface at the UI.
- +Code symmetry: `add()` and `add_batch()` behave the same on failure.
- −One-time noise: operators whose graph builds were "succeeding" only because of the silent fallback will now see a failed task. This is the intended correction; mention in PR body.
- **Follow-up**:
- If transient blips become an operational issue, revisit Option B in a separate ticket using `retry_with_backoff` against `_g.add_episode`.
### Decision: Use `logger.exception(...)` not `logger.error(...)`
- **Context**: R2.4 requires ERROR-level logging of the underlying exception.
- **Alternatives Considered**: `logger.error(str(e))` (no traceback), `logger.warning(...)` (current behavior).
- **Selected Approach**: `logger.exception("Episode add failed (group_id=%s)", graph_id)` then `raise`.
- **Rationale**: `logger.exception` logs at ERROR with the full traceback, which is what the steering doc prescribes for unrecoverable adapter failures.
- **Trade-offs**: A small amount of duplication if `_build_graph_worker` also logs via `logger.exception`. Acceptable — the two log lines describe different layers (adapter vs. task) and have different identifying context.
### Decision: Document Ollama under the existing OpenAI provider, not as a separate provider literal
- **Context**: The ticket lists "per-provider embedder factory" as out of scope; Ollama is already reachable via the existing `openai` branch.
- **Selected Approach**: Document Ollama as a configuration *choice* of the existing `openai` Graphiti provider (set the three `EMBEDDING_*` env vars).
- **Rationale**: Avoids code duplication and matches the ticket's scope.
## Risks & Mitigations
- **Risk**: Operators currently relying on the silent fallback see new failed tasks. **Mitigation**: PR body calls this out explicitly with a "what changed" note pointing at the embedder env vars.
- **Risk**: The `except` is removed but a transient timeout intermittently fails the entire graph build. **Mitigation**: Documented as a known follow-up (Option B). Acceptable today because the alternative was an empty graph that *looked* successful.
- **Risk**: Documentation drifts between `.env.example`, `CLAUDE.md`, `README.md`. **Mitigation**: Keep all four edits in this PR and reference the same env-var triple verbatim.
## References
- Ticket #18 — `.ticket/18.md` (snapshot in this repo)
- Steering — `.kiro/steering/error-handling.md` § Background Task Errors and § Logging
- Steering — `.kiro/steering/tech.md` § Key Libraries (`graphiti-core` adapter rule)
- Code — `backend/app/services/graphiti_adapter.py:92–115, :441–475`
- Code — `backend/app/services/graph_builder.py:143–234, :256–310`

23
.kiro/specs/graphiti-ollama-embedder/spec.json Normal file
View File

@ -0,0 +1,23 @@
{
"feature_name": "graphiti-ollama-embedder",
"created_at": "2026-05-07T20:24:55Z",
"updated_at": "2026-05-07T20:35:00Z",
"language": "en",
"phase": "tasks-generated",
"ticket": 18,
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}

91
.kiro/specs/graphiti-ollama-embedder/tasks.md Normal file
View File

@ -0,0 +1,91 @@
# Implementation Tasks — graphiti-ollama-embedder
> Source spec: `.kiro/specs/graphiti-ollama-embedder/`
> Ticket: #18
## Plan
This feature has two narrowly scoped deliverables:
1. **Code change** — remove the silent placeholder-UUID fallback in `_GraphNamespace.add_batch` so embedding failures propagate and the surrounding graph-build `Task` ends in `FAILED`.
2. **Configuration documentation** — describe the existing-but-undocumented Ollama embedder configuration in `.env.example`, `CLAUDE.md`, `README.md`, and `docker-compose.yml`.
The code change is self-contained in one method. The configuration-file edits do not depend on the code change and can run in parallel with each other.
## Tasks
- [x] 1. Make embedding-batch failures loud (adapter fix)
- [x] 1.1 Replace the silent placeholder-UUID fallback in `_GraphNamespace.add_batch` with ERROR-level logging plus exception propagation
- Open the per-episode `try/except Exception` around the synchronous `add_episode` call in the batch ingestion path of the Graphiti adapter and remove the placeholder-UUID branch entirely.
- Replace the existing `WARNING`-level log line with a `logger.exception(...)` call that captures the `graph_id` and the index of the failing episode in its message; do not include the episode body, API keys, or full traceback duplication beyond what `logger.exception` emits.
- Re-raise the original exception so it bubbles up to `GraphBuilderService.add_text_batches` (which already re-raises) and on to `_build_graph_worker` (which already calls `fail_task`).
- Preserve the happy-path contract: a successful episode still produces exactly one `_EpisodeResult` whose `uuid_` matches the Graphiti-assigned episode UUID, and the returned list keeps input order.
- Leave the single-episode `add(...)` method untouched (it already raises naturally) and leave `_GraphNamespace.search(...)` untouched (its log-and-return-empty contract is documented in steering and out of scope).
- Observable completion: when the embedder is misconfigured (e.g. `EMBEDDING_MODEL` set to an unknown model on the configured base URL), starting a graph build through the UI causes the `Task` to transition to `FAILED` with `Task.error` populated by the underlying Graphiti exception message, and the backend log includes an ERROR-level entry from the Graphiti adapter naming the failing `graph_id`.
- _Requirements: 2.1, 2.2, 2.4, 2.6_
- _Boundary: graphiti_adapter._GraphNamespace.add_batch_
- [x] 2. Document the Ollama embedder configuration
- [x] 2.1 (P) Add a commented Ollama embedder block to `.env.example`
- Append three commented environment-variable lines configuring the existing `EMBEDDING_BASE_URL`, `EMBEDDING_API_KEY`, and `EMBEDDING_MODEL` for an Ollama deployment with `mxbai-embed-large`.
- Include a short comment explaining the prerequisite step (`ollama pull mxbai-embed-large`) and the rationale for `mxbai-embed-large` over `nomic-embed-text` (1024-dim vs 768-dim, must match Graphiti's default `EMBEDDING_DIM`).
- Use `http://host.docker.internal:11434/v1` as the base URL example so the snippet works from inside the `mirofish` container; mention that host-mode (`npm run dev`) operators can substitute `http://localhost:11434/v1`.
- Set the example `EMBEDDING_API_KEY` to a non-empty placeholder string (Ollama ignores the value but `OpenAIEmbedderConfig` requires it to be non-empty).
- Leave the existing OpenAI/Gemini commented examples untouched — the Ollama block is additive.
- Observable completion: a fresh `cp .env.example .env` followed by uncommenting only the three Ollama lines and pulling the model in Ollama is sufficient to point the existing `openai`-provider Graphiti embedder at the local Ollama daemon.
- _Requirements: 1.1_
- _Boundary: .env.example_
- [x] 2.2 (P) Extend the "Required Environment Variables" section in `CLAUDE.md`
- Update the `EMBEDDING_MODEL` notes to enumerate the three supported embedder configurations: OpenAI (`text-embedding-3-small`), Gemini (`text-embedding-004`), and Ollama (`mxbai-embed-large`).
- Document the 1024-dim constraint imposed by Graphiti's default `EMBEDDING_DIM` and explicitly note that 768-dim models such as `nomic-embed-text` are unsupported until `EMBEDDING_DIM` is made configurable.
- Cross-reference `.env.example` for the Ollama-specific `EMBEDDING_BASE_URL`/`EMBEDDING_API_KEY` triple instead of duplicating the values inline.
- Observable completion: a new contributor reading only `CLAUDE.md` § "Required Environment Variables" can identify all three supported embedder providers and the dim constraint without consulting external sources.
- _Requirements: 1.2, 3.3_
- _Boundary: CLAUDE.md_
- [x] 2.3 (P) Add an Ollama section and `curl` smoke test to `README.md`
- In the "Required Environment Variables" block, add an Ollama example alongside the existing Gemini hint covering `EMBEDDING_BASE_URL`, `EMBEDDING_API_KEY`, and `EMBEDDING_MODEL`.
- Append a one-line `curl` snippet that POSTs to `$EMBEDDING_BASE_URL/embeddings` with the configured model and a trivial input, then pipes through `jq '.data[0].embedding | length'` to verify a `1024` response — explicitly framed as a pre-build smoke test.
- Use the same `host.docker.internal:11434` convention as `.env.example` and `docker-compose.yml`, with a short note on the `localhost` substitution for host-mode operators.
- Keep the existing copy/install steps untouched; this edit is additive within the same `## Configure Environment Variables` (or equivalent) subsection.
- Observable completion: an operator running the new `curl` snippet against a correctly configured Ollama daemon sees `1024` printed to stdout and can use that as a go/no-go signal before kicking off the graph build.
- _Requirements: 1.3, 1.4_
- _Boundary: README.md_
- [x] 2.4 (P) Add a `host.docker.internal` comment to the `mirofish` service in `docker-compose.yml`
- Add a single comment line above (or alongside) the existing `NEO4J_URI` override in the `mirofish` service noting that an Ollama daemon running on the host is reachable from this container via `host.docker.internal:11434` and that this is the value to use for `EMBEDDING_BASE_URL` when running the Compose stack.
- Do not introduce any new service, environment variable, or volume; the change is comment-only.
- Observable completion: a reader of `docker-compose.yml` who sets up Ollama on the host can derive the correct `EMBEDDING_BASE_URL` value without consulting external Docker networking documentation.
- _Requirements: 1.3_
- _Boundary: docker-compose.yml_
- [ ] 3. Manual end-to-end verification (deferred to reviewer — requires running Neo4j + LLM stack)
- [ ] 3.1 Verify the happy and failure paths through the graph-build pipeline (deferred to reviewer)
- Run `npm run dev` against the existing OpenAI/Qwen-style embedder configuration to confirm the graph-build flow still completes with real nodes/edges in Neo4j (regression check for R3.1).
- Set `EMBEDDING_MODEL` to a deliberately invalid value (e.g. `text-embedding-3-small-typo`) against the same base URL, trigger a graph build through the UI, and confirm the project exits `GRAPH_BUILDING`, the backing `Task` reaches `status = FAILED`, and `Task.error` carries the underlying 404/unknown-model message (R2.3, R2.5). Inspect the backend logs for the new ERROR-level entry from the Graphiti adapter (R2.4).
- If an Ollama daemon with `mxbai-embed-large` is available, follow the documented `.env.example` snippet plus the `curl` smoke test, then run a full graph build and confirm Neo4j has nodes/edges scoped to the project's `group_id` (R1.5).
- Note in the PR body that, on a partial-batch failure, episodes successfully written before the failure remain committed in Neo4j (post-condition documented in design.md); a re-run appends rather than overwrites because Graphiti episode UUIDs are unique.
- Observable completion: PR description records the three scenarios (OpenAI happy path, deliberate-typo failure path, optional Ollama happy path) with the resulting `Task` status, an excerpt of `Task.error` for the failure case, and a link to (or extract from) the ERROR-level adapter log.
- _Depends: 1.1, 2.1, 2.2, 2.3, 2.4_
- _Requirements: 1.5, 2.3, 2.4, 2.5, 3.1, 3.2_
- _Boundary: end-to-end pipeline (verification only, no code change)_
## Requirements Coverage
| Requirement | Tasks |
|-------------|-------|
| 1.1 | 2.1 |
| 1.2 | 2.2 |
| 1.3 | 2.3, 2.4 |
| 1.4 | 2.3 |
| 1.5 | 3.1 |
| 2.1 | 1.1 |
| 2.2 | 1.1 |
| 2.3 | 3.1 (verification — already implemented in `_build_graph_worker`) |
| 2.4 | 1.1, 3.1 |
| 2.5 | 3.1 (verification — already implemented in frontend task polling) |
| 2.6 | 1.1 |
| 3.1 | 3.1 |
| 3.2 | 3.1 |
| 3.3 | 2.2 |

5
.kiro/specs/i18n-ci-guard/baseline.txt Normal file
View File

@ -0,0 +1,5 @@
# Per-path CJK baseline for the i18n CI guard.
# Format: <path>\t<count>. Sorted lexicographically.
# Refresh via: python scripts/ci/i18n_cjk_guard.py --update-baseline
backend/app 307
frontend/src 124

544
.kiro/specs/i18n-ci-guard/design.md Normal file
View File

@ -0,0 +1,544 @@
# Design — i18n-ci-guard
## Overview
This feature installs a permanent, PR-time CI guard that blocks
regressions of the project's English-by-default state. It performs two
checks: `locales/en.json` must contain zero CJK characters, and the
total CJK match count under `backend/app/` and `frontend/src/` must not
exceed a committed per-path baseline. The guard is a single Python
script invoked by a single GitHub Actions workflow.
**Purpose**: This feature delivers an automatic regression gate to the
i18n initiative so reviewers do not have to spot CJK reintroductions
by eye.
**Users**: Project maintainers and PR authors. Maintainers gain a
hard regression gate; PR authors gain a script they can run locally to
catch regressions before pushing.
**Impact**: Adds the project's first `pull_request`-triggered CI
workflow. No production source under `backend/app/`, `frontend/src/`,
or `locales/` is modified by this spec — only new files are added.
### Goals
- Fail any PR that introduces a CJK character into `locales/en.json`.
- Fail any PR whose CJK match count under `backend/app/` or
`frontend/src/` exceeds the committed baseline.
- Print a single actionable failure message that includes the exact
command a contributor must run if the regression is intentional.
- Run end-to-end under sixty seconds on `ubuntu-latest`.
- Be reproducible verbatim on a developer machine with Python ≥3.11
and `git`.
### Non-Goals
- Re-implementing the full classification pipeline from
`.kiro/specs/i18n-e2e-english-verification/` (that work belongs to
PR #27).
- Auto-updating the baseline on `main`.
- Translating any production source to satisfy a higher baseline. The
initial baseline is recorded against `main` and only ratchets down
over time.
- Gating commits at pre-commit time. The guard is CI-only; a future
spec may wrap it in a hook.
## Boundary Commitments
### This Spec Owns
- The guard script `scripts/ci/i18n_cjk_guard.py` and its CLI
contract.
- The workflow `.github/workflows/i18n-cjk-guard.yml` and its
trigger configuration.
- The baseline file `.kiro/specs/i18n-ci-guard/baseline.txt` and its
format.
- The pass/fail semantics of both checks.
### Out of Boundary
- Any change to files under `backend/app/`, `frontend/src/`, or
`locales/` — except `locales/en.json` if it is found to contain CJK
during initial baseline calibration (a remediation translation would
be a separate spec/PR).
- The classification heuristics in PR #27's `classify.py`.
- Pre-commit hooks; IDE integrations; alternative scoped paths beyond
`backend/app/` and `frontend/src/`.
### Allowed Dependencies
- Python ≥3.11 standard library.
- `git` (for `git grep -nIP` invocation).
- `actions/checkout@v4` and `actions/setup-python@v5` from the
GitHub Actions Marketplace.
### Revalidation Triggers
- Adding a third scoped path → baseline file format changes; consumers
(none today) re-check.
- Changing the regex range → audit pipeline alignment must be
re-confirmed.
- Switching from `pull_request` to `merge_group` or other event →
required-status-check rules in branch protection must be re-checked.
## Architecture
### Existing Architecture Analysis
- **Repo layout**: monorepo split by runtime (`backend/`, `frontend/`)
with shared `locales/` at root. The guard scopes its scan to
`backend/app/`, `frontend/src/`, and `locales/en.json`, matching the
audit pipeline's canonical scope.
- **Existing scripts pattern**: `scripts/<purpose>.py` for developer
tools. The new `scripts/ci/` subdirectory introduces a clear,
CI-only home without disturbing the existing developer scripts.
- **Existing CI**: `.github/workflows/docker-image.yml` is tag-only.
No `pull_request` workflow exists. The new workflow is additive and
does not affect the docker-image workflow.
### Architecture Pattern & Boundary Map
```mermaid
flowchart LR
PR[Pull Request to main] -->|trigger| WF[.github/workflows/i18n-cjk-guard.yml]
WF -->|setup-python + checkout| RUN[python scripts/ci/i18n_cjk_guard.py]
RUN -->|read| EN[locales/en.json]
RUN -->|git grep -nIP| BAPP[backend/app/]
RUN -->|git grep -nIP| FSRC[frontend/src/]
RUN -->|read| BL[.kiro/specs/i18n-ci-guard/baseline.txt]
RUN -->|exit 0 or 1| WF
WF -->|status| PR
DEV[Developer terminal] -->|python scripts/ci/i18n_cjk_guard.py| RUN
DEV -->|--update-baseline| RUN
RUN -.->|writes| BL
```
**Architecture Integration**:
- **Selected pattern**: single-purpose script + thin workflow.
Matches the project's existing `scripts/<purpose>.py` convention.
- **Domain boundaries**: the guard is a pure verification tool with no
side effects on production code. Its only writeable surface is the
baseline file, and only when explicitly invoked with
`--update-baseline`.
- **Existing patterns preserved**: stdlib-only Python tooling
(precedent: `scripts/check_i18n_logs.py`); single-file workflows in
`.github/workflows/`.
- **New components rationale**: a new file rather than an extension of
an existing script — the existing script is scoped to a fixed
module list and is not a regression gate.
- **Steering compliance**: respects layer-based structure (script
lives at repo root in `scripts/ci/`, not under `backend/` or
`frontend/`), no new heavy dependencies, no `os.getenv` calls
outside `backend/app/config.py`.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Frontend / CLI | Python 3.11 stdlib (`argparse`, `json`, `re`, `subprocess`, `pathlib`, `sys`) | Guard CLI | Stdlib only — Req 5.5 |
| Backend / Services | n/a | — | Guard does not touch backend services |
| Data / Storage | Plain-text baseline file under `.kiro/specs/` | Per-path count store | One line per path, `<path>\t<count>` |
| Messaging / Events | n/a | — | — |
| Infrastructure / Runtime | GitHub Actions `ubuntu-latest`, `actions/checkout@v4`, `actions/setup-python@v5` | PR-time runner | `fetch-depth: 1` is sufficient |
## File Structure Plan
### Directory Structure
```
scripts/
└── ci/
└── i18n_cjk_guard.py # Guard CLI (new)
.github/
└── workflows/
└── i18n-cjk-guard.yml # PR-time workflow (new)
.kiro/specs/i18n-ci-guard/
├── spec.json # (existing, updated)
├── requirements.md # (existing)
├── gap-analysis.md # (existing)
├── research.md # (existing)
├── design.md # (this file)
├── tasks.md # (created in next phase)
└── baseline.txt # Per-path CJK match counts (new)
```
### Modified Files
- `.kiro/specs/i18n-ci-guard/spec.json` — phase / approval fields
updated by Kiro flow only.
- No production source files are modified by this spec.
## System Flows
### Guard execution (default mode)
```mermaid
sequenceDiagram
participant CI as GitHub Actions
participant Script as i18n_cjk_guard.py
participant Repo as Working tree
participant BL as baseline.txt
CI->>Script: python scripts/ci/i18n_cjk_guard.py
Script->>Repo: read locales/en.json
Script->>Script: scan for CJK chars
alt en.json has CJK
Script-->>CI: exit 1 + per-key findings
else en.json clean
Script->>Repo: git grep -nIP backend/app/
Script->>Repo: git grep -nIP frontend/src/
Script->>BL: read baseline counts
alt any current count > baseline
Script-->>CI: exit 1 + per-path delta + refresh hint
else within baseline
Script-->>CI: exit 0 + summary
end
end
```
### Baseline refresh
```mermaid
sequenceDiagram
participant Dev as Developer
participant Script as i18n_cjk_guard.py
participant Repo as Working tree
participant BL as baseline.txt
Dev->>Script: python scripts/ci/i18n_cjk_guard.py --update-baseline
Script->>Repo: git grep -nIP backend/app/
Script->>Repo: git grep -nIP frontend/src/
Script->>BL: write per-path counts (sorted)
Script-->>Dev: exit 0 + new counts
```
The two checks run in fixed order: en.json first (cheap, decisive),
then per-path counts. Both run under all conditions; the script does
not short-circuit after the first failure so the contributor sees the
complete diagnostic in one CI log.
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | Scan en.json for CJK | `i18n_cjk_guard.py` | CLI default mode | Guard execution |
| 1.2 | Fail with key:line per offender | `i18n_cjk_guard.py` | CLI stderr output | Guard execution |
| 1.3 | Report clean state | `i18n_cjk_guard.py` | CLI stdout summary | Guard execution |
| 1.4 | Hard error if file missing | `i18n_cjk_guard.py` | CLI stderr + exit 1 | Guard execution |
| 2.1 | Count CJK matches per scoped path | `i18n_cjk_guard.py` | `git grep -nIP` invocation | Guard execution |
| 2.2 | Read baseline counts | `i18n_cjk_guard.py`, `baseline.txt` | File read | Guard execution |
| 2.3 | Fail on regression | `i18n_cjk_guard.py` | Exit 1 | Guard execution |
| 2.4 | Pass when within baseline | `i18n_cjk_guard.py` | Exit 0 | Guard execution |
| 2.5 | Skip binary files | `git grep -I` | — | Guard execution |
| 2.6 | Tracked-only scope | `git grep` default | — | Guard execution |
| 3.1 | Per-key locale failure detail | `i18n_cjk_guard.py` | CLI stderr lines | Guard execution |
| 3.2 | Per-path regression detail | `i18n_cjk_guard.py` | CLI stderr lines | Guard execution |
| 3.3 | Print refresh command | `i18n_cjk_guard.py` | CLI stderr footer | Guard execution |
| 3.4 | Success summary lines | `i18n_cjk_guard.py` | CLI stdout | Guard execution |
| 4.1 | Baseline under spec dir | `baseline.txt` | File path | — |
| 4.2 | Diff-friendly text format | `baseline.txt` | File format | — |
| 4.3 | Refresh via flag | `i18n_cjk_guard.py` | `--update-baseline` | Baseline refresh |
| 4.4 | No implicit baseline writes | `i18n_cjk_guard.py` | CLI default mode | Guard execution |
| 4.5 | Hard error if baseline missing | `i18n_cjk_guard.py` | Exit 1 + message | Guard execution |
| 5.1 | PR-only trigger to main | `i18n-cjk-guard.yml` | `on.pull_request.branches` | — |
| 5.2 | Checkout PR head | `i18n-cjk-guard.yml` | `actions/checkout@v4` | — |
| 5.3 | Surface output on failure | `i18n-cjk-guard.yml` | Default GH log | — |
| 5.4 | Pass on exit 0 | `i18n-cjk-guard.yml` | Default | — |
| 5.5 | Stdlib-only, no third-party | `i18n_cjk_guard.py`, `i18n-cjk-guard.yml` | — | — |
| 5.6 | ≤60s runtime | `i18n-cjk-guard.yml` | `timeout-minutes: 1` | — |
| 6.1 | Same result locally | `i18n_cjk_guard.py` | CLI | — |
| 6.2 | Single stable entry point | `scripts/ci/i18n_cjk_guard.py` | Path | — |
| 6.3 | No env vars / secrets | `i18n_cjk_guard.py` | CLI | — |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|--------------|--------|--------------|------------------|-----------|
| `i18n_cjk_guard.py` | CI script | Two-check guard CLI | 1.1–6.3 | `git`, Python stdlib | Service (CLI) |
| `i18n-cjk-guard.yml` | CI workflow | Run guard on every PR to main | 5.1–5.6 | `actions/checkout@v4`, `actions/setup-python@v5` | Batch / Job |
| `baseline.txt` | Data | Per-path baseline counts | 4.1, 4.2, 2.2 | — | State (file) |
### CI Script
#### `i18n_cjk_guard.py`
| Field | Detail |
|-------|--------|
| Intent | Run two CJK-regression checks; optionally refresh the baseline |
| Requirements | 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 3.1, 3.2, 3.3, 3.4, 4.1, 4.3, 4.4, 4.5, 5.5, 6.1, 6.2, 6.3 |
| Owner / Reviewers | i18n maintainers |
**Responsibilities & Constraints**
- Owns the canonical guard semantics: which paths are scoped, which
regex is canonical, what counts as a regression.
- Runs in pure Python 3.11 stdlib + a single `git` subprocess per
scoped path.
- Never modifies any file other than the baseline file, and only when
invoked with `--update-baseline`.
- Always runs both checks (does not short-circuit), so a single CI log
shows every failure mode at once.
**Dependencies**
- Inbound: `i18n-cjk-guard.yml` workflow; developers running locally.
- Outbound: `git` subprocess (`git grep`, `git rev-parse`).
- External: none.
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x]
##### Service Interface (CLI)
```text
i18n_cjk_guard.py [--update-baseline] [--baseline PATH] [--repo-root PATH]
```
Type-annotated module signature (Python type hints, public functions
only):
```python
def main(argv: list[str]) -> int: ...
def run_check(repo_root: pathlib.Path, baseline_path: pathlib.Path) -> int:
"""Run both checks; return 0 on success, 1 on any failure."""
def update_baseline(repo_root: pathlib.Path, baseline_path: pathlib.Path) -> int:
"""Refresh the baseline file with current per-path counts; return 0."""
def scan_locale_cjk(en_json_path: pathlib.Path) -> list[LocaleFinding]:
"""Return a list of (key, line_number, snippet) tuples for every
CJK occurrence in locales/en.json. Empty list when clean."""
def count_path_cjk(repo_root: pathlib.Path, scoped_path: str) -> int:
"""Return the number of CJK match lines under scoped_path,
using `git grep -nIP '[\\x{4e00}-\\x{9fff}]' -- <scoped_path>`."""
def read_baseline(baseline_path: pathlib.Path) -> dict[str, int]:
"""Parse the baseline file. Each non-empty, non-comment line is
'<path>\\t<count>'. Raise BaselineError on any malformed input
or missing file."""
def write_baseline(baseline_path: pathlib.Path, counts: dict[str, int]) -> None:
"""Atomically overwrite the baseline file with sorted entries
and a single trailing newline."""
```
Where:
```python
LocaleFinding = tuple[str, int, str] # (dotted_key, line_number, snippet)
SCOPED_PATHS: tuple[str, ...] = ("backend/app", "frontend/src")
EN_JSON_REL_PATH: str = "locales/en.json"
CJK_PATTERN: str = "[\\x{4e00}-\\x{9fff}]" # passed to git grep -P
CJK_RE: re.Pattern[str] = re.compile(r"[一-鿿]")
SNIPPET_MAX_LEN: int = 80
```
- **Preconditions**: invoked with CWD at the repo root or
`--repo-root` set; `git` is on `$PATH`; the working tree is the
intended scan target.
- **Postconditions** (default mode): exit 0 iff both checks pass;
exit 1 otherwise. Stdout receives the success summary; stderr
receives findings on failure. The baseline file is unchanged.
- **Postconditions** (`--update-baseline`): the baseline file is
rewritten to current per-path counts and exit 0 is returned.
- **Invariants**: regex range, scoped paths, and baseline file path
are constants — no env-var override.
##### State Management
- **State model**: a dict `{<scoped_path>: <count>}` parsed from
the baseline file.
- **Persistence**: plain-text file at
`.kiro/specs/i18n-ci-guard/baseline.txt`. Atomic write via
`tmp + os.replace`.
- **Concurrency**: single-writer (developer running
`--update-baseline`); CI workers only read.
**Implementation Notes**
- Output format mirrors `scripts/check_i18n_logs.py`:
`<file>:<line>: <reason>: <snippet>` on stderr, summary on stdout,
trailing `OK` or `N issues`.
- The exact refresh command printed on regression failure is:
`python scripts/ci/i18n_cjk_guard.py --update-baseline`.
- `count_path_cjk` invokes `git grep` via `subprocess.run` with
`check=False`; `git grep` exits 1 when there are zero matches, so
the function treats exit codes 0 and 1 as success and any other
code as a hard error.
- Localised key extraction for `en.json` walks the parsed JSON dict;
line numbers are obtained by re-reading the file as text and
matching the value's first textual occurrence.
- Risks: see `research.md` § Risks & Mitigations.
### CI Workflow
#### `i18n-cjk-guard.yml`
| Field | Detail |
|-------|--------|
| Intent | Run the guard on every PR to `main` |
| Requirements | 5.1, 5.2, 5.3, 5.4, 5.5, 5.6 |
| Owner / Reviewers | i18n maintainers |
**Contracts**: Batch / Job [x]
##### Batch / Job Contract
- **Trigger**: `on: pull_request: branches: [main]`.
- **Input / validation**: PR head ref checkout via
`actions/checkout@v4` with `fetch-depth: 1`. Python set up via
`actions/setup-python@v5` with `python-version: '3.11'`.
- **Output / destination**: pass/fail status surfaced as a GitHub
Actions check on the PR. Script stdout/stderr appears in the
workflow log.
- **Idempotency & recovery**: re-running the workflow re-evaluates the
same working tree; no persistent side effects on the runner.
##### Workflow shape (sketch)
```yaml
name: i18n CJK Guard
on:
pull_request:
branches: [main]
jobs:
guard:
runs-on: ubuntu-latest
timeout-minutes: 1
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- run: python scripts/ci/i18n_cjk_guard.py
```
### Baseline Data File
#### `baseline.txt`
| Field | Detail |
|-------|--------|
| Intent | Persist the per-path CJK match-count baseline |
| Requirements | 2.2, 4.1, 4.2 |
**Contracts**: State [x]
##### Format
```text
# Per-path CJK baseline for the i18n CI guard.
# Format: <path>\t<count>. Sorted lexicographically.
# Refresh via: python scripts/ci/i18n_cjk_guard.py --update-baseline
backend/app <int>
frontend/src <int>
```
- One header block of `#`-prefixed comments (parser ignores).
- Blank lines ignored.
- Lines must match `^(?P<path>[^\t\n]+)\t(?P<count>\d+)$`.
- Trailing newline mandatory.
## Data Models
### Domain Model
- `LocaleFinding` — value object
`(dotted_key: str, line_number: int, snippet: str)`.
- `PathCount` — pair `(scoped_path: str, count: int)`. The full
baseline is a `dict[str, int]` keyed by scoped path.
Invariants:
- `count` is a non-negative integer.
- `scoped_path` is one of `SCOPED_PATHS`.
- `LocaleFinding.snippet` is at most `SNIPPET_MAX_LEN` characters,
truncated with an ellipsis when needed.
## Error Handling
### Error Strategy
- All non-zero exits are accompanied by a stderr message identifying
the failing check, the offending file or path, and (for regressions)
the refresh command. The script never raises uncaught exceptions
past `main()` in normal flow; unexpected I/O errors propagate as
`OSError` with a clear traceback so CI logs surface them clearly.
### Error Categories and Responses
- **Locale failure** (Req 1.2): one stderr line per offending key
(`locales/en.json:<line>: cjk-in-en: <key> = <snippet>`), then a
trailing `N issues` summary.
- **Regression failure** (Req 3.2): one stderr line per regressed
path (`<path>: cjk-regression: baseline=<b> current=<c> delta=+<d>`)
followed by a one-line refresh hint:
`# refresh via: python scripts/ci/i18n_cjk_guard.py --update-baseline`.
- **Missing en.json** (Req 1.4): stderr `locales/en.json: missing
catalogue file`, exit 1.
- **Missing or malformed baseline** (Req 4.5): stderr
`<baseline-path>: missing or malformed; refresh via …`, exit 1.
- **`git grep` unavailable / non-PCRE**: stderr
`git grep failed: <stderr>`, exit 1.
### Monitoring
- The guard is a single short-lived script. All observability is
delegated to GitHub Actions logs (stdout/stderr, run duration).
No external telemetry.
## Testing Strategy
### Unit Tests (Python)
Place tests under `scripts/ci/tests/test_i18n_cjk_guard.py` (or invoke
the script directly via subprocess in a tmp git repo). The project's
test runner is `pytest` (already used by `backend/`), but the new
tests must be runnable with `python -m pytest` from the repo root
without backend dependencies. Tests are scoped to:
1. `scan_locale_cjk` — clean catalogue returns empty list; planted CJK
value returns a single `LocaleFinding` with the correct key and
line number.
2. `count_path_cjk` — given a tmp git repo with N planted CJK lines,
returns N; binary file matches are excluded; untracked file
matches are excluded.
3. `read_baseline` / `write_baseline` round-trip — write counts,
re-read, equal.
4. `read_baseline` malformed input — non-tab line → `BaselineError`.
5. `run_check` end-to-end — passing baseline → exit 0; regressed
baseline → exit 1 and stderr contains the refresh command.
### Integration Tests
1. Workflow shape — `actionlint` (optional, if installed locally) on
`i18n-cjk-guard.yml`. At minimum, `python -c "import yaml;
yaml.safe_load(open('.github/workflows/i18n-cjk-guard.yml'))"` for
YAML validity.
2. Local end-to-end — run
`python scripts/ci/i18n_cjk_guard.py` from the repo root with the
committed baseline; expect exit 0 on a clean checkout of `main`.
3. Refresh end-to-end — run with `--update-baseline`; verify
baseline file is rewritten and a second default run is exit 0.
### Performance / Load
- Single-pass `git grep` over the scoped paths runs in <2 s on the
current repo. The workflow's `timeout-minutes: 1` is a hard ceiling
per Req 5.6.
## Optional Sections
### Security Considerations
- The guard reads only tracked text files; no secrets are accessed.
- The workflow uses `GITHUB_TOKEN` only implicitly via
`actions/checkout`; no additional permissions are requested
(`permissions:` block omitted relies on the repo default of
`contents: read`, which is sufficient).

169
.kiro/specs/i18n-ci-guard/gap-analysis.md Normal file
View File

@ -0,0 +1,169 @@
# Gap Analysis — i18n-ci-guard
Comparison of the approved requirements against the current MiroFish
codebase, focused on what already exists, what is missing, and what
options the design phase should choose between.
## 1. Current State Investigation
### Domain assets already in the repo
- **`scripts/check_i18n_logs.py`** — Python-stdlib-only, exit-code-based
i18n verification script. Uses the same canonical CJK regex
`[一-鿿]` (`U+4E00..U+9FFF`) the new guard needs, prints findings as
`<file>:<line>: <reason>: <snippet>`, and was written for ticket #6.
Strong precedent for the new guard's CLI surface and output format.
- **`scripts/_apply_translations.py`, `scripts/_codemod_i18n.py`,
`scripts/_merge_locale_keys.py`** — i18n tooling sibling scripts.
Convention is to keep auxiliary i18n scripts under `scripts/` at the
repo root.
- **`.github/workflows/docker-image.yml`** — only existing GH Actions
workflow; triggers on tag pushes and `workflow_dispatch`. No PR-time
workflow exists yet, so the new guard introduces the project's first
PR-blocking CI check.
- **PR #27 / branch `chore/i18n-10-e2e-english-verification`** — defines
the audit methodology referenced by the ticket. Its `audit_cjk.sh`
uses `git grep -nIP '[\x{4e00}-\x{9fff}]' -- backend/app frontend/src
locales/en.json` — the canonical scoped scan command. PR #27 is open;
the new guard must work with or without it merged.
- **`.kiro/specs/<feature>/`** — established home for spec artefacts.
`i18n-externalize-backend-logs/` is the closest precedent for an
i18n-flavoured spec.
- **`locales/en.json`, `locales/zh.json`, `locales/languages.json`** —
shared i18n source consumed by both runtimes.
### Conventions extracted
- Auxiliary scripts: `scripts/<purpose>.py`, Python ≥3.11 stdlib only,
shebang `#!/usr/bin/env python3`, double-quoted strings, snake_case,
Google-style docstrings on the module and public functions.
- Output format: `<file>:<line>: <reason>: <snippet>`, summary line
`OK` or `N issues`, exit `0`/`1`.
- Reuse the canonical regex `[一-鿿]` rather than re-deriving range
literals.
- 4-space indent, ≤120 cols, no trailing whitespace, single trailing
newline (`.claude/rules/dev-guidelines.md`).
### Integration surfaces
- **CI**: GitHub Actions, `.github/workflows/`. `ubuntu-latest` runner,
Python 3.11+ via `actions/setup-python@v5` (use the same version
pin already present in the docker-image workflow ecosystem if any).
- **Repo layout boundaries** scoped by the audit: `backend/app/`,
`frontend/src/`, `locales/en.json` — all live at repo root or two
levels deep.
- **Git working tree**: the guard relies on `git grep -I` for tracked,
text-only matches; this binds the guard to a runner that has `git`
available (true on `ubuntu-latest` and on developer machines).
## 2. Requirement-to-Asset Map
| Req | Need | Existing asset | Gap |
| --- | --------------------------------- | ----------------------------------------------------------------------------------------------- | ----------- |
| 1 | CJK scan of `locales/en.json` | `scripts/check_i18n_logs.py` already loads `locales/*.json` and runs the canonical regex. | Missing — new guard must scan en.json specifically and emit `key:line` per offender. |
| 2 | CJK count under `backend/app/` and `frontend/src/` against baseline | Audit `audit_cjk.sh` (PR #27) demonstrates `git grep -nIP` is the canonical scan; no baseline file exists yet on main. | Missing — no per-path counter, no baseline file. |
| 3 | Actionable failure messaging | `check_i18n_logs.py` output format reusable. | Missing — need refresh-baseline command in failure text. |
| 4 | Baseline file lifecycle | None. | Missing — file format and refresh subcommand to design. |
| 5 | GH Actions PR integration | `.github/workflows/` directory exists; one tag-only workflow. | Missing — new `pull_request` workflow. |
| 6 | Local reproducibility | Existing scripts run locally with stdlib; same pattern reusable. | None — covered by following the existing pattern. |
## 3. Implementation Approach Options
### Option A — Extend `scripts/check_i18n_logs.py`
Add a new `--cjk-guard` mode (catalogue scan + per-path baseline diff)
to the existing script, then call it from the new workflow.
- ✅ One file to maintain; reuses the regex constant and CLI.
- ❌ The existing script is tightly scoped to the in-scope backend
modules and the parity check. Mixing a PR-gating regression check into
it dilutes its intent and grows it past the SRP line that the
surrounding scripts respect.
- ❌ The existing script targets a fixed list of backend modules; the
new guard scans whole subtrees. The two scopes don't fit one CLI.
### Option B — New, focused script `scripts/ci/i18n_cjk_guard.py` + new workflow (recommended)
A new directory `scripts/ci/` holds CI-only scripts; the guard is a
single file that performs both checks and supports a `--refresh-baseline`
flag. New workflow `.github/workflows/i18n-cjk-guard.yml` runs it on
every PR to `main`.
- ✅ Clean separation: production-i18n script (`check_i18n_logs.py`)
and CI-gating script (`i18n_cjk_guard.py`) live side by side without
overlapping responsibilities.
- ✅ Mirrors the established convention of one script per
responsibility under `scripts/`.
- ✅ The baseline file lives under the spec dir
(`.kiro/specs/i18n-ci-guard/baseline.txt`), matching the ticket's
"baseline must be committed and reviewable" requirement.
- ❌ One more file in the repo, but the file is small (~150 LoC).
### Option C — Hybrid: shared `cjk_scan.py` helper + thin guard script
Factor the regex + git-grep logic into a tiny shared helper consumed by
both `check_i18n_logs.py` and the new guard.
- ✅ DRY for the regex constant.
- ❌ Premature abstraction: today the only shared element is one
one-line regex. The two scripts have different scopes, output
formats, and consumers. Pulling a helper out now satisfies
consistency without paying for itself; defer until a third caller
appears.
### Recommendation
**Option B**. It matches the project's established "one focused script
per responsibility" convention, isolates the new CI surface from
existing i18n scripts, and keeps the baseline file collocated with
spec metadata where reviewers expect to find it.
## 4. Research Items for Design Phase
- **Baseline file format**: prefer a stable, line-oriented text format
over JSON to minimize diff churn (e.g., `path<TAB>count` per line,
trailing newline). Confirm in design.
- **`git grep` invocation portability**: `git grep -nIP` works on all
modern git builds (≥2.4 ships PCRE2). `ubuntu-latest` ships ≥2.40.
No portability concern; record the assumption explicitly.
- **`fetch-depth`** for the `actions/checkout@v4` step: `git grep`
scans the working tree, not history, so a shallow clone (`fetch-depth:
1`) is sufficient.
- **Workflow timeout budget**: capture the empirical runtime of the
full scan locally (already measured: a single `git grep` over the
scoped paths runs in <2 seconds with ~3.6k matches). The 60-second
ceiling in Req 5 is comfortable.
- **Failure-message refresh command** wording: the design should pin
the exact command shown to contributors so it stays one stable
string developers can copy.
- **Initial baseline values**: with `git grep -nIP '[\x{4e00}-\x{9fff}]'`
on the current branch — `backend/app` = 2707, `frontend/src` = 902,
`locales/en.json` = 0. The committed baseline must be regenerated
against `main` at implementation time so it reflects the merge target.
## 5. Effort & Risk
- **Effort**: **S** (1–3 days). Small, self-contained additions
(one Python script, one workflow file, one baseline file, plus the
spec). All patterns already exist in the repo.
- **Risk**: **Low**. No production-source changes, no new dependencies,
no architectural shifts. The only failure mode is a noisy guard
blocking unrelated PRs — mitigated by the per-path baseline ratchet.
## 6. Recommendations for Design Phase
- Adopt **Option B** (new focused script + new workflow + baseline file
under spec dir).
- Lock in the canonical regex `[一-鿿]` and the canonical scan command
`git grep -nIP '[\x{4e00}-\x{9fff}]' -- <path>` to keep this guard
bytewise-aligned with the audit pipeline.
- Use a line-oriented baseline format keyed by scoped path; explicit
`--refresh-baseline` (or equivalent) subcommand updates it; no
implicit overwrite.
- Output: machine-friendly findings on stderr, summary on stdout,
exit `0`/`1`.
- The workflow should run only on `pull_request` to `main` (Req 5.1)
with `fetch-depth: 1` and `actions/setup-python@v5`. No third-party
packages.
- Baseline counts must be recomputed against `main` before the PR
ships; do not commit baselines from a feature branch's working tree.

189
.kiro/specs/i18n-ci-guard/requirements.md Normal file
View File

@ -0,0 +1,189 @@
# Requirements Document
## Project Description (Input)
Add a permanent CI guard that runs an i18n CJK audit on every pull request.
Linked GitHub issue: #26 (.ticket/26.md).
The guard must fail a PR build when:
1. locales/en.json contains any CJK character (range U+4E00..U+9FFF), or
2. The total count of CJK matches across backend/app/ and frontend/src/ regresses (i.e. exceeds) a committed baseline value.
## Introduction
The i18n initiative has driven the project toward English-by-default UI, logs,
prompts, and documentation. Manual audits (see PR #27, the
`i18n-e2e-english-verification` spec) have repeatedly surfaced regressions
where Chinese strings re-enter the codebase. This spec installs a permanent,
self-contained CI guard that runs on every pull request and fails the build
when (a) `locales/en.json` is no longer CJK-clean, or (b) the total CJK match
count under `backend/app/` and `frontend/src/` regresses against a committed
baseline.
The guard is intentionally minimal: it captures the two highest-signal checks
from the larger audit pipeline so it can run on every PR with a sub-minute
budget and without depending on the (currently unmerged) verification spec.
The committed baseline lets the project ratchet down gaps over time without
blocking unrelated PRs on pre-existing CJK content.
## Boundary Context
- **In scope**:
- A locally runnable Python script that performs both guard checks on the
current working tree.
- A baseline file committed under the spec directory recording the
accepted CJK match counts per scoped path.
- A GitHub Actions workflow that runs the script on every pull request
targeting `main` and fails the build when either check fails.
- A clear, actionable failure message (which path regressed, baseline
value, current value, command to update the baseline).
- **Out of scope**:
- The full classification pipeline (`classify.py`, `render_report.py`,
`post_comment.sh`) from the unmerged `i18n-e2e-english-verification`
spec — those scripts perform deeper audit work and are not required
for the PR-time guard.
- Auto-updating the baseline on `main` (the baseline is a normal
reviewable file).
- Translation work itself; this spec only enforces a regression gate.
- Any change to production source under `backend/app/`, `frontend/src/`,
or `locales/` apart from translations needed to satisfy the guard
against its own initial baseline.
- **Adjacent expectations**:
- PR #27 (`chore/i18n-10-e2e-english-verification`) provides the
methodology referenced here. This spec must remain functional whether
PR #27 has been merged or not.
- The guard reuses the canonical CJK regex range
`[一-鿿]` already established by that audit.
## Requirements
### Requirement 1: Locale-catalogue CJK cleanliness check
**Objective:** As a maintainer of the English locale catalogue, I want every
PR to fail when `locales/en.json` reintroduces any CJK character, so that the
English catalogue stays CJK-free.
#### Acceptance Criteria
1. When the guard script is run from the repository root, the i18n CI Guard
shall scan the contents of `locales/en.json` for any character in the
range `U+4E00..U+9FFF`.
2. If `locales/en.json` contains at least one such character, the i18n CI
Guard shall exit with a non-zero status and report each offending
`key:line` pair on standard output.
3. While `locales/en.json` contains zero such characters, the i18n CI Guard
shall report the catalogue as CJK-clean.
4. If `locales/en.json` is missing or unreadable, the i18n CI Guard shall
exit with a non-zero status and emit an explicit error message naming
the missing file.
### Requirement 2: Backend/frontend CJK regression check against committed baseline
**Objective:** As a maintainer of English support across the codebase, I
want every PR to fail when the total CJK match count under `backend/app/`
or `frontend/src/` exceeds a committed baseline, so that the codebase
ratchets monotonically toward English-only without blocking PRs on
pre-existing CJK content.
#### Acceptance Criteria
1. When the guard script is run, the i18n CI Guard shall count the total
number of CJK matches (range `U+4E00..U+9FFF`, line-level, text files
only) under each of the scoped paths `backend/app/` and `frontend/src/`.
2. The i18n CI Guard shall read the baseline counts from a single
committed baseline file under the spec directory.
3. If the current count for any scoped path exceeds the baseline count for
that path, the i18n CI Guard shall exit with a non-zero status.
4. While the current count for every scoped path is less than or equal to
the baseline, the i18n CI Guard shall exit with status zero for this
check.
5. The i18n CI Guard shall ignore matches inside binary files
(image, font, archive, lockfile, or other non-text formats) by relying
on `git grep -I` semantics.
6. The i18n CI Guard shall scope its scan to tracked files only (matches
in untracked or ignored files shall not contribute to the count).
### Requirement 3: Actionable failure messaging
**Objective:** As a contributor whose PR was rejected by the guard, I want
the failure message to tell me exactly what regressed and how to fix it,
so that I can either translate the offending content or — when intentional —
update the baseline through normal review.
#### Acceptance Criteria
1. If the locale-catalogue check fails, the i18n CI Guard shall print, for
each offending entry: the dotted catalogue key, the line number in
`locales/en.json`, and a truncated snippet of the value.
2. If the regression check fails, the i18n CI Guard shall print, for each
regressed scoped path: the path name, the baseline count, the current
count, and the delta.
3. If the regression check fails, the i18n CI Guard shall print the exact
shell command a contributor must run locally to refresh the baseline
file so the PR can be re-reviewed against the new value.
4. The i18n CI Guard shall print, on success, a one-line summary per check
confirming the catalogue is CJK-clean and the per-path counts are at or
below baseline.
### Requirement 4: Baseline file lifecycle
**Objective:** As a reviewer enforcing English support, I want the baseline
to live in the repository as a small, human-readable file that only changes
through code review, so that downward ratcheting is intentional and
auditable.
#### Acceptance Criteria
1. The i18n CI Guard shall store the baseline as a single committed file
under `.kiro/specs/i18n-ci-guard/`.
2. The baseline file shall record one count per scoped path, in a stable,
diff-friendly text format (no JSON line shuffling, no trailing
whitespace).
3. When the guard script is invoked with an explicit "refresh baseline"
subcommand or flag, the i18n CI Guard shall overwrite the baseline file
with the current per-path counts and exit with status zero.
4. While no refresh flag is supplied, the i18n CI Guard shall never modify
the baseline file.
5. If the baseline file is missing at check time, the i18n CI Guard shall
exit with a non-zero status and instruct the contributor to refresh it.
### Requirement 5: GitHub Actions PR integration
**Objective:** As a project maintainer, I want every pull request targeting
`main` to be gated by the guard, so that no merge silently regresses the
English-only state of the catalogue or codebase.
#### Acceptance Criteria
1. The i18n CI Guard workflow shall trigger on every `pull_request` event
whose base ref is `main`.
2. While the workflow runs, the i18n CI Guard shall check out the PR head
commit with full history sufficient for `git grep` to scan tracked
files.
3. When the guard script exits with non-zero status, the workflow shall
fail and surface the script's standard output and standard error in the
GitHub Actions log.
4. When the guard script exits with status zero, the workflow shall pass.
5. The workflow shall use only Python from the standard
`actions/setup-python` distribution and tools already available on the
GitHub-hosted `ubuntu-latest` runner (`bash`, `git`); it shall not
install third-party Python packages.
6. The workflow shall complete within sixty seconds of wall-clock time on
a clean `ubuntu-latest` runner.
### Requirement 6: Local reproducibility
**Objective:** As a developer preparing a PR, I want to run the same guard
locally before pushing, so that I can catch regressions before CI does.
#### Acceptance Criteria
1. When the guard script is invoked from a developer machine that has
Python 3.11 or newer and `git` available, the i18n CI Guard shall
produce the same pass/fail result and the same per-path counts that
it would produce in CI for the same working tree.
2. The i18n CI Guard shall expose a single, stable invocation entry point
(a script under `scripts/ci/`) documented in the spec's design and
README touchpoints.
3. The i18n CI Guard shall require zero environment variables or secrets
to run locally.

175
.kiro/specs/i18n-ci-guard/research.md Normal file
View File

@ -0,0 +1,175 @@
# Research & Design Decisions — i18n-ci-guard
## Summary
- **Feature**: `i18n-ci-guard`
- **Discovery Scope**: Simple Addition (one Python script + one GH Actions
workflow + one baseline file). Extension-flavoured because it builds on
established `scripts/` conventions and the canonical CJK regex used by
the larger audit pipeline.
- **Key Findings**:
- The canonical CJK match command `git grep -nIP '[\x{4e00}-\x{9fff}]'
-- <path>` is already used by the unmerged audit pipeline (PR #27)
and is portable on every git ≥2.4 (`ubuntu-latest` ships ≥2.40).
- `scripts/check_i18n_logs.py` is a strong CLI/style precedent:
Python-stdlib-only, exit `0`/`1`, output as `<file>:<line>:
<reason>: <snippet>`, canonical regex `[一-鿿]`.
- The repository has no existing `pull_request`-triggered GH Actions
workflow; this guard introduces the first one. The only existing
workflow (`.github/workflows/docker-image.yml`) runs on tag pushes
only.
- Current per-path counts on this branch:
`backend/app=2707, frontend/src=902, locales/en.json=0`. These are
sample counts; the committed baseline must be regenerated against
`main` at implementation time.
## Research Log
### Canonical scan command
- **Context**: Requirement 2 needs a stable per-path CJK count and
Requirement 5.5 forbids third-party packages.
- **Sources Consulted**:
- `audit_cjk.sh` from PR #27 commit `3481408`.
- `git grep` man page.
- **Findings**:
- `git grep -nIP '[\x{4e00}-\x{9fff}]' -- <path>` returns one match
per matching line in tracked, text-only files. `-I` excludes binary
files; `-P` enables PCRE2 so the `\x{...}` Unicode range works.
- This matches the input format consumed by the existing audit
classifier, so the guard's match counts are directly comparable
across pipelines.
- **Implications**:
- The guard re-uses this exact command; no new dependencies.
- Because `-I` skips binary files and tracked-only is the default,
Requirements 2.5 and 2.6 are satisfied by the command itself
rather than by additional script logic.
### Baseline file format
- **Context**: Requirement 4 needs a diff-friendly committed baseline.
- **Sources Consulted**:
- Diff churn behaviour of JSON vs. line-oriented text in this repo's
history (e.g. `locales/*.json` PR diffs frequently re-key, while
plain-text `parity.txt` from PR #27 reads cleanly).
- **Findings**:
- Line-oriented `<path>\t<count>` files produce minimal diffs and
require no JSON parser.
- A two-line file (one per scoped path) is large enough to be
self-explanatory and small enough to never line-shuffle.
- **Implications**:
- Use plain text, sorted by path, single trailing newline. Reject
the file as malformed if the script cannot parse it (Req 4.5).
### Locale-catalogue scan path
- **Context**: Requirement 1 wants `key:line` per CJK offender in
`locales/en.json`.
- **Sources Consulted**:
- `scripts/check_i18n_logs.py` (`flatten_keys` reuse pattern).
- `check_parity.py` from PR #27 (`flatten`, `[cjk-in-en]` block).
- **Findings**:
- Both precedents flatten the locale dict and run the canonical
regex against each leaf string value. Line numbers are derivable
by re-reading the file as text and matching the value's first
occurrence (good enough for an actionable error message).
- Empty-string values and non-string leaf values (booleans, null)
are skipped.
- **Implications**:
- Implement a tiny flatten-then-scan helper inside the guard
script; do not add a new shared utility module.
### GH Actions trigger and budget
- **Context**: Requirements 5.1, 5.5, 5.6.
- **Sources Consulted**:
- GitHub-hosted runners reference (`ubuntu-latest`).
- `actions/setup-python@v5` README.
- **Findings**:
- `ubuntu-latest` has Python 3.10+ pre-installed; `actions/setup-python@v5`
pins to 3.11 in <5 s.
- A single `git grep` over the scoped paths runs in <2 s on this
repo (~3.6k matches). End-to-end the workflow comfortably fits
inside the 60 s ceiling.
- **Implications**:
- Use `actions/checkout@v4` with `fetch-depth: 1`,
`actions/setup-python@v5` with `python-version: '3.11'`, and run
the script directly. No caching layer needed.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| A. Extend `check_i18n_logs.py` | Add `--cjk-guard` mode to existing script | Reuses one file | Conflates two scopes; existing script is module-scoped, guard is subtree-scoped | Rejected |
| B. New `scripts/ci/i18n_cjk_guard.py` + new workflow | Single-purpose script + workflow + baseline file | Clean SRP; matches "one script per responsibility" precedent | One additional file | **Selected** |
| C. Shared `cjk_scan.py` helper + thin guard | Factor regex/git-grep into helper | DRY for regex constant | Premature abstraction; only one shared symbol today | Rejected |
## Design Decisions
### Decision: Single-purpose CI script + GH Actions workflow (Option B)
- **Context**: Requirements 1–6 demand a small, self-contained guard.
- **Alternatives Considered**: A (extend), C (shared helper).
- **Selected Approach**: New script `scripts/ci/i18n_cjk_guard.py`,
new workflow `.github/workflows/i18n-cjk-guard.yml`, baseline file
`.kiro/specs/i18n-ci-guard/baseline.txt`.
- **Rationale**: Matches the project's "one focused script per
responsibility" convention; isolates a CI-blocking surface from the
existing i18n developer scripts; keeps the baseline collocated with
the spec for review traceability.
- **Trade-offs**: One more file in `scripts/` vs. tighter cohesion.
- **Follow-up**: When a third caller wants the canonical regex, factor
it out then.
### Decision: Plain-text baseline format
- **Context**: Requirement 4.2 demands stable, diff-friendly format.
- **Alternatives Considered**: JSON, YAML.
- **Selected Approach**: One line per scoped path: `<path>\t<count>`,
sorted lexicographically by path, single trailing newline.
- **Rationale**: Zero parser dependency; predictable diffs; trivial
to refresh atomically.
- **Trade-offs**: Less expressive than JSON (no nested structure), but
the data model is two integers — nesting is unnecessary.
### Decision: Refresh via `--update-baseline` subcommand-style flag
- **Context**: Requirement 4.3 needs an explicit refresh path.
- **Alternatives Considered**: Separate `update_baseline.py` script;
Makefile target.
- **Selected Approach**: Single script with two modes: default (check
+ exit 0/1) and `--update-baseline` (overwrite baseline + exit 0).
- **Rationale**: One CLI surface to remember; the failure message
prints the exact command to run.
- **Trade-offs**: Slightly more conditional logic in one script;
acceptable given the small total LoC.
### Decision: Workflow runs only on `pull_request` to `main`
- **Context**: Requirement 5.1.
- **Alternatives Considered**: Run on `push` to all branches as well;
run on `pull_request` to any base branch.
- **Selected Approach**: `on.pull_request.branches: [main]` only.
- **Rationale**: Aligns with how the existing project uses `main` as
the protected branch (see `gh pr list` history; every feature PR
targets `main`). Avoids redundant runs on intra-branch chains.
- **Trade-offs**: A direct push to `main` would not be guarded — but
branch protection already discourages that path (per
`dev-guidelines.md`).
## Risks & Mitigations
- **Risk**: Baseline drifts upward unintentionally during
`--update-baseline` runs, hiding real regressions.
- *Mitigation*: Failure message instructs contributors to refresh
*only when intentional*; the baseline file is reviewed in the same
PR diff. Acceptance Criteria 3.3 makes this explicit.
- **Risk**: `git grep -P` not built with PCRE on a developer's local
git build (rare on Linux/macOS, possible on minimal Windows builds).
- *Mitigation*: The guard prints a clear error if `git grep` exits
non-zero with PCRE mode; documents Python ≥3.11 + git ≥2.20 as
prerequisites.
- **Risk**: Baseline counts captured on a feature branch include
changes not yet on `main`, mis-anchoring the ratchet.
- *Mitigation*: The implementation task explicitly recomputes
baseline against `origin/main` before committing; documented in
`tasks.md`.
## References
- PR #27 audit pipeline (`audit_cjk.sh`, `check_parity.py`,
`classify.py`) — methodology source of truth.
- `scripts/check_i18n_logs.py` — CLI/style precedent.
- `git grep` man page — `-n`, `-I`, `-P` flag semantics.
- GitHub Actions `actions/setup-python@v5` and `actions/checkout@v4`
README pages.

24
.kiro/specs/i18n-ci-guard/spec.json Normal file
View File

@ -0,0 +1,24 @@
{
"feature_name": "i18n-ci-guard",
"created_at": "2026-05-08T00:25:37Z",
"updated_at": "2026-05-08T00:40:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true,
"ticket": "26",
"ticket_url": "https://github.com/salestech-group/MiroFish/issues/26"
}

157
.kiro/specs/i18n-ci-guard/tasks.md Normal file
View File

@ -0,0 +1,157 @@
# Implementation Tasks — i18n-ci-guard
> Approved spec: see `requirements.md`, `design.md`, `research.md`,
> `gap-analysis.md` in this directory.
## Tasks
- [x] 1. Foundation: scaffold the CI guard script with stable CLI surface and stdlib-only dependencies
- [x] 1.1 Create the empty guard script and CLI skeleton
- Place the new script at the path designated by the design (`scripts/ci/`).
- Establish the module docstring, the canonical CJK regex constant, the
scoped-paths constant tuple, and the `argparse` parser exposing default
check mode plus an explicit `--update-baseline` flag and a
`--baseline` path override.
- Confirm the script exits 0 on a smoke `--help` invocation and rejects
unknown flags with non-zero exit.
- Observable: running `python scripts/ci/i18n_cjk_guard.py --help` from
the repo root prints usage text containing every documented flag and
exits 0; running with an unknown flag exits non-zero.
- _Requirements: 5.5, 6.2, 6.3_
- _Boundary: i18n_cjk_guard.py_
- [x] 2. Core: implement the two CJK checks
- [x] 2.1 Implement the locale-catalogue scan
- Recursively walk the parsed `locales/en.json` dict, applying the
canonical regex to every string leaf to gather offending entries.
- Compute the source line number by re-reading the file as text and
matching the value's first textual occurrence; truncate snippets to
the documented snippet length.
- On a missing or unreadable catalogue file, emit a clear stderr
message and exit non-zero.
- Observable: against a synthetic clean catalogue, the function returns
an empty list; against a synthetic catalogue with one CJK value, it
returns exactly one finding tuple with the correct dotted key and
line number.
- _Requirements: 1.1, 1.2, 1.3, 1.4, 3.1_
- _Boundary: i18n_cjk_guard.py_
- [x] 2.2 (P) Implement the per-path CJK count via `git grep`
- Invoke `git grep -nIP '[\x{4e00}-\x{9fff}]' -- <scoped_path>` for each
scoped path; treat exit codes 0 (matches found) and 1 (no matches) as
success, any other exit code as a hard error reported on stderr.
- Count lines of stdout; the result for a zero-match path must be the
integer `0`, never an exception.
- Reject working-tree states where `git` is not available or PCRE is
not enabled, with a clear stderr message.
- Observable: against a tmp git repository with N planted CJK lines
under a scoped path, the function returns N; with zero CJK content,
it returns 0; binary files and untracked files do not contribute.
- _Requirements: 2.1, 2.4, 2.5, 2.6_
- _Boundary: i18n_cjk_guard.py_
- [x] 2.3 Implement baseline file read/write with strict format
- Parse the baseline file as `<path>\t<count>` lines, ignoring `#`
comments and blank lines, raising a typed error on malformed input
or missing file.
- Write atomically (`tmp + os.replace`) with sorted entries, a single
header comment block, and a single trailing newline.
- Observable: a round-trip write/read of a deterministic counts dict
yields the same dict; a baseline file containing a non-tab line is
rejected with a clear error; the baseline file ends with exactly one
`\n`.
- _Requirements: 4.2, 4.3_
- _Boundary: i18n_cjk_guard.py_
- [x] 3. Integration: wire the two checks into the default and refresh modes
- [x] 3.1 Compose the default check mode
- Run both checks under all conditions (do not short-circuit), so a
single CI log shows every failure in one pass.
- Print a one-line success summary per check on stdout when both pass.
- On locale failure, print `<file>:<line>: <reason>: <snippet>` lines
on stderr and a trailing `N issues` summary; on regression failure,
print `<path>: cjk-regression: baseline=<b> current=<c> delta=+<d>`
lines plus the exact verbatim refresh command.
- Surface a non-zero exit when either check fails and exit 0 only when
both pass.
- Observable: against a working tree with the committed baseline at or
above the current count and a CJK-clean en.json, exit code is 0 and
stdout contains the success summary; planting one CJK char in
en.json or planting enough new CJK lines to break the baseline
yields exit 1 and the documented stderr text.
- _Requirements: 1.2, 1.3, 1.4, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.4, 4.5_
- _Boundary: i18n_cjk_guard.py_
- [x] 3.2 Compose the `--update-baseline` mode
- When the flag is provided, recompute current per-path counts and
overwrite the baseline file via the atomic writer; print the new
counts on stdout; exit 0.
- When the flag is absent, never write the baseline file under any
code path.
- Observable: invoking with `--update-baseline` rewrites the baseline
file's contents to match current counts and exits 0; running the
default mode immediately afterward exits 0.
- _Requirements: 4.3, 4.4_
- _Boundary: i18n_cjk_guard.py_
- [x] 4. Establish the committed baseline anchored to `main`
- [x] 4.1 Capture initial baseline counts against `main`
- Operate from a tree that reflects `origin/main`'s state for the
scoped paths (e.g., a fresh checkout, a worktree at `origin/main`,
or `git checkout origin/main -- backend/app frontend/src` followed
by a clean revert) so the committed baseline does not over- or
under-count relative to the merge target.
- Run `--update-baseline` to materialize the counts; confirm the
resulting file is exactly two non-comment data lines (one per
scoped path) sorted lexicographically.
- Observable: the baseline file is committed to
`.kiro/specs/i18n-ci-guard/baseline.txt` and `python scripts/ci/i18n_cjk_guard.py`
against the same `main`-aligned tree exits 0.
- _Requirements: 4.1, 4.2_
- _Boundary: baseline.txt_
- [x] 5. Wire the guard into GitHub Actions on every PR to `main`
- [x] 5.1 Add the PR-time workflow
- Create the workflow file at the path designated by the design,
triggered on `pull_request` whose base ref is `main`.
- Set explicit minimal permissions (`contents: read`), a one-minute
job timeout, `actions/checkout@v4` with `fetch-depth: 1`, and
`actions/setup-python@v5` pinned to Python 3.11.
- The single executable step invokes the guard script with no
arguments; the workflow surfaces the script's stdout and stderr in
the GitHub Actions log without filtering.
- Observable: the workflow YAML parses cleanly; on a PR with no CJK
regression, the job passes; on a PR that introduces a CJK regression
or CJK in en.json, the job fails and the log shows the documented
failure messages.
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6_
- _Boundary: i18n-cjk-guard.yml_
- [x] 6. Validation: tests and end-to-end checks
- [x] 6.1 Add unit and integration tests for the guard script
- Cover the locale scan against a synthetic clean catalogue and a
synthetic CJK-tainted catalogue, asserting findings tuples match.
- Cover the per-path counter against a tmp git repo with both N>0
and N=0 planted CJK lines, asserting the zero-match path exits
cleanly with a count of 0.
- Cover the baseline read/write round-trip and the malformed-input
rejection path.
- Cover the default mode end-to-end (pass and fail paths) with the
expected exit codes and stderr fragments, including the verbatim
refresh command on regression failure.
- Observable: `python -m pytest scripts/ci/tests/test_i18n_cjk_guard.py`
from the repo root passes locally with stdlib-only Python.
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.4, 2.5, 2.6, 3.3, 4.3, 4.5, 6.1, 6.3_
- _Boundary: scripts/ci/tests/_
- [x] 6.2 Run the guard locally to confirm reproducibility against the committed baseline
- From a clean working tree at `main` (or a worktree at `origin/main`
+ this branch's new files merged on top), invoke the guard with no
arguments and confirm exit code 0 and the success summary.
- Confirm the same command is the documented developer entry point
referenced from the failure-message refresh hint.
- Observable: terminal session shows exit code 0 and the documented
one-line per-check success summary; the same script path (`scripts/ci/i18n_cjk_guard.py`)
appears verbatim in the regression-failure refresh hint.
- _Requirements: 6.1, 6.2, 6.3_
- _Boundary: i18n_cjk_guard.py, baseline.txt_

2924
.kiro/specs/i18n-e2e-english-verification/audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/cjk-grep-bucketed.txt Normal file
View File

File diff suppressed because it is too large Load Diff

2916
.kiro/specs/i18n-e2e-english-verification/audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/cjk-grep.txt Normal file
View File

File diff suppressed because it is too large Load Diff

2917
.kiro/specs/i18n-e2e-english-verification/audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/classified.csv Normal file
View File

File diff suppressed because it is too large Load Diff

60
.kiro/specs/i18n-e2e-english-verification/audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/comment-body.md Normal file
View File

@ -0,0 +1,60 @@
### Verification report - run on commit `9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd`
This run was produced by `.kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh`.
Captured artefacts live under `.kiro/specs/i18n-e2e-english-verification/audit/<commit-sha>/`.
**Audit summary:** 2916 CJK matches across the auditable paths.
- 237 `gap` (actionable, see follow-ups)
- 380 `review-needed` (soft signal; needs human eyeball)
- 2299 `deliberate` (mostly backend docstrings/comments - covered by issue #7)
- 0 `non-applicable` (binary file false positives - excluded)
**Gap-category breakdown:** backend-prompt-label=143, frontend-ui-string=49, frontend-regex-parser=36, backend-log=9
---
#### Issue checklist mapping
## Section 5 - Issue #10 checklist mapping
Each line below is taken from the ticket body, with an explicit status.
- [ ] **GAP** - **Frontend UI** — every label, button, modal, error toast, and tooltip in EN. No Chinese strings on screen. - 29 hard-coded CJK literal(s) in `frontend/src/views|components/`
- [ ] **GAP** - **Step 1 — Graph Build** - 5 gap(s) classified, see Section 1/3
- MANUAL-PENDING: Status messages in EN - not verifiable statically; awaiting live run
- GAP: Ontology JSON descriptions in EN (depends on #2) - 14 gap(s) classified, see Section 1/3
- GAP: Backend logs in EN (depends on #6) - 9 gap(s) classified, see Section 1/3
- [ ] **GAP** - **Step 2 — Env Setup** - 61 gap(s) classified, see Section 1/3
- GAP: Generated agent profiles (`bio`, `persona`, `profession`, `interested_topics`) in EN (depends on #3) - 61 gap(s) classified, see Section 1/3
- MANUAL-PENDING: `gender` still the English enum (`male` / `female` / `other`) - not verifiable statically; awaiting live run
- [ ] **GAP** - **Step 3 — Simulation** - 14 gap(s) classified, see Section 1/3
- GAP: Sim config `content`, `narrative_direction`, `hot_topics`, `reasoning` in EN (depends on #4) - 14 gap(s) classified, see Section 1/3
- MANUAL-PENDING: `poster_type` still PascalCase English - not verifiable statically; awaiting live run
- MANUAL-PENDING: `stance` still one of `supportive` / `opposing` / `neutral` / `observer` - not verifiable statically; awaiting live run
- GAP: Generated tweets / Reddit posts in EN (depends on #3 personas + #4 sim config) - 14 gap(s) classified, see Section 1/3
- [ ] **GAP** - **Step 4 — Report** - 70 gap(s) classified, see Section 1/3
- GAP: Report sections, headings, prose in EN (depends on #5) - 70 gap(s) classified, see Section 1/3
- MANUAL-PENDING: ReACT thinking trace in EN - requires live walkthrough
- MANUAL-PENDING: Tool-call results render correctly - requires live walkthrough
- [ ] **GAP** - **Step 5 — Interaction** - 2 gap(s) classified, see Section 1/3
- GAP: Interview chat replies in EN (depends on #3) - 2 gap(s) classified, see Section 1/3
- GAP: Report Agent chat replies in EN (depends on #5) - 72 gap(s) classified, see Section 1/3
- [ ] **GAP** - **Backend logs** — full pipeline-run logs in EN (depends on #6) - 9 gap(s) classified, see Section 1/3
- [ ] **GAP** - **Locale propagation** — confirm `Accept-Language: en` (or thread-local locale set via `set_locale`) reaches background tasks and survives the OASIS subprocess boundary. - 9 CJK log strings on EN code path
- [ ] **MANUAL-PENDING** - Every touchpoint above renders in Chinese; no English regressions. - requires live walkthrough
- [ ] **MANUAL-PENDING** - zh.json backfill (#8) covered: Step 3, Step 4, Step 5, and graph panel labels are all Chinese. - not verifiable statically; awaiting live run
---
#### How to re-run
```bash
# from the repository root, on any commit:
bash .kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh
# artefacts at .kiro/specs/i18n-e2e-english-verification/audit/<HEAD-sha>/
```
If `gh` is not authenticated when re-running, the comment body and follow-up bodies are written to `PENDING-issue-10-comment.md` / `PENDING-followups/` for a human to post.
Out of scope for this run (per R5.3 / R7.3): live UI walkthrough, full Docker-Compose pipeline run, and any inline gap fixes.

1
.kiro/specs/i18n-e2e-english-verification/audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/comment-url.txt Normal file
View File

@ -0,0 +1 @@
https://github.com/salestech-group/MiroFish/issues/10#issuecomment-4400060417

4
.kiro/specs/i18n-e2e-english-verification/audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/followup-urls.txt Normal file
View File

@ -0,0 +1,4 @@
https://github.com/salestech-group/MiroFish/issues/23
https://github.com/salestech-group/MiroFish/issues/24
https://github.com/salestech-group/MiroFish/issues/25
https://github.com/salestech-group/MiroFish/issues/26

143
.kiro/specs/i18n-e2e-english-verification/audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/gap-report.md Normal file
View File

@ -0,0 +1,143 @@
# Verification gap report - i18n-e2e-english-verification
**Commit:** `9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd`
## Overview
- Total CJK matches audited: **2916**
- Class distribution: deliberate=2299, review-needed=380, gap=237
- Gap categories: backend-prompt-label=143, frontend-ui-string=49, frontend-regex-parser=36, backend-log=9
- Gap pipeline steps: Report=70, Env Setup=61, n/a=47, UI=29, Simulation=14, Logs=9, Graph Build=5, Interaction=2
## Section 1 - Static CJK audit
Canonical command (PCRE):
```
git grep -nIP "[\x{4e00}-\x{9fff}]" -- backend/app frontend/src locales/en.json
```
Raw output captured at `audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/cjk-grep.txt` and bucketed at `audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/cjk-grep-bucketed.txt`.
`locales/en.json` CJK matches: **0** (acceptance: zero).
Top files by gap count:
| File | Gap count |
|------|-----------|
| `backend/app/services/oasis_profile_generator.py` | 60 |
| `frontend/src/components/Step4Report.vue` | 50 |
| `backend/app/services/zep_graph_memory_updater.py` | 47 |
| `frontend/src/views/Process.vue` | 29 |
| `backend/app/services/report_agent.py` | 20 |
| `backend/app/services/simulation_config_generator.py` | 13 |
| `backend/app/services/ontology_generator.py` | 5 |
| `backend/app/utils/retry.py` | 4 |
| `backend/app/api/graph.py` | 3 |
| `frontend/src/components/Step2EnvSetup.vue` | 3 |
| `frontend/src/components/Step5Interaction.vue` | 2 |
| `frontend/src/components/Step3Simulation.vue` | 1 |
## Section 2 - Locale catalogue parity
```
# Locale parity for HEAD
# en keys: 953
# zh keys: 953
[missing-keys]
# (none)
[cjk-in-en]
# (none)
[identical-values]
# (none)
```
## Section 3 - LLM-prompt locale verification
Backend prompt-label gaps (CJK string literals inside services that compose LLM prompts): **143**
First 10 examples (file:line - match):
- `backend/app/services/oasis_profile_generator.py:65` - "username": self.user_name, # OASIS 库要求字段名为 username（无下划线）
- `backend/app/services/oasis_profile_generator.py:93` - "username": self.user_name, # OASIS 库要求字段名为 username（无下划线）
- `backend/app/services/oasis_profile_generator.py:194` - raise ValueError("LLM_API_KEY 未配置")
- `backend/app/services/oasis_profile_generator.py:384` - all_summaries.add(f"相关实体: {node.name}")
- `backend/app/services/oasis_profile_generator.py:390` - context_parts.append("事实信息:\n" + "\n".join(f"- {f}" for f in results["facts"][:20]))
- `backend/app/services/oasis_profile_generator.py:392` - context_parts.append("相关实体:\n" + "\n".join(f"- {s}" for s in results["node_summaries"][:10]))
- `backend/app/services/oasis_profile_generator.py:422` - context_parts.append("### 实体属性\n" + "\n".join(attrs))
- `backend/app/services/oasis_profile_generator.py:438` - relationships.append(f"- {entity.name} --[{edge_name}]--> (相关实体)")
- `backend/app/services/oasis_profile_generator.py:440` - relationships.append(f"- (相关实体) --[{edge_name}]--> {entity.name}")
- `backend/app/services/oasis_profile_generator.py:443` - context_parts.append("### 相关事实和关系\n" + "\n".join(relationships))
- ... and 133 more (see `classified.csv`)
These prompts feed the LLM verbatim; CJK labels bias the model toward Chinese output even when the requested locale is English.
## Section 4 - Locale propagation surface
| Boundary | Status | Evidence |
|----------|--------|----------|
| HTTP -> Flask handler | manual-pending | runtime not exercised in sandbox; static review showed no per-request locale carrier |
| Flask handler -> Task worker | manual-pending | thread-local `set_locale` referenced in CLAUDE.md but not statically verified end-to-end |
| Task worker -> OASIS subprocess | manual-pending | subprocess boundary requires live run |
| Backend logger | gap | 9 hard-coded CJK log line(s) on EN code path |
First 10 backend-log gap examples:
- `backend/app/api/graph.py:385` - build_logger.info(f"[{task_id}] 开始构建图谱...")
- `backend/app/api/graph.py:494` - build_logger.info(f"[{task_id}] 图谱构建完成: graph_id={graph_id}, 节点={node_count}, 边={edge_count}")
- `backend/app/api/graph.py:513` - build_logger.error(f"[{task_id}] 图谱构建失败: {str(e)}")
- `backend/app/services/oasis_profile_generator.py:945` - print(f"开始生成Agent人设 - 共 {total} 个实体，并行数: {parallel_count}")
- `backend/app/services/oasis_profile_generator.py:1001` - print(f"人设生成完成！共生成 {len([p for p in profiles if p])} 个Agent")
- `backend/app/utils/retry.py:55` - logger.error(f"函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}")
- `backend/app/utils/retry.py:108` - logger.error(f"异步函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}")
- `backend/app/utils/retry.py:179` - logger.error(f"API调用在 {self.max_retries} 次重试后仍失败: {str(e)}")
- `backend/app/utils/retry.py:227` - logger.error(f"处理第 {idx + 1} 项失败: {str(e)}")
## Section 5 - Issue #10 checklist mapping
Each line below is taken from the ticket body, with an explicit status.
- [ ] **GAP** - **Frontend UI** — every label, button, modal, error toast, and tooltip in EN. No Chinese strings on screen. - 29 hard-coded CJK literal(s) in `frontend/src/views|components/`
- [ ] **GAP** - **Step 1 — Graph Build** - 5 gap(s) classified, see Section 1/3
- MANUAL-PENDING: Status messages in EN - not verifiable statically; awaiting live run
- GAP: Ontology JSON descriptions in EN (depends on #2) - 14 gap(s) classified, see Section 1/3
- GAP: Backend logs in EN (depends on #6) - 9 gap(s) classified, see Section 1/3
- [ ] **GAP** - **Step 2 — Env Setup** - 61 gap(s) classified, see Section 1/3
- GAP: Generated agent profiles (`bio`, `persona`, `profession`, `interested_topics`) in EN (depends on #3) - 61 gap(s) classified, see Section 1/3
- MANUAL-PENDING: `gender` still the English enum (`male` / `female` / `other`) - not verifiable statically; awaiting live run
- [ ] **GAP** - **Step 3 — Simulation** - 14 gap(s) classified, see Section 1/3
- GAP: Sim config `content`, `narrative_direction`, `hot_topics`, `reasoning` in EN (depends on #4) - 14 gap(s) classified, see Section 1/3
- MANUAL-PENDING: `poster_type` still PascalCase English - not verifiable statically; awaiting live run
- MANUAL-PENDING: `stance` still one of `supportive` / `opposing` / `neutral` / `observer` - not verifiable statically; awaiting live run
- GAP: Generated tweets / Reddit posts in EN (depends on #3 personas + #4 sim config) - 14 gap(s) classified, see Section 1/3
- [ ] **GAP** - **Step 4 — Report** - 70 gap(s) classified, see Section 1/3
- GAP: Report sections, headings, prose in EN (depends on #5) - 70 gap(s) classified, see Section 1/3
- MANUAL-PENDING: ReACT thinking trace in EN - requires live walkthrough
- MANUAL-PENDING: Tool-call results render correctly - requires live walkthrough
- [ ] **GAP** - **Step 5 — Interaction** - 2 gap(s) classified, see Section 1/3
- GAP: Interview chat replies in EN (depends on #3) - 2 gap(s) classified, see Section 1/3
- GAP: Report Agent chat replies in EN (depends on #5) - 72 gap(s) classified, see Section 1/3
- [ ] **GAP** - **Backend logs** — full pipeline-run logs in EN (depends on #6) - 9 gap(s) classified, see Section 1/3
- [ ] **GAP** - **Locale propagation** — confirm `Accept-Language: en` (or thread-local locale set via `set_locale`) reaches background tasks and survives the OASIS subprocess boundary. - 9 CJK log strings on EN code path
- [ ] **MANUAL-PENDING** - Every touchpoint above renders in Chinese; no English regressions. - requires live walkthrough
- [ ] **MANUAL-PENDING** - zh.json backfill (#8) covered: Step 3, Step 4, Step 5, and graph panel labels are all Chinese. - not verifiable statically; awaiting live run
## Section 6 - ZH regression check
- Locale catalogues at full key parity (953 EN keys / 953 ZH keys, symmetric difference 0 - see Section 2).
- No ZH-specific regression detected in static review. Live ZH walkthrough is `manual-pending`.
## Section 7 - Follow-up plan
Per R7.2, gaps are grouped into the following follow-up issues (placeholder bodies in `PENDING-followups/`):
1. **Frontend hard-coded UI strings** (49 matches + 36 regex parsers depending on CJK backend output).
2. **Backend log strings** (9 matches).
3. **Backend LLM-prompt context labels** (143 matches).
4. **Permanent CI guard** (preventative - re-run this audit on every PR).
Backend docstring/comment matches (the bulk of `deliberate` rows) are covered by the existing issue #7 and are not re-filed here.

13
.kiro/specs/i18n-e2e-english-verification/audit/9dcaecd2d27e6325bae0c53b9ab41eb86d0269cd/parity.txt Normal file
View File

@ -0,0 +1,13 @@
# Locale parity for HEAD
# en keys: 953
# zh keys: 953
[missing-keys]
# (none)
[cjk-in-en]
# (none)
[identical-values]
# (none)

62
.kiro/specs/i18n-e2e-english-verification/audit/scripts/audit_cjk.sh Executable file
View File

@ -0,0 +1,62 @@
#!/usr/bin/env bash
# Run the canonical CJK grep with PCRE, then write the raw output and a
# bucketed summary partitioned by top-level path. Excludes binary file
# matches (e.g. .jpeg) since ripgrep / git grep can otherwise score them.
set -euo pipefail
if [ "$#" -ne 1 ]; then
printf 'usage: %s <sha-dir>\n' "$0" >&2
exit 64
fi
sha_dir="$1"
mkdir -p "${sha_dir}"
raw="${sha_dir}/cjk-grep.txt"
bucketed="${sha_dir}/cjk-grep-bucketed.txt"
# Canonical PCRE grep against the three top-level paths owned by this audit.
# git grep -P uses PCRE2 - ranges like \x{4e00}-\x{9fff} are valid here.
# `-I` (--no-binary) excludes binary-file matches outright so the audit
# reports only text content.
git grep -nIP '[\x{4e00}-\x{9fff}]' \
-- backend/app frontend/src locales/en.json \
> "${raw}" \
|| true
awk_script='
function bucket(path) {
if (path ~ /^backend\/app\//) return "backend/app"
if (path ~ /^frontend\/src\//) return "frontend/src"
if (path ~ /^locales\/en\.json/) return "locales/en.json"
return "other"
}
{
split($0, parts, ":")
path = parts[1]
b = bucket(path)
counts[b]++
lines[b] = (b in lines ? lines[b] "\n" : "") $0
}
END {
order[1] = "backend/app"
order[2] = "frontend/src"
order[3] = "locales/en.json"
order[4] = "other"
for (i = 1; i <= 4; i++) {
b = order[i]
c = (b in counts ? counts[b] : 0)
printf("[%s] (%d lines)\n", b, c)
if (c > 0) {
print lines[b]
}
print ""
}
}
'
awk "${awk_script}" "${raw}" > "${bucketed}"
raw_lines=$(wc -l < "${raw}" | tr -d ' ')
printf ' cjk-grep.txt: %s lines\n' "${raw_lines}"
printf ' cjk-grep-bucketed.txt: written\n'

128
.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py Executable file
View File

@ -0,0 +1,128 @@
#!/usr/bin/env python3
"""Diff locales/en.json against locales/zh.json and emit parity.txt.
Three labelled blocks are written:
* `[missing-keys]` - keys present on one side but not the other.
* `[cjk-in-en]` - EN catalogue values that contain CJK characters.
* `[identical-values]` - keys whose EN and ZH value are identical AND the
value is non-empty AND has more than two ASCII words.
These are review-needed signals, not gaps.
Run from the repository root.
"""
from __future__ import annotations
import json
import re
import sys
from pathlib import Path
from typing import Dict, Iterator, Tuple
CJK_RANGE = re.compile(r"[一-鿿]")
def flatten(d: Dict[str, object], prefix: str = "") -> Iterator[Tuple[str, object]]:
"""Recursively yield (dotted-key, value) pairs from a nested dict."""
for key, value in d.items():
path = f"{prefix}.{key}" if prefix else key
if isinstance(value, dict):
yield from flatten(value, path)
else:
yield path, value
def is_non_trivial_english_prose(value: object) -> bool:
"""Heuristic for the identical-value 'review-needed' signal.
True when:
* value is a string,
* value is non-empty after strip,
* value contains more than two whitespace-separated tokens,
* value contains no CJK characters (otherwise it's just an untranslated
ZH original which is not a review-needed signal here).
"""
if not isinstance(value, str):
return False
text = value.strip()
if not text:
return False
if CJK_RANGE.search(text):
return False
return len(text.split()) > 2
def main(argv: list[str]) -> int:
if len(argv) != 2:
print(f"usage: {argv[0]} <sha-dir>", file=sys.stderr)
return 64
sha_dir = Path(argv[1])
sha_dir.mkdir(parents=True, exist_ok=True)
out_path = sha_dir / "parity.txt"
en_path = Path("locales/en.json")
zh_path = Path("locales/zh.json")
if not en_path.exists() or not zh_path.exists():
print(f"missing locale files: {en_path}, {zh_path}", file=sys.stderr)
return 1
en = json.loads(en_path.read_text(encoding="utf-8"))
zh = json.loads(zh_path.read_text(encoding="utf-8"))
en_flat = dict(flatten(en))
zh_flat = dict(flatten(zh))
en_only = sorted(set(en_flat) - set(zh_flat))
zh_only = sorted(set(zh_flat) - set(en_flat))
cjk_in_en = []
for key, value in sorted(en_flat.items()):
if isinstance(value, str) and CJK_RANGE.search(value):
cjk_in_en.append((key, value))
identical = []
for key in sorted(set(en_flat) & set(zh_flat)):
en_val = en_flat[key]
zh_val = zh_flat[key]
if en_val == zh_val and is_non_trivial_english_prose(en_val):
identical.append((key, en_val))
lines: list[str] = []
lines.append(f"# Locale parity for HEAD")
lines.append(f"# en keys: {len(en_flat)}")
lines.append(f"# zh keys: {len(zh_flat)}")
lines.append("")
lines.append("[missing-keys]")
if not en_only and not zh_only:
lines.append("# (none)")
for key in en_only:
lines.append(f"en-only: {key}")
for key in zh_only:
lines.append(f"zh-only: {key}")
lines.append("")
lines.append("[cjk-in-en]")
if not cjk_in_en:
lines.append("# (none)")
for key, value in cjk_in_en:
snippet = value if len(value) <= 80 else value[:77] + "..."
lines.append(f"{key}: {snippet}")
lines.append("")
lines.append("[identical-values]")
if not identical:
lines.append("# (none)")
for key, value in identical:
snippet = value if len(value) <= 80 else value[:77] + "..."
lines.append(f"{key}: {snippet}")
lines.append("")
out_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
print(
f" parity.txt written: missing={len(en_only) + len(zh_only)}, "
f"cjk-in-en={len(cjk_in_en)}, identical-values={len(identical)}"
)
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))

182
.kiro/specs/i18n-e2e-english-verification/audit/scripts/classify.py Executable file
View File

@ -0,0 +1,182 @@
#!/usr/bin/env python3
"""Classify each CJK match into a 4-class label and a category tag.
Inputs (read from <sha-dir>):
cjk-grep.txt - raw `git grep -nP` output, one match per line.
parity.txt - output of check_parity.py (used to harvest cjk-in-en gaps).
Output (written to <sha-dir>/classified.csv):
CSV columns: file, line, match, class, category, pipeline_step
Classes are a closed set: deliberate / gap / non-applicable / review-needed.
Categories and pipeline-step tags are likewise closed sets - see classify_match.
Run from the repository root.
"""
from __future__ import annotations
import csv
import re
import sys
from pathlib import Path
from typing import Iterable, Tuple
CJK_RANGE = re.compile(r"[一-鿿]")
PROMPT_FILES = (
"backend/app/services/ontology_generator.py",
"backend/app/services/oasis_profile_generator.py",
"backend/app/services/simulation_config_generator.py",
"backend/app/services/report_agent.py",
"backend/app/services/zep_graph_memory_updater.py",
)
LOG_HINTS = ("logger.", "log.", "print(", "build_logger.", "logging.")
BINARY_EXTS = (
".jpg", ".jpeg", ".png", ".gif", ".pdf",
".woff", ".woff2", ".ttf", ".eot", ".ico",
)
def classify_match(file: str, raw_line: str) -> Tuple[str, str, str]:
"""Return (class, category, pipeline_step) for one grep match line."""
if any(file.lower().endswith(ext) for ext in BINARY_EXTS):
return ("non-applicable", "binary-false-positive", "n/a")
if file == "locales/en.json":
return ("gap", "catalogue-parity", "UI")
stripped = raw_line.lstrip()
pipeline_step = pipeline_step_for(file)
if file.endswith(".vue"):
if re.search(r"\.match\s*\(\s*/", raw_line):
return ("gap", "frontend-regex-parser", pipeline_step)
if re.search(r"['\"`].*[一-鿿].*['\"`]", raw_line):
return ("gap", "frontend-ui-string", pipeline_step)
if stripped.startswith("//") or stripped.startswith("/*") or stripped.startswith("*"):
return ("deliberate", "frontend-comment", pipeline_step)
return ("review-needed", "frontend-other", pipeline_step)
if file.endswith(".py"):
if stripped.startswith("#"):
return ("deliberate", "backend-comment", pipeline_step)
if stripped.startswith('"""') or stripped.startswith("'''"):
return ("deliberate", "backend-docstring", pipeline_step)
if not re.search(r"['\"]", raw_line):
# bare CJK on a non-string line: most likely an unterminated docstring
# body. Treat as a docstring continuation.
return ("deliberate", "backend-docstring", pipeline_step)
if any(hint in raw_line for hint in LOG_HINTS):
return ("gap", "backend-log", "Logs")
if file in PROMPT_FILES:
return ("gap", "backend-prompt-label", pipeline_step)
return ("review-needed", "backend-string", pipeline_step)
if file.endswith(".js") or file.endswith(".ts"):
if stripped.startswith("//") or stripped.startswith("*"):
return ("deliberate", "frontend-comment", pipeline_step)
return ("review-needed", "frontend-other", pipeline_step)
return ("review-needed", "uncategorised", pipeline_step)
def pipeline_step_for(file: str) -> str:
"""Map a path to one of the closed-set pipeline-step tags."""
if "ontology_generator" in file or "graph_builder" in file or "graph.py" in file:
return "Graph Build"
if "oasis_profile_generator" in file or "Step2" in file:
return "Env Setup"
if "simulation_config_generator" in file or "simulation" in file or "Step3" in file:
return "Simulation"
if "report_agent" in file or "Step4" in file:
return "Report"
if "Step5" in file or "interaction" in file.lower() or "interview" in file.lower():
return "Interaction"
if "logger" in file or "retry" in file:
return "Logs"
if file.startswith("frontend/src/views/") or file.startswith("frontend/src/components/"):
return "UI"
return "n/a"
def parse_grep_line(line: str) -> Tuple[str, str, str]:
"""Split a `git grep -n` line into (file, line-number, match-text)."""
parts = line.split(":", 2)
if len(parts) < 3:
return ("", "", line)
return (parts[0], parts[1], parts[2])
def parity_to_rows(parity_path: Path) -> Iterable[Tuple[str, str, str, str, str, str]]:
"""Promote `[cjk-in-en]` block entries from parity.txt into classified rows."""
if not parity_path.exists():
return
in_block = False
for raw in parity_path.read_text(encoding="utf-8").splitlines():
if raw.startswith("["):
in_block = raw.strip() == "[cjk-in-en]"
continue
if not in_block:
continue
if not raw or raw.startswith("#"):
continue
yield (
"locales/en.json",
"0",
raw,
"gap",
"catalogue-parity",
"UI",
)
def main(argv: list[str]) -> int:
if len(argv) != 2:
print(f"usage: {argv[0]} <sha-dir>", file=sys.stderr)
return 64
sha_dir = Path(argv[1])
grep_path = sha_dir / "cjk-grep.txt"
parity_path = sha_dir / "parity.txt"
out_path = sha_dir / "classified.csv"
if not grep_path.exists():
print(f"missing input: {grep_path}", file=sys.stderr)
return 1
rows: list[Tuple[str, str, str, str, str, str]] = []
grep_lines = grep_path.read_text(encoding="utf-8").splitlines()
for raw_line in grep_lines:
if not raw_line:
continue
file, lineno, match = parse_grep_line(raw_line)
if not file:
continue
cls, category, step = classify_match(file, match)
rows.append((file, lineno, match.strip(), cls, category, step))
rows.extend(parity_to_rows(parity_path))
raw_count = sum(1 for line in grep_lines if line.strip())
grep_rows = [r for r in rows if r[0] != "locales/en.json" or r[1] != "0"]
if len(grep_rows) != raw_count:
print(
f"row-count drift: input={raw_count}, classified={len(grep_rows)}",
file=sys.stderr,
)
return 1
with out_path.open("w", encoding="utf-8", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(["file", "line", "match", "class", "category", "pipeline_step"])
writer.writerows(rows)
summary: dict[str, int] = {}
for row in rows:
summary[row[3]] = summary.get(row[3], 0) + 1
summary_str = ", ".join(f"{cls}={n}" for cls, n in sorted(summary.items()))
print(f" classified.csv: {len(rows)} rows ({summary_str})")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))

79
.kiro/specs/i18n-e2e-english-verification/audit/scripts/file_followups.sh Executable file
View File

@ -0,0 +1,79 @@
#!/usr/bin/env bash
# Iterate <sha-dir>/PENDING-followups/*.md and file each non-empty body
# as a GitHub issue. The first markdown heading line (`# title`) becomes
# the issue title; any `<!-- labels: a,b,c -->` line at the bottom of the
# body becomes the --label argument.
#
# On per-category failure the body is left in place and the script exits
# non-zero at the end (after attempting all categories).
set -uo pipefail
if [ "$#" -ne 1 ]; then
printf 'usage: %s <sha-dir>\n' "$0" >&2
exit 64
fi
sha_dir="$1"
pending_dir="${sha_dir}/PENDING-followups"
urls_path="${sha_dir}/followup-urls.txt"
if [ ! -d "${pending_dir}" ]; then
printf 'missing PENDING-followups dir: %s\n' "${pending_dir}" >&2
exit 1
fi
# Append-only URL log so retries on the same sha-dir preserve previous filings.
touch "${urls_path}"
if ! command -v gh >/dev/null 2>&1; then
printf ' gh not available; leaving all bodies in PENDING-followups/\n'
exit 2
fi
if ! gh auth status >/dev/null 2>&1; then
printf ' gh not authenticated; leaving all bodies in PENDING-followups/\n'
exit 2
fi
partial=0
for body in "${pending_dir}"/[0-9]*-*.md; do
[ -f "${body}" ] || continue
if [ ! -s "${body}" ]; then
# Empty placeholder - the corresponding category had zero gaps in this run.
continue
fi
title="$(awk 'NR==1 && /^# /{sub(/^# /, ""); print; exit}' "${body}")"
if [ -z "${title}" ]; then
title="i18n: follow-up from issue #10 verification ($(basename "${body}" .md))"
fi
label_line="$(grep -oE '<!-- labels: [^>]+-->' "${body}" | head -1 || true)"
labels="$(printf '%s' "${label_line}" | sed -E 's/<!-- labels: //; s/ *-->//' || true)"
label_args=()
if [ -n "${labels}" ]; then
IFS=',' read -ra parts <<< "${labels}"
for label in "${parts[@]}"; do
label_args+=( --label "$(echo "${label}" | tr -d ' ')" )
done
fi
printf ' filing: %s\n' "${title}"
if url="$(gh issue create --repo salestech-group/MiroFish \
--title "${title}" \
--body-file "${body}" \
"${label_args[@]}" 2>&1)"; then
printf '%s\n' "${url}" >> "${urls_path}"
printf ' -> %s\n' "${url}"
rm -f "${body}"
else
printf ' !! gh issue create failed: %s\n' "${url}" >&2
partial=1
fi
done
if [ "${partial}" -eq 1 ]; then
exit 2
fi
exit 0

42
.kiro/specs/i18n-e2e-english-verification/audit/scripts/post_comment.sh Executable file
View File

@ -0,0 +1,42 @@
#!/usr/bin/env bash
# Post comment-body.md as a comment on issue #10.
#
# Falls back to writing PENDING-issue-10-comment.md when gh is unavailable
# or the post fails - exits non-zero in that case so the orchestrator can
# downgrade its overall status.
set -euo pipefail
if [ "$#" -ne 1 ]; then
printf 'usage: %s <sha-dir>\n' "$0" >&2
exit 64
fi
sha_dir="$1"
body="${sha_dir}/comment-body.md"
if [ ! -f "${body}" ]; then
printf 'missing comment body: %s\n' "${body}" >&2
exit 1
fi
if ! command -v gh >/dev/null 2>&1; then
printf ' gh not available; writing PENDING-issue-10-comment.md\n'
cp "${body}" "${sha_dir}/PENDING-issue-10-comment.md"
exit 2
fi
if ! gh auth status >/dev/null 2>&1; then
printf ' gh not authenticated; writing PENDING-issue-10-comment.md\n'
cp "${body}" "${sha_dir}/PENDING-issue-10-comment.md"
exit 2
fi
if url="$(gh issue comment 10 --repo salestech-group/MiroFish --body-file "${body}" 2>&1)"; then
printf '%s\n' "${url}" > "${sha_dir}/comment-url.txt"
printf ' posted: %s\n' "${url}"
rm -f "${sha_dir}/PENDING-issue-10-comment.md"
exit 0
fi
printf ' gh post failed; writing PENDING-issue-10-comment.md\n'
cp "${body}" "${sha_dir}/PENDING-issue-10-comment.md"
exit 2

419
.kiro/specs/i18n-e2e-english-verification/audit/scripts/render_report.py Executable file
View File

@ -0,0 +1,419 @@
#!/usr/bin/env python3
"""Render the gap report and the issue-#10 comment body.
Inputs (from <sha-dir>):
classified.csv - per-match classification rows.
parity.txt - en/zh catalogue parity output.
cjk-grep-bucketed.txt - human-readable bucketed grep output.
Inputs (from repo):
.ticket/10.md - snapshot of issue #10's body (used to mirror its checklist).
Outputs (to <sha-dir>):
gap-report.md - full structured report (seven sections).
comment-body.md - markdown comment to be posted on issue #10.
PENDING-followups/01..04-*.md - one body per gap category (placeholders allowed).
Usage:
python3 render_report.py <sha-dir> <commit-sha>
"""
from __future__ import annotations
import csv
import re
import sys
from collections import Counter, defaultdict
from pathlib import Path
from typing import Dict, List
ISSUE_NUMBER = 10
REPO_SLUG = "salestech-group/MiroFish"
def load_rows(csv_path: Path) -> list[dict]:
with csv_path.open(encoding="utf-8", newline="") as fh:
return list(csv.DictReader(fh))
def load_ticket_body(ticket_path: Path) -> str:
"""Strip the YAML frontmatter and return the markdown body."""
text = ticket_path.read_text(encoding="utf-8")
if text.startswith("---\n"):
end = text.find("\n---\n", 4)
if end != -1:
return text[end + 5 :]
return text
CHECKBOX_RE = re.compile(r"^(\s*)- \[ \] (.+)$")
SUBBULLET_RE = re.compile(r"^(\s+)- (.+)$")
def evidence_for_step(rows: list[dict], step: str) -> list[dict]:
"""Return gap rows whose pipeline_step matches the given UI tag."""
return [r for r in rows if r["class"] == "gap" and r["pipeline_step"] == step]
def render_section_5(ticket_body: str, rows: list[dict]) -> str:
"""Map every checklist item from the ticket body to a status."""
gaps_by_step = defaultdict(list)
for row in rows:
if row["class"] == "gap":
gaps_by_step[row["pipeline_step"]].append(row)
out: list[str] = []
out.append("## Section 5 - Issue #10 checklist mapping\n")
out.append("Each line below is taken from the ticket body, with an explicit status.\n")
in_checklist = False
for line in ticket_body.splitlines():
match = CHECKBOX_RE.match(line)
if match:
in_checklist = True
indent, text = match.group(1), match.group(2)
status, note = status_for_checklist_item(text, gaps_by_step)
out.append(f"{indent}- [{('x' if status == 'pass' else ' ')}] **{status.upper()}** - {text}{note}")
continue
sub = SUBBULLET_RE.match(line)
if in_checklist and sub:
indent, text = sub.group(1), sub.group(2)
status, note = status_for_checklist_item(text, gaps_by_step)
out.append(f"{indent}- {status.upper()}: {text}{note}")
continue
if line.startswith("##") or line.startswith("---"):
in_checklist = False
return "\n".join(out) + "\n"
def status_for_checklist_item(text: str, gaps_by_step: Dict[str, list]) -> tuple[str, str]:
"""Return (status, suffix-note) for one checklist line.
Pure-UI items default to manual-pending in this run; items with a
backing pipeline-step that has gaps are reported as gap with a count.
"""
lower = text.lower()
candidates: list[str] = []
if "graph build" in lower or "ontology" in lower:
candidates.append("Graph Build")
if "env setup" in lower or "agent profile" in lower or "profession" in lower:
candidates.append("Env Setup")
if "simulation" in lower or "tweet" in lower or "reddit" in lower or "sim config" in lower:
candidates.append("Simulation")
if "report" in lower:
candidates.append("Report")
if "interaction" in lower or "interview" in lower or "chat repl" in lower:
candidates.append("Interaction")
if "log" in lower:
candidates.append("Logs")
relevant_gaps = []
for step in candidates:
relevant_gaps.extend(gaps_by_step.get(step, []))
if "frontend ui" in lower or "no chinese strings on screen" in lower or "every label" in lower:
ui_gaps = gaps_by_step.get("UI", [])
if ui_gaps:
return ("gap", f" - {len(ui_gaps)} hard-coded CJK literal(s) in `frontend/src/views|components/`")
return ("manual-pending", " - live UI walkthrough not run in this sandbox")
if "locale propagation" in lower or "set_locale" in lower:
prop = gaps_by_step.get("Logs", [])
if prop:
return ("gap", f" - {len(prop)} CJK log strings on EN code path")
return ("manual-pending", " - locale-propagation runtime check not run in this sandbox")
if relevant_gaps:
return ("gap", f" - {len(relevant_gaps)} gap(s) classified, see Section 1/3")
if any(c in lower for c in ("ui", "screenshot", "chat", "modal", "tooltip", "render", "trace", "thinking")):
return ("manual-pending", " - requires live walkthrough")
return ("manual-pending", " - not verifiable statically; awaiting live run")
def render_gap_report(rows: list[dict], ticket_body: str, parity_text: str, sha: str) -> str:
classes = Counter(r["class"] for r in rows)
gap_rows = [r for r in rows if r["class"] == "gap"]
gap_categories = Counter(r["category"] for r in gap_rows)
gap_steps = Counter(r["pipeline_step"] for r in gap_rows)
out: list[str] = []
out.append(f"# Verification gap report - i18n-e2e-english-verification\n")
out.append(f"**Commit:** `{sha}`\n")
out.append("")
out.append("## Overview\n")
out.append(f"- Total CJK matches audited: **{len(rows)}**")
out.append(f"- Class distribution: {format_counter(classes)}")
out.append(f"- Gap categories: {format_counter(gap_categories)}")
out.append(f"- Gap pipeline steps: {format_counter(gap_steps)}")
out.append("")
out.append("## Section 1 - Static CJK audit\n")
out.append("Canonical command (PCRE):\n")
out.append("```")
out.append('git grep -nIP "[\\x{4e00}-\\x{9fff}]" -- backend/app frontend/src locales/en.json')
out.append("```")
out.append("")
out.append(f"Raw output captured at `audit/{sha}/cjk-grep.txt` and bucketed at `audit/{sha}/cjk-grep-bucketed.txt`.")
out.append("")
out.append(f"`locales/en.json` CJK matches: **{sum(1 for r in rows if r['file'] == 'locales/en.json')}** (acceptance: zero).")
out.append("")
out.append("Top files by gap count:")
out.append("")
out.append("| File | Gap count |")
out.append("|------|-----------|")
by_file = Counter(r["file"] for r in gap_rows)
for file, count in by_file.most_common(15):
out.append(f"| `{file}` | {count} |")
out.append("")
out.append("## Section 2 - Locale catalogue parity\n")
out.append("```")
out.append(parity_text.strip())
out.append("```")
out.append("")
out.append("## Section 3 - LLM-prompt locale verification\n")
prompt_gaps = [r for r in gap_rows if r["category"] == "backend-prompt-label"]
out.append(f"Backend prompt-label gaps (CJK string literals inside services that compose LLM prompts): **{len(prompt_gaps)}**")
out.append("")
if prompt_gaps:
out.append("First 10 examples (file:line - match):")
out.append("")
for row in prompt_gaps[:10]:
out.append(f"- `{row['file']}:{row['line']}` - {row['match']}")
if len(prompt_gaps) > 10:
out.append(f"- ... and {len(prompt_gaps) - 10} more (see `classified.csv`)")
out.append("")
out.append(
"These prompts feed the LLM verbatim; CJK labels bias the model toward Chinese output even when "
"the requested locale is English."
)
out.append("")
out.append("## Section 4 - Locale propagation surface\n")
log_gaps = [r for r in gap_rows if r["category"] == "backend-log"]
out.append("| Boundary | Status | Evidence |")
out.append("|----------|--------|----------|")
out.append(
"| HTTP -> Flask handler | manual-pending | runtime not exercised in sandbox; static review showed no per-request locale carrier |"
)
out.append(
"| Flask handler -> Task worker | manual-pending | thread-local `set_locale` referenced in CLAUDE.md but not statically verified end-to-end |"
)
out.append(
f"| Task worker -> OASIS subprocess | manual-pending | subprocess boundary requires live run |"
)
out.append(
f"| Backend logger | {'gap' if log_gaps else 'pass'} | {len(log_gaps)} hard-coded CJK log line(s) on EN code path |"
)
out.append("")
if log_gaps:
out.append("First 10 backend-log gap examples:")
out.append("")
for row in log_gaps[:10]:
out.append(f"- `{row['file']}:{row['line']}` - {row['match']}")
out.append("")
out.append(render_section_5(ticket_body, rows))
out.append("## Section 6 - ZH regression check\n")
out.append(
"- Locale catalogues at full key parity (953 EN keys / 953 ZH keys, symmetric difference 0 - "
"see Section 2).\n"
"- No ZH-specific regression detected in static review. Live ZH walkthrough is `manual-pending`.\n"
)
out.append("## Section 7 - Follow-up plan\n")
out.append("Per R7.2, gaps are grouped into the following follow-up issues (placeholder bodies in `PENDING-followups/`):")
out.append("")
out.append(
f"1. **Frontend hard-coded UI strings** ({len(by_category(rows, 'frontend-ui-string'))} matches + "
f"{len(by_category(rows, 'frontend-regex-parser'))} regex parsers depending on CJK backend output)."
)
out.append(f"2. **Backend log strings** ({len(by_category(rows, 'backend-log'))} matches).")
out.append(f"3. **Backend LLM-prompt context labels** ({len(by_category(rows, 'backend-prompt-label'))} matches).")
out.append("4. **Permanent CI guard** (preventative - re-run this audit on every PR).")
out.append("")
out.append(
"Backend docstring/comment matches (the bulk of `deliberate` rows) are covered by the existing issue #7 and are not re-filed here."
)
return "\n".join(out) + "\n"
def by_category(rows: list[dict], category: str) -> list[dict]:
return [r for r in rows if r["category"] == category and r["class"] == "gap"]
def format_counter(c: Counter) -> str:
return ", ".join(f"{k}={v}" for k, v in c.most_common())
def render_comment_body(rows: list[dict], ticket_body: str, sha: str) -> str:
classes = Counter(r["class"] for r in rows)
gap_rows = [r for r in rows if r["class"] == "gap"]
gap_categories = Counter(r["category"] for r in gap_rows)
out: list[str] = []
out.append(f"### Verification report - run on commit `{sha}`\n")
out.append("This run was produced by `.kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh`.")
out.append("Captured artefacts live under `.kiro/specs/i18n-e2e-english-verification/audit/<commit-sha>/`.\n")
out.append("")
out.append(f"**Audit summary:** {sum(classes.values())} CJK matches across the auditable paths.")
out.append(f"- {classes.get('gap', 0)} `gap` (actionable, see follow-ups)")
out.append(f"- {classes.get('review-needed', 0)} `review-needed` (soft signal; needs human eyeball)")
out.append(f"- {classes.get('deliberate', 0)} `deliberate` (mostly backend docstrings/comments - covered by issue #7)")
out.append(
f"- {classes.get('non-applicable', 0)} `non-applicable` (binary file false positives - excluded)"
)
out.append("")
out.append(f"**Gap-category breakdown:** {format_counter(gap_categories)}")
out.append("")
out.append("---")
out.append("")
out.append("#### Issue checklist mapping")
out.append("")
out.append(render_section_5(ticket_body, rows))
out.append("---")
out.append("")
out.append("#### How to re-run")
out.append("")
out.append("```bash")
out.append("# from the repository root, on any commit:")
out.append("bash .kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh")
out.append("# artefacts at .kiro/specs/i18n-e2e-english-verification/audit/<HEAD-sha>/")
out.append("```")
out.append("")
out.append(
"If `gh` is not authenticated when re-running, the comment body and follow-up bodies are written to "
"`PENDING-issue-10-comment.md` / `PENDING-followups/` for a human to post."
)
out.append("")
out.append("Out of scope for this run (per R5.3 / R7.3): live UI walkthrough, full Docker-Compose pipeline run, and any inline gap fixes.")
return "\n".join(out) + "\n"
def render_followup_bodies(rows: list[dict], sha_dir: Path, sha: str) -> None:
pending_dir = sha_dir / "PENDING-followups"
pending_dir.mkdir(parents=True, exist_ok=True)
ui_gaps = by_category(rows, "frontend-ui-string") + by_category(rows, "frontend-regex-parser")
log_gaps = by_category(rows, "backend-log")
prompt_gaps = by_category(rows, "backend-prompt-label")
files = [
(
"01-frontend-ui-strings.md",
"i18n: replace hard-coded chinese ui strings in process and step components with i18n keys",
ui_gaps,
(
"Several `.vue` templates in `frontend/src/views/` and `frontend/src/components/` still emit "
"Chinese strings directly instead of routing them through `vue-i18n` keys. Some `Step4Report.vue` "
"regex parsers also rely on Chinese tokens emitted by the backend (so they will silently break "
"once the backend prompts are translated)."
),
["i18n", "bug"],
),
(
"02-backend-log-strings.md",
"i18n: externalise remaining chinese log strings in flask api and utils",
log_gaps,
(
"After issue #6 externalised most backend log messages, a handful of `logger.info` / "
"`logger.error` call sites in `backend/app/api/graph.py` and `backend/app/utils/retry.py` "
"still hard-code Chinese strings, so backend logs leak Chinese under EN locale."
),
["i18n"],
),
(
"03-backend-prompt-labels.md",
"i18n: translate chinese context labels inside llm-prompt assembly in backend services",
prompt_gaps,
(
"Several `services/*_generator.py` files compose LLM prompts that still embed Chinese "
"context labels (e.g. `\"事实信息:\"`, `\"相关实体:\"`) into the prompt string verbatim. These "
"labels bias the LLM toward Chinese output even when the requested locale is English."
),
["i18n"],
),
(
"04-permanent-ci-guard.md",
"i18n: add a permanent ci guard that runs the e2e cjk audit on every pr",
[],
(
"Promote the audit pipeline at `.kiro/specs/i18n-e2e-english-verification/audit/scripts/` to "
"a permanent CI check. The guard should fail when `locales/en.json` contains any CJK character "
"and when the gap count regresses against a committed baseline."
),
["i18n", "enhancement"],
),
]
for name, title, gaps, summary, labels in files:
if not gaps and not name.startswith("04-"):
(pending_dir / name).write_text("", encoding="utf-8")
continue
body = [
f"# {title}",
"",
"## Summary",
"",
summary,
"",
"## Linked from",
"",
f"- Issue #{ISSUE_NUMBER} (verification report comment).",
f"- Spec: `.kiro/specs/i18n-e2e-english-verification/` at commit `{sha}`.",
"",
"## Evidence",
"",
]
if gaps:
for row in gaps[:50]:
body.append(f"- `{row['file']}:{row['line']}` - {row['match']}")
if len(gaps) > 50:
body.append(f"- ... and {len(gaps) - 50} more (see `classified.csv` in the spec dir)")
else:
body.append("- (No gaps in this run; this is a preventative follow-up only.)")
body.append("")
body.append("## Acceptance")
body.append("")
body.append("- [ ] Each `file:line` above is fixed (or explicitly classified as `deliberate`).")
body.append("- [ ] Re-running `bash .kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh` shows zero gaps in this category.")
body.append("")
body.append(f"<!-- labels: {','.join(labels)} -->")
body.append("")
(pending_dir / name).write_text("\n".join(body), encoding="utf-8")
def main(argv: list[str]) -> int:
if len(argv) != 3:
print(f"usage: {argv[0]} <sha-dir> <commit-sha>", file=sys.stderr)
return 64
sha_dir = Path(argv[1])
sha = argv[2]
rows = load_rows(sha_dir / "classified.csv")
parity_text = (sha_dir / "parity.txt").read_text(encoding="utf-8")
ticket_body = load_ticket_body(Path(".ticket/10.md"))
gap_report = render_gap_report(rows, ticket_body, parity_text, sha)
(sha_dir / "gap-report.md").write_text(gap_report, encoding="utf-8")
comment_body = render_comment_body(rows, ticket_body, sha)
(sha_dir / "comment-body.md").write_text(comment_body, encoding="utf-8")
render_followup_bodies(rows, sha_dir, sha)
print(f" gap-report.md, comment-body.md, PENDING-followups/ written under {sha_dir}")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))

71
.kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh Executable file
View File

@ -0,0 +1,71 @@
#!/usr/bin/env bash
# Orchestrate the i18n end-to-end verification audit.
#
# Reads working-tree state via git (no production-source modifications),
# captures classified output under audit/<commit-sha>/, and posts the
# verification report comment + follow-up issues via gh when available.
#
# Exit codes:
# 0 - audit succeeded and all GitHub side effects applied
# 1 - audit step failed (read-only producer aborted)
# 2 - audit succeeded but at least one GitHub side effect was deferred to PENDING
set -euo pipefail
repo_root="$(git rev-parse --show-toplevel)"
cd "$repo_root"
spec_root=".kiro/specs/i18n-e2e-english-verification"
scripts_dir="${spec_root}/audit/scripts"
sha="$(git rev-parse HEAD)"
sha_dir="${spec_root}/audit/${sha}"
mkdir -p "${sha_dir}"
printf 'Verification audit\n repo: %s\n sha: %s\n out: %s\n\n' \
"${repo_root}" "${sha}" "${sha_dir}"
ghs_exit=0
step() {
local label="$1"
shift
printf '== %s ==\n' "${label}"
"$@"
}
step "audit_cjk.sh" bash "${scripts_dir}/audit_cjk.sh" "${sha_dir}"
step "check_parity.py" python3 "${scripts_dir}/check_parity.py" "${sha_dir}"
step "classify.py" python3 "${scripts_dir}/classify.py" "${sha_dir}"
step "render_report.py" python3 "${scripts_dir}/render_report.py" "${sha_dir}" "${sha}"
# GitHub side effects: failures here downgrade the run to exit 2 but
# do not abort the rest of the side effects.
set +e
step "post_comment.sh" bash "${scripts_dir}/post_comment.sh" "${sha_dir}"
[ $? -ne 0 ] && ghs_exit=2
step "file_followups.sh" bash "${scripts_dir}/file_followups.sh" "${sha_dir}"
[ $? -ne 0 ] && ghs_exit=2
set -e
printf '\n== summary ==\n'
printf 'sha-dir: %s\n' "${sha_dir}"
if [ -f "${sha_dir}/comment-url.txt" ]; then
printf 'comment: %s\n' "$(cat "${sha_dir}/comment-url.txt")"
else
printf 'comment: PENDING (see %s/PENDING-issue-10-comment.md)\n' "${sha_dir}"
fi
if [ -f "${sha_dir}/followup-urls.txt" ]; then
printf 'follow-ups posted:\n'
sed 's/^/ /' "${sha_dir}/followup-urls.txt"
fi
if compgen -G "${sha_dir}/PENDING-followups/[0-9]*-*.md" > /dev/null; then
printf 'follow-ups PENDING:\n'
for body in "${sha_dir}"/PENDING-followups/[0-9]*-*.md; do
if [ -s "${body}" ]; then
printf ' %s\n' "${body}"
fi
done
fi
exit "${ghs_exit}"

560
.kiro/specs/i18n-e2e-english-verification/design.md Normal file
View File

@ -0,0 +1,560 @@
# Design — i18n-e2e-english-verification
## Overview
**Purpose**: This spec produces a deterministic, re-runnable verification pass that proves (or disproves) the MiroFish 5-step pipeline runs cleanly in English, and posts a structured report on issue #10 with a `pass` / `gap` / `manual-pending` status per checklist item.
**Users**: i18n maintainers reviewing the epic (#11), and any future verifier re-running the audit after subsequent merges. The deliverable is read by humans on GitHub (issue comment) and re-run by humans (or CI in a future iteration) to confirm parity.
**Impact**: No production code is modified. The repository gains one new directory tree (`.kiro/specs/i18n-e2e-english-verification/`) containing the spec, the audit scripts, and the captured outputs. One GitHub comment is posted on #10. Up to four follow-up issues are filed.
### Goals
- Static-audit `backend/app`, `frontend/src`, `locales/en.json` for CJK characters; classify every match.
- Verify EN / ZH locale catalogue parity and flag suspect untranslated entries.
- Verify LLM-prompt assets respect the requested locale.
- Document locale-propagation gaps across Flask → `Task` → OASIS subprocess → ReACT agent.
- Post a single canonical comment on issue #10 with per-checklist statuses.
- File follow-up issues for every gap (no inline fixes).
- Make the audit re-runnable by capturing artefacts under `.kiro/specs/.../audit/<commit-sha>/`.
### Non-Goals
- Patching any `gap` discovered (R7.3 — strictly verification).
- Performance / load testing.
- Adding new locales beyond EN / ZH.
- Building a permanent CI guard (filed as a follow-up issue, not implemented here).
- Live UI / Docker walkthrough — captured as `manual-pending` in this run's report.
## Boundary Commitments
### This Spec Owns
- The audit scripts and the captured audit outputs under `.kiro/specs/i18n-e2e-english-verification/audit/`.
- The `gap-report.md` artefact and the comment body posted on issue #10.
- The grouping rule for follow-up issues (one per category — UI strings, backend log strings, backend LLM-prompt labels, suggested CI guard).
- The `pass` / `gap` / `manual-pending` / `review-needed` classification scheme.
### Out of Boundary
- Any modification of files under `backend/app/`, `frontend/src/`, or `locales/`.
- Fixing the gaps the audit discovers — those land in their own follow-up issues.
- Live UI walkthrough, Docker run, or LLM execution.
- A permanent CI check — filed as a separate follow-up issue.
### Allowed Dependencies
- `git` (for `git grep`, capturing HEAD sha).
- `gh` CLI (for the comment + follow-up issues; with documented fallback when unavailable).
- `python3` (for the catalogue parity diff).
- The repo working tree at HEAD of the working branch.
### Revalidation Triggers
- Any merge to `main` that touches `locales/`, `backend/app/`, or `frontend/src/` invalidates the captured audit; a re-run should produce a new `audit/<commit-sha>/` directory.
- A change to issue #10's checklist body (e.g. a new sub-item) requires re-mapping in `gap-report.md`.
- A change to the four follow-up categories (e.g. project decides to file one issue per file) requires re-running the issue-filing script with new grouping.
## Architecture
### Existing Architecture Analysis
- The MiroFish backend is Flask + Python `Task` workers + an OASIS subprocess (per CLAUDE.md). i18n surfaces are: `vue-i18n` for the SPA, `locales/*.json` shared by both ends, a backend logger that resolves keys per locale, and inline LLM prompts in `backend/app/services/*.py`.
- The verification pass does **not** hook into any of these — it reads files only. No Flask blueprint, no `Task` model, no Neo4j query.
### Architecture Pattern & Boundary Map
```mermaid
graph TB
Verifier[Verifier shell entrypoint]
Audit[audit_cjk.sh]
Parity[check_parity.py]
Classify[classify.py]
Report[render_report.py]
Comment[post_comment.sh]
FollowUp[file_followups.sh]
Repo[Working tree]
Captures[audit slash sha slash]
GH[GitHub via gh CLI]
Verifier --> Audit
Verifier --> Parity
Audit --> Classify
Parity --> Classify
Classify --> Report
Report --> Captures
Report --> Comment
Report --> FollowUp
Audit --> Repo
Parity --> Repo
Comment --> GH
FollowUp --> GH
```
**Architecture Integration**:
- **Selected pattern**: Linear pipeline of read-only scripts that each emit a single artefact, composed by a thin shell entrypoint. No mutable state outside `audit/<sha>/`.
- **Domain boundaries**: `audit_cjk.sh` owns the raw grep; `check_parity.py` owns the catalogue diff; `classify.py` owns the four-class labels; `render_report.py` owns the comment body; `post_comment.sh` and `file_followups.sh` own GitHub side effects.
- **Existing patterns preserved**: Shell + Python script pair (matches the project's existing `setup`/`run` style); no new test runner, no new linter.
- **New components rationale**: Each script is single-purpose so failures (e.g. `gh` permission issues) are isolated and the pipeline can resume from the failed step.
- **Steering compliance**: No production-code touch (R7.3); 4-space indent in any committed Python; double quotes; `snake_case`; reserved Bash exits with a non-zero status on any uncaught error.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| CLI / Audit runner | Bash 5+, `git grep -P` (PCRE) | Run the canonical CJK audit | `\x{...}` ranges require PCRE — `git grep -E` will fail on this regex (verified). |
| Static checks | Python 3.11 (project minimum per CLAUDE.md) | Catalogue parity + classification + report rendering | Standard library only — no new deps. |
| GitHub integration | `gh` CLI | Post the comment, file follow-ups | Falls back to `audit/<sha>/PENDING-*` files when missing. |
| Output formats | Plain text + Markdown | Captures + comment body | No HTML, no JSON beyond `gh`'s own. |
## File Structure Plan
### Directory Structure
```
.kiro/specs/i18n-e2e-english-verification/
├── spec.json
├── requirements.md
├── gap-analysis.md
├── research.md
├── design.md
├── tasks.md
├── HANDOFF.md # only if implementation hits the 3-cycle remediation cap
└── audit/
├── scripts/
│ ├── run_audit.sh # entrypoint - chains the steps below
│ ├── audit_cjk.sh # git grep PCRE + bucket counts
│ ├── check_parity.py # locales/en.json vs zh.json key + identical-value diff
│ ├── classify.py # apply 4-class labels to grep matches
│ ├── render_report.py # produce gap-report.md + comment-body.md
│ ├── post_comment.sh # gh issue comment 10 with comment-body.md (or PENDING-*)
│ └── file_followups.sh # gh issue create per category (or PENDING-*)
└── <commit-sha>/ # captured outputs of one verification run
├── cjk-grep.txt # raw `git grep -nP ...` output
├── cjk-grep-bucketed.txt # the same, partitioned by top-level path
├── parity.txt # en/zh diff summary
├── classified.csv # match-by-match label
├── gap-report.md # the canonical structured report
├── comment-body.md # the markdown posted to issue #10
├── PENDING-issue-10-comment.md # only if gh comment failed
└── PENDING-followups/ # only if gh issue create failed
├── 01-frontend-ui-strings.md
├── 02-backend-log-strings.md
├── 03-backend-prompt-labels.md
└── 04-permanent-ci-guard.md
```
### Modified Files
- *(None.)* The spec explicitly forbids touching production source.
## System Flows
```mermaid
sequenceDiagram
participant V as Verifier
participant Run as run_audit.sh
participant FS as Working tree
participant GH as GitHub
V->>Run: bash run_audit.sh
Run->>FS: git grep -nP, git rev-parse HEAD
FS-->>Run: cjk-grep.txt + sha
Run->>FS: read locales json
FS-->>Run: en/zh dicts
Run->>Run: classify
Run->>FS: write audit slash sha slash artefacts
Run->>GH: gh issue comment 10
alt gh succeeds
GH-->>Run: comment URL
Run->>GH: gh issue create x N follow-ups
GH-->>Run: issue URLs
else gh fails
Run->>FS: write PENDING markdown to audit slash sha slash
end
Run-->>V: exit 0 success or exit 2 PENDING
```
**Key flow decisions**:
- The audit always writes the captured artefacts to disk first (idempotent, re-runnable). The GitHub side effects are the *last* steps so any earlier failure leaves a complete capture for inspection.
- A non-zero `gh` exit shifts the pipeline to PENDING mode rather than failing the whole run; the script exits `2` to flag "audit ran but GitHub side-effects didn't apply".
## Requirements Traceability
| Requirement | Summary | Components | Interfaces / Artefacts | Flows |
|-------------|---------|------------|------------------------|-------|
| 1.1 | Run canonical `git grep` | audit_cjk.sh | `cjk-grep.txt` | Audit step |
| 1.2 | Classify each match | classify.py | `classified.csv` | Audit step |
| 1.3 | Record file:line + step tag for `gap` | classify.py | `classified.csv` (`step` column) | Audit step |
| 1.4 | No file modifications during audit | run_audit.sh | scripts are read-only | — |
| 1.5 | `en.json` CJK = always `gap` | classify.py | hard rule in classifier | Audit step |
| 2.1 | Enumerate keys recursively | check_parity.py | `parity.txt` | Audit step |
| 2.2 | Missing-key gaps recorded | check_parity.py | `parity.txt` (missing-key block) | Audit step |
| 2.3 | EN catalogue CJK = `gap` | check_parity.py | `parity.txt` (cjk-in-en block) | Audit step |
| 2.4 | EN/ZH identical = `review-needed` | check_parity.py | `parity.txt` (identical-value block) | Audit step |
| 2.5 | No catalogue edits | check_parity.py | read-only stdlib JSON load | — |
| 3.1 | Enumerate prompt files | classify.py (heuristic — known files list) | `gap-report.md` Section 3 | — |
| 3.2 | Confirm locale-aware or EN-only | classify.py | `gap-report.md` Section 3 | — |
| 3.3 | Hard-coded ZH directive = `gap` | classify.py | `classified.csv` (`category=prompt-label`) | — |
| 3.4 | #3, #4, #5 prompts post-merge check | classify.py | `gap-report.md` Section 3 | — |
| 4.1 | Identify handoff boundaries | render_report.py | `gap-report.md` Section 4 | — |
| 4.2 | Confirm explicit or re-derived locale | render_report.py | `gap-report.md` Section 4 | — |
| 4.3 | Silent default = `gap` | classify.py | `classified.csv` (`category=propagation`) | — |
| 4.4 | Backend logger EN under EN | classify.py | `classified.csv` (`category=backend-log`) | — |
| 5.1 | Comment lists every checklist item | render_report.py | `comment-body.md` | Comment-post |
| 5.2 | Each `gap` includes file:line + follow-up link | render_report.py | `comment-body.md` | Comment-post |
| 5.3 | `manual-pending` items state repro steps | render_report.py | `comment-body.md` | Comment-post |
| 5.4 | Comment includes raw audit (or path) | render_report.py | `comment-body.md` (path reference) | Comment-post |
| 5.5 | Post via `gh issue comment 10` | post_comment.sh | `comment-body.md` | Comment-post |
| 6.1 | ZH covers every EN key | check_parity.py | (already passes per gap-analysis) | — |
| 6.2 | Locale-aware prompts symmetric | render_report.py | `gap-report.md` Section 6 | — |
| 6.3 | EN-only ZH value = `review-needed` | check_parity.py | `parity.txt` (identical-value block) | — |
| 6.4 | ZH regression filed as gap | classify.py | `classified.csv` | — |
| 7.1 | File issue per gap | file_followups.sh | `gh issue create` | Follow-up |
| 7.2 | Group by category | file_followups.sh | one body per category in `PENDING-followups/` | Follow-up |
| 7.3 | No production-code edits | run_audit.sh | only writes under `.kiro/specs/.../` | — |
| 7.4 | Label follow-ups `i18n` | file_followups.sh | `gh issue create --label i18n` | Follow-up |
| 7.5 | Fallback inline list when no `gh` | file_followups.sh | `PENDING-followups/*.md` | Follow-up |
| 8.1 | Capture raw output | run_audit.sh | `audit/<sha>/` directory | Audit step |
| 8.2 | Preserve previous run | run_audit.sh | `<sha>` subdirectory naming | Audit step |
| 8.3 | Record HEAD sha | run_audit.sh | `git rev-parse HEAD` | Audit step |
| 8.4 | Idempotent re-run | run_audit.sh | re-running on same sha overwrites that sha's dir | Audit step |
## Components and Interfaces
| Component | Domain | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------|--------|--------------|--------------------------|-----------|
| run_audit.sh | Verification pipeline | Compose the audit and route artefacts | 1.4, 7.3, 8.1, 8.2, 8.3, 8.4 | git (P0), python3 (P0), gh (P1) | Batch |
| audit_cjk.sh | Static audit | Run `git grep -nP` and bucket | 1.1, 1.5 | git (P0) | Batch |
| check_parity.py | Catalogue diff | Diff en/zh + identical-value heuristic | 2.1, 2.2, 2.3, 2.4, 2.5, 6.1, 6.3 | python3 stdlib (P0) | Batch |
| classify.py | Classification | Apply the 4-class label per match | 1.2, 1.3, 1.5, 3.1, 3.2, 3.3, 3.4, 4.3, 4.4, 6.4 | cjk-grep.txt (P0), parity.txt (P0) | Batch |
| render_report.py | Report assembly | Produce gap-report.md + comment-body.md | 4.1, 4.2, 5.1, 5.2, 5.3, 5.4, 6.2 | classified.csv (P0) | Batch |
| post_comment.sh | GitHub side-effect | Post the comment on #10 | 5.5 | gh (P0), comment-body.md (P0) | Service |
| file_followups.sh | GitHub side-effect | Open follow-up issues | 7.1, 7.2, 7.4, 7.5 | gh (P0), PENDING-followups/* (P0) | Service |
### Verification pipeline
#### `run_audit.sh`
| Field | Detail |
|-------|--------|
| Intent | Single shell entrypoint that runs every step in order and persists artefacts under `audit/<commit-sha>/` |
| Requirements | 1.4, 7.3, 8.1, 8.2, 8.3, 8.4 |
**Responsibilities & Constraints**
- Must NOT modify any file outside `.kiro/specs/i18n-e2e-english-verification/`.
- Must capture HEAD sha before any other step (so the artefact path is set).
- Must exit `0` on full success (audit + GitHub side effects) and `2` on PENDING (audit succeeded, side effects didn't).
- Must be safely re-runnable on the same sha (overwriting that sha's directory is acceptable).
**Dependencies**
- Inbound: invoked manually by the verifier (`bash run_audit.sh`) — Criticality: P0.
- Outbound: `audit_cjk.sh`, `check_parity.py`, `classify.py`, `render_report.py`, `post_comment.sh`, `file_followups.sh` — Criticality: P0 each.
- External: `git`, `python3`, `gh` (P1 — fallback supported).
**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [x] / State [ ]
##### Batch / Job Contract
- **Trigger**: manual `bash .kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh`.
- **Input / validation**: working tree at any commit; rejects detached non-clean trees? — no, the audit reads tracked files only via `git grep`, so unstaged edits are ignored deliberately.
- **Output / destination**: `.kiro/specs/i18n-e2e-english-verification/audit/<commit-sha>/`.
- **Idempotency & recovery**: Re-running on the same sha overwrites that sha's directory. PENDING outputs survive across runs until a `gh`-enabled run replaces them.
**Implementation Notes**
- Integration: invoked by humans only — no CI hookup in this spec.
- Validation: confirm `gh auth status` before attempting comment/issue posts; on failure, branch to PENDING.
- Risks: shell quoting around the PCRE pattern (`[\x{4e00}-\x{9fff}]`) — use single-quoted argument to `git grep -P`.
#### `audit_cjk.sh`
| Field | Detail |
|-------|--------|
| Intent | Run the canonical PCRE grep + per-bucket counts |
| Requirements | 1.1, 1.5 |
**Responsibilities & Constraints**
- Output: `cjk-grep.txt` (raw `git grep -nP` lines) and `cjk-grep-bucketed.txt` (one section per top-level path: `backend/app`, `frontend/src`, `locales/en.json`).
- Excludes binary file matches (e.g. `.jpeg` false positives).
**Dependencies**
- Inbound: `run_audit.sh` (P0).
- External: `git` 2.x (P0 — must support `-P` for PCRE).
**Contracts**: Batch [x]
##### Batch / Job Contract
- **Trigger**: invoked by `run_audit.sh`.
- **Input / validation**: receives the target output directory as argv[1]; aborts if missing.
- **Output / destination**: `cjk-grep.txt`, `cjk-grep-bucketed.txt` in `<sha>/`.
- **Idempotency & recovery**: deterministic — same tree → same output.
**Implementation Notes**
- Integration: pure read-only against `git`.
- Validation: `git --version` precondition; abort with a clear error if PCRE unsupported.
- Risks: ripgrep is NOT used (avoids a hard `rg` dependency); `git grep -P` is built-in to git's PCRE2 binding.
#### `check_parity.py`
| Field | Detail |
|-------|--------|
| Intent | Compare `locales/en.json` and `locales/zh.json`: key parity, CJK in EN, identical-value heuristic |
| Requirements | 2.1, 2.2, 2.3, 2.4, 2.5, 6.1, 6.3 |
**Responsibilities & Constraints**
- Recursively flattens nested-dict keys with dotted paths.
- Reports three blocks: `missing-keys`, `cjk-in-en`, `identical-values`.
- Treats values as `review-needed` only if (a) en value == zh value, (b) value is non-empty, (c) value is more than two ASCII words.
**Dependencies**
- Inbound: `run_audit.sh` (P0).
- External: `json` from Python stdlib (P0).
**Contracts**: Batch [x]
##### Batch / Job Contract
- **Trigger**: invoked by `run_audit.sh` with the `<sha>` directory as argv[1].
- **Input / validation**: reads `locales/en.json` and `locales/zh.json` from cwd (must be invoked from repo root); fails fast on JSON parse error.
- **Output / destination**: `parity.txt` in `<sha>/`.
- **Idempotency & recovery**: pure function of catalogue contents.
**Implementation Notes**
- Integration: invoked from repo root so relative paths resolve.
- Validation: parse-on-load, both files must be objects.
- Risks: the "more than two ASCII words" heuristic may produce noise — `review-needed` is intentionally a soft label not a `gap`.
#### `classify.py`
| Field | Detail |
|-------|--------|
| Intent | Apply the 4-class label (`deliberate` / `gap` / `non-applicable` / `review-needed`) and a category tag per match |
| Requirements | 1.2, 1.3, 1.5, 3.1, 3.2, 3.3, 3.4, 4.3, 4.4, 6.4 |
**Responsibilities & Constraints**
- Reads `cjk-grep.txt` and `parity.txt`; emits `classified.csv` with columns: `file`, `line`, `match`, `class`, `category`, `pipeline_step`.
- Categories (closed set): `frontend-ui-string`, `frontend-regex-parser`, `backend-docstring`, `backend-comment`, `backend-log`, `backend-prompt-label`, `propagation`, `catalogue-parity`, `binary-false-positive`.
- Pipeline-step tags (closed set): `Graph Build`, `Env Setup`, `Simulation`, `Report`, `Interaction`, `Logs`, `UI`, `n/a`.
- Classification rules:
- `locales/en.json` CJK → always `gap` / `catalogue-parity` / `n/a` (R1.5).
- File path under `frontend/src/views/` or `frontend/src/components/` AND match is inside a string literal (heuristic: enclosed in `'…'`/`"…"`/`` `…` ``) → `gap` / `frontend-ui-string`.
- Match inside a `text.match(/.../)` call in a `.vue` file → `frontend-regex-parser` / `gap` (cause: backend emits CJK).
- Backend `.py` file, line starts with `#` or appears inside a triple-quoted docstring → `deliberate-blocked-by-#7` / `backend-docstring` (or `backend-comment`) — counted but not filed as a fresh follow-up since #7 already covers it.
- Backend `.py` file, line contains `logger.`, `log.`, `print(` and CJK in a string literal → `gap` / `backend-log` / appropriate step tag.
- Backend `.py` file in `services/{ontology,oasis_profile,simulation_config,report_agent}_generator.py` and CJK appears inside an LLM-prompt context label (heuristic: a string literal not preceded by `#`) → `gap` / `backend-prompt-label`.
- Binary files (e.g. `.jpeg` ripgrep matches): `non-applicable` / `binary-false-positive`.
- Anything else: `review-needed` (forces a human look).
**Dependencies**
- Inbound: `audit_cjk.sh`, `check_parity.py` (P0).
- External: `csv` from Python stdlib.
**Contracts**: Batch [x]
##### Batch / Job Contract
- **Trigger**: invoked by `run_audit.sh` after the two preceding steps.
- **Input / validation**: `cjk-grep.txt` and `parity.txt` must exist in `<sha>/`.
- **Output / destination**: `classified.csv`.
- **Idempotency & recovery**: deterministic — same inputs → same csv.
**Implementation Notes**
- Integration: classification rules are heuristics, not a parser; correctness is bounded by careful regexes and an explicit "fallthrough = `review-needed`" rule.
- Validation: every input row produces an output row (no silent drops); a count-equality assertion runs at the end.
- Risks: false negatives (e.g. a Chinese log string that doesn't contain `logger.` on the same line) — `review-needed` fallthrough catches these.
#### `render_report.py`
| Field | Detail |
|-------|--------|
| Intent | Produce `gap-report.md` and `comment-body.md` |
| Requirements | 4.1, 4.2, 5.1, 5.2, 5.3, 5.4, 6.2 |
**Responsibilities & Constraints**
- `gap-report.md`: Sections: Overview, Section 1 (static audit), Section 2 (parity), Section 3 (prompt verification), Section 4 (propagation), Section 5 (issue-#10 checklist mapping), Section 6 (ZH regression), Section 7 (follow-up plan).
- `comment-body.md`: Markdown comment for issue #10 — mirrors the issue's checklist with `pass` / `gap` / `manual-pending` for each line, plus a "How to re-run" footer.
- Reads `classified.csv` and the issue body (snapshot at `.ticket/10.md`).
**Dependencies**
- Inbound: `classify.py` (P0), `.ticket/10.md` (P0).
- External: Python stdlib only.
**Contracts**: Batch [x]
##### Batch / Job Contract
- **Trigger**: `run_audit.sh` after `classify.py`.
- **Input / validation**: `classified.csv` and `.ticket/10.md` must exist.
- **Output / destination**: `gap-report.md`, `comment-body.md` in `<sha>/`.
- **Idempotency & recovery**: deterministic.
**Implementation Notes**
- Integration: the comment body must include a `Run on commit <sha>` header so the comment is traceable.
- Validation: confirm every issue-body checkbox has been mapped (count check).
- Risks: rendering CJK characters in markdown — Python writes UTF-8 by default; comment body is verified to round-trip via `gh`.
#### `post_comment.sh`
| Field | Detail |
|-------|--------|
| Intent | Post `comment-body.md` as a comment on issue #10 |
| Requirements | 5.5 |
**Responsibilities & Constraints**
- `gh issue comment 10 --repo salestech-group/MiroFish --body-file <sha>/comment-body.md`.
- On non-zero exit, copies the body to `<sha>/PENDING-issue-10-comment.md` and exits non-zero.
**Dependencies**
- External: `gh` (P0; degrades to PENDING when missing).
**Contracts**: Service [x]
##### Service Interface
```text
post_comment.sh <sha-dir>
precondition: <sha-dir>/comment-body.md exists
postcondition (success): comment posted; URL printed to stdout
postcondition (failure): <sha-dir>/PENDING-issue-10-comment.md present; exit code 2
```
**Implementation Notes**
- Integration: must be the second-to-last step (so failures don't block the issue-filing fallback).
- Validation: parses `gh`'s URL output and writes it to `<sha>/comment-url.txt` on success.
- Risks: PR-time rate limits — unlikely for a single comment.
#### `file_followups.sh`
| Field | Detail |
|-------|--------|
| Intent | Open one follow-up issue per gap category |
| Requirements | 7.1, 7.2, 7.4, 7.5 |
**Responsibilities & Constraints**
- Iterates `<sha>/PENDING-followups/*.md` (which `render_report.py` always writes; the ones whose category had zero gaps stay empty placeholders).
- For each non-empty body, runs `gh issue create --repo salestech-group/MiroFish --title <title> --body-file <body> --label i18n`.
- On `gh` failure for any single category, leaves the corresponding `PENDING-followups/<n>-*.md` in place and exits non-zero at the end (after attempting all categories).
**Dependencies**
- External: `gh` (P0; degrades to PENDING).
**Contracts**: Service [x]
##### Service Interface
```text
file_followups.sh <sha-dir>
precondition: <sha-dir>/PENDING-followups/*.md exist (possibly empty placeholders)
postcondition (success): all non-empty bodies posted; URLs appended to <sha-dir>/followup-urls.txt; bodies removed from PENDING-followups/
postcondition (partial): URLs in followup-urls.txt for the ones that posted; the rest stay in PENDING-followups/; exit code 2
```
**Implementation Notes**
- Integration: must be the last step.
- Validation: post-hoc count check (`gh` URLs + remaining PENDING bodies = total categories).
- Risks: a category that the spec already considers covered (e.g. backend docstrings → blocked by #7) is not re-filed; the spec's category list is closed and excludes that case.
## Data Models
### Domain Model
The audit operates on three logical concepts:
- **Match** — a single line of `git grep` output. `(file, line, raw_text)`.
- **Classification** — `(match, class ∈ {deliberate, gap, non-applicable, review-needed}, category ∈ closed-set, pipeline_step ∈ closed-set)`.
- **Follow-up** — `(category, title, body, status ∈ {posted, pending}, url?)`.
Invariant: every `Match` produces exactly one `Classification`; every `Classification` with `class == gap` belongs to exactly one `Follow-up` category (which may aggregate multiple gaps).
### Logical Data Model
**`classified.csv` schema** (CSV, UTF-8, header row):
| Column | Type | Notes |
|--------|------|-------|
| `file` | string | repo-relative path |
| `line` | int | 1-indexed |
| `match` | string | trimmed grep line |
| `class` | enum | `deliberate` / `gap` / `non-applicable` / `review-needed` |
| `category` | enum | closed set listed in classify.py rules |
| `pipeline_step` | enum | closed set listed in classify.py rules |
Natural key: `(file, line)`.
**`parity.txt` structure** (text, three labelled blocks):
```
[missing-keys]
en-only: <key.path>
zh-only: <key.path>
[cjk-in-en]
<key.path>: <value snippet>
[identical-values]
<key.path>: <value> # review-needed if non-trivial English prose
```
### Data Contracts & Integration
- **`comment-body.md`** must be valid GitHub-flavoured Markdown; checkbox lines preserve the issue's original ordering.
- **Follow-up issue body** must be valid GitHub-flavoured Markdown; first line is a one-sentence summary; subsequent sections are: `## Evidence` (file:line list), `## Linked from` (#10 + comment URL), `## Acceptance` (a small checklist).
## Error Handling
### Error Strategy
- **Read-only operations** (steps 1–4): on any uncaught error (missing file, JSON parse error), the script aborts with a non-zero exit before any artefact is half-written. The orchestrator uses `set -euo pipefail`.
- **GitHub side effects** (steps 5–6): wrapped — failure routes to PENDING outputs and the orchestrator exits `2`.
### Error Categories and Responses
- **User errors**: invoked from wrong directory → fail fast with "must be run from repo root".
- **System errors**: `git`/`python3`/`gh` missing → fail fast with "install <tool>"; `gh auth status` not OK → branch to PENDING.
- **Business errors**: classification produces 0 matches but `cjk-grep.txt` non-empty → assertion failure (count-equality bug).
### Monitoring
- The orchestrator prints a one-line status per step.
- Final summary block to stdout: total matches, gaps, `manual-pending`, follow-ups posted vs PENDING.
## Testing Strategy
- **Unit tests**: not introduced — the scripts are simple enough that a one-shot dry run on the live tree is the canonical validation.
- **Integration test**: a single `bash run_audit.sh` against the working tree; success criteria below.
- **Validation checklist** (run during implementation):
- The audit produces a non-empty `cjk-grep.txt`.
- `parity.txt` reports 0 missing keys (matches the live state at HEAD).
- `classified.csv` row count == `cjk-grep.txt` line count.
- `gap-report.md` and `comment-body.md` parse as valid markdown (manual eyeball — no toolchain required).
- The classifier marks every `locales/en.json` CJK as `gap` (currently zero such matches, so this asserts the negative).
- With `gh` available: a comment is posted on #10 and follow-up issues are created.
- With `gh` simulated as absent (e.g. `PATH=/dev/null`): PENDING outputs appear under `<sha>/`.
### Out of scope for testing
- The live UI walkthrough is `manual-pending` (R5.3) and not part of the test plan.
- Performance, scalability, security: nothing to test — read-only single-shot scripts.

136
.kiro/specs/i18n-e2e-english-verification/gap-analysis.md Normal file
View File

@ -0,0 +1,136 @@
# Gap Analysis — i18n-e2e-english-verification
## 1. Current state investigation
### Domain-relevant assets in the repo
| Concern | Location | Notes |
|---|---|---|
| Locale catalogues | `locales/en.json`, `locales/zh.json`, `locales/languages.json` | Flat-namespaced JSON, loaded by `vue-i18n` and the backend logger. |
| Frontend i18n loader | `frontend/src/i18n/` | Provides `useI18n()` to components. |
| Frontend UI surface | `frontend/src/views/`, `frontend/src/components/` | Step1–5 components + `Process.vue` orchestrator. |
| Backend logger | `backend/app/utils/logger.py` (per CLAUDE.md) | Externalised log messages (#6 work). |
| Locale helpers | `backend/app/utils/` | Per CLAUDE.md, locale propagation lives here. |
| Prompt assets that emit user-visible text | `backend/app/services/ontology_generator.py` (#2, #3?), `oasis_profile_generator.py` (#3), `simulation_config_generator.py` (#4), `report_agent.py` (#5) | Prompts are inline Python strings, not separate files. |
| Pipeline boundaries | `backend/app/api/*.py` (Flask), `services/simulation_runner.py` + `simulation_ipc.py` (subprocess), `services/report_agent.py` (ReACT) | Locale must propagate across all of these. |
### Project conventions surfaced
- `Task` model used for any long-running operation (CLAUDE.md). Verification doesn't introduce one — it is a one-shot batch.
- Reasoning-model output stripping convention exists, irrelevant here.
- Per-project `group_id` isolation in Neo4j — verification queries should NOT touch Neo4j; we run a static audit only.
- "Match the surrounding file's style" (no enforced formatter).
### Live audit baseline (commit `9dcaecd`)
```
git grep -nP "[\x{4e00}-\x{9fff}]" -- backend/app frontend/src locales/en.json | wc -l
→ 2918 lines across 36 files
```
Bucketed:
| Bucket | Files | Lines | Notes |
|---|---|---|---|
| `locales/en.json` | 0 | 0 | ✅ clean |
| `frontend/src/views/Process.vue` | 1 | 65 | hard-coded UI strings (template + JS literals), not i18n keys |
| `frontend/src/components/Step{2,3,4,5}*.vue` | 4 | ~50 (mostly Step4Report.vue regex parsers) | depends-on-backend regex parsers + a few literals |
| `backend/app/services/*.py` | 13 | majority | docstrings + comments + a few prompt assembly fragments + agent context labels (e.g. `"事实信息:"` in `oasis_profile_generator.py`) |
| `backend/app/api/*.py` | 4 | many | docstrings + comments + log-message Chinese (`build_logger.info(f"[{task_id}] 开始构建图谱...")` etc) |
| `backend/app/utils/*.py` | 7 | many | docstrings + comments + log strings (e.g. `retry.py` "函数 {func} 在 N 次重试后仍失败") |
| `backend/app/models/*.py` | 3 | docstrings | docstrings only (probably) |
### Locale catalogue parity (Python check)
```
en keys: 953
zh keys: 953
symmetric diff: 0
```
→ R2 (parity) passes. ZH backfill (#8) closed the gap and en/zh are now lock-step.
### Boundary review surface (R4)
- `backend/app/api/graph.py` `build_logger.info(f"[{task_id}] 开始构建图谱...")` shows the backend logger is still emitting Chinese on the build path — this is exactly the kind of leak #6 was supposed to externalise.
- `backend/app/utils/retry.py` `logger.error(f"函数 {func.__name__} 在 {max_retries} 次重试后仍失败...")` — same: log strings remain hard-coded Chinese.
- ReACT/agent context labels in `oasis_profile_generator.py` (`"事实信息:"`, `"相关实体:"`) feed directly into the LLM prompt — these will bias the model toward Chinese output.
## 2. Requirements feasibility
### Mapping requirements → existing assets
| Req | Need | Existing asset | Gap tag |
|---|---|---|---|
| R1 (static audit) | run `git grep` and capture output | git, ripgrep | None — straightforward |
| R1.5 (`en.json` CJK check) | inspect catalogue | already at 0 hits | None — passes |
| R2 (parity) | enumerate keys recursively, diff | small Python script | None — already passes |
| R3 (prompt verification) | read prompt strings in `services/*.py` | inline Python strings | **Constraint** — prompts are inline, not standalone files; verification must read source not assets |
| R4 (propagation) | trace locale across Flask → Task → OASIS → ReACT | source code review | **Research needed** in design phase: where exactly is locale stored today? CLAUDE.md hints `set_locale` thread-local exists but path not yet read |
| R5 (post comment) | `gh issue comment 10` | `gh` CLI | None |
| R6 (ZH regression) | confirm zh values are non-English | small Python script | None |
| R7 (file follow-ups) | `gh issue create` | `gh` CLI | None |
| R8 (capture & idempotence) | write under `.kiro/specs/.../audit/` | filesystem | None |
### Complexity signals
- Algorithmic: trivial — grep + count + diff.
- Workflow: post a comment + open follow-up issues — one-shot.
- External integrations: GitHub via `gh`. No DB, no Neo4j, no LLM calls.
### Constraints from existing architecture
- **No code edits to `backend/app/`, `frontend/src/`, `locales/`** — the spec is verification-only. The change-set is confined to `.kiro/specs/i18n-e2e-english-verification/` (audit captures, gap report, follow-up issue list) and any commit message / PR description.
- Manual UI walkthrough is not feasible in a sandboxed CLI — must be marked `manual-pending` per R5.3.
- Live `docker-compose up` likewise unavailable — same handling.
## 3. Implementation approach options
### Option A — Pure shell + Python script kept under `.kiro/specs/.../audit/`
- A single Bash + Python pipeline that emits `audit/cjk-grep.txt`, `audit/parity.txt`, `audit/gap-report.md`.
- Posts the comment via `gh` and opens follow-ups via `gh issue create`.
- Scripts are read-only against production source.
✅ Simplest, no production-code touch.
✅ Easy to re-run.
❌ Scripts only relevant to this ticket — scoped to `.kiro/specs/.../audit/scripts/`, not promoted to a reusable `tools/`.
### Option B — Build a reusable `tools/i18n-audit/` checker
- Create a permanent CLI under `tools/` so future verifiers can re-run.
- Integrates with CI (could become a check that fails when `en.json` contains CJK).
❌ Adds a tool & directory the project doesn't have. Scope creep — the spec is for one verification pass, not a CI check.
❌ A reusable tool wants its own ticket; ramming it in here violates the "no inline fixes" rule.
### Option C — Hybrid: ad-hoc script for this run, plus open a follow-up issue requesting the reusable CI check
- Run the verification with disposable scripts (Option A) AND file a follow-up issue asking for the reusable CI check (Option B as a future ticket).
✅ Keeps current ticket scoped.
✅ Captures the value of B without bloating this PR.
## 4. Out-of-scope items deferred
- Any **production code edits** that would close gaps. R7 makes this explicit.
- Live UI walkthrough / dynamic verification — captured as `manual-pending` in the report.
## 5. Effort & risk
- **Effort**: S (1 day) — auditing scripts + report writing + issue filings.
- **Risk**: Low — read-only operations, no architectural change, the failure mode (`gh` lacking permissions) is handled by R7.5 (fallback inline list).
## 6. Recommendations for design phase
- **Preferred approach**: Option C (hybrid).
- **Key decisions to make in design**:
- Concrete script layout under `.kiro/specs/i18n-e2e-english-verification/audit/`.
- Format of `audit/gap-report.md` (the artefact echoed into the issue comment).
- Exact follow-up issue grouping rule (R7.2): one issue per pipeline step? per file? per category (UI / logs / prompts / docstrings)?
- Reproducibility (R8.2): do we keep `audit/<commit-sha>/` per run, or `audit/latest/` + `audit/previous/`?
- Whether the scripts are committed to the repo (they live under `.kiro/specs/...` — yes by default) or only the captured outputs.
- **Research items to carry forward**:
- Read `backend/app/utils/` to confirm whether a locale helper / `set_locale` exists today (R4 detail).
- Read `backend/app/utils/logger.py` to confirm where externalised log keys live and how the locale is selected at log time (R4 + Step-1 logs checklist item).
- Confirm whether any `services/*.py` Chinese match is part of an LLM **prompt** vs a comment — only prompt matches block R3.

122
.kiro/specs/i18n-e2e-english-verification/requirements.md Normal file
View File

@ -0,0 +1,122 @@
# Requirements Document
## Project Description (Input)
Issue #10: i18n end-to-end verification of full pipeline. Run a verification pass to prove the entire 5-step pipeline (Graph Build, Env Setup, Simulation, Report, Interaction) works cleanly in English, with locale propagating across Flask routes, background tasks, OASIS subprocess, Graphiti/Neo4j, and the ReACT report agent. Produce a verification report (posted as a comment on issue #10) summarising pass/fail per checklist item and listing any leftover Chinese strings as `file:line` refs. Run the static audit `git grep -nE "[\\x{4e00}-\\x{9fff}]" -- backend/app frontend/src locales/en.json` and confirm only deliberately-kept Chinese remains. File any newly discovered gaps as follow-up issues (do NOT patch silently in this ticket). Acceptance: all checklist items pass for both EN and ZH; report posted; no surprise Chinese in EN paths. Out of scope: fixing newly discovered gaps inline; perf/load testing; new locales beyond EN/ZH.
## Introduction
This spec covers the final verification pass for the i18n epic (#11). After issues #2–#9, #12 land, the entire 5-step MiroFish pipeline must demonstrably run in English — UI, background work, LLM-generated artifacts (ontologies, agent profiles, sim configs, reports, chat replies), and backend logs — without any unintended Chinese leaking into English-locale paths. The pass also regression-checks that switching locale back to Chinese still produces fully Chinese output. Because the pipeline crosses a Flask app, background `Task` workers, an OASIS subprocess, Graphiti/Neo4j, and a ReACT report agent, the verification has both a static (grep + locale-file) component and a dynamic (live walkthrough of Step 1 → 5) component.
The deliverables are: (a) a static audit + categorization of any remaining Chinese strings under English paths, (b) a verification report posted as a comment on issue #10 summarising pass/fail per checklist item with `file:line` evidence, and (c) follow-up GitHub issues for every gap found — fixes are explicitly **out of scope** here.
## Boundary Context
- **In scope**:
- Static audit (`git grep` for CJK Unified Ideographs) of `backend/app/`, `frontend/src/`, and `locales/en.json`.
- Inspection of locale catalogues (`locales/en.json`, `locales/zh.json`) for parity, key coverage, and accidental Chinese in the EN catalogue.
- Inspection of LLM-prompt assets that drive Step 1–5 outputs (ontology, profile, sim-config, report-agent prompts) to confirm they emit English under EN locale.
- Inspection of locale propagation paths: HTTP request → Flask handler → `Task` background worker → OASIS subprocess → ReACT agent.
- Verification report posted as a comment on issue #10.
- Follow-up issues filed for every gap found.
- **Out of scope**:
- Fixing any newly discovered gaps inline in this ticket — they are filed as separate issues.
- Performance or load testing.
- Adding new locales beyond EN/ZH.
- The live UI walkthrough with screenshots, when no human or browser is available — the static audit results plus prompt/locale-catalogue evidence stand in. The verification report explicitly marks UI-only checklist items as "manual-pending" if not run live.
- **Adjacent expectations**:
- Closes the i18n epic #11 once #12 also lands.
- Depends on (and re-verifies) the work in #2, #3, #4, #5, #6, #8, #9, #12.
## Requirements
### Requirement 1: Static CJK audit of English code paths
**Objective:** As an i18n verifier, I want a deterministic grep-based audit of files that should be English-only, so that any Chinese leaking into the EN-locale code path is detected and recorded.
#### Acceptance Criteria
1. The Verification System shall execute `git grep -nE "[\x{4e00}-\x{9fff}]" -- backend/app frontend/src locales/en.json` and capture every match with `file:line` precision.
2. The Verification System shall classify each match as one of: (a) `deliberate` (e.g. test fixture demonstrating ZH input, doc example, comment explicitly retained per project convention), (b) `gap` (unintended Chinese in EN-facing code), or (c) `non-applicable` (false positive such as a regex character class).
3. When a match is classified as `gap`, the Verification System shall record `file:line`, the Chinese substring, and the affected pipeline step (Graph Build / Env Setup / Simulation / Report / Interaction / Logs / UI).
4. The Verification System shall not modify any matched file as part of this audit; remediation is filed as a follow-up issue per Requirement 7.
5. While the audit is running, the Verification System shall additionally inspect `locales/en.json` for entries whose value contains CJK characters and report those separately (an EN catalogue value containing Chinese is always a `gap`).
### Requirement 2: Locale catalogue parity check
**Objective:** As an i18n verifier, I want to confirm that the EN and ZH catalogues stay in lockstep, so that switching locale never falls back to a missing key or leaks the other locale.
#### Acceptance Criteria
1. The Verification System shall enumerate the key set of `locales/en.json` and `locales/zh.json` (recursively across nested objects) and compute the symmetric difference.
2. If a key is present in `en.json` but missing from `zh.json` (or vice versa), the Verification System shall record the missing key path and treat it as a `gap`.
3. If any value in `en.json` contains a CJK character, the Verification System shall record it as a `gap` (as in Requirement 1.5).
4. If any value in `zh.json` is identical to its `en.json` counterpart and the EN value is non-trivial English prose (more than two ASCII words), the Verification System shall flag it as a candidate untranslated entry — these are reported as `review-needed`, not auto-classified `gap`, since some technical terms (URLs, identifiers, single tokens) legitimately stay identical.
5. The Verification System shall not edit either catalogue file as part of this check.
### Requirement 3: LLM-prompt locale verification
**Objective:** As an i18n verifier, I want to confirm that every LLM prompt that drives a Step 1–5 output respects the requested locale, so that ontology entries, agent profiles, simulation configs, report prose, and chat replies render in the user's selected language.
#### Acceptance Criteria
1. The Verification System shall enumerate the prompt files that produce user-visible output for Steps 1–5 (e.g. ontology generator, OASIS profile generator, simulation-config generator, report agent prompts, interview chat).
2. For each prompt file, the Verification System shall confirm that it either (a) is fully English with an explicit "respond in ${locale}" directive, or (b) is rendered through a locale-aware template that injects the active locale.
3. If a prompt file hard-codes a Chinese-only directive (e.g. "请用中文回答") on the EN code path, the Verification System shall record it as a `gap`.
4. The Verification System shall confirm that the prompt files referenced by issues #3, #4, #5 are no longer Chinese-only post-merge; if any still are, they are recorded as `gap` blocking #10.
### Requirement 4: Locale propagation surface review
**Objective:** As an i18n verifier, I want to confirm that the active locale survives every process boundary, so that an EN request still produces EN output after it crosses into a `Task` worker, the OASIS subprocess, or the ReACT agent.
#### Acceptance Criteria
1. The Verification System shall identify each handoff boundary: HTTP → Flask handler, Flask handler → `Task` worker, `Task` worker → OASIS subprocess, ReACT agent → tool calls.
2. For each handoff, the Verification System shall confirm that the locale is either (a) carried explicitly in the call payload / kwargs, or (b) re-derived deterministically (e.g. from per-project config, `Accept-Language` header, or `set_locale` thread-local equivalent) on the receiving side.
3. If a boundary discards the locale and the receiving side defaults silently to Chinese (or any non-EN locale) under an EN request, the Verification System shall record the boundary as a `gap`.
4. The Verification System shall examine the backend logger to confirm that log messages on the EN code path resolve to English templates (depends on #6).
### Requirement 5: Verification report comment on issue #10
**Objective:** As the issue owner, I want a single canonical verification report posted as a comment on issue #10, so that reviewers can see pass/fail per checklist item and trace every `gap` to a `file:line` and a follow-up issue.
#### Acceptance Criteria
1. When the static audit, parity check, prompt verification, and propagation review are complete, the Verification System shall compose a markdown comment on issue #10 that lists every checklist item from the ticket body with one of the statuses `pass` / `gap` / `manual-pending`.
2. For each `gap` status, the comment shall include `file:line` references and a link to the follow-up issue filed per Requirement 7.
3. For each `manual-pending` status, the comment shall state explicitly that the item requires a live UI walkthrough (or full-stack run) which was not performed in this verification environment, and shall list the exact reproduction steps the next reviewer needs to run.
4. The comment shall include the raw output (or a path to the captured output) of the `git grep` audit so future verifiers can diff against the baseline.
5. The Verification System shall post the comment using `gh issue comment 10 --repo salestech-group/MiroFish` and shall record the resulting comment URL in the spec / commit message.
### Requirement 6: ZH regression check
**Objective:** As an i18n verifier, I want to confirm that the ZH locale still renders fully Chinese, so that the EN work has not regressed the original-language experience.
#### Acceptance Criteria
1. The Verification System shall confirm that `locales/zh.json` covers every key present in `locales/en.json` (Requirement 2) so that no UI string falls back to English under ZH.
2. The Verification System shall confirm that prompts rendered through locale-aware templates produce a Chinese variant when locale=zh (i.e. the templating mechanism is symmetric between EN and ZH).
3. If a UI string is English-only under ZH (i.e. `zh.json` value is identical to the EN value and the value is non-trivial English prose), the Verification System shall flag it per Requirement 2.4 as `review-needed`.
4. The Verification System shall record any ZH-specific regression as a separate `gap` and file a follow-up issue per Requirement 7.
### Requirement 7: Follow-up issues for every discovered gap
**Objective:** As the project owner, I want every gap discovered in this verification pass tracked as its own GitHub issue, so that fixes are sequenced separately and #10 stays scoped to verification only.
#### Acceptance Criteria
1. When a `gap` is recorded by Requirements 1–6, the Verification System shall file a GitHub issue against `salestech-group/MiroFish` containing: a one-sentence summary, the affected pipeline step, the `file:line` evidence, and a link back to issue #10 and to the verification report comment.
2. If grouping is sensible (e.g. five `gap`s in a single locale-catalogue file), the Verification System shall consolidate them into a single follow-up issue with a checklist body, instead of filing five micro-issues.
3. The Verification System shall not patch any gap inline in this ticket; the spec change-set must be limited to the verification artefacts (spec docs + report capture under `.kiro/specs/i18n-e2e-english-verification/`) and must not modify production source files under `backend/app/`, `frontend/src/`, or `locales/`.
4. The Verification System shall label every follow-up issue with the `i18n` label (and `bug` if the gap is regressing existing behaviour) so they aggregate under the i18n epic.
5. If the verification environment cannot file issues (e.g. no `gh` permissions), the Verification System shall list the would-be issues inline in the verification report as a fallback so a human can file them, and shall mark the corresponding checklist item `gap-pending-issue` instead of `gap`.
### Requirement 8: Reproducibility and idempotence
**Objective:** As a future verifier, I want this verification pass to be re-runnable, so that we can re-baseline after each subsequent merge to the i18n epic.
#### Acceptance Criteria
1. The Verification System shall capture the raw audit output to `.kiro/specs/i18n-e2e-english-verification/audit/` so the next verifier can diff against the previous run.
2. While a previous capture exists, the Verification System shall preserve it (timestamped or under a `previous/` subdirectory) rather than overwriting it silently.
3. The Verification System shall record the commit SHA at the time of the audit so the report comment can be tied to a specific tree state.
4. If the audit is re-run and the gap set is unchanged, the Verification System shall produce a no-op report comment that confirms parity rather than spamming a new gap list.

112
.kiro/specs/i18n-e2e-english-verification/research.md Normal file
View File

@ -0,0 +1,112 @@
# Research & Design Decisions — i18n-e2e-english-verification
## Summary
- **Feature**: `i18n-e2e-english-verification`
- **Discovery Scope**: Extension (verification-only against existing i18n surface)
- **Key Findings**:
- `locales/en.json` is already CJK-clean (0 hits) and `locales/zh.json` is at perfect parity (953/953 keys).
- Bulk of remaining CJK is in backend Python source (~26 files across `services/`, `api/`, `utils/`, `models/`) — overwhelmingly docstrings, comments, and a non-trivial number of log strings + LLM-prompt context labels. This is blocked by issue #7 (translate Chinese docstrings/comments).
- Frontend `Process.vue` still has ~65 hard-coded Chinese strings in template/JS literals (not routed through `t()` keys); 4 step components have a smaller surface (mainly Step4Report's regex parsers that match Chinese backend output).
- Live UI/full-stack walkthrough is not feasible in this sandboxed CLI environment — that portion of the verification will be reported as `manual-pending` with reproduction steps.
## Research Log
### Audit baseline
- **Context**: R1 requires running the canonical `git grep` audit and bucketing the matches.
- **Sources consulted**: ripgrep / `git grep -P` against the working tree at `9dcaecd` (HEAD of `docs/i18n-9-translate-frontend-comments`).
- **Findings**:
- Total CJK lines: **2918** across **36** files (counting 2 binary `.jpeg` false positives that ripgrep matches when scanning the assets folder).
- Bucket distribution: `locales/en.json` 0 / `frontend/src` 7 files (5 source + 2 binary) / `backend/app` 29 files.
- The shell-style regex `[\x{4e00}-\x{9fff}]` in the issue body must be passed to `git grep` with `-P` (PCRE) — POSIX ERE rejects `\x{...}` ranges. The verification scripts must use `-P` or document the deviation.
- **Implications**: The audit script must use PCRE; binary files should be excluded explicitly so the `.jpeg` false positives do not pollute the gap report.
### Locale-catalogue parity
- **Context**: R2 demands key-set parity between `en.json` and `zh.json`.
- **Sources consulted**: small Python diff over the catalogues (recursive nested-dict key flattening).
- **Findings**: 953 keys each, symmetric difference 0. Already passing.
- **Implications**: R2.1, R2.2 will trivially pass; R2.4 (untranslated-but-identical entries) still needs running.
### Locale propagation surface
- **Context**: R4 requires confirming that locale survives Flask handler → `Task` → OASIS subprocess → ReACT agent.
- **Sources consulted**: `backend/app/api/graph.py`, `backend/app/services/` skim, CLAUDE.md (mentions `set_locale` thread-local).
- **Findings**:
- `backend/app/api/graph.py` line 385 etc still emit Chinese log strings inline (`build_logger.info(f"[{task_id}] 开始构建图谱...")`) — the log externalisation work (#6) didn't reach these call sites.
- `backend/app/utils/retry.py` log strings are still hard-coded Chinese (`logger.error(f"函数 {func.__name__} ...")`).
- `oasis_profile_generator.py` LLM-prompt context labels (`"事实信息:"`, `"相关实体:"`) feed into the agent prompt verbatim — these will bias the LLM toward Chinese output even under EN locale.
- **Implications**: R4.3 (locale discarded silently → defaults non-EN) has live evidence; multiple `gap` items will be filed.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Pure shell + Python script (Option A) | One-shot scripts in `.kiro/specs/.../audit/scripts/` produce `audit/<sha>/*.txt` and `audit/<sha>/gap-report.md` | Simplest; no production-code touch; easy to re-run; fits R8 capture format | Scoped to this ticket — not a permanent CI guard | Selected |
| Reusable `tools/i18n-audit/` CLI (Option B) | Promote the audit to a permanent project tool wired into CI | Long-term safety net; future PRs would fail on regressions | Out of scope per R7.3 (verification-only); adds new top-level directory | Filed as a follow-up issue, not implemented here |
| Hybrid (Option C) | Run Option A now; file an issue requesting Option B as future work | Captures B's value without bloating this PR | None material | Adopted |
## Design Decisions
### Decision: Audit lives entirely under `.kiro/specs/i18n-e2e-english-verification/`
- **Context**: R7.3 forbids modifying production source in this ticket; the verification artefacts (scripts and captures) need a home.
- **Alternatives considered**:
1. Top-level `tools/i18n-audit/` — rejected (creates a long-lived asset out of a one-shot ticket).
2. `scripts/` next to existing project scripts — rejected (project has no convention for verification scripts; `.kiro/specs/` is the canonical home for spec-scoped work).
3. `.kiro/specs/.../audit/` — selected.
- **Selected approach**: Scripts at `.kiro/specs/i18n-e2e-english-verification/audit/scripts/` and outputs at `.kiro/specs/.../audit/<commit-sha>/`.
- **Rationale**: Co-locates spec, requirements, design, and the artefacts a future verifier needs to re-run the pass. Honours the steering rule that the spec dir is the source of truth for spec-scoped state.
- **Trade-offs**: Scripts aren't reused beyond this ticket. Re-runs require checking out the spec dir (which is committed).
- **Follow-up**: File a follow-up issue suggesting Option B (a permanent CI guard) for the next iteration of the i18n epic.
### Decision: Manual UI walkthrough → `manual-pending`, not `gap`
- **Context**: R5.3 already permits `manual-pending` when a checklist item requires running the live stack. This run is sandboxed CLI — no browser, no Docker.
- **Alternatives considered**:
1. Mark UI items `gap` because they weren't proven — rejected (a `gap` is a *known* failure; UI items are simply untested in this run).
2. Skip them silently — rejected (R5.1 requires every checklist item to have a status).
3. Mark `manual-pending` with reproduction steps — selected.
- **Rationale**: Honest about the verification environment's limits. Future verifiers can flip `manual-pending` to `pass` or `gap` after running the live walkthrough.
- **Trade-offs**: Issue #10 cannot be fully closed by this run alone; the verification-pass comment will say so explicitly.
### Decision: Gap classification = (deliberate / gap / non-applicable / review-needed)
- **Context**: R1.2 lists three classes; R2.4 introduces a fourth (`review-needed`).
- **Alternatives considered**:
1. Three-class only — rejected (forces premature decisions on identical en/zh values).
2. Four-class with explicit semantics — selected.
- **Rationale**: A four-class scheme keeps the `gap` count truthful (it counts only known-bad lines), and `review-needed` is a soft signal that a human should re-check.
- **Trade-offs**: Slightly more complex schema; mitigated by documenting the four labels at the top of `gap-report.md`.
### Decision: Follow-up grouping by category, not by file
- **Context**: R7.2 allows consolidation. There are too many CJK-bearing files (29) to file one issue each.
- **Alternatives considered**:
1. One issue per file — rejected (29 micro-issues).
2. One issue per pipeline step (R1.3 step tag) — feasible but cross-cuts existing per-component issues like #7.
3. One issue per **gap category** — selected: (a) frontend hard-coded UI strings, (b) backend log strings, (c) backend LLM-prompt context labels, (d) recommend a permanent CI check.
- **Rationale**: Categories already align with how the i18n epic broke down work (#3, #4, #5, #6 = LLM-prompts; #7 = docstrings/comments; #9 = frontend comments). Categories also map cleanly to single PRs, which is how subsequent fixes will land.
- **Trade-offs**: Some files appear in multiple categories. Mitigated by listing `file:line` evidence inside each category issue.
### Decision: Issue-comment fallback when `gh` is unavailable
- **Context**: R7.5 mandates a fallback if `gh` permissions are missing.
- **Selected approach**: If `gh` posts fail, the script writes the comment body to `audit/<sha>/PENDING-issue-10-comment.md` and the would-be follow-up issue bodies to `audit/<sha>/PENDING-followups/*.md` so a human can paste them.
- **Rationale**: Keeps the audit re-runnable offline; keeps the artefact set faithful to what *would* have been posted.
- **Trade-offs**: Verification doesn't truly close until a human posts. Surfaced loudly in the run-summary.
## Risks & Mitigations
- **Risk**: A `gap` is mis-classified as `non-applicable` (e.g. a regex character class versus a real Chinese label) → Mitigation: classification tracked in a small CSV alongside the raw grep, so re-classification is auditable.
- **Risk**: `gh` rate limits hit when filing follow-ups → Mitigation: file at most 4 follow-ups (one per category) — far below any rate limit.
- **Risk**: Re-running the audit on a divergent branch produces a noisy diff → Mitigation: `audit/<commit-sha>/` directories preserve history; comparison is opt-in via `diff -ru`.
- **Risk**: Live walkthrough never happens, leaving #10 in `manual-pending` indefinitely → Mitigation: the verification report comment names a concrete "next reviewer" reproduction script; `manual-pending` items have explicit acceptance criteria.
## References
- Issue #10 — https://github.com/salestech-group/MiroFish/issues/10
- Epic #11 — https://github.com/salestech-group/MiroFish/issues/11
- `gap-analysis.md` — bucketed audit baseline
- `requirements.md` — EARS acceptance criteria for this spec

24
.kiro/specs/i18n-e2e-english-verification/spec.json Normal file
View File

@ -0,0 +1,24 @@
{
"feature_name": "i18n-e2e-english-verification",
"created_at": "2026-05-07T18:25:18Z",
"updated_at": "2026-05-07T18:25:18Z",
"language": "en",
"phase": "tasks-generated",
"ticket": 10,
"ticket_url": "https://github.com/salestech-group/MiroFish/issues/10",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}

87
.kiro/specs/i18n-e2e-english-verification/tasks.md Normal file
View File

@ -0,0 +1,87 @@
# Tasks — i18n-e2e-english-verification
## 1. Foundation — audit workspace and entrypoint
- [x] 1.1 Create the audit script directory and the read-only orchestrator skeleton
- Establish `.kiro/specs/i18n-e2e-english-verification/audit/scripts/` with a `run_audit.sh` skeleton that uses `set -euo pipefail`.
- The orchestrator captures HEAD sha (`git rev-parse HEAD`) and creates `.kiro/specs/i18n-e2e-english-verification/audit/<sha>/` as the artefact root.
- Observable completion: running `bash .kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh` from repo root creates an empty `audit/<sha>/` directory and exits `0`.
- _Requirements: 1.4, 7.3, 8.1, 8.2, 8.3, 8.4_
- _Boundary: run_audit.sh_
## 2. Core — read-only audit producers
- [x] 2.1 (P) Implement the canonical CJK grep with PCRE
- `audit_cjk.sh` runs `git grep -nP '[\x{4e00}-\x{9fff}]' -- backend/app frontend/src locales/en.json` and writes the raw output to `<sha>/cjk-grep.txt`.
- Produces a partitioned `<sha>/cjk-grep-bucketed.txt` with one section per top-level path (`backend/app`, `frontend/src`, `locales/en.json`).
- Excludes binary file matches (e.g. `.jpeg`) by skipping paths whose `git check-attr` reports `binary` (or by file-extension allowlist if check-attr is unset).
- Observable completion: `<sha>/cjk-grep.txt` contains exactly the same lines as a manual `git grep -nP …` run, and `<sha>/cjk-grep-bucketed.txt` has the three labelled sections with line counts.
- _Requirements: 1.1, 1.5_
- _Boundary: audit_cjk.sh_
- [x] 2.2 (P) Implement the locale-catalogue parity diff
- `check_parity.py` loads `locales/en.json` and `locales/zh.json`, recursively flattens nested-dict keys with dotted paths, and writes `<sha>/parity.txt` with three labelled blocks: `[missing-keys]`, `[cjk-in-en]`, `[identical-values]`.
- The `[identical-values]` block flags entries only when EN value equals ZH value AND the value is non-empty AND has more than two ASCII words.
- Observable completion: `<sha>/parity.txt` exists; on the current tree `[missing-keys]` is empty and `[cjk-in-en]` is empty (matching the gap-analysis baseline).
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 6.1, 6.3_
- _Boundary: check_parity.py_
- [x] 2.3 Implement the four-class classifier
- `classify.py` consumes `<sha>/cjk-grep.txt` and `<sha>/parity.txt` and writes `<sha>/classified.csv` with columns `file,line,match,class,category,pipeline_step`.
- Implements the closed-set rules from design.md "classify.py": `locales/en.json` CJK → `gap`/`catalogue-parity`; `frontend/src/{views,components}/*.vue` string literal → `gap`/`frontend-ui-string`; `text.match(/.../)` regex pattern with CJK → `gap`/`frontend-regex-parser`; `.py` line starting with `#` or inside a triple-quoted block → `deliberate`/`backend-{comment,docstring}`; `.py` `logger.|log.|print(` line with CJK in a string literal → `gap`/`backend-log` with appropriate step tag; `.py` LLM-prompt label in `services/{ontology,oasis_profile,simulation_config,report_agent}_generator.py` → `gap`/`backend-prompt-label`; binary file → `non-applicable`/`binary-false-positive`; everything else → `review-needed`.
- Asserts row-count equality with the input grep (no silent drops).
- Observable completion: `<sha>/classified.csv` row count == `cjk-grep.txt` line count, and at least one row of each non-empty class is present (verified by counting per-class rows in stdout summary).
- _Requirements: 1.2, 1.3, 1.5, 3.1, 3.2, 3.3, 3.4, 4.3, 4.4, 6.4_
- _Boundary: classify.py_
- _Depends: 2.1, 2.2_
## 3. Core — report assembly
- [x] 3.1 Render the gap report and the issue-#10 comment body
- `render_report.py` reads `<sha>/classified.csv` and `.ticket/10.md`; writes `<sha>/gap-report.md` (with the seven sections from design.md) and `<sha>/comment-body.md` (mirroring the issue's checklist with `pass`/`gap`/`manual-pending` per line + a "How to re-run" footer + a `Run on commit <sha>` header).
- Section 4 of `gap-report.md` enumerates the four propagation boundaries and reports each as `pass`/`gap`/`unknown`, with file:line evidence drawn from `classified.csv`.
- Section 5 maps every checklist item from `.ticket/10.md` to a `pass` / `gap` / `manual-pending` status. UI-checklist items default to `manual-pending` (live walkthrough not feasible in sandbox) and include a concrete reproduction script.
- Always writes the four follow-up issue body templates to `<sha>/PENDING-followups/`: `01-frontend-ui-strings.md`, `02-backend-log-strings.md`, `03-backend-prompt-labels.md`, `04-permanent-ci-guard.md` — empty placeholder if the corresponding category had zero `gap` rows.
- Observable completion: `<sha>/gap-report.md`, `<sha>/comment-body.md`, and `<sha>/PENDING-followups/01..04-*.md` all exist; opening `<sha>/comment-body.md` shows every checkbox from `.ticket/10.md` mapped to a status.
- _Requirements: 4.1, 4.2, 5.1, 5.2, 5.3, 5.4, 6.2_
- _Boundary: render_report.py_
## 4. Integration — orchestrator and GitHub side effects
- [x] 4.1 Wire run_audit.sh to the four producer steps and add the GitHub posting hooks
- `run_audit.sh` invokes (in order) `audit_cjk.sh`, `check_parity.py`, `classify.py`, `render_report.py`, then `post_comment.sh` and `file_followups.sh`.
- On any error in steps 1-4 the orchestrator aborts (`set -euo pipefail`) before any subsequent step runs.
- On `gh` failure in steps 5 or 6, the orchestrator continues to the next step but exits `2` at the end (audit succeeded, side effects didn't fully apply).
- Observable completion: a clean run on the current tree creates a complete `<sha>/` directory; if `gh` is forced absent (e.g. `PATH=$(pwd)/empty bash run_audit.sh`), the orchestrator still produces all four producer artefacts and the `PENDING-followups/` and exits with `2`.
- _Requirements: 1.4, 7.3, 8.1, 8.2, 8.3, 8.4_
- _Boundary: run_audit.sh_
- _Depends: 2.3, 3.1_
- [x] 4.2 Implement post_comment.sh and file_followups.sh with PENDING fallback
- `post_comment.sh` calls `gh issue comment 10 --repo salestech-group/MiroFish --body-file <sha>/comment-body.md`; on failure it copies the body to `<sha>/PENDING-issue-10-comment.md` and exits non-zero. On success it writes the resulting URL to `<sha>/comment-url.txt`.
- `file_followups.sh` iterates `<sha>/PENDING-followups/*.md`; for each non-empty body it calls `gh issue create --repo salestech-group/MiroFish --title <title-from-body-first-line> --body-file <body> --label i18n` (and `--label bug` when the body's frontmatter declares regression). On per-category failure it leaves that body in place; on success it removes the body and appends the issue URL to `<sha>/followup-urls.txt`.
- Observable completion: with `gh` available, the comment URL appears in `<sha>/comment-url.txt` and any non-empty follow-up body produces an issue URL in `<sha>/followup-urls.txt`; with `gh` absent, both bodies stay under `<sha>/PENDING-*` and exit codes are non-zero.
- _Requirements: 5.5, 7.1, 7.2, 7.4, 7.5_
- _Boundary: post_comment.sh, file_followups.sh_
- _Depends: 3.1_
## 5. Validation — execute the verification pass
- [x] 5.1 Execute the audit on the current tree and capture a baseline run
- Run `bash .kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh` from repo root.
- Confirm `<sha>/cjk-grep.txt`, `cjk-grep-bucketed.txt`, `parity.txt`, `classified.csv`, `gap-report.md`, `comment-body.md`, and `PENDING-followups/01..04-*.md` all exist and are non-empty (the placeholders for empty categories may be empty by design).
- Confirm `parity.txt` `[missing-keys]` and `[cjk-in-en]` blocks are empty (matches the gap-analysis baseline).
- Confirm `classified.csv` row count matches `cjk-grep.txt` line count exactly.
- Observable completion: the baseline `<sha>/` directory is committed under `.kiro/specs/i18n-e2e-english-verification/audit/`.
- _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 8.1, 8.3_
- _Boundary: run_audit.sh and producer scripts_
- _Depends: 4.1_
- [x] 5.2 Post the comment on issue #10 and file the follow-up issues
- Run `post_comment.sh <sha-dir>` and `file_followups.sh <sha-dir>` (or rely on `run_audit.sh` to invoke them) so the verification report comment is posted and follow-up issues are filed for non-empty categories.
- Capture `comment-url.txt` and `followup-urls.txt` under `<sha>/` so the PR description can link to them.
- If `gh` lacks permissions for any of the calls, the corresponding `PENDING-*` file is left in place per R7.5; the run summary surfaces the partial state.
- Observable completion: a comment appears on https://github.com/salestech-group/MiroFish/issues/10 mirroring `comment-body.md`; follow-up issues for non-empty categories exist and carry the `i18n` label.
- _Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 6.4, 7.1, 7.2, 7.4, 7.5_
- _Boundary: post_comment.sh, file_followups.sh_
- _Depends: 4.2, 5.1_

286
.kiro/specs/i18n-externalize-remaining-backend-logs/design.md Normal file
View File

@ -0,0 +1,286 @@
# Design Document
## Overview
**Purpose**: Replace the last nine hard-coded Chinese log/print strings in three backend modules (`backend/app/api/graph.py`, `backend/app/services/oasis_profile_generator.py`, `backend/app/utils/retry.py`) with calls to the existing `t("log.<domain>.<key>", **kwargs)` helper, and add the corresponding entries to `locales/en.json` and `locales/zh.json`. The result is locale-correct backend logs with zero behavioural drift.
**Users**: Backend operators reading logs in English deployments; existing Chinese-locale operators (preserved verbatim).
**Impact**: Removes the last sources of Chinese-text leakage in backend logs under the `en` locale, completing the i18n coverage started by ticket #6.
### Goals
- Replace the nine f-string arguments listed in ticket #24 with `t("log.<domain>.<key>", **kwargs)` calls.
- Add eleven new locale entries (3 in `log.graph_api`, 2 in `log.profile_generator`, 4 in new `log.retry`) to both `locales/en.json` and `locales/zh.json` with key parity.
- Preserve all interpolated values, all log levels, all control flow, and all `print(...)` console banners.
### Non-Goals
- Translating other Chinese strings in the same files (docstrings, comments, `update_task` messages, `progress_callback` messages, `logger.warning` retry messages) — out of scope for ticket #24.
- Modifying the `t()` helper, the locale resolution logic, or the locale dictionary structure (other than adding the listed keys).
- Frontend `vue-i18n` translation work or schema changes to `locales/{en,zh}.json`.
- Adding test infrastructure, the `run_audit.sh` script, or any new dev dependency.
## Boundary Commitments
### This Spec Owns
- The string-literal contents of nine specific `logger.{info,error}` and `print(...)` call sites (exact `file:line` listed in Requirement 1).
- Eleven new translation entries in `locales/en.json` and `locales/zh.json`.
- The new `log.retry` sub-namespace under the existing top-level `log` key.
### Out of Boundary
- Other Chinese strings in the three modified files.
- Any change to public API contracts, log levels, or response payloads.
- Any change to the `t()` helper or the per-thread / per-request locale resolution logic.
- Frontend `zh.json` entries beyond the ones this spec must add for backend parity (i.e., none — frontend keys are untouched).
### Allowed Dependencies
- `backend/app/utils/locale.py` (`t`) — already in use, just import it where needed.
- The existing locale dictionaries `locales/{en,zh}.json` — extend, don't re-organise.
- `get_logger` from `backend/app/utils/logger.py` — already imported by `retry.py`.
### Revalidation Triggers
- Renaming `t()` or moving it to a different module.
- Changing the placeholder syntax in `t()` from `{name}` to anything else.
- Restructuring `locales/en.json` / `zh.json` (e.g., flattening `log.<domain>.m###` into a flat key tree).
## Architecture
### Existing Architecture Analysis
This spec extends a pattern already established by ticket #6 (`i18n-externalize-backend-logs`). The convention is:
1. Source-code call sites use `t("log.<domain>.m###", placeholder=value, …)` instead of `f"…{value}…"`.
2. Each `t()` key has matching entries in `locales/en.json` (English copy) and `locales/zh.json` (verbatim original Chinese).
3. Placeholders use `{name}` (replaced via `str.replace` inside `t()`).
4. The locale is resolved per request (`Accept-Language`) or per thread (`set_locale`); `'zh'` is the default fallback; missing keys return the key string and emit a deduped warning.
The constraint: only the nine listed call sites change. No new architecture, no new component, no new integration point.
### Architecture Pattern & Boundary Map
The change is a **pure string-externalisation extension** of the existing localisation pattern. No new components, no new flows, no new dependencies. The only structural addition is a new `log.retry` sub-namespace inside the existing top-level `log` key in the locale dictionaries.
```mermaid
flowchart LR
A[graph.py:385/494/513<br/>build_logger.{info,error}] -->|t("log.graph_api.mNNN", ...)| L[t() helper<br/>backend/app/utils/locale.py]
B[oasis_profile_generator.py:945/1001<br/>print(...)] -->|t("log.profile_generator.mNNN", ...)| L
C[retry.py:55/108/179/227<br/>logger.error] -->|t("log.retry.mNNN", ...)| L
L --> EN[locales/en.json<br/>log.graph_api.m027-m029<br/>log.profile_generator.m024-m025<br/>log.retry.m001-m004]
L --> ZH[locales/zh.json<br/>same key paths<br/>verbatim Chinese values]
```
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | Python ≥3.11 | Source-language change site | No version change |
| Backend / Services | `backend/app/utils/locale.py` (project-internal) | Provides `t(key, **kwargs)` | Reused as-is |
| Data / Storage | `locales/en.json`, `locales/zh.json` | Adds 11 new key/value pairs | Flat JSON, UTF-8 |
| Infrastructure / Runtime | Flask 3.0 / asyncio | Locale resolution context | No runtime change |
## File Structure Plan
### Modified Files
- `backend/app/api/graph.py` — Replace the f-string argument of three `build_logger.{info,error}` calls (lines 385, 494, 513) with `t("log.graph_api.<key>", **kwargs)`. No new imports (already imports `t` on line 21).
- `backend/app/services/oasis_profile_generator.py` — Replace the f-string argument of two `print(...)` calls (lines 945, 1001) with `t("log.profile_generator.<key>", **kwargs)`. No new imports (already imports `t` on line 23).
- `backend/app/utils/retry.py` — Add `from .locale import t` (or `from ..utils.locale import t`, matching the project's existing relative-import style). Replace the f-string argument of four `logger.error` calls (lines 55, 108, 179, 227) with `t("log.retry.<key>", **kwargs)`.
- `locales/en.json` — Append three keys to `log.graph_api`, two to `log.profile_generator`, and a new `log.retry` sub-namespace with four keys.
- `locales/zh.json` — Mirror the same key paths with verbatim original Chinese strings.
No new files. No deleted files.
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | Replace `graph.py` log strings via `t()` | `graph.py` build-task closure | `t("log.graph_api.<key>", ...)` | Build pipeline log emission |
| 1.2 | Replace `oasis_profile_generator.py` banner prints via `t()` | `OasisProfileGenerator.generate_profiles_parallel` | `t("log.profile_generator.<key>", ...)` | Profile-generation banner |
| 1.3 | Replace `retry.py` errors via `t()` (new `log.retry` namespace) | `retry_with_backoff`, `retry_with_backoff_async`, `RetryableAPIClient` | `t("log.retry.<key>", ...)` | Retry-failure path |
| 1.4 | Preserve interpolated values via kwargs | All three modules | `t(key, name=value, ...)` with `{name}` placeholders | All log emission |
| 1.5 | Zero CJK in the listed lines after change | Same as 1.1–1.3 | n/a | n/a |
| 2.1, 2.2 | Add 11 new keys to `en.json` and `zh.json` | Locale dictionaries | JSON file edits | n/a |
| 2.3 | Use next available `m###` slot per namespace | Locale dictionaries | n/a | n/a |
| 2.4 | Structural parity across both files | Locale dictionaries | Verification script | n/a |
| 2.5 | No new top-level keys; no existing keys touched | Locale dictionaries | n/a | n/a |
| 3.1 | Graph build pipeline behaves identically | `graph.py` build-task closure | n/a | Build pipeline |
| 3.2 | Profile generator continues to print exactly two banners | `oasis_profile_generator.py` | n/a | Banner emission |
| 3.3 | Retry semantics unchanged (raise, sleep, level, position) | `retry.py` | n/a | Retry path |
| 3.4 | HTTP responses unchanged | All API endpoints | n/a | n/a |
| 4.1, 4.2, 4.3, 4.4 | Locale resolution works in all contexts | `t()` helper (unchanged) | n/a | n/a |
| 5.1 | CJK regex audit on the nine lines passes | Verification procedure | `grep -P "[一-鿿]"` | n/a |
| 5.2 | Key-parity audit passes | Verification procedure | Python `json.load` walk | n/a |
| 5.3 | Placeholder-integrity audit passes | Verification procedure | Python regex check | n/a |
| 5.4 | Only stock tooling | Verification procedure | `grep`, `python3` | n/a |
| 5.5 | `pytest` continues to pass | Backend test suite | `uv run python -m pytest` | n/a |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|--------------|--------------------------|-----------|
| `graph.py` build-task closure | Backend / API | Log graph-build start/complete/fail in active locale | 1.1, 1.4, 1.5, 3.1 | `t()` (P0), `build_logger` (P0) | Behaviour-only |
| OASIS banner prints | Backend / Services | Print banner around parallel profile generation | 1.2, 1.4, 1.5, 3.2 | `t()` (P0) | Console-output |
| Retry error logs | Backend / Utils | Log final-failure errors after retry exhaustion | 1.3, 1.4, 1.5, 3.3 | `t()` (P0), `logger` (P0) | Behaviour-only |
| Locale dictionaries | Backend / Data | Provide en/zh strings for new keys | 2.1–2.5 | JSON parse (P0) | Data |
### Backend / Services
#### `graph.py` build-task closure
| Field | Detail |
|-------|--------|
| Intent | Emit "build started", "build completed", "build failed" log records using `t()` |
| Requirements | 1.1, 1.4, 1.5, 3.1 |
**Responsibilities & Constraints**
- Replace three f-string log arguments only.
- Do not change log level, log handler, control flow, or surrounding `task_manager.update_task(...)` calls.
**Dependencies**
- Inbound: called from `task_manager.run_task` (P0)
- Outbound: `t()` (P0), `build_logger.{info,error}` (P0)
**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ] ← (none — purely behavioural)
**Key Mapping**
| Line | Existing source | New key | EN translation | ZH translation |
|------|-----------------|---------|----------------|----------------|
| 385 | `f"[{task_id}] 开始构建图谱..."` | `log.graph_api.m027` | `[{task_id}] Starting graph build...` | `[{task_id}] 开始构建图谱...` |
| 494 | `f"[{task_id}] 图谱构建完成: graph_id={graph_id}, 节点={node_count}, 边={edge_count}"` | `log.graph_api.m028` | `[{task_id}] Graph build completed: graph_id={graph_id}, nodes={node_count}, edges={edge_count}` | `[{task_id}] 图谱构建完成: graph_id={graph_id}, 节点={node_count}, 边={edge_count}` |
| 513 | `f"[{task_id}] 图谱构建失败: {str(e)}"` | `log.graph_api.m029` | `[{task_id}] Graph build failed: {e}` | `[{task_id}] 图谱构建失败: {e}` |
**Implementation Notes**
- `t` is already imported at `graph.py:21`.
- Use `e=str(e)` to maintain the existing exception-string semantics.
#### OASIS banner prints (`oasis_profile_generator.py`)
| Field | Detail |
|-------|--------|
| Intent | Wrap the two banner-print arguments in `t()` while leaving the surrounding `'='*60` separator prints intact |
| Requirements | 1.2, 1.4, 1.5, 3.2 |
**Responsibilities & Constraints**
- Replace only the *content* line of each banner (the line at 945 and the line at 1001). The two `'='*60` separator prints around them (lines 944/946 and 1000/1002) contain only ASCII and stay verbatim.
- Do not remove either `print(...)` call.
- Do not modify the existing `logger.info(t("log.profile_generator.m017", …))` at line 943.
**Key Mapping**
| Line | Existing source | New key | EN translation | ZH translation |
|------|-----------------|---------|----------------|----------------|
| 945 | `f"开始生成Agent人设 - 共 {total} 个实体，并行数: {parallel_count}"` | `log.profile_generator.m024` | `Starting agent profile generation — {total} entities, parallelism: {parallel_count}` | `开始生成Agent人设 - 共 {total} 个实体，并行数: {parallel_count}` |
| 1001 | `f"人设生成完成！共生成 {len([p for p in profiles if p])} 个Agent"` | `log.profile_generator.m025` | `Profile generation complete — generated {count} agents` | `人设生成完成！共生成 {count} 个Agent` |
**Implementation Notes**
- The expression `len([p for p in profiles if p])` becomes a kwarg: `count=len([p for p in profiles if p])`. This is a single name, easier for the locale dictionaries.
- `t` is already imported at `oasis_profile_generator.py:23`.
#### Retry error logs (`retry.py`)
| Field | Detail |
|-------|--------|
| Intent | Localise the four "final-failure" `logger.error` strings; introduce `log.retry` sub-namespace |
| Requirements | 1.3, 1.4, 1.5, 3.3, 4.1–4.4 |
**Responsibilities & Constraints**
- Add `from ..utils.locale import t` at the top of `retry.py` (matching the relative-import depth used by other `backend/app/utils/*` files).
- Replace four f-string `logger.error(...)` arguments only.
- Do not touch the `logger.warning(...)` retry-attempt messages (out of scope per ticket #24).
- Do not change exception handling, control flow, or the public decorator/class signatures.
**Key Mapping**
| Line | Existing source | New key | EN translation | ZH translation |
|------|-----------------|---------|----------------|----------------|
| 55 | `f"函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}"` | `log.retry.m001` | `Function {func_name} still failing after {max_retries} retries: {e}` | `函数 {func_name} 在 {max_retries} 次重试后仍失败: {e}` |
| 108 | `f"异步函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}"` | `log.retry.m002` | `Async function {func_name} still failing after {max_retries} retries: {e}` | `异步函数 {func_name} 在 {max_retries} 次重试后仍失败: {e}` |
| 179 | `f"API调用在 {self.max_retries} 次重试后仍失败: {str(e)}"` | `log.retry.m003` | `API call still failing after {max_retries} retries: {e}` | `API调用在 {max_retries} 次重试后仍失败: {e}` |
| 227 | `f"处理第 {idx + 1} 项失败: {str(e)}"` | `log.retry.m004` | `Failed processing item #{index}: {e}` | `处理第 {index} 项失败: {e}` |
**Implementation Notes**
- Use kwargs `func_name=func.__name__`, `max_retries=max_retries` (or `self.max_retries`), `index=idx + 1`, `e=str(e)`.
- Locale resolution at the call site: in Flask request scope → `Accept-Language`; in background tasks → `set_locale` per-thread; in async coroutines → per-thread (asyncio shares the OS thread). Default fallback is `'zh'`. No new wiring needed (Requirement 4).
### Backend / Data
#### Locale dictionaries
| Field | Detail |
|-------|--------|
| Intent | Provide en/zh strings for the eleven new keys with structural parity |
| Requirements | 2.1, 2.2, 2.3, 2.4, 2.5 |
**Responsibilities & Constraints**
- Append to existing `log.graph_api` and `log.profile_generator` sub-namespaces.
- Add a new `log.retry` sub-namespace as a sibling of the others.
- No top-level key additions; no modifications to any pre-existing key.
- Maintain UTF-8 encoding and the file's existing 2-space indent style.
**Implementation Notes**
- Use `python3 -m json.tool` (or equivalent) to round-trip the JSON files after editing, to ensure formatting consistency.
- Validate parity with a small Python script that recursively compares key paths.
## System Flows
(Skipped — no non-trivial flow change. The build / profile / retry call paths execute as before; only the message text source language differs.)
## Error Handling
### Error Strategy
This spec changes only message-string sources. Error-handling semantics in the touched code are preserved:
- `graph.py:513` continues to set `project.status = ProjectStatus.FAILED` and call `task_manager.update_task(..., status=TaskStatus.FAILED, ...)` after the `build_logger.error(...)` call.
- `retry.py` continues to `raise` the underlying exception after the final `logger.error(...)`.
- The `t()` helper does not raise on missing keys — it returns the key string and emits a deduped warning. This contract is unchanged.
### Error Categories and Responses
Out of scope — no new error category is introduced.
## Testing Strategy
### Unit / Integration Tests
The project does not currently maintain a comprehensive backend unit-test suite for these modules. The change is verified mechanically rather than via new pytest tests:
1. **CJK absence on the touched lines** — `grep -nP "[一-鿿]"` against the nine specific lines must return no matches.
2. **JSON parse + key parity** — a small inline Python check that loads `locales/{en,zh}.json` and asserts every newly-added key path exists in both files.
3. **Placeholder integrity** — for each new key, every `{name}` placeholder in the `zh` value must also appear in the `en` value (and vice versa).
4. **Existing test suite** — `uv run python -m pytest` continues to pass; ticket #6's tests at `backend/scripts/test_profile_format.py` are not affected by this work.
### Manual Smoke Test
After implementation:
- Set `Accept-Language: en` and run an end-to-end graph build via the local Flask app (`npm run dev`); confirm the start / complete / fail log lines render in English.
- Run a profile generation flow and observe the banner prints in English.
- Force a retry exhaustion (e.g., temporarily lower `max_retries=0` and trigger an error) and confirm the `log.retry` message renders in English.
(Manual smoke is documentation-only; not a blocker for merging.)
## Optional Sections
### Security Considerations
None. No auth, no PII, no external integration changes. The exception text in log messages was already exposed via the previous f-string formatting; routing it through `t()` does not change the surface.
### Performance & Scalability
Negligible. `t()` is an in-memory dict lookup with `str.replace` for placeholders; cost is below noise floor for log emission.

124
.kiro/specs/i18n-externalize-remaining-backend-logs/gap-analysis.md Normal file
View File

@ -0,0 +1,124 @@
# Implementation Gap Analysis
## 1. Codebase Findings
### 1.1 Existing infrastructure already covers the i18n mechanics
- `backend/app/utils/locale.py` already exports `t(key, **kwargs)` with:
- per-thread locale (`set_locale` writes `_thread_local.locale`)
- per-request locale (`get_locale` checks Flask `has_request_context()` then `Accept-Language`)
- `zh` fallback when the active locale is missing a key, then key-string fallback if `zh` is missing too
- dedup'd warning on missing keys (`_warn_missing_key_once`), no exceptions raised
- All wiring required by Requirement 4 is therefore already in place. **No `locale.py` change is needed for ticket #24.**
### 1.2 The two files we touch already use `t()`
- `backend/app/api/graph.py:21` — `from ..utils.locale import t`
- `backend/app/services/oasis_profile_generator.py:23` — `from ..utils.locale import get_language_instruction, get_locale, set_locale, t`
The third file does NOT yet import `t`:
- `backend/app/utils/retry.py` — no `from ..utils.locale import t`. Need to add the import.
### 1.3 Existing locale namespace shape (from `locales/en.json`)
- `log.graph_api` — populated `m006`–`m019, m026`. Next free slots that are *contiguous* would be `m027`, `m028`, `m029`. (Could also reuse `m009, m010, m012, m020–m025` since they are absent, but it is safer to append at the tail to avoid colliding with any unmerged work assuming a particular reservation.)
- `log.profile_generator` — populated `m001`–`m023` densely. Next free: `m024`, `m025`.
- `log.retry` — does NOT exist. Will be created with `m001`–`m004`.
The `log.profile_generator.m017` key already covers a *similar* message ("Starting parallel generation of {total} agent profiles (parallelism: {parallel_count})…"). The `print(...)` at `oasis_profile_generator.py:945` and the `logger.info(t("log.profile_generator.m017", ...))` at line 943 are emitting the same logical event in two channels — log + console banner. The cleanest move is **not** to reuse `m017` (which would lose the banner-style separator/centring) but to introduce dedicated `m024` / `m025` keys for the banner text, so the banner has its own copy decoupled from the log line.
### 1.4 Translation pattern already established by ticket #6
Per the prior spec at `.kiro/specs/i18n-externalize-backend-logs/`, the project's convention is:
- `t("log.<domain>.m###", placeholder=value, …)` inside `logger.{info,warning,error,debug,exception}` calls.
- Placeholders use `{name}` syntax (replaced via `str.replace` inside `t()`); positional `{0}`/`{}` are not supported.
- f-string formatting must be removed entirely from the call argument; values are passed as kwargs.
- The Chinese source string is preserved verbatim in `zh.json`, with `f"…{var}…"` rewritten as `"…{var}…"`.
This work strictly extends the existing pattern. **No new convention is introduced.**
### 1.5 `build_logger` vs. module logger
In `graph.py`, the affected calls use a locally-created `build_logger = get_logger('mirofish.build')` inside the `build_task` background function (lines 383). This is a different logger handle, but `t()` is logger-agnostic — it returns a string that any logger can format. No special handling needed.
### 1.6 `print(...)` calls in `oasis_profile_generator.py`
The two banner prints (lines 945 and 1001) are deliberate console-output decorations (visible on stdout for the Flask process), separate from the structured log emitted by `logger.info` on lines 943 and earlier. The task is to keep them as `print(...)` but route the message text through `t(...)`:
```python
print(t("log.profile_generator.m024", total=total, parallel_count=parallel_count))
```
This preserves the user-visible banner cosmetics (`'='*60` separators on lines 944, 946, 1000, 1002) and only changes the text content.
### 1.7 Locale resolution for `retry.py`
`retry.py` is invoked from three contexts:
1. **Flask request handlers (sync)** — `has_request_context()` is true; `get_locale()` reads `Accept-Language`. Works.
2. **Background tasks** — the existing background-task entry points (e.g., `task_manager.run_task`) already call `set_locale(...)` per `i18n-externalize-backend-logs` (verified by reading `oasis_profile_generator.py` which uses the same pattern with `set_locale` imported on line 23). Works.
3. **Async coroutines (`retry_with_backoff_async`)** — `get_locale()` falls back to `_thread_local.locale`. Asyncio runs coroutines on the same thread by default, so the per-thread locale propagates. If the coroutine is dispatched onto a fresh executor thread without `set_locale`, the helper falls back to `zh` (the default) — still a valid string, just defaulting to Chinese. The default-fallback is acceptable here because (a) the helper still returns a non-None string, and (b) the audit only requires the *source code* to be free of Chinese literals, not that every emitted log record be English regardless of caller context.
**Decision:** No new locale-propagation wiring needed. Document the async fallback in the design and tasks.
## 2. Out-of-scope items (encountered during research)
These were observed in the same files but are explicitly **not** part of ticket #24 and will not be addressed:
- `backend/app/api/graph.py` — Chinese in `task_manager.update_task(..., message="初始化图谱构建服务...")` and similar (#24 lists only the three log calls).
- `backend/app/utils/retry.py` — Chinese in `logger.warning(...)` retry messages (lines 63–66, 115–117, 185–187) and Chinese docstrings (lines 1–3, 25–35, 36–39, 90, 156–166, 200–212).
- `backend/app/services/oasis_profile_generator.py` — Chinese in `progress_callback(... f"已完成 …")` (line 976) and Chinese docstrings/comments throughout.
These are tracked under sibling tickets (#7 for docstrings/comments; the residual `logger.warning` in `retry.py` is a candidate for a future audit ticket).
## 3. Implementation Approaches Considered
### Approach A — Append-at-tail with new `log.retry` namespace (recommended)
- New keys: `log.graph_api.m027`, `m028`, `m029`; `log.profile_generator.m024`, `m025`; new `log.retry.m001`–`m004`.
- Add `from ..utils.locale import t` to `retry.py`.
- Replace each f-string in the nine call sites with a `t(...)` call.
- Update `locales/en.json` and `locales/zh.json` in lock-step.
- **Pros:** Mirrors the conventions of #6 exactly; no risk of overwriting existing keys; minimal diff.
- **Cons:** Numbering gaps under `log.graph_api` remain (cosmetic).
### Approach B — Fill numbering gaps in `log.graph_api`
- Reuse missing slots `m009`, `m010`, `m012`, `m020`–`m025`.
- **Pros:** Tighter numbering.
- **Cons:** Risk of colliding with reserved-but-not-yet-merged keys from another branch; harder to review (mixed insertion sites in JSON).
- **Verdict:** Reject. The cost of conflict review is not worth the cosmetic gain.
### Approach C — Consolidate the `print(...)` banners into the existing `log.profile_generator.m017`
- Remove the two `print(...)` calls; rely solely on `logger.info(t(...))`.
- **Pros:** One fewer key to add.
- **Cons:** Deletes user-visible console banner behaviour (a behaviour change), violates Requirement 3.2 ("continue to print exactly two banner messages"), and is out-of-scope per ticket #24 which says "fixed (or explicitly classified as `deliberate`)" — i.e., translate, don't remove.
- **Verdict:** Reject.
## 4. Recommendation
Proceed with **Approach A**.
Implementation will:
1. Add four entries to `log.retry` (new sub-namespace) — one per `logger.error` line in `retry.py`.
2. Add three entries to `log.graph_api` — one per `build_logger` line in `graph.py`.
3. Add two entries to `log.profile_generator` — one per `print(...)` banner in `oasis_profile_generator.py`.
4. Replace all nine f-strings with `t(...)` calls; pass interpolated values as kwargs.
5. Add `from ..utils.locale import t` to `retry.py`.
6. Mirror every new key in `zh.json` with the verbatim original Chinese text.
7. Run a regex / Python audit to confirm parity and absence of CJK on the touched lines.
## 5. Risks / open questions
| Risk | Severity | Mitigation |
|---|---|---|
| `retry.py` async path running on a fresh thread without `set_locale` returns Chinese | Low | Documented; not a blocker for #24 acceptance, which targets *source-code* CJK absence. Any improvement is a separate ticket. |
| Adding `from ..utils.locale import t` introduces a new module import into `retry.py` (low-level utility) | Low | The `locale` module has no transitive imports of `retry.py`, so no circular-import risk. Verified by reading `locale.py`. |
| Existing test that asserts Chinese log text breaks | Low | Searched for `"开始构建图谱"` / `"图谱构建完成"` / `"图谱构建失败"` / `"开始生成Agent人设"` / `"人设生成完成"` / `"重试后仍失败"` / `"处理第"` test fixtures — none found in `backend/`. |
## 6. Conclusion
**Ready to proceed to design.** The gap is small: nine string-literal replacements, eleven new locale entries, one new import. The mechanics are identical to the already-merged ticket #6 work. No design uncertainty remains; design phase will simply formalise the key-naming and the per-file edit plan.

91
.kiro/specs/i18n-externalize-remaining-backend-logs/requirements.md Normal file
View File

@ -0,0 +1,91 @@
# Requirements Document
## Introduction
After ticket #6 externalised most backend log/print messages into the project's `t()` localization helper, a small set of call sites in three modules still emit hard-coded Chinese strings. As a result, English operators reading backend logs under the `en` locale see Chinese text leaking from these residual sites. This spec finishes the job for ticket #24 by routing every remaining hard-coded Chinese log/print string in `backend/app/api/graph.py`, `backend/app/services/oasis_profile_generator.py`, and `backend/app/utils/retry.py` through `t("log.<domain>.<key>", **fmt)` and adding the corresponding entries to `locales/en.json` and `locales/zh.json`. The goal is locale-correct backend logs with zero behavioural drift in HTTP responses, control flow, or interpolated values.
## Boundary Context
- **In scope**:
- Replace the Chinese string literals in the nine call sites listed by ticket #24:
- `backend/app/api/graph.py:385` — `build_logger.info(f"[{task_id}] 开始构建图谱...")`
- `backend/app/api/graph.py:494` — `build_logger.info(f"[{task_id}] 图谱构建完成: graph_id={graph_id}, 节点={node_count}, 边={edge_count}")`
- `backend/app/api/graph.py:513` — `build_logger.error(f"[{task_id}] 图谱构建失败: {str(e)}")`
- `backend/app/services/oasis_profile_generator.py:945` — `print(f"开始生成Agent人设 - 共 {total} 个实体，并行数: {parallel_count}")`
- `backend/app/services/oasis_profile_generator.py:1001` — `print(f"人设生成完成！共生成 {len([p for p in profiles if p])} 个Agent")`
- `backend/app/utils/retry.py:55` — `logger.error(f"函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}")`
- `backend/app/utils/retry.py:108` — `logger.error(f"异步函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}")`
- `backend/app/utils/retry.py:179` — `logger.error(f"API调用在 {self.max_retries} 次重试后仍失败: {str(e)}")`
- `backend/app/utils/retry.py:227` — `logger.error(f"处理第 {idx + 1} 项失败: {str(e)}")`
- Add new locale keys for the externalised strings to both `locales/en.json` (English) and `locales/zh.json` (verbatim original Chinese) under the existing top-level `log.<domain>` namespaces (`log.graph_api`, `log.profile_generator`, and a new `log.retry`).
- Pass interpolated values (`task_id`, `graph_id`, `node_count`, `edge_count`, `total`, `parallel_count`, `func_name`, `max_retries`, `idx`, exception text, etc.) through `t()` keyword arguments using the helper's `{name}` placeholder syntax.
- **Out of scope**:
- Other Chinese strings in the same files that are not on the ticket's evidence list (Chinese docstrings, Chinese inline comments, the `task_manager.update_task(... message="...")` Chinese values in `graph.py`, the `logger.warning("…重试…")` calls in `retry.py`, and the in-loop `progress_callback(... f"已完成 …")` and `print(f"-" * 70 …)` decorations in `oasis_profile_generator.py`). Those are tracked elsewhere (#7 for docstrings/comments; #25 for prompt/context labels; future audit may pick up the remaining warning-level retry strings under a separate ticket).
- Any change to log levels, response status codes, control flow, public API surface, or to the `t()` helper itself.
- Adding a new locale or changing the per-thread / per-request locale resolution.
- Frontend `vue-i18n` files; this spec touches only backend usage of `t()` and the shared `locales/{en,zh}.json`.
- **Adjacent expectations**:
- The `t()` helper at `backend/app/utils/locale.py` already covers `set_locale`, `get_locale`, missing-key fallback, and per-thread locale (verified by ticket #6). New code reuses it without modification.
- The two top-level `log` sub-namespaces `log.graph_api` and `log.profile_generator` already exist in `locales/en.json` / `locales/zh.json` with `m###` numeric suffixes; new keys must use the next available `m###` slot in each existing namespace and must not collide with or overwrite any existing key.
- `retry.py` is module-level shared infrastructure used from request handlers, background tasks, and async coroutines — locale resolution must continue to work in each of these contexts without new wiring (Requirement 4 below documents this explicitly so behaviour is mechanically verified).
- Ticket #24's acceptance criterion mentions a verification script under `.kiro/specs/i18n-e2e-english-verification/audit/scripts/run_audit.sh`. That script is not present in the repository at this commit; this spec substitutes a deterministic regex audit (see Requirement 5) that is runnable from the repo root with `grep` + `python` only and that any future `run_audit.sh` can incorporate.
## Requirements
### Requirement 1: Externalise Remaining Chinese Log/Print Strings via `t()`
**Objective:** As a backend operator viewing logs under the `en` locale, I want every Chinese log/print string in the nine listed call sites to be emitted via the existing `t()` helper, so that backend logs no longer leak Chinese text in English deployments.
#### Acceptance Criteria
1. The Backend Logging Layer shall replace the f-string argument of each of the three `build_logger.{info,error}` calls in `backend/app/api/graph.py` at lines 385, 494, and 513 with `t("log.graph_api.<key>", task_id=task_id, ...)`, where the key is a new entry under the existing `log.graph_api` namespace.
2. The Backend Logging Layer shall replace the f-string argument of each of the two `print(...)` calls in `backend/app/services/oasis_profile_generator.py` at lines 945 and 1001 with `print(t("log.profile_generator.<key>", ...))`, keeping the `print` call (so console-output behaviour is preserved) but routing the message text through `t()` under the existing `log.profile_generator` namespace.
3. The Backend Logging Layer shall replace the f-string argument of each of the four `logger.error` calls in `backend/app/utils/retry.py` at lines 55, 108, 179, and 227 with `t("log.retry.<key>", **kwargs)`, introducing a new top-level sub-namespace `log.retry` that mirrors the structure of the other `log.<domain>` sub-namespaces.
4. The Backend Logging Layer shall preserve every interpolated value (`task_id`, `graph_id`, `node_count`, `edge_count`, `total`, `parallel_count`, `func.__name__`, `max_retries`, `idx`, exception text) by passing them as keyword arguments to `t(...)` and referencing them via `{name}` placeholders inside the locale dictionaries; no `f"..."` formatting, `%`-formatting, or string concatenation shall remain around the call.
5. The Backend Logging Layer shall not contain any Chinese character (Unicode range `U+4E00`–`U+9FFF`) inside the string-literal argument of any `logger.{info,warning,error,debug,exception}`, `build_logger.{info,warning,error,debug,exception}`, or `print(...)` call at the nine listed line locations after the change.
### Requirement 2: Locale Dictionary Parity for the New Keys
**Objective:** As a translator or developer adding a new locale, I want every newly externalised key to exist in both `locales/en.json` and `locales/zh.json` with identical nested structure, so that the locale files remain mechanically diffable.
#### Acceptance Criteria
1. The Locale Dictionary shall add, in `locales/en.json`, an English translation for every key introduced by Requirement 1, placed under the relevant `log.<domain>` sub-namespace (`log.graph_api`, `log.profile_generator`, or the new `log.retry`).
2. The Locale Dictionary shall add, in `locales/zh.json`, the original Chinese text (verbatim, with `{placeholder}` substitutions where the source had `f"…{var}…"`) for every key introduced by Requirement 1, under the same key path used in `en.json`.
3. The Locale Dictionary shall use the next available `m###` numeric suffix per existing sub-namespace (so it does not overwrite or shadow any pre-existing `log.graph_api.m###` or `log.profile_generator.m###` key); the new `log.retry` sub-namespace shall start its keys at `m001`.
4. The Locale Dictionary shall expose a structurally identical key tree across `locales/en.json` and `locales/zh.json` for every newly added key path: a recursive comparison of the two files' key paths (ignoring values) shall produce an empty difference for the keys this spec introduces.
5. The Locale Dictionary shall not introduce a new top-level key (the only addition is the new `log.retry` sub-key under the existing top-level `log` namespace) and shall not modify, remove, or re-order any existing key already present in `locales/{en,zh}.json`.
### Requirement 3: Behavioural and Functional Equivalence
**Objective:** As a reviewer, I want to confirm that swapping the message strings does not change runtime behaviour, so that this PR is purely a localisation change.
#### Acceptance Criteria
1. The Graph Build Pipeline shall, after the change, continue to: update `project.status` to `GRAPH_BUILDING` then `GRAPH_COMPLETED` (or `FAILED` on error), call `task_manager.update_task(...)` with the same status/progress/result payloads, and emit one log record at each of the three pre-existing log points (start, completion, failure) with identical level (`info`/`info`/`error`) and identical interpolated values; only the human-readable text and its language source shall differ.
2. The Profile Generator shall, after the change, continue to print exactly two banner messages around `concurrent.futures.ThreadPoolExecutor`-driven generation (one before, one after), retain the surrounding `'='*60` separator lines verbatim, and not emit additional log records or alter the order of `logger.info`/`logger.warning` calls.
3. The Retry Utility shall, after the change, continue to: raise the original exception after the final retry, sleep for the same backoff durations, and emit exactly one `logger.error` per call site at the same control-flow position; the helper's signature, decorator behaviour, and async/sync split shall be unchanged.
4. The Backend HTTP Layer shall return the same HTTP status code, response key set, and (for non-translated keys) value structure for `/api/graph/build` and any other endpoint that transitively triggers the touched code paths; no `jsonify(...)` payload shape shall change as a side-effect of this work.
### Requirement 4: Locale Resolution in Background and Async Contexts
**Objective:** As a backend service author, I want the new `t()` calls to resolve to the correct locale even when invoked from background threads or async coroutines, so that operators see consistent log language regardless of where the call originates.
#### Acceptance Criteria
1. When `t("log.graph_api.<key>", ...)` is called from the `build_task` background thread inside `backend/app/api/graph.py` (started via `task_manager.run_task`), the Locale Helper shall resolve to the locale that was established for that thread (per the existing per-thread / `set_locale` mechanism), not silently fall back to the default `zh`.
2. When `t("log.retry.<key>", ...)` is called from the synchronous `retry_with_backoff` decorator wrapping a Flask request handler, the Locale Helper shall resolve via the active Flask request context (`Accept-Language` header), consistent with how request-scoped `t()` calls behave elsewhere in the codebase.
3. When `t("log.retry.<key>", ...)` is called from the asynchronous `retry_with_backoff_async` decorator under `asyncio`, the Locale Helper shall resolve via whichever locale source is in scope for that coroutine (request context if present; otherwise the per-thread fallback set by the caller), without raising and without requiring any new locale-propagation wiring inside `retry.py`.
4. If a `t()` call introduced by this spec references a key that is missing from both the active locale and the `zh` fallback, the Locale Helper shall continue to behave per the existing contract: emit a single deduped warning naming the key and locale, and return the key string itself (never `None`, never raise).
### Requirement 5: Verification and Regression Guards
**Objective:** As a reviewer of this PR, I want repeatable mechanical checks that prove the in-scope files are clean of stray hard-coded Chinese log/print strings on those nine lines, so that the acceptance criteria can be re-validated on every future change.
#### Acceptance Criteria
1. The Verification Procedure shall, when run against the repository, report zero matches of any Unicode CJK character (range `U+4E00`–`U+9FFF`) on the nine specific lines covered by Requirement 1 in their post-change form (i.e., `grep -P "[一-鿿]"` against the replaced lines returns no hits).
2. The Verification Procedure shall, when run against `locales/en.json` and `locales/zh.json`, confirm via a Python `json.load` + recursive key walk that every newly introduced key path exists in both files, and exit non-zero if a key path is present in only one of them.
3. The Verification Procedure shall confirm via Python that for each new key in `locales/zh.json` whose source f-string contained an `{var}` placeholder, the same `{var}` placeholder appears in the new English translation in `locales/en.json` (so interpolation is not silently dropped during translation).
4. The Verification Procedure shall require only tools already available in the dev environment (`grep`, `python3`, optional `jq`) — no new runtime or dev dependencies shall be added by this spec.
5. The Backend Test Suite shall continue to pass (`uv run python -m pytest`) after the change, with no new failures introduced; in particular, any pre-existing tests that assert the prior Chinese log/print text shall be updated to assert via the same `t()` lookup or an English translation rather than removed.

91
.kiro/specs/i18n-externalize-remaining-backend-logs/research.md Normal file
View File

@ -0,0 +1,91 @@
# Research & Design Decisions
## Summary
- **Feature**: `i18n-externalize-remaining-backend-logs`
- **Discovery Scope**: Simple Addition (extending an established convention from ticket #6)
- **Key Findings**:
- The `t()` helper, per-thread locale, and missing-key fallback are already in place in `backend/app/utils/locale.py` and require no changes.
- The convention `t("log.<domain>.m###", **kwargs)` with `{name}` placeholders is already used by all sibling modules; this spec strictly extends it.
- No existing test fixtures reference any of the nine Chinese strings to be replaced.
## Research Log
### Existing locale namespace structure
- **Context**: Need to add new keys without colliding with existing entries.
- **Sources Consulted**: `locales/en.json`, `locales/zh.json`, `.kiro/specs/i18n-externalize-backend-logs/requirements.md`.
- **Findings**:
- `log.graph_api` is densely populated `m006`–`m019` plus `m026`. Free contiguous slots starting at the tail: `m027`, `m028`, `m029`.
- `log.profile_generator` is densely populated `m001`–`m023`. Free slots: `m024`, `m025`.
- `log.retry` does not exist; introducing it as a sibling to other `log.<domain>` namespaces matches the existing pattern.
- **Implications**: New keys append at the tail per existing namespace; `log.retry` is created fresh starting at `m001`.
### Locale resolution in async / background contexts
- **Context**: `retry.py` is shared infrastructure invoked from sync request handlers, background tasks, and async coroutines.
- **Sources Consulted**: `backend/app/utils/locale.py`, `backend/app/services/oasis_profile_generator.py` (uses `set_locale`), Flask docs (request-context behaviour).
- **Findings**:
- `get_locale()` returns the request-context `Accept-Language` header when a Flask request is active, the per-thread locale otherwise, and `'zh'` as the default.
- Asyncio coroutines run on the same OS thread by default, so the per-thread locale set by the parent function propagates into `await`-driven calls.
- Missing-key fallback returns the key string and emits a deduped warning — never raises.
- **Implications**: No new locale-propagation wiring needed inside `retry.py`. Adding `from ..utils.locale import t` is sufficient.
### `print(...)` vs `logger` for the OASIS banners
- **Context**: Two `print(...)` banner statements at `oasis_profile_generator.py:945` and `:1001` decorate stdout. Should we keep them as `print` or fold them into existing `logger.info` calls?
- **Sources Consulted**: `backend/app/services/oasis_profile_generator.py:943` (existing `logger.info(t("log.profile_generator.m017", …))`), ticket #24 acceptance ("each `file:line` is fixed").
- **Findings**:
- The existing `logger.info` and the `print(...)` are emitting the same logical event in two channels. The banner adds `'='*60` separators on the surrounding lines, which is purely a console-cosmetic; replacing the print with a logger call would lose the visual banner.
- Ticket #24 wants externalisation, not removal.
- **Implications**: Keep both calls. Wrap the `print(f"...")` argument with `t(...)`. Introduce dedicated keys (`m024`, `m025`) so the banner copy is decoupled from the structured log copy at `m017`.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Append-at-tail (selected) | Add new `m###` keys at the next contiguous slot per namespace; create `log.retry` fresh | Mirrors #6 convention; minimal diff; no overwrite risk | Numbering gaps under `log.graph_api` remain | Aligns with steering principle of preserving established conventions |
| Fill numbering gaps | Reuse missing slots `m009`, `m010`, etc. | Tighter numbering | Risk of colliding with reserved-but-not-yet-merged keys; mixed insertion sites complicate review | Rejected |
| Consolidate banner prints into logger | Remove the `print(...)` calls; use only `logger.info(t(...))` | One fewer key | Behaviour change (loses console banner); violates Requirement 3.2 | Rejected |
## Design Decisions
### Decision: Add a new `log.retry` sub-namespace rather than reusing `log.bootstrap` or `log.graph_api`
- **Context**: `retry.py` is a generic utility used by many callers; it does not belong to a single domain.
- **Alternatives Considered**:
1. Place keys under `log.bootstrap` — wrong domain (bootstrap is for app startup logs).
2. Place keys under each caller's namespace — would require dynamic key resolution, adding complexity.
3. New `log.retry` sub-namespace — clean and self-describing.
- **Selected Approach**: Introduce `log.retry.m001`–`m004` as a peer of `log.graph_api`, `log.profile_generator`, etc.
- **Rationale**: Matches the per-domain naming scheme already in use; locates retry-specific copy in one place.
- **Trade-offs**: Adds one new sub-namespace under `log`, but does not change the top-level key set.
- **Follow-up**: Verify that no other module already defines `log.retry` (verified: it does not exist).
### Decision: Wrap `print(...)` arguments rather than removing the prints
- **Context**: Ticket #24 mandates externalisation of the listed call sites; behaviour preservation is in scope.
- **Alternatives Considered**:
1. Keep `print(t("..."))` — preserves console banner, externalises text.
2. Remove `print(...)`; rely on `logger.info` only — drops banner.
- **Selected Approach**: Option 1. The `'='*60` separator lines stay; only the message text routes through `t(...)`.
- **Rationale**: Minimum change; respects Requirement 3.2.
- **Trade-offs**: None significant.
- **Follow-up**: Confirm during validation that the surrounding separator prints (`print(f"\n{'='*60}")`) are not on the ticket's evidence list (they are not — they contain only ASCII).
### Decision: Pass exception text as a keyword argument named `e` (not `error`)
- **Context**: Existing `log.profile_generator` keys use `e=str(e)` and `error=...` inconsistently. Need to pick one convention to remain consistent.
- **Alternatives Considered**:
1. Use `e` — matches `log.profile_generator.m003`, `m005`, `m008`, `m012`.
2. Use `error` — matches `log.profile_generator.m018`.
- **Selected Approach**: Use `e` for raw exception strings (the more common pattern). Where a separate label is more readable, use a domain-specific name (e.g. `error` is fine when it carries semantic weight).
- **Rationale**: Match the dominant existing convention.
- **Trade-offs**: None.
- **Follow-up**: Use `e` throughout the new keys.
## Risks & Mitigations
- **Async retry on a fresh thread without `set_locale`** — Falls back to `'zh'`. Acceptable: ticket #24 acceptance targets *source-code* CJK absence. Documented for future ticket if needed.
- **Circular imports when adding `from ..utils.locale import t` to `retry.py`** — `locale.py` imports only `json`, `logging`, `os`, `threading`, and `flask` (no project modules). No circular risk.
- **Test-suite breakage from changed log text** — No fixtures match the Chinese strings. Verified by grep of `backend/`. Low risk.
## References
- Sibling spec: `.kiro/specs/i18n-externalize-backend-logs/requirements.md` — established convention.
- Ticket #6 (closed) and ticket #24 (this work).
- `backend/app/utils/locale.py` — `t()` contract.

24
.kiro/specs/i18n-externalize-remaining-backend-logs/spec.json Normal file
View File

@ -0,0 +1,24 @@
{
"feature_name": "i18n-externalize-remaining-backend-logs",
"created_at": "2026-05-07T22:24:20Z",
"updated_at": "2026-05-07T22:50:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true,
"ticket": 24,
"related_tickets": [10, 6]
}

63
.kiro/specs/i18n-externalize-remaining-backend-logs/tasks.md Normal file
View File

@ -0,0 +1,63 @@
# Implementation Plan
- [x] 1. Add three new keys to `log.graph_api` in both locale files
- In `locales/en.json`, append `m027`, `m028`, `m029` under `log.graph_api` with the English translations from the design's key-mapping table
- In `locales/zh.json`, append the same three keys under `log.graph_api` with the verbatim original Chinese text (rewriting `f"...{var}..."` as `"...{var}..."`)
- Confirm via `python3 -m json.tool` that both files round-trip without reformatting other keys
- Observable completion: `python3 -c "import json; en=json.load(open('locales/en.json'))['log']['graph_api']; zh=json.load(open('locales/zh.json'))['log']['graph_api']; assert {'m027','m028','m029'} <= set(en) <= set(zh) | set(en); print('ok')"` exits zero
- _Requirements: 2.1, 2.2, 2.3, 2.5_
- [x] 2. Replace the three Chinese f-strings in `backend/app/api/graph.py` with `t()` calls
- Line 385: replace `f"[{task_id}] 开始构建图谱..."` with `t("log.graph_api.m027", task_id=task_id)`
- Line 494: replace the build-completion f-string with `t("log.graph_api.m028", task_id=task_id, graph_id=graph_id, node_count=node_count, edge_count=edge_count)`
- Line 513: replace the build-failure f-string with `t("log.graph_api.m029", task_id=task_id, e=str(e))`
- Do not change log levels, surrounding `task_manager.update_task` calls, or control flow
- Observable completion: `grep -nP "[一-鿿]" backend/app/api/graph.py | grep -E "^(385|494|513):"` returns no matches; `python3 -c "import ast; ast.parse(open('backend/app/api/graph.py').read())"` succeeds
- _Requirements: 1.1, 1.4, 1.5, 3.1, 3.4_
- _Depends: 1_
- [x] 3. Add two new keys to `log.profile_generator` in both locale files
- In `locales/en.json`, append `m024` and `m025` under `log.profile_generator` per the design table
- In `locales/zh.json`, mirror with the verbatim original Chinese banner text (using `{count}` placeholder where the source had `len([p for p in profiles if p])`)
- Observable completion: same key-presence assertion as Task 1 but for `m024`, `m025`
- _Requirements: 2.1, 2.2, 2.3, 2.5_
- [x] 4. Replace the two `print(...)` banner strings in `backend/app/services/oasis_profile_generator.py` with `t()` calls
- Line 945: replace `f"开始生成Agent人设 - 共 {total} 个实体，并行数: {parallel_count}"` with `t("log.profile_generator.m024", total=total, parallel_count=parallel_count)`
- Line 1001: replace `f"人设生成完成！共生成 {len([p for p in profiles if p])} 个Agent"` with `t("log.profile_generator.m025", count=len([p for p in profiles if p]))`
- Keep the surrounding `print(f"\n{'='*60}")` separator lines exactly as they are; keep both `print(...)` calls (do not collapse into the existing `logger.info` at line 943)
- Observable completion: `grep -nP "[一-鿿]" backend/app/services/oasis_profile_generator.py | grep -E "^(945|1001):"` returns no matches; the file still parses with `ast.parse`
- _Requirements: 1.2, 1.4, 1.5, 3.2_
- _Depends: 3_
- [x] 5. Add a new `log.retry` sub-namespace with four keys to both locale files
- In `locales/en.json`, add `log.retry` as a peer of the other `log.<domain>` sub-namespaces, with keys `m001`–`m004` per the design table
- In `locales/zh.json`, mirror the same `log.retry` sub-namespace with verbatim original Chinese
- Use placeholder names `func_name`, `max_retries`, `index`, `e` consistently across both files (note: the source `idx + 1` is bound to `index=idx + 1` at the call site — placeholder names cannot contain `+`)
- Observable completion: `python3 -c "import json; en=json.load(open('locales/en.json'))['log']['retry']; zh=json.load(open('locales/zh.json'))['log']['retry']; assert set(en)==set(zh)=={'m001','m002','m003','m004'}; print('ok')"` exits zero
- _Requirements: 2.1, 2.2, 2.3, 2.5_
- [x] 6. Externalise the four `logger.error` strings in `backend/app/utils/retry.py`
- Add `from .locale import t` at the top of `retry.py` (use the same relative-import depth as `from ..utils.logger import get_logger` already in the file — i.e., `from .locale import t`)
- Line 55: replace `f"函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}"` with `t("log.retry.m001", func_name=func.__name__, max_retries=max_retries, e=str(e))`
- Line 108: replace `f"异步函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}"` with `t("log.retry.m002", func_name=func.__name__, max_retries=max_retries, e=str(e))`
- Line 179: replace `f"API调用在 {self.max_retries} 次重试后仍失败: {str(e)}"` with `t("log.retry.m003", max_retries=self.max_retries, e=str(e))`
- Line 227: replace `f"处理第 {idx + 1} 项失败: {str(e)}"` with `t("log.retry.m004", index=idx + 1, e=str(e))`
- Do not modify the `logger.warning(...)` retry-attempt messages or the docstrings (out of scope for #24)
- Observable completion: `grep -nP "[一-鿿]" backend/app/utils/retry.py | grep -E "^(55|108|179|227):"` returns no matches; `python3 -c "import ast; ast.parse(open('backend/app/utils/retry.py').read())"` succeeds; `python3 -c "from backend.app.utils import retry; print(retry.t)"` resolves the import
- _Requirements: 1.3, 1.4, 1.5, 3.3, 4.1, 4.2, 4.3, 4.4_
- _Depends: 5_
- [x] 7. Run mechanical verification across the change
- From the repo root, verify zero CJK on the nine affected lines:
```
grep -nP "[一-鿿]" backend/app/api/graph.py | grep -E "^(385|494|513):" || echo OK_graph
grep -nP "[一-鿿]" backend/app/services/oasis_profile_generator.py | grep -E "^(945|1001):" || echo OK_profile
grep -nP "[一-鿿]" backend/app/utils/retry.py | grep -E "^(55|108|179|227):" || echo OK_retry
```
Each should print `OK_*`.
- Run a Python parity check that asserts every newly-added key path exists in both `locales/en.json` and `locales/zh.json` and that every `{name}` placeholder in the `zh` value also appears in the `en` value (and vice versa).
- Run `cd backend && uv run python -m pytest` and confirm no new failures relative to the pre-change baseline.
- Observable completion: all three grep assertions print `OK_*`; the parity Python check exits zero; the pytest run reports the same pass/fail count as on `main` for these files.
- _Requirements: 1.5, 2.4, 5.1, 5.2, 5.3, 5.4, 5.5_
- _Depends: 2, 4, 6_

229
.kiro/specs/i18n-frontend-comments/design.md Normal file
View File

@ -0,0 +1,229 @@
# Design Document — i18n-frontend-comments
## Overview
**Purpose**: Translate Chinese developer comments in `frontend/src/` to English so non-Chinese-reading maintainers can understand intent without translation tooling. Strictly documentation-only; no behavior change.
**Users**: Frontend maintainers and reviewers of MiroFish — developers who read and modify `frontend/src/` but do not read Chinese.
**Impact**: 20 files in `frontend/src/` change; the compiled bundle is byte-equivalent modulo source-map comment lines. The `vue-i18n` user-facing translation surface (`/locales/*.json`) is unaffected.
### Goals
- Eliminate Chinese characters (U+4E00–U+9FFF) from `frontend/src/` comments and dev-facing string literals (`console.*`).
- Preserve every comment's *why* (semantic intent) when translating; delete comments that merely restate the code per `dev-guidelines.md`.
- Append `(#9)` ticket reference to any TODO/FIXME marker that lacks one.
- Keep `npm run build` green and the rendered UI byte-equivalent on a smoke check.
### Non-Goals
- Translating user-facing strings (those live in `/locales/*.json`; tracked separately).
- Translating LLM prompt template strings (translation would change model input — retained and documented in PR per Requirement 1.5).
- Restructuring comments into JSDoc (only keep JSDoc when already JSDoc-shaped).
- Reformatting code, renaming identifiers, or any change to `<script>` / `<template>` semantics.
- Touching backend Python comments (covered by ticket #7) or repo-root configuration files.
## Boundary Commitments
### This Spec Owns
- All comment text inside files under `frontend/src/`: line comments (`//`), block comments (`/* */`), JSDoc (`/** */`), and Vue template comments (`<!-- -->`).
- The natural-language portion of JSDoc tags (`@param`, `@returns`, etc.) — not the tag syntax itself.
- Chinese-content string literals passed to `console.error`, `console.warn`, and `console.log` (developer-facing, not in i18n locales).
- The PR-level documentation listing any deliberately-retained bilingual content.
### Out of Boundary
- Any change inside `/locales/*.json` (covered by issues #8 and #11).
- Any change in `backend/`, `static/`, repo root, or anywhere outside `frontend/src/`.
- LLM prompt template string literals (e.g. `Step5Interaction.vue:725-727`) — retained as documented exceptions.
- New tooling (linters, formatters, translation scripts).
- Any executable change: identifier names, import paths, expression edits, Vue template structure outside `<!-- -->` text, or `<style>` selectors / values.
### Allowed Dependencies
- Existing Vite build (`npm run build`) and Vue dev server (`npm run dev`) for verification.
- `ripgrep` for the verification command.
- No runtime dependencies — this is text-only editing.
### Revalidation Triggers
- Discovery during implementation that a category of Chinese content beyond comments + `console.*` strings exists in `frontend/src/` → update the design's String-Literal Decision Matrix and add residuals to the PR description rather than silently expanding scope.
- Discovery that a JSDoc block carries semantically-load-bearing Chinese (e.g. an idiom that does not have a 1:1 English rendering) → keep both languages, document in PR per Req 1.5.
## Architecture
### Existing Architecture Analysis
Per `structure.md`, `frontend/src/` is layered into `views/`, `components/`, `api/`, `store/`, plus `App.vue`. This spec does not change the layering. Per `tech.md`, the project uses no enforced linter/formatter and existing files mix English and Chinese comments — this spec is the explicit ask to normalize the comment language to English in this directory.
### Architecture Pattern & Boundary Map
This is a documentation-only change — no architectural pattern to choose. The relevant boundary is purely *which textual regions of which files are eligible for edit*. The decision matrix below is the architecture for this spec.
#### Region eligibility matrix
| Region | Action |
| --- | --- |
| `//` line comment | Translate; delete if it restates the code per Req 2.1 |
| `/* */` block comment | Translate; delete if redundant per Req 2.1 |
| `/** */` JSDoc block | Translate the natural-language content; preserve tag syntax (`@param`, `@returns`, etc.) per Req 1.4 |
| `<!-- -->` Vue template comment | Translate per Req 1.3 |
| `console.error|warn|log('… 中文 …')` | Translate the string content (developer-facing, not in i18n locales) |
| LLM prompt template string literal | **Do not translate**; document in PR per Req 1.5 |
| Any other string literal containing Chinese | **Do not translate** (Req 4.4); document if non-empty |
| Identifiers, imports, exports, expressions | **Do not change** (Req 4.2) |
| Vue template structure (tags, attributes, bindings) | **Do not change** (Req 4.2) |
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|---|---|---|---|
| Frontend | Vue 3.5 + Vite 7 (existing) | Build target — must continue to compile | No version change |
| Verification | `ripgrep` (already present in repo workflows) | Acceptance gate via `rg '[\x{4e00}-\x{9fff}]' frontend/src/` | No new dependency |
| No new tooling | — | — | Per `tech.md` steering: "No enforced linter or formatter… match the surrounding file's style" |
## File Structure Plan
No directory or file additions. All edits are in-place inside the 20 files identified by ripgrep:
```
frontend/src/
├── App.vue # 4 hits — translate
├── api/
│ ├── graph.js # 10 hits
│ ├── index.js # 8 hits (incl. JSDoc-light line comments)
│ ├── report.js # 8 hits
│ └── simulation.js # 29 hits (JSDoc-heavy)
├── components/
│ ├── GraphPanel.vue # 84 hits — D3 logic comments + template
│ ├── HistoryDatabase.vue # 124 hits
│ ├── Step1GraphBuild.vue # 5 hits + 3 console.error strings
│ ├── Step2EnvSetup.vue # 76 hits
│ ├── Step3Simulation.vue # 52 hits
│ ├── Step4Report.vue # 176 hits
│ └── Step5Interaction.vue # 34 hits + LLM prompt strings (RETAIN)
├── store/
│ └── pendingUpload.js # 2 hits
└── views/
├── Home.vue # 43 hits
├── InteractionView.vue # 6 hits
├── MainView.vue # 4 hits
├── Process.vue # 191 hits — largest file (2067 lines)
├── ReportView.vue # 6 hits
├── SimulationRunView.vue # 18 hits
└── SimulationView.vue # 22 hits
```
### Modified Files
All 20 files above receive comment translation (and, for `Step1GraphBuild.vue` and any others discovered during implementation, `console.*` string translation). No file is created, deleted, or moved.
## System Flows
### Per-file translation sequence
```mermaid
flowchart TD
A[Open file] --> B[Locate Chinese region with rg or editor]
B --> C{Region type?}
C -->|Comment| D{Restates the code?}
D -->|Yes| E[Delete comment]
D -->|No| F[Translate, preserve intent]
C -->|JSDoc| G[Translate natural-language content<br/>preserve tag syntax]
C -->|Vue template comment| H[Translate inside &lt;!-- --&gt;]
C -->|console.* string| I[Translate string content]
C -->|LLM prompt string| J[Skip; record for PR description]
C -->|Other string literal| K[Skip per Req 4.4]
E --> L[Next region]
F --> L
G --> L
H --> L
I --> L
J --> L
K --> L
L --> M{File done?}
M -->|No| B
M -->|Yes| N[Run rg on file: confirm zero remaining hits<br/>OR all remaining are intentional retentions]
N --> O[File complete]
```
### TODO/FIXME sweep
```mermaid
flowchart LR
A[rg 'TODO|FIXME' frontend/src/] --> B{Any hits?}
B -->|None| C[Document in PR: no markers found]
B -->|Has hits| D[For each hit]
D --> E{Already has #N reference?}
E -->|Yes| F[Leave unchanged]
E -->|No, was Chinese| G[Translate description AND append #9]
E -->|No, was already English| H[Out of scope; leave unchanged]
```
## Requirements Traceability
| Requirement | Summary | Realized by |
|---|---|---|
| 1.1 | Zero Chinese in `frontend/src/` per ripgrep | Per-file translation pass; verification command in PR |
| 1.2 | Preserve semantic intent | Translator judgment per region; Req 2.3 enforces conservative-on-ambiguity |
| 1.3 | Handle SFC blocks correctly | Region eligibility matrix (`<script>` / `<template>` / `<style>` rows) |
| 1.4 | Preserve JSDoc structure | Region matrix: "Translate the natural-language content; preserve tag syntax" |
| 1.5 | Document retained bilingual content | PR description lists `Step5Interaction.vue` LLM prompts (and any others) |
| 2.1 | Delete redundant comments | Per-file flowchart `D → E` branch |
| 2.2 | Translate intent-bearing comments | Per-file flowchart `D → F` branch |
| 2.3 | Conservative on ambiguity | Translator rule encoded in research.md Decision; default is *translate, not delete* |
| 2.4 | No new explanatory comments | Translation rule: never add comments not present in original (except `(#9)` ticket ref) |
| 3.1 | Keep TODO/FIXME marker, translate trailing text | TODO sweep flowchart `G` branch |
| 3.2 | Append `(#9)` ticket ref where missing | TODO sweep flowchart `G` branch |
| 3.3 | Preserve existing ticket refs | TODO sweep flowchart `E → F` branch |
| 4.1 | `npm run build` exit 0 | Build run as part of acceptance check |
| 4.2 | No executable change | Region matrix: identifiers/imports/expressions are *not eligible* |
| 4.3 | UI smoke-check identical | Manual smoke after build |
| 4.4 | Leave string literals untouched (except `console.*`) | Region matrix; documented exception for `console.*` is the sole carve-out |
| 5.1 | Verification command in PR | PR template hand-off |
| 5.2 | List retained bilingual files | PR template hand-off |
| 5.3 | Branch + commit naming | `docs/i18n-9-translate-frontend-comments` and `docs(i18n): translate chinese comments in frontend src to english` |
| 5.4 | No edits outside `frontend/src/` | `git diff --name-only main..HEAD` review at PR time |
## Components and Interfaces
This is a documentation-only change — there are no software components, services, or APIs to design. The "interfaces" of this spec are textual:
| Interface | Owner | Contract |
| --- | --- | --- |
| `frontend/src/**/*.{vue,js}` comments | This spec | All comment text is English. Chinese is permitted only when explicitly listed in the PR description as a deliberately-retained bilingual case. |
| `frontend/src/**/*.{vue,js}` `console.*` string literals | This spec | All `console.error|warn|log` argument strings are English. |
| `frontend/src/**/*.{vue,js}` non-`console` string literals | Out of scope | Unchanged from baseline. Any Chinese in these strings (e.g. LLM prompt templates) is documented in the PR. |
| `frontend/src/**/*.{vue,js}` executable code | Out of scope | Byte-identical except for surrounding comment lines. |
## Data Models
Not applicable — no data structures change.
## Error Handling
Not applicable — no runtime code path changes. The "errors" of this spec are reviewer-detectable issues:
| Issue | Detection | Response |
|---|---|---|
| Translation drift (wrong meaning) | Reviewer reads English comment against surrounding code | Reviewer flags; translator revises |
| Accidental edit to executable code | `git diff` review filtered to non-comment lines | Revert; restart that file |
| Residual Chinese in non-LLM string | Verification ripgrep returns unexpected file | Either translate (if `console.*`) or move LLM exception to PR description |
| Build failure on `npm run build` | CI / local build | Bisect: most likely accidental edit to a `<script>` or `<template>` block; revert |
## Testing Strategy
No automated tests added. The spec's verification surface is:
- **Acceptance ripgrep**: `rg '[\x{4e00}-\x{9fff}]' frontend/src/` returns no files (or only files listed as retained in the PR description).
- **Vite build**: `npm run build` exits 0.
- **Manual UI smoke**: `npm run dev`, navigate Home → Process → each Step component → Interaction → Report; confirm rendering matches pre-change baseline. (Cannot be fully proven; explicit acknowledgment of "manual smoke" per the steering note that "type-check/test passes do not prove feature correctness here".)
- **Diff hygiene check**: `git diff --stat main..HEAD` shows only `frontend/src/` files modified.
## Implementation Notes
- Per the project's manual-style ethos, do this in an editor with rg-driven navigation. No new scripts.
- For each file, do all edits in one pass, then re-run `rg '[\x{4e00}-\x{9fff}]' <file>` to confirm zero residual (or only the deliberately-retained string literals, which the implementer should know about ahead of time per the design's eligibility matrix).
- The largest 6 files (`Process.vue`, `Step4Report.vue`, `HistoryDatabase.vue`, `GraphPanel.vue`, `Step2EnvSetup.vue`, `Step3Simulation.vue`) account for ~80% of the work; budget time accordingly.
- Reviewer aid: the PR description should list, in order, the verification command, the verification result, the file count, and any retained-bilingual exceptions. Keep the description short — the diff itself carries the work.

133
.kiro/specs/i18n-frontend-comments/gap-analysis.md Normal file
View File

@ -0,0 +1,133 @@
# Gap Analysis — i18n-frontend-comments
## 1. Current State Investigation
### Scope discovery (ground truth)
Ripgrep `[\x{4e00}-\x{9fff}]` over `frontend/src/` returns **20 files, 902 occurrences**:
| File | Hits |
| --- | ---: |
| `views/Process.vue` | 191 |
| `components/Step4Report.vue` | 176 |
| `components/HistoryDatabase.vue` | 124 |
| `components/GraphPanel.vue` | 84 |
| `components/Step2EnvSetup.vue` | 76 |
| `components/Step3Simulation.vue` | 52 |
| `views/Home.vue` | 43 |
| `components/Step5Interaction.vue` | 34 |
| `api/simulation.js` | 29 |
| `views/SimulationView.vue` | 22 |
| `views/SimulationRunView.vue` | 18 |
| `api/graph.js` | 10 |
| `api/index.js` | 8 |
| `api/report.js` | 8 |
| `views/InteractionView.vue` | 6 |
| `views/ReportView.vue` | 6 |
| `components/Step1GraphBuild.vue` | 5 |
| `App.vue` | 4 |
| `views/MainView.vue` | 4 |
| `store/pendingUpload.js` | 2 |
No `.css` files exist under `frontend/src/`; styles live inside Vue SFC `<style>` blocks.
### Comment shapes encountered
Sampling representative files confirms three syntactic forms — all already English-syntax, only the natural-language content is Chinese:
- **JS line comments**: `// 创建axios实例`, `timeout: 300000, // 5分钟超时（本体生成可能需要较长时间）`
- **JSDoc blocks** in `api/simulation.js`: `/** * 创建模拟 */`, `* @returns {Promise} 返回配置信息，包含元数据和配置内容`
- **Vue template comments** in `views/Home.vue`: `<!-- 顶部导航栏 -->`, `<!-- 上半部分：Hero 区域 -->`
### String literals containing Chinese (NOT comments)
A naive regex for Chinese inside quoted strings flags **8 files**. Spot-checks reveal two distinct categories that the ticket body did not explicitly anticipate:
- **Developer-facing log strings** — e.g. `Step1GraphBuild.vue:216` `console.error('缺少项目或图谱信息')`. These print to the browser dev console and are not part of the i18n locale surface. Translating them does not change runtime behavior.
- **LLM prompt template strings** — e.g. `Step5Interaction.vue:725-727` `\`以下是我们之前的对话：\n${historyContext}\n\n现在我的新问题是：${message}\``. These are sent to a Chinese-tuned LLM (default Qwen). Translating them *would* change the model's input and could shift output behavior.
The ticket says **"no UI string changes (those are already in `locales/en.json`)"** and **"Out of scope: Translating user-facing strings"**. Neither category above is user-facing UI text — `locales/*.json` already covers user-facing strings via `vue-i18n`. The ticket's acceptance criterion #1 (`grep returns no files, or only files with deliberately-kept bilingual comments listed in PR`) leaves room to retain the LLM prompt strings as documented exceptions.
### Conventions to respect (from steering)
- `tech.md`: 4-space indent, no enforced linter, "match the surrounding file's style". Existing files mix English and Chinese in comments/docstrings — preserve both *unless asked*. **This ticket is the explicit ask.**
- `structure.md`: `frontend/src/api/*.js` services use Axios with 5-min timeout + exponential retry. The translation pass must not touch the retry/timeout logic.
- `dev-guidelines.md` (project-level): "Don't comment the obvious — comment the *why*." JSDoc on all exported functions, classes, interfaces (so JSDoc blocks must be **kept** in JSDoc form when translating, not deleted as redundant).
- `commits.md`: Conventional Commits, lowercase, imperative, max 72 chars, no `Co-Authored-By:` footer. Branch `<type>/<ticket>-<desc>` — ticket dictates `docs/i18n-9-translate-frontend-comments`.
### Existing i18n-related precedent
Recent merged PRs in the same epic (#11):
- `feat/i18n-2-translate-ontology-generator-prompts` → backend prompt translation, full content swap.
- `feat/i18n-4-translate-sim-config-prompts`, `feat/i18n-5-translate-report-agent-prompts` → similar backend prompt swaps.
- `feat/i18n-6-externalize-backend-logs` → moved log strings out of code into i18n keys.
- `fix/i18n-8-backfill-zh-json` (current branch base) → backfilled missing zh translations.
**Pattern**: prior i18n work changed both content *and* infrastructure (locale-keying logs). This ticket explicitly does not — it is a documentation-only pass without re-keying anything.
## 2. Requirements ↔ Asset Map
| Req | Asset to change | Gap tag | Note |
| --- | --- | --- | --- |
| 1.1–1.4 (translate comments incl. JSDoc) | All 20 files listed above | — (clear) | Largely mechanical; respect SFC block boundaries (`<script>` vs `<template>` vs `<style>`). |
| 1.5 (deliberately bilingual) | LLM prompt strings in `Step5Interaction.vue` (and any others discovered) | **Constraint** | Keep Chinese, document in PR. Behavior-risk if translated. |
| 2.x (drop redundant) | Files with `// 获取数据`-style restate-the-code comments | — | Apply per case during the pass; conservative when ambiguous. |
| 3.x (TODO/FIXME ticket refs) | Search `frontend/src/` for `TODO\|FIXME` | **Unknown** | No matches noted in spot checks; will sweep during implementation. If none found, requirement is satisfied vacuously. |
| 4.x (no behavior change) | Confirmed by `npm run build` exit 0 + manual smoke | — | Vite build is the reference; keep all string-literal content (other than developer-log strings) untouched; identifiers and imports are off-limits. |
| 5.x (PR hand-off) | PR description, branch name, commit message | — | Branch name from ticket: `docs/i18n-9-translate-frontend-comments`. |
### Discovered scope ambiguity → decision needed
Two boundary calls that the requirements should sharpen before design:
- **`console.error` / `console.warn` / `console.log` strings with Chinese content** — translate (developer-facing, not in locales) or leave (string-literal change risks scope creep)? Recommended: **translate**, since they are dev-facing comments-by-other-means and the ticket's spirit is "English-readable code". This is a design decision to be encoded in the design doc, not a new requirement.
- **LLM prompt template strings** — leave as-is and list in PR (per Req 1.5). This is the safer call: the LLM is Chinese-tuned by default and translating a system prompt is a behavior change.
Both decisions stay inside the requirements as currently written (specifically Req 1.5 + Req 4.4, which already excludes string literals from the translation pass except where developer-log strings are concerned). The design phase will document the rule explicitly.
## 3. Implementation Approach Options
### Option A — Single-pass translation per file, no tooling
**Approach**: Open each of the 20 files, translate every Chinese comment in place, drop redundant ones, append `(#9)` to bare TODO/FIXME, leave Chinese string literals (LLM prompts) and translate `console.*` Chinese strings. Verify with `rg [\x{4e00}-\x{9fff}] frontend/src/`.
- ✅ Lowest overhead, no new tools or scripts
- ✅ Fits a one-shot doc-only PR
- ✅ Maximally aligns with `dev-guidelines.md` "comment the *why*" — judgment per comment
- ❌ ~900 occurrences spread across 20 files — most concentrated in 6 files (>50 hits each) which are large (`Process.vue` is 2067 lines, `Step4Report.vue`, `HistoryDatabase.vue`)
- ❌ Manual judgment for redundant-vs-meaningful adds reviewer load
### Option B — Automated translation script + manual pass
**Approach**: Write a Node/Python script that walks files, extracts Chinese comments, runs them through an LLM, and writes back. Then a manual pass on the diff.
- ✅ Faster on long files
- ❌ Adds a dependency (LLM call) and a scratch script, neither delivered
- ❌ The translation needs *judgment* (drop vs translate per Req 2) — automation undercuts the "comment the *why*" rule
- ❌ Risk of touching string literals or identifiers if regex is loose
- ❌ Out of step with the steering "no enforced tooling without discussion" principle
### Option C — File-by-file with task batching
**Approach**: Group the 20 files into work units by size: (a) high-touch (Process, Step4Report, HistoryDatabase, GraphPanel, Step2EnvSetup, Step3Simulation), (b) mid-touch (Home, Step5Interaction, simulation.js, SimulationView, SimulationRunView), (c) light (api/{graph,index,report}.js, the 4–8 hit views, App.vue, store/pendingUpload.js, Step1GraphBuild.vue). Implementation tasks mirror these groups. Verify after each group with the ripgrep check.
- ✅ Same translation effort as A but with checkpointable progress (matches the project's task-tracking pattern from steering — "background tasks expose progress")
- ✅ Reviewer can read the PR file-group-by-file-group instead of all-at-once
- ✅ If the PR needs to land partial (rare), the light + mid groups still ship a valuable subset
- ❌ A few extra task headings in `tasks.md` vs Option A's "do the thing"
## 4. Effort & Risk
- **Effort**: **S (1–2 days)**. Mechanical translation, plus judgment calls. ~900 occurrences but no architectural work.
- **Risk**: **Low**. Doc-only change. The only real risks are (a) accidentally editing a string literal that affects the LLM prompt or a hardcoded user-visible string, and (b) deleting a comment whose intent the translator misread. Both are mitigated by Req 4.4 ("leave string literals unchanged") and Req 2.3 (conservative-when-ambiguous).
## 5. Recommendations for Design Phase
- **Preferred approach**: **Option C** — file-grouped translation pass, no tooling, no script. It matches the project's manual-style ethos and the existing pipeline-aligned task structure, and produces a reviewable PR.
- **Encode in design**:
- The translation rule for each comment shape (`//`, `/* */`, JSDoc, `<!-- -->`).
- The decision matrix for string literals: translate `console.*` Chinese strings; retain LLM prompt strings (in `Step5Interaction.vue`) and list them in the PR per Req 1.5.
- The TODO/FIXME sweep approach (single ripgrep pass before the file loop).
- The verification command and acceptance check sequence.
- **Research items carried forward**: none — the codebase has been inspected enough to commit to Option C without further investigation.

70
.kiro/specs/i18n-frontend-comments/requirements.md Normal file
View File

@ -0,0 +1,70 @@
# Requirements Document
## Introduction
This spec covers a pure-documentation cleanup pass: translate Chinese developer comments in `frontend/src/` to English so English-speaking maintainers can read the code. The change is documentation-only — no runtime behavior changes, no UI string changes (those live in `/locales/*.json`), and no architectural refactor. Tracked as GitHub issue #9, the lowest user-impact ticket in the i18n epic (#11).
The work targets developer-facing comments in 20 known files: 7 views, 7 components, 4 `api/*.js` modules, `App.vue`, and `store/pendingUpload.js`. The discovery method is `grep -rln '[一-鿿]' frontend/src/` (or the ripgrep equivalent), which must return zero matches at completion (or only files explicitly listed as deliberately bilingual in the PR description).
## Boundary Context
- **In scope**: Translating Chinese developer comments (line comments, block comments, JSDoc, and Vue `<!-- ... -->` template comments) to English in `frontend/src/`. Dropping comments that merely restate the code, per `dev-guidelines.md`. Appending ticket references to TODO/FIXME markers that lack one.
- **Out of scope**: Any user-facing string, label, placeholder, toast, or template-rendered text — these live in `/locales/en.json` and `/locales/zh.json` and are tracked separately (see #8). Restructuring comments into JSDoc unless they are already JSDoc-shaped. Reformatting code, renaming identifiers, or any non-comment change. Backend Python comments (covered by ticket #7).
- **Adjacent expectations**: The Vite build (`npm run build`) and the Vue dev server (`npm run dev`) must continue to compile and run. The `vue-i18n` translation surface in `/locales/*.json` is unaffected. The frontend `api/` services keep their existing behavior — the 5-min Axios timeout and exponential retry described in steering remain unchanged.
## Requirements
### Requirement 1: Comment Translation Coverage
**Objective:** As a frontend maintainer who does not read Chinese, I want every developer comment in `frontend/src/` to be in English, so that I can understand intent without translation tooling.
#### Acceptance Criteria
1. The Frontend Source Tree shall contain no Chinese characters (Unicode range U+4E00–U+9FFF) in any `.vue`, `.js`, or `.css` file under `frontend/src/`, as verified by ripgrep `[\x{4e00}-\x{9fff}]` returning zero matching files.
2. When a Chinese comment is translated, the Translation Pass shall preserve the original semantic intent (the *why* the comment was written) without paraphrasing into a different meaning.
3. Where a comment exists in `<script>`, `<template>`, or `<style>` blocks of a Single-File Component, the Translation Pass shall translate it in-place using the syntax appropriate to that block (`//` / `/* */` for script and style, `<!-- -->` for template).
4. If a Chinese comment is part of a JSDoc block (`/** ... */`), the Translation Pass shall keep the JSDoc structure intact and translate only the natural-language content.
5. Where a deliberately-bilingual comment must be retained (e.g. a quotation, a domain term needing the original), the Translation Pass shall list the file in the PR description and shall keep an English explanation alongside the Chinese.
### Requirement 2: Drop Redundant Comments
**Objective:** As a code reviewer, I want comments that merely restate the code to be removed during the translation pass, so that the codebase aligns with `dev-guidelines.md` ("comment the *why*, not the *what*").
#### Acceptance Criteria
1. When a Chinese comment only paraphrases the immediately following statement in different words (e.g. `// 获取数据` above `fetchData()`), the Translation Pass shall delete the comment rather than translate it.
2. When a Chinese comment encodes non-obvious intent (a constraint, an invariant, a workaround, a reason behind a magic number), the Translation Pass shall translate it rather than delete it.
3. If a comment's value cannot be judged from local context alone, the Translation Pass shall translate it conservatively (preserve rather than delete) and shall not delete a comment merely because the maintainer is unsure of its purpose.
4. The Translation Pass shall not introduce new comments beyond those required to translate or to add a TODO ticket reference; gratuitous explanatory comments are not added.
### Requirement 3: Preserve TODO/FIXME Markers and Add Ticket References
**Objective:** As a project maintainer tracking work-in-progress markers, I want every TODO and FIXME comment to carry a ticket reference, so that future cleanup can be triaged systematically.
#### Acceptance Criteria
1. When a Chinese TODO or FIXME comment is encountered, the Translation Pass shall keep the `TODO` / `FIXME` marker (uppercase, English) and translate the trailing description.
2. Where a TODO or FIXME marker lacks a ticket reference, the Translation Pass shall append a reference in the form `TODO(#<n>): …` or `FIXME(#<n>): …`, using `#9` if no more specific ticket exists for the underlying work.
3. If a TODO or FIXME marker already references a ticket (e.g. `TODO(#42)`), the Translation Pass shall preserve that reference unchanged.
### Requirement 4: No Runtime Behavior Change
**Objective:** As a release engineer, I want the translated branch to produce a behaviorally identical bundle, so that I can ship the change without retesting feature surfaces.
#### Acceptance Criteria
1. When `npm run build` runs against the translated branch, the Vite Build shall complete successfully with the same exit code (0) as the pre-translation baseline.
2. The Translation Pass shall not change any executable code: no identifier renames, no expression edits, no import or export changes, no Vue template structure changes outside `<!-- -->` comment text.
3. While the application is running in `npm run dev`, the User Interface shall render identically to the pre-translation baseline for the Home, Process, and each Step component flow on a manual smoke check.
4. If a translation pass risks ambiguity between a comment and a string literal (Chinese characters in a quoted string), the Translation Pass shall leave the string literal unchanged — string content is out of scope and belongs to `/locales/*.json`.
### Requirement 5: Verifiability and PR Hand-off
**Objective:** As a reviewer of this PR, I want a single command and a short checklist to confirm acceptance, so that review effort is bounded and reproducible.
#### Acceptance Criteria
1. The PR Description shall include the verification command and its expected output: `rg '[\x{4e00}-\x{9fff}]' frontend/src/` returns no matches (or only the deliberately-bilingual files listed in the PR).
2. The PR Description shall list any deliberately-retained bilingual comments with the file path and a one-line rationale.
3. The Branch Name shall be `docs/i18n-9-translate-frontend-comments` and the Commit Message shall start with `docs(i18n): translate chinese comments in frontend src to english` per the ticket's stated convention and the project's Conventional Commits rule.
4. The Translation Pass shall not modify files outside `frontend/src/` (notably no edits under `/locales/`, `/backend/`, or repo-root configuration).

97
.kiro/specs/i18n-frontend-comments/research.md Normal file
View File

@ -0,0 +1,97 @@
# Research & Design Decisions — i18n-frontend-comments
## Summary
- **Feature**: `i18n-frontend-comments`
- **Discovery Scope**: Simple Addition (documentation-only translation pass; no architectural change)
- **Key Findings**:
- 20 files in `frontend/src/` contain Chinese characters (902 total occurrences). Concentration follows file size: `Process.vue` (191), `Step4Report.vue` (176), `HistoryDatabase.vue` (124), `GraphPanel.vue` (84), `Step2EnvSetup.vue` (76), `Step3Simulation.vue` (52). The remaining 14 files have ≤43 hits each.
- Chinese appears in three comment shapes (JS line `//`, JSDoc `/** */`, Vue `<!-- -->`) and — unexpectedly — inside two flavors of string literal: `console.error('…')` developer logs (low risk to translate) and LLM prompt template strings in `Step5Interaction.vue` (behavior risk if translated, since the default LLM is Chinese-tuned).
- The codebase has no enforced linter/formatter (per `tech.md`) and `dev-guidelines.md` already states "comment the *why*, not the *what*". The existing comment density skews toward restating-the-code in Chinese; a meaningful share will be deleted rather than translated.
## Research Log
### Inventory and shape of Chinese content
- **Context**: Need to decide whether one pass can mechanically translate or whether per-file judgment is required.
- **Sources Consulted**: `rg [\x{4e00}-\x{9fff}] frontend/src/` (full count) and content-mode samples of `api/index.js`, `api/simulation.js`, `views/Home.vue`, `components/Step1GraphBuild.vue`, `components/Step5Interaction.vue`.
- **Findings**:
- Comments are syntactically standard (`//`, `/** */`, `<!-- -->`); no inline-Chinese identifiers.
- JSDoc blocks in `api/simulation.js` (and likely `api/graph.js`, `api/report.js`) include `@returns`, `@param` annotations with Chinese descriptions — translate only the natural-language portion, keep tag structure.
- `console.error` strings in `components/Step1GraphBuild.vue` (3 hits at lines 216, 237, 241) are dev-facing only, not user-facing.
- LLM prompt template strings in `components/Step5Interaction.vue` (lines 725–727) are sent to a Chinese-tuned model; translation is a behavior change.
- **Implications**: Per-file judgment pass is required. String literals are out of scope by default (Req 4.4); only `console.*` Chinese strings are in scope as a documented exception (developer-facing).
### Tooling decision: manual vs scripted
- **Context**: ~900 occurrences across 20 files — would automation help?
- **Sources Consulted**: Steering `tech.md` ("No enforced linter or formatter in this repo by design… Discuss with the user before introducing ESLint/Prettier/Ruff/Black"); `dev-guidelines.md` ("comment the *why*"); gap-analysis Option B trade-offs.
- **Findings**: Automation undercuts Req 2 (drop redundant comments requires human judgment). The project explicitly disallows new tooling without discussion. The work fits an S-effort manual pass.
- **Implications**: No new scripts; no new dependencies; manual translation file-by-file.
### Verification path
- **Context**: How does the reviewer confirm acceptance?
- **Sources Consulted**: Ticket body acceptance criteria; project's Vite build (`npm run build`).
- **Findings**: A single ripgrep command confirms Req 1.1; `npm run build` confirms Req 4.1; manual smoke confirms Req 4.3. No new test harness is justified for a doc-only change (per steering "Don't add a heavy test harness without discussing scope").
- **Implications**: PR description carries the verification one-liner; the build is the proof.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| A. Single-pass translation, no tooling | Translate all 20 files in one PR; manual judgment per comment | Simple, low overhead | Long diff for the largest 6 files | Matches "manual style" steering ethos |
| B. Automated LLM-driven script + manual review | Script extracts Chinese comments, LLM translates, dev reviews diff | Faster on long files | Adds dependency; undercuts judgment requirement; risk of touching strings/identifiers | Rejected — clashes with "no new tooling" steering |
| C. File-grouped manual pass (selected) | Same translation effort as A, but tasks split into file groups: high-touch / mid-touch / light | Reviewable progress, matches project's task-tracking pattern | A few extra task headings | Selected — pairs cleanly with `tasks.md` structure |
## Design Decisions
### Decision: Manual file-grouped translation, no tooling
- **Context**: 20 files, ~900 occurrences, mixed comment shapes plus a small set of in-scope dev-log strings.
- **Alternatives Considered**:
1. Single mass pass (Option A) — workable but reviewer-unfriendly for the largest files
2. Automated LLM translation script (Option B) — fast but loses per-comment judgment and adds tooling
3. File-grouped manual pass (Option C) — same effort as A with clearer task decomposition
- **Selected Approach**: Group files into three batches by occurrence count and translate each batch as one task. After each batch, run the verification ripgrep to check progress.
- **Rationale**: Aligns with `tech.md` steering ("match the surrounding file's style"), `dev-guidelines.md` ("comment the *why*"), and lets `tasks.md` mirror the existing project task-tracking pattern. The S-effort estimate fits one work session.
- **Trade-offs**: A few extra task headings vs. cleaner reviewability. No infrastructure cost.
- **Follow-up**: Confirm `console.*` Chinese strings are translated; confirm LLM prompts in `Step5Interaction.vue` are documented as retained in PR description.
### Decision: String-literal scope rule
- **Context**: Some Chinese appears in string literals, not just comments.
- **Alternatives Considered**:
1. Strict: comments only — leaves dev-facing `console.*` Chinese which any maintainer reading dev console would still see in Chinese
2. Permissive: all string literals — translates LLM prompt templates, changing model behavior
3. Targeted: comments + dev-facing log strings (`console.*`); retain LLM prompts as documented exceptions
- **Selected Approach**: Targeted (option 3). Translate `console.error`, `console.warn`, `console.log` strings whose content is Chinese. Leave LLM prompt template strings alone and list them in the PR description per Req 1.5.
- **Rationale**: Honors the spirit of the ticket ("English-readable code") while preserving Req 4 ("no runtime behavior change") for the LLM-bound strings. Matches Req 4.4 (string literals untouched *except* where dev-log translation is unambiguous).
- **Trade-offs**: Reviewer needs to verify the exception list in the PR description against the residual ripgrep matches. Mitigated by Req 5.1 (PR description must document residuals).
- **Follow-up**: During implementation, confirm there are no other categories of Chinese-string-literal beyond `console.*` and LLM prompts. If discovered, add to the documented exception list rather than expanding scope.
### Decision: TODO/FIXME ticket reference policy
- **Context**: Req 3 mandates ticket references on TODO/FIXME markers.
- **Alternatives Considered**:
1. Skip the sweep entirely if no markers exist
2. Sweep `frontend/src/` for `TODO|FIXME` once at the start; append `(#9)` only where missing
- **Selected Approach**: Run a single `rg 'TODO|FIXME' frontend/src/` sweep before the file-translation loop; record any matches; apply Req 3.1–3.3 inline with each file's translation.
- **Rationale**: Lightest-weight implementation of Req 3. If no markers exist (likely for `frontend/src/`), the requirement is satisfied vacuously and noted in the PR description.
- **Trade-offs**: None.
- **Follow-up**: If markers exist in non-Chinese form (English TODOs without ticket refs), the requirement says only to act on *Chinese* markers; out of scope to retrofit unrelated existing English TODOs.
## Risks & Mitigations
- **Risk**: Accidentally translating an LLM prompt string and shifting model behavior. **Mitigation**: Req 4.4 + Decision "String-literal scope rule"; document retained Chinese strings in PR.
- **Risk**: Misinterpreting a Chinese comment and translating to wrong meaning. **Mitigation**: Req 2.3 (conservative when ambiguous; keep + translate rather than delete).
- **Risk**: Reviewer churn over which comments to delete vs. translate. **Mitigation**: `dev-guidelines.md` is the rubric; Decision documents the rule (delete only when comment paraphrases the next statement; translate when the comment encodes intent).
- **Risk**: PR is too large to review (Process.vue alone has ~191 hits). **Mitigation**: File-grouped tasks + per-group ripgrep checkpoint; each group is reviewable as a unit.
## References
- `dev-guidelines.md` (project) — comment philosophy and Conventional Commits.
- `tech.md` (steering) — "No enforced linter or formatter… match the surrounding file's style."
- `structure.md` (steering) — `frontend/src/` directory layout (views/components/api/store).
- Ticket #9 body — acceptance criteria, branch and commit naming.
- Gap analysis (`gap-analysis.md`) — Option C trade-offs and effort/risk estimate.

23
.kiro/specs/i18n-frontend-comments/spec.json Normal file
View File

@ -0,0 +1,23 @@
{
"feature_name": "i18n-frontend-comments",
"created_at": "2026-05-07T16:24:12Z",
"updated_at": "2026-05-07T16:35:00Z",
"language": "en",
"phase": "tasks-generated",
"ticket": 9,
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": false
}
},
"ready_for_implementation": false
}

53
.kiro/specs/i18n-frontend-comments/tasks.md Normal file
View File

@ -0,0 +1,53 @@
# Implementation Plan
## Foundation
- [x] 1. Sweep TODO/FIXME markers and capture pre-translation baseline
- Run `rg 'TODO|FIXME' frontend/src/` and record all matches with file/line; for each, note whether the description is in Chinese (in scope for translation) or already English (out of scope per Boundary Commitments).
- Capture the pre-translation ripgrep baseline so the verification command output can be compared after the translation pass.
- Observable completion: a working note (not committed) listing every TODO/FIXME in `frontend/src/`, classified as "translate-and-tag", "already-tagged", or "English-out-of-scope", and a recorded count of files matching `[\x{4e00}-\x{9fff}]` in `frontend/src/` (expected: 20 files, ~902 occurrences).
- _Requirements: 3.1, 3.2, 3.3, 5.1_
## Core Translation Pass
- [x] 2. Translate light-touch files (≤10 hits)
- Translate Chinese comments to English in `App.vue`, `store/pendingUpload.js`, `views/MainView.vue`, `views/InteractionView.vue`, `views/ReportView.vue`, `components/Step1GraphBuild.vue`, `api/index.js`, `api/graph.js`, `api/report.js`. Apply the region-eligibility matrix from design.md: translate line/block/JSDoc/template comments; preserve JSDoc tag syntax; delete comments that merely restate the next statement; keep comments that encode intent.
- Translate Chinese content inside `console.error|warn|log` string literals in `components/Step1GraphBuild.vue` (3 known hits at lines 216, 237, 241). Leave all other string literals unchanged.
- For any TODO/FIXME marker that was Chinese and lacked a ticket reference, append `(#9)`; preserve existing references.
- Observable completion: `rg '[\x{4e00}-\x{9fff}]' frontend/src/{App.vue,store,views/MainView.vue,views/InteractionView.vue,views/ReportView.vue,components/Step1GraphBuild.vue,api}` returns no matches (no retained-bilingual cases expected in this group).
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 4.2, 4.4, 5.4_
- [x] 3. Translate mid-touch files (10–60 hits)
- Translate `api/simulation.js` (29 hits, JSDoc-heavy: keep `@param`, `@returns`, etc., translate only natural-language content), `views/SimulationRunView.vue` (18 hits), `views/SimulationView.vue` (22 hits), `views/Home.vue` (43 hits), `components/Step5Interaction.vue` (34 hits), `components/Step3Simulation.vue` (52 hits).
- In `components/Step5Interaction.vue`, retain Chinese inside the LLM prompt template strings (around lines 725–727) per Requirement 1.5; record file/line in a working note for the PR description. Translate all comments and any non-LLM-prompt Chinese content in this file as normal.
- For any other Chinese-content string literal encountered in this group, leave the literal unchanged and record file/line for the PR description.
- Observable completion: `rg '[\x{4e00}-\x{9fff}]' <files-in-group>` returns matches only for `components/Step5Interaction.vue` (LLM prompt strings) and any other documented retained-bilingual literals; no comment-region match remains in any of these files.
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 4.2, 4.4, 5.4_
- [x] 4. Translate high-touch files (>60 hits)
- Translate `components/Step2EnvSetup.vue` (76 hits), `components/GraphPanel.vue` (84 hits, mixed D3 logic comments and `<template>` comments), `components/HistoryDatabase.vue` (124 hits), `components/Step4Report.vue` (176 hits), `views/Process.vue` (191 hits, the 2067-line workflow orchestrator).
- These files concentrate ~80% of total occurrences; budget time accordingly. Apply the same region-eligibility matrix as task 2: translate comments, preserve JSDoc tag syntax, delete redundant comments, keep intent-bearing ones.
- Translate `console.*` Chinese strings if encountered; leave LLM prompts and other string literals unchanged and record for the PR description.
- Observable completion: `rg '[\x{4e00}-\x{9fff}]' frontend/src/components/{Step2EnvSetup,GraphPanel,HistoryDatabase,Step4Report}.vue frontend/src/views/Process.vue` returns no comment-region matches; any residuals are documented retained-bilingual string literals.
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 4.2, 4.4, 5.4_
## Integration & Validation
- [x] 5. Run final acceptance verification
- Run `rg '[\x{4e00}-\x{9fff}]' frontend/src/` on the full directory and confirm output is empty or contains only the pre-recorded retained-bilingual files (LLM prompt strings in `components/Step5Interaction.vue` and any others documented during tasks 2–4).
- Run `npm run build` and confirm exit code 0 and successful Vite build output.
- Run `git diff --stat main..HEAD` (or against the branch base) and confirm only `frontend/src/**` paths are modified — no edits under `/locales/`, `/backend/`, or repo root.
- Observable completion: all three checks pass; if any check fails, return to the relevant translation task before proceeding to PR.
- _Requirements: 1.1, 4.1, 4.2, 5.4_
- [x] 6. Manual UI smoke check
- Run `npm run dev`; in a browser, navigate Home → Process → each Step component (1–5) → Interaction → Report; confirm rendering matches the pre-translation baseline (no missing text, no broken bindings, no console errors that did not exist before).
- Per `tech.md` steering, the manual smoke is the only practical proof that no executable change crept in; type-check or build pass alone is not sufficient.
- Observable completion: every page renders identically to baseline; no new console errors; the implementer can confirm "UI unchanged" in the PR description.
- _Requirements: 4.3_
- [x] 7. Compose PR description with verification artifacts
- Draft the PR body listing: (a) the verification command `rg '[\x{4e00}-\x{9fff}]' frontend/src/` and its post-translation output, (b) the file count and any retained-bilingual files with one-line rationale per Requirement 5.2, (c) confirmation that the manual UI smoke passed, (d) confirmation that no files outside `frontend/src/` were modified.
- Use branch name `docs/i18n-9-translate-frontend-comments` and commit message `docs(i18n): translate chinese comments in frontend src to english` per Requirement 5.3 and the project's Conventional Commits rule.
- Observable completion: the PR description, branch name, and commit subject are ready to use by `/done`; all five Requirement 5 acceptance criteria are visibly satisfied in the PR body.
- _Requirements: 5.1, 5.2, 5.3, 5.4_

517
.kiro/specs/i18n-locale-parity-guard/design.md Normal file
View File

@ -0,0 +1,517 @@
# Design — i18n-locale-parity-guard
## Overview
This feature extends the project's PR-time i18n CI guard so that any pull request which introduces a key in only one of `locales/en.json` / `locales/zh.json` fails. It satisfies acceptance criterion #4 of epic #11 (locale-key parity) with a permanent automated check.
**Purpose**: Lock in locale-catalogue key parity as a permanent CI invariant so that AC #4 of epic #11 cannot regress as new strings are added.
**Users**: Project maintainers and PR authors. Maintainers gain a hard regression gate; PR authors gain a script they can run locally to confirm parity before pushing.
**Impact**: Adds a third check to the existing PR-time guard `scripts/ci/i18n_cjk_guard.py`. No production source under `backend/app/`, `frontend/src/`, or `locales/` is modified by this spec.
### Goals
- Fail any PR whose flattened-key set in `locales/en.json` differs from that of `locales/zh.json`.
- Print actionable failure lines (`<file>:<line>: parity-<en|zh>-only: <dotted-key>`) and a summary count.
- Compose with the existing CJK-clean and per-path-ratchet checks in a single CLI invocation, with a single exit code, no short-circuit.
- Run end-to-end in well under one second on the live catalogues; stdlib-only.
- Pass on `main` at the moment this spec ships (live catalogues are already parity-clean).
### Non-Goals
- Re-implementing the manual audit pipeline at `.kiro/specs/i18n-e2e-english-verification/audit/scripts/`. The new check is the CI extract; the audit retains its own copy of `check_parity.py`.
- Cross-locale value-equality, identical-value heuristics, or ICU-placeholder-shape checks.
- Auto-creating missing keys, suggesting translations, or reformatting the catalogues.
- Modifying the `locales/` schema, the `vue-i18n` runtime, or `backend/app/utils/locale.py`.
- Adding a new GitHub Actions workflow or workflow step.
## Boundary Commitments
### This Spec Owns
- The new parity-check helpers (`_flatten_keys`, `_locate_key_line`, `_format_parity_finding`, `run_parity_check`) and constants (`ZH_JSON_REL_PATH`) inside `scripts/ci/i18n_cjk_guard.py`.
- The new third block of `run_check` that invokes `run_parity_check` and integrates its result into the existing `failed` accumulator and `success_summary` collector.
- The pass/fail semantics of the locale-key parity check.
- New unit / integration tests under `scripts/ci/tests/` covering the parity check and its composition.
### Out of Boundary
- The audit pipeline at `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py` (independent, manual-only).
- The structure or format of the baseline file `.kiro/specs/i18n-ci-guard/baseline.txt` (parity is binary; no baseline needed).
- The workflow file `.github/workflows/i18n-cjk-guard.yml` (unchanged; same `python scripts/ci/i18n_cjk_guard.py` invocation already covers the new check).
- Any change to `locales/en.json` or `locales/zh.json` content.
- Open follow-up issues #7, #23, #25 (out-of-scope translation work).
### Allowed Dependencies
- Python ≥3.11 standard library (`json`, `os`, `pathlib`, `re`, `subprocess`, `sys`, `argparse`, `unittest`).
- The existing helpers `_flatten`, `_value_line_number`, `_truncate`, the `EN_JSON_REL_PATH` constant, and the `run_check`/`update_baseline` functions in `scripts/ci/i18n_cjk_guard.py`.
- `git` (for the existing CJK-counting block, untouched here).
### Revalidation Triggers
- Adding a third locale catalogue → parity becomes pairwise; design must be revisited.
- Changing the `flatten` contract (e.g. encoding non-dict containers like lists) → the parity check's "exact match with `check_parity.py`" clause must be re-asserted against the new contract.
- Splitting the guard into multiple CLI scripts → Requirement 3 ("one invocation") must be re-anchored.
## Architecture
### Existing Architecture Analysis
The guard is a single-file Python CLI: `scripts/ci/i18n_cjk_guard.py` (~393 lines, stdlib-only) invoked by one workflow step in `.github/workflows/i18n-cjk-guard.yml`. Its `run_check(repo_root, baseline_path) -> int` function is the orchestrator; today it composes two checks without short-circuit:
1. `scan_locale_cjk(en_json_path)` — fail when `locales/en.json` contains any CJK character.
2. Per-path baseline ratchet — fail when `count_path_cjk(repo_root, p)` exceeds `read_baseline(...)[p]` for any `p` in `("backend/app", "frontend/src")`.
A `failed: bool` accumulator is set independently by each block; a `success_summary: list[str]` collects "OK …" lines that print only on full success. This design extends it with a third block.
The audit pipeline at `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py` already implements the algorithm we need (recursive `flatten` + symmetric difference). Its logic is the canonical reference for Requirement 1.1.
### Architecture Pattern & Boundary Map
```mermaid
graph TB
Workflow[GitHub Actions step]
Main[main entry]
UpdateBaseline[update_baseline]
RunCheck[run_check orchestrator]
CjkClean[scan_locale_cjk]
Ratchet[count_path_cjk + read_baseline]
Parity[run_parity_check NEW]
EnJson[locales en.json]
ZhJson[locales zh.json]
BaselineFile[baseline.txt]
Workflow --> Main
Main -->|--update-baseline| UpdateBaseline
Main --> RunCheck
RunCheck --> CjkClean
RunCheck --> Ratchet
RunCheck --> Parity
CjkClean --> EnJson
Ratchet --> BaselineFile
Parity --> EnJson
Parity --> ZhJson
```
**Architecture Integration**:
- **Selected pattern**: Composed checks inside a single orchestrator (`run_check`). Each check is an independent function that returns a pass/fail signal and a list of human-readable lines; the orchestrator accumulates them.
- **Domain/feature boundaries**: Parity logic is internal to the guard module. It does not depend on the audit pipeline, the per-path ratchet, or the locale runtime.
- **Existing patterns preserved**: No-short-circuit composition, stderr-for-failure / stdout-for-success, lexicographic ordering for determinism, atomic-write / tmp-rename for any new persistence (none added here).
- **New components rationale**: `run_parity_check` is the only new orchestrator-level function; small private helpers (`_flatten_keys`, `_locate_key_line`, `_format_parity_finding`) keep `run_parity_check`'s body short and individually testable.
- **Steering compliance**: Stdlib-only; explicit type hints (PEP 604 union syntax already in use in this module); single-responsibility helpers; module dependency direction unchanged (still no imports from `backend/`, `frontend/`, or `locales/` runtime code).
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | n/a | n/a | This is a CI tool; no backend or service code is touched. |
| Infrastructure / Runtime | Python 3.11 stdlib (`json`, `pathlib`, `re`, `subprocess`, `sys`, `argparse`); GitHub Actions `ubuntu-latest`; `actions/checkout@v4`; `actions/setup-python@v5` | Runtime for the guard script and its new parity check. | Versions match the existing guard. No new dependencies; `pyproject.toml` and CI image unchanged. |
| Test Tooling | Python `unittest` (stdlib) | Drives parity check unit + integration tests. | Same framework as existing tests in `scripts/ci/tests/test_i18n_cjk_guard.py`. |
## File Structure Plan
### Directory Structure
```
scripts/
└── ci/
├── i18n_cjk_guard.py # Extended: adds parity helpers + third block in run_check
└── tests/
└── test_i18n_cjk_guard.py # Extended: adds ParityCheckTests + composition test
```
### Modified Files
- `scripts/ci/i18n_cjk_guard.py`
- Add module-level constants: `ZH_JSON_REL_PATH = "locales/zh.json"`.
- Add private helpers: `_flatten_keys`, `_locate_key_line`, `_format_parity_finding`.
- Add public function: `run_parity_check(repo_root: Path) -> ParityResult`.
- Add a new `NamedTuple` (or `@dataclass(frozen=True, slots=True)`) `ParityResult` with fields `(passed: bool, failure_lines: list[str], success_summary: str | None)`.
- Edit `run_check`: insert the parity block after the per-path-ratchet block, before the final `if not failed: print(success_summary)` block. Match the existing accumulator idiom.
- Update the module docstring to list three checks.
- `scripts/ci/tests/test_i18n_cjk_guard.py`
- Extend `_make_full_repo` (or add a sibling `_make_full_repo_with_zh`) to write a `locales/zh.json` alongside the existing `locales/en.json`. Keep the default ZH a parity-clean mirror of the EN fixture so existing tests do not need to change semantically.
- Add new test class `ParityCheckTests` covering Requirements 1.1, 1.2, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3, 2.5.
- Add one composition test (Requirement 5.1.f) inside `RunCheckEndToEndTests` (or a new `RunCheckCompositionTests` class) that plants a CJK string and a parity divergence in the same repo and asserts both failure lines + exit 1.
- Update existing `RunCheckEndToEndTests.test_*` to either commit a parity-clean `locales/zh.json` or assert the parity check now also runs but does not flip the test outcome.
### Files Not Created
- No new source file is created. Option C (separate `locale_parity.py` helper module) was rejected in `gap-analysis.md` and `research.md`.
- No new workflow file. The existing `.github/workflows/i18n-cjk-guard.yml` is invoked unchanged.
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | Flatten EN/ZH into matching dotted-key sets | `i18n_cjk_guard._flatten_keys` (new), reuses `_flatten` | `_flatten_keys(data: dict) -> set[str]` | n/a |
| 1.2 | Pass on identical key sets, success line includes shared count | `run_parity_check`, `run_check` | `ParityResult.success_summary` | Run-Check Composition |
| 1.3 / 1.4 | Fail on en-only or zh-only keys | `run_parity_check` | `ParityResult.passed`, `ParityResult.failure_lines` | Run-Check Composition |
| 1.5 | Dict leaves are non-leaves; scalar leaves are leaves | `_flatten_keys` (no type narrowing) | n/a | n/a |
| 2.1 | `<file>:<line>: parity-<side>-only: <key>` lines | `_format_parity_finding`, `_locate_key_line` | `_format_parity_finding(file, line, key, side) -> str` | n/a |
| 2.2 | Line-1 fallback when key not located | `_locate_key_line` | `_locate_key_line(text_lines, key) -> int` (returns 1 on miss) | n/a |
| 2.3 | Final `parity: en-only=N, zh-only=M` summary | `run_parity_check` | Last entry of `ParityResult.failure_lines` on failure | n/a |
| 2.4 | All parity output to stderr | `run_check` integration block | `print(..., file=sys.stderr)` | Run-Check Composition |
| 2.5 | Lexicographic ordering | `run_parity_check` | `sorted(...)` over symmetric difference | n/a |
| 3.1 | All checks run, no short-circuit | `run_check` (existing accumulator pattern) | `failed: bool` accumulator | Run-Check Composition |
| 3.2 / 3.3 | Single exit code: 1 on any fail, 0 otherwise | `run_check` | Returns `1 if failed else 0` | Run-Check Composition |
| 3.4 / 3.5 | `--update-baseline`, `--baseline`, `--repo-root` flags unchanged | `main`, `_build_parser` | Existing argparse surface | n/a |
| 3.6 | Workflow file unchanged | `.github/workflows/i18n-cjk-guard.yml` | n/a (no edit) | n/a |
| 4.1 | Stdlib-only | `i18n_cjk_guard` imports | No new imports | n/a |
| 4.2 | Sub-second runtime | `_flatten_keys` is O(keys); set-diff is O(keys) | n/a | n/a |
| 4.3 | Deterministic output | All sorts lexicographic | n/a | n/a |
| 5.1 (a–f) | Tests for success, en-only, zh-only, both, scalar-leaf, composition | `scripts/ci/tests/test_i18n_cjk_guard.py:ParityCheckTests` + composition test | n/a | n/a |
| 5.2 / 5.3 / 5.4 | Match existing test style; isolated fixtures; clean run on parity-clean repo | Same test file | n/a | n/a |
| 6.1 | Guard passes on live catalogues at HEAD | Manual run at implementation time | `python scripts/ci/i18n_cjk_guard.py` exit 0 | n/a |
| 6.2 | If divergence found, document in tasks.md and fix | n/a (does not trigger; live parity holds) | n/a | n/a |
## System Flows
### Run-Check Composition
```mermaid
sequenceDiagram
participant CLI as main
participant Orch as run_check
participant CjkChk as scan_locale_cjk
participant RatChk as ratchet block
participant ParChk as run_parity_check
participant Out as stderr/stdout
CLI->>Orch: run_check repo baseline
Orch->>CjkChk: scan en.json
CjkChk-->>Orch: findings list
alt findings non-empty
Orch->>Out: stderr cjk-in-en lines
Note over Orch: failed = True
else
Note over Orch: success summary append
end
Orch->>RatChk: count + read baseline
RatChk-->>Orch: regressions list
alt regressions non-empty
Orch->>Out: stderr cjk-regression lines + refresh hint
Note over Orch: failed = True
else
Note over Orch: success summary append
end
Orch->>ParChk: run parity check
ParChk-->>Orch: ParityResult
alt parity failed
Orch->>Out: stderr parity lines + parity summary
Note over Orch: failed = True
else
Note over Orch: success summary append
end
alt failed false
Orch->>Out: stdout success lines
end
Orch-->>CLI: 1 if failed else 0
```
**Key decisions**:
- The parity block is appended last so its (potentially long) failure list is contiguous in the failure stream.
- The `failed` accumulator is shared with the prior two blocks; this is the only mechanism for cross-block signalling.
- The summary line `parity: en-only=N, zh-only=M` is appended to `ParityResult.failure_lines` (last entry) so the orchestrator can print all failure lines uniformly without a special-case branch.
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|--------------|--------------------------|-----------|
| `_flatten_keys` | Guard / helper | Return the dotted-key set of a parsed JSON catalogue, mirroring `check_parity.py.flatten`. | 1.1, 1.5 | `_flatten` (P0, existing) | Service |
| `_locate_key_line` | Guard / helper | Best-effort line-number resolution for a dotted key in raw JSON text, with line-1 fallback. | 2.1, 2.2 | none | Service |
| `_format_parity_finding` | Guard / helper | Format one failure line as `<file>:<line>: parity-<side>-only: <key>`. | 2.1 | none | Service |
| `ParityResult` | Guard / DTO | Carry parity-check outcome (passed flag, failure lines, success-summary line). | 1.2, 2.3, 2.5 | none | State |
| `run_parity_check` | Guard / orchestrator-leaf | Read both catalogues, compute symmetric difference, build `ParityResult`. | 1.1–1.5, 2.1–2.5 | `_flatten_keys` (P0), `_locate_key_line` (P0), `_format_parity_finding` (P0) | Service |
| `run_check` (modified) | Guard / orchestrator | Compose the three checks with a single `failed` accumulator and exit code. | 3.1–3.3 | All three checks (P0) | Service |
| `ParityCheckTests` (test) | Tests | Unit + integration coverage for parity. | 5.1 (a–f), 5.2–5.4 | `run_parity_check`, `run_check` (P0) | Service |
### Guard / helper layer
#### `_flatten_keys`
| Field | Detail |
|-------|--------|
| Intent | Return the set of dotted-key paths of a parsed JSON object, mirroring `check_parity.py.flatten`. |
| Requirements | 1.1, 1.5 |
**Responsibilities & Constraints**
- Iterate via the existing `_flatten(prefix, value, out)` helper to guarantee identical path semantics.
- Descend only into `dict`. Any non-dict (string, number, bool, null, list) at a leaf produces a key.
- Return a `set[str]` so the parity caller can compute symmetric differences without re-deduplicating.
**Dependencies**
- Inbound: `run_parity_check` (P0).
- Outbound: `_flatten` (P0, existing private helper in same module).
**Contracts**: Service [x]
##### Service Interface
```python
def _flatten_keys(data: dict[str, object]) -> set[str]:
...
```
- Preconditions: `data` is the result of `json.loads` over a catalogue file (i.e., a `dict` at the top level).
- Postconditions: every dotted path returned corresponds to a non-`dict` leaf in `data`. The set is unordered; callers must sort before formatting output (Requirement 2.5).
- Invariants: `_flatten_keys({}) == set()`. For any catalogue `c`, `_flatten_keys(c)` is identical to the set of keys produced by `check_parity.py.flatten(c)`.
**Implementation Notes**
- Integration: One call site (`run_parity_check`).
- Validation: Unit-test against a hand-rolled fixture with mixed leaf types (string, number, bool, null) and at least three nesting levels (Requirement 5.1.e).
- Risks: None. Reuses the existing flatten primitive verbatim.
#### `_locate_key_line`
| Field | Detail |
|-------|--------|
| Intent | Best-effort line-number resolution for a dotted key in the raw JSON source text, with a deterministic line-1 fallback. |
| Requirements | 2.1, 2.2 |
**Responsibilities & Constraints**
- Accept the splitlines view of a JSON file (`text_lines: list[str]`) and a dotted key (`dotted_key: str`).
- Search for the leaf segment of the dotted key (after the last `.`) wrapped in JSON quotes, e.g. `"missingKey"`. Return the 1-based line number of the first match.
- Fall back to `1` when no match is found (mirrors `_value_line_number`).
- Performance must remain linear in the number of lines.
**Dependencies**
- Inbound: `run_parity_check` (P0).
- Outbound: none.
**Contracts**: Service [x]
##### Service Interface
```python
def _locate_key_line(text_lines: list[str], dotted_key: str) -> int:
...
```
- Preconditions: `dotted_key` non-empty; `text_lines` is the result of `Path.read_text(...).splitlines()`.
- Postconditions: returns an integer ≥ 1.
- Invariants: When the leaf segment appears in `text_lines` wrapped in `"..."`, the return is the (1-based) line number of the first occurrence. Otherwise the return is `1`.
**Implementation Notes**
- Integration: One call site (`run_parity_check`).
- Validation: Unit-test the exact-match path, the multi-occurrence path (first match wins), and the not-found fallback.
- Risks: A leaf segment that also appears as part of another (unrelated) key or in a value text could yield a slightly misleading line number. Acceptable: the dotted key in the failure message is the source of truth; the line is a navigation aid. Documented in the docstring.
#### `_format_parity_finding`
| Field | Detail |
|-------|--------|
| Intent | Format a single parity-failure line in the canonical layout used by the guard. |
| Requirements | 2.1 |
**Responsibilities & Constraints**
- Produce strings of the exact form `<file>:<line>: parity-en-only: <dotted-key>` or `<file>:<line>: parity-zh-only: <dotted-key>`.
- Mirror the existing `_format_locale_finding` style (`<file>:<line>: <category>: <payload>`).
**Dependencies**
- Inbound: `run_parity_check` (P0).
- Outbound: none.
**Contracts**: Service [x]
##### Service Interface
```python
def _format_parity_finding(file_rel_path: str, line_no: int, dotted_key: str, side: str) -> str:
...
```
- Preconditions: `side in {"en-only", "zh-only"}`; `file_rel_path` is one of `EN_JSON_REL_PATH` / `ZH_JSON_REL_PATH`; `line_no >= 1`.
- Postconditions: returns a single line with no embedded newline.
- Invariants: The category token in the line is exactly `parity-en-only` or `parity-zh-only` so log greps match deterministically.
### Guard / DTO layer
#### `ParityResult`
| Field | Detail |
|-------|--------|
| Intent | Immutable carrier for parity-check outcome consumed by `run_check`. |
| Requirements | 1.2, 2.3, 2.5 |
**Contracts**: State [x]
##### State Management
- State model:
```python
class ParityResult(NamedTuple):
passed: bool
failure_lines: list[str] # already-formatted lines, including the trailing "parity: en-only=N, zh-only=M" summary on failure
success_summary: str | None # populated only when passed is True
```
- Persistence & consistency: in-memory only; constructed by `run_parity_check` and consumed by `run_check`.
- Concurrency strategy: n/a (single-process, single-call).
### Guard / orchestrator-leaf
#### `run_parity_check`
| Field | Detail |
|-------|--------|
| Intent | Compute the locale-key parity outcome and produce a `ParityResult`. |
| Requirements | 1.1–1.5, 2.1–2.5 |
**Responsibilities & Constraints**
- Read both `locales/en.json` and `locales/zh.json` from `repo_root`.
- Flatten each via `_flatten_keys` and compute the symmetric difference.
- For each en-only key (sorted lexicographically): resolve its line via `_locate_key_line` over the EN catalogue's source-text lines, and emit a `parity-en-only` line via `_format_parity_finding`.
- For each zh-only key (sorted lexicographically, after en-only): resolve its line via `_locate_key_line` over the ZH catalogue's source-text lines, and emit a `parity-zh-only` line.
- On failure, append a final `parity: en-only=N, zh-only=M` summary line to `failure_lines`.
- On success, build the success summary `OK locale-parity: <count> keys per side`.
- If either catalogue file is missing, return a `ParityResult(passed=False, failure_lines=[<single error line>], success_summary=None)` and let `run_check` fold the error into the global `failed` flag.
**Dependencies**
- Inbound: `run_check` (P0).
- Outbound: `_flatten_keys`, `_locate_key_line`, `_format_parity_finding` (all P0).
**Contracts**: Service [x]
##### Service Interface
```python
def run_parity_check(repo_root: Path) -> ParityResult:
...
```
- Preconditions: `repo_root` is a valid working-tree directory; `locales/en.json` and `locales/zh.json` are expected at the relative paths defined by `EN_JSON_REL_PATH` and `ZH_JSON_REL_PATH`.
- Postconditions: returns a `ParityResult`. When `passed`, `failure_lines == []` and `success_summary` is non-`None`. When not `passed`, `failure_lines` is non-empty and ends with a `parity: en-only=…` summary line; `success_summary` is `None`.
- Invariants: Flattened-key-set computation matches `check_parity.py.flatten` byte-for-byte for any input. Output is deterministic across runs for identical inputs.
**Implementation Notes**
- Integration: Called once per `run_check` invocation. Skipped entirely in `--update-baseline` mode (covered by Requirement 3.4 — `update_baseline` is invoked from `main` instead of `run_check`).
- Validation: Unit-test all required outcomes (Requirement 5.1 a–e); integration-test composition (5.1 f).
- Risks: A malformed JSON catalogue raises `json.JSONDecodeError`. The function should treat this the same as a missing file (return `ParityResult(passed=False, …)`), so the guard reports a clean failure rather than crashing CI with a Python traceback.
### Guard / orchestrator (modified)
#### `run_check` (modification)
| Field | Detail |
|-------|--------|
| Intent | Compose all three checks (CJK-clean, per-path ratchet, parity) into one exit code. |
| Requirements | 3.1, 3.2, 3.3 |
**Responsibilities & Constraints**
- After the existing per-path-ratchet block (existing line ~258–293) and before the final `if not failed` block (existing line ~295–298), call `run_parity_check(repo_root)`.
- If the result is not passed, set `failed = True`, print every entry of `result.failure_lines` to `sys.stderr`, one line per `print(...)` call.
- If passed, append `result.success_summary` to `success_summary`.
- Return `1 if failed else 0` (unchanged).
**Dependencies**
- Inbound: `main` (P0, via either standalone CLI or test invocation).
- Outbound: `scan_locale_cjk`, per-path ratchet helpers, `run_parity_check` (all P0).
**Contracts**: Service [x] / State [x]
##### Service Interface
Unchanged signature: `def run_check(repo_root: Path, baseline_path: Path) -> int`.
- Preconditions: unchanged.
- Postconditions: exit code reflects all three checks (was: two checks).
- Invariants: still no short-circuit between checks.
**Implementation Notes**
- Integration: One inserted block of ~10 lines in the existing function.
- Validation: Existing CLI smoke tests continue to pass; new `RunCheckEndToEndTests` cases assert correct fail/pass propagation when only the parity check fails, only an existing check fails, or both fail.
- Risks: A future maintainer could accidentally short-circuit by inserting an early `return` between blocks. Mitigated by the composition test (Requirement 5.1.f) which fails if any block is skipped.
### Tests
#### `ParityCheckTests`
| Field | Detail |
|-------|--------|
| Intent | Unit + integration coverage for the parity check, matching the style of existing `RunCheckEndToEndTests`. |
| Requirements | 5.1 (a–f), 5.2, 5.3, 5.4 |
**Responsibilities & Constraints**
- Use `unittest`, `tempfile.TemporaryDirectory`, and the existing `_make_repo` / `_commit_file` test helpers.
- Each test owns its own ephemeral repo. No reliance on the live `locales/` content for negative paths (Requirement 5.3).
- Assertions check exit code AND substring presence of the failure category tokens (`parity-en-only`, `parity-zh-only`) AND that the summary line is the last failure line.
**Dependencies**
- Inbound: `unittest.main`.
- Outbound: `i18n_cjk_guard.run_parity_check`, `i18n_cjk_guard.run_check` (both P0).
**Implementation Notes**
- Test cases (one per Requirement 5.1 sub-bullet):
- (a) `test_passes_when_keys_match` — both catalogues identical → `run_parity_check` returns `passed=True`; `run_check` returns 0.
- (b) `test_fails_on_en_only_key` — `en.json` has an extra key → `run_parity_check` returns `passed=False`, failure includes `parity-en-only`, summary is `parity: en-only=1, zh-only=0`.
- (c) `test_fails_on_zh_only_key` — symmetric of (b).
- (d) `test_fails_on_both_sided_divergence` — failure list contains both `parity-en-only` and `parity-zh-only` lines, ordered en-first then zh, each lex-sorted within its group.
- (e) `test_passes_with_scalar_leaves_at_same_path` — both catalogues have a scalar (e.g. `null`, `42`, `false`) at the same dotted path → parity passes (Requirement 1.5).
- (f) `test_run_check_no_short_circuit` — one repo plants both a CJK in `en.json` and a parity-divergent key. Expect: exit 1; stderr contains both `cjk-in-en` and `parity-en-only` (or `parity-zh-only`); the per-path-ratchet success summary is suppressed (since failed).
- Risks: Test fixtures must use `ensure_ascii=False` JSON to match the live catalogue style.
## Error Handling
### Error Strategy
- **Missing catalogue file** → `run_parity_check` returns `ParityResult(passed=False, failure_lines=[<missing-file-line>], success_summary=None)`. `run_check` flips `failed`, prints the line to stderr, returns 1.
- **Malformed JSON** → same path as missing catalogue. `json.JSONDecodeError` is caught inside `run_parity_check`; the line printed names the offending file and the parser's `msg`.
- **Parity divergence** (the expected unhappy path) → fail per Requirements 1.3 / 1.4 / 2.1–2.5.
- **`_locate_key_line` cannot find the key** → fall back to line 1 (Requirement 2.2). Not an error; the caller proceeds.
- **No-short-circuit invariant** → enforced by the orchestrator's accumulator pattern; covered by Requirement 5.1.f.
### Monitoring
CI workflow logs (GitHub Actions) are the sole observability surface. Failure lines are designed to be greppable: `parity-en-only`, `parity-zh-only`, `parity: en-only=`, `parity: zh-only=` are stable tokens.
## Testing Strategy
### Unit Tests
- `_flatten_keys`: empty input, flat input, mixed-type leaves, three-level nesting, `null` and scalar leaves.
- `_locate_key_line`: exact match, multi-occurrence (first wins), not found (line-1 fallback).
- `_format_parity_finding`: en-only and zh-only sides, embedded special characters in key names (e.g. underscores, digits).
- `ParityResult`: pass-shape and fail-shape construction.
### Integration Tests
- All six `ParityCheckTests` sub-cases listed above.
- The composition case (Requirement 5.1.f) inside `RunCheckCompositionTests` (or appended to `RunCheckEndToEndTests`).
- A regression of the existing `RunCheckEndToEndTests` cases after extending `_make_full_repo` to write a default parity-clean `locales/zh.json`.
### Performance / Load
- One sanity case: parity check on a synthetic 10 000-key catalogue completes in well under one second on the CI runner. Asserted by a `time.perf_counter()` budget of 1.0 s in the integration test.
## Performance & Scalability
- Catalogue size: ~1000 keys today; growth bounded by the number of UI strings + log keys. Even at 10× the current size, `_flatten` + set-diff remains negligible (<100 ms).
- The CI workflow timeout is 1 minute (`.github/workflows/i18n-cjk-guard.yml:timeout-minutes: 1`); the new check adds at most tens of milliseconds.
## Supporting References
- `gap-analysis.md` (this spec) — implementation-approach options A/B/C with rationale.
- `research.md` (this spec) — design decision records.
- `.kiro/specs/i18n-ci-guard/design.md` — prior CI guard's design doc (style and boundary precedents).
- `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py` — reference parity algorithm.

154
.kiro/specs/i18n-locale-parity-guard/gap-analysis.md Normal file
View File

@ -0,0 +1,154 @@
# Gap Analysis — i18n-locale-parity-guard
## Current State Investigation
### Domain assets
| Asset | Path | Role |
|------|------|------|
| Existing PR-time guard | `scripts/ci/i18n_cjk_guard.py` (393 lines) | Runs (a) zero-CJK-in-`en.json`, (b) per-path CJK ratchet on `backend/app` + `frontend/src`. CLI: `--update-baseline`, `--baseline`, `--repo-root`. Stdlib-only. |
| Workflow | `.github/workflows/i18n-cjk-guard.yml` | `pull_request` trigger; single step `python scripts/ci/i18n_cjk_guard.py`. 1-minute timeout. Python 3.11. |
| Existing tests | `scripts/ci/tests/test_i18n_cjk_guard.py` (358 lines) | `unittest`, stdlib-only. Per-function test classes (`ScanLocaleCjkTests`, `CountPathCjkTests`, `BaselineRoundTripTests`, `RunCheckEndToEndTests`, `UpdateBaselineTests`, `CliSmokeTests`). Synthetic git repos via `tempfile.TemporaryDirectory` + `git init`. |
| Reference parity logic | `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py` (128 lines) | Already implements `flatten()` (recursive dotted-key generator) and the EN/ZH symmetric-difference computation. Used only by the manual audit pipeline; not in CI. |
| Locale catalogues | `locales/en.json`, `locales/zh.json` | Two-space-indented JSON, `ensure_ascii=False`. 962 keys per side at HEAD; symmetric difference 0. Multi-level nesting (e.g. `common.confirm`, `step1.upload.title`, `log.api.graph.startBuild`). |
| Prior spec | `.kiro/specs/i18n-ci-guard/{design.md,baseline.txt}` | Documents the CJK-guard's design, format, and "scope ratchets only" rationale. The new check should compose, not replace. |
### Conventions extracted
- **Module layout**: One CLI script per check class; checks compose inside a `run_check(...)` orchestrator that returns 0/1.
- **Output discipline**: Stderr for failures, stdout for success summaries. Each failure line is self-contained (`<file>:<line>: <category>: <key/payload>`). Refresh hints (when applicable) printed once at the end.
- **No-short-circuit composition**: `run_check` evaluates every check before exiting (existing pattern at lines 230, 258, 271 in `i18n_cjk_guard.py`).
- **Stdlib-only, deterministic**: existing module imports only `argparse`, `json`, `os`, `re`, `subprocess`, `sys`, `pathlib`. All sorts use lexicographic order.
- **Test-fixture isolation**: Each test owns a `tempfile.TemporaryDirectory()` and writes its own JSON / source files. Negative-path tests never depend on the live `locales/`.
- **Atomic writes**: `write_baseline` uses tmp-file + `os.replace`; if any new persistence is added, mirror that pattern.
- **JSON line-resolution helper**: `_value_line_number(text_lines, value)` already implements the line-fallback semantics required by R2.2 (returns 1 when value not found). Reusable for parity reporting if we resolve by **key name** rather than by **value**.
### Integration surfaces
- The workflow file invokes the guard exactly once: `python scripts/ci/i18n_cjk_guard.py`. Anything done inside `run_check` is automatically picked up — **no workflow change needed** if we extend the existing script (R3.6).
- `--update-baseline` short-circuits inside `main()` *before* `run_check` is called; the new parity check naturally won't run in that mode (R3.4).
- The audit pipeline at `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py` is independent and stays untouched (R6's "spec for prior CI guard" boundary).
- Baseline file format is single-purpose (CJK counts) and does not need to grow to accommodate parity (parity has no baseline — divergence is binary).
## Requirement-to-Asset Map
| # | Requirement | Existing asset(s) | Gap tag | Notes |
|---|-------------|------------------|---------|-------|
| 1.1 | Flatten EN/ZH into dotted keys matching `check_parity.py` | `audit/scripts/check_parity.py:flatten` (reference); existing `_flatten` in guard also flattens but only collects (key, value) pairs into a list | **Constraint** | Two `_flatten` flavours exist. Need ONE canonical function inside the guard module that mirrors `check_parity.py.flatten` (recursive, descends into dicts only, emits leaf scalars). The existing private `_flatten(prefix, value, out)` in the guard is already key-value-emitting and will work; the parity check just consumes its keys. |
| 1.2 | Pass when key sets identical, emit success summary with key count | `success_summary` list in `run_check` | **Missing** | Add a parity success line in the same idiom: `"OK locale-parity: 962 keys per side"`. |
| 1.3 / 1.4 | Fail on en-only or zh-only keys | None — no parity check exists | **Missing** | Compute symmetric difference. |
| 1.5 | Treat dict leaves as non-leaves; treat scalar leaves the same as string leaves for parity | `_flatten` already descends only into dicts and emits any non-dict as a leaf; `scan_locale_cjk` then narrows to strings, but parity should NOT narrow | **Constraint** | Use `_flatten` directly (no narrowing). |
| 2.1 | Print `<file>:<line>: <key>: en-only|zh-only` | `_value_line_number` resolves a value's line; needs adaptation for keys | **Missing** | Search for the JSON key token (e.g. `"missingKey"`) in the source-text lines using a substring scan; reuse the line-1 fallback from `_value_line_number`. |
| 2.2 | Fall back to line 1 when location not found | `_value_line_number` already returns 1 in this case | **Reuse** | |
| 2.3 | Final summary `parity: en-only=<n>, zh-only=<m>` | None | **Missing** | One line, stderr. |
| 2.4 | All parity output to stderr | `print(..., file=sys.stderr)` pattern used everywhere | **Reuse** | |
| 2.5 | Lexicographic sort | Existing patterns use `sorted(...)` | **Reuse** | |
| 3.1 / 3.2 / 3.3 | Compose with existing checks; one exit code | `run_check` already composes (a) and (b) without short-circuit | **Constraint** | Insert (c) at the end of `run_check`, after the per-path block but before the final return. Each check toggles the same `failed` flag. |
| 3.4 | `--update-baseline` does not run parity | `main()` short-circuits to `update_baseline()` and never enters `run_check` | **Reuse** | Untouched. |
| 3.5 | `--baseline` and `--repo-root` semantics unchanged | `_build_parser` and `_detect_repo_root` | **Reuse** | Untouched. |
| 3.6 | Workflow file unchanged | `.github/workflows/i18n-cjk-guard.yml` | **Reuse** | No edit needed. |
| 4.1 | Stdlib-only | Existing module is stdlib-only | **Reuse** | `json` is the only library needed for ZH loading. |
| 4.2 | Sub-second runtime | ~1k keys; flatten + set diff is O(n) | **Constraint** | Trivially holds. |
| 4.3 | Deterministic output | All sorts lexicographic | **Reuse** | |
| 5.1–5.4 | Tests under `scripts/ci/tests/` for success / en-only / zh-only / both / scalar-leaves / no-short-circuit | `test_i18n_cjk_guard.py:RunCheckEndToEndTests` is the integration class | **Missing** | Add either a new `ParityCheckTests` class or extend `RunCheckEndToEndTests`. Reuse `_make_full_repo` style; need a `zh_json` argument or a new helper that writes both locale files. |
| 6.1 | Guard passes on live catalogues at merge target | EN/ZH parity verified manually (962/962, 0 diff) | **Reuse** | Manual run after implementation. |
| 6.2 | Document any blocking divergence in tasks.md | n/a | **Conditional** | Only relevant if 6.1 fails — currently does not. |
### Complexity signal
- **Algorithmic logic** only: load two JSON files, recursive flatten, set diff, sort, format, print. No external integrations, no I/O contention, no perf concerns at the catalogue size.
## Implementation Approach Options
### Option A — Extend `scripts/ci/i18n_cjk_guard.py` *(recommended)*
**What changes**:
- Add private helpers to the existing module:
- `_flatten_keys(data) -> set[str]` — wrapper over the existing `_flatten` that returns just the dotted-key set.
- `_locate_key_line(text_lines, dotted_key) -> int` — substring scan for the leaf segment (after the last `.`) wrapped in JSON quotes; returns 1 on miss (mirrors `_value_line_number`'s fallback).
- `_format_parity_finding(file_rel_path, line_no, dotted_key, side) -> str` — single-line formatter.
- Add a function `run_parity_check(repo_root) -> tuple[bool, list[str], str]` returning `(passed, failure_lines, success_summary_line)`. Callable independently for tests.
- In `run_check`, after the per-path baseline block and before the final return:
- Call `run_parity_check(repo_root)`.
- If failed, set `failed = True`, print all failure lines + the `parity: ...` summary to stderr.
- If passed, append the success line to `success_summary`.
- Add a `ZH_JSON_REL_PATH` constant alongside `EN_JSON_REL_PATH`.
**Compatibility assessment**:
- All existing CLI flags, exit codes, and stdout/stderr patterns preserved.
- No new top-level dependencies. `json` already imported.
- The module grows to ~470 lines, comparable to similar single-purpose CLI scripts in the repo (`oasis_profile_generator.py` is much larger). Single-responsibility is preserved: the responsibility is "PR-time i18n catalogue health," and parity is a sub-instance of that.
- Existing tests continue to pass unmodified (none of the changed functions break their contract).
**Trade-offs**:
- ✅ Zero workflow churn, single CI job, single CLI surface.
- ✅ Reuses `_flatten`, line-resolution fallback, sort/print idioms.
- ✅ All checks fail/pass together — easier to read in CI logs.
- ❌ Module name (`i18n_cjk_guard`) is now slightly misleading: it also enforces parity, not just CJK presence. Mitigated by docstring update.
### Option B — New parallel script `scripts/ci/i18n_locale_parity_guard.py` + new workflow step
**What changes**:
- New script that implements the parity check standalone.
- Either (i) add a second job to `.github/workflows/i18n-cjk-guard.yml`, or (ii) add a new workflow file `i18n-locale-parity-guard.yml`.
- New test file `scripts/ci/tests/test_i18n_locale_parity_guard.py`.
**Compatibility assessment**:
- Both scripts duplicate `_flatten`, line-resolution helper, JSON loader, repo-root detection, argparse boilerplate.
- Two CI runs (or two steps) to read and ack on every PR.
**Trade-offs**:
- ✅ Single-responsibility script per file (matches one literal reading of project conventions).
- ❌ Code duplication ~80 lines.
- ❌ Two CI surfaces; PR review fatigue.
- ❌ Violates the spirit of R3 ("compose with the existing checks") — composing across two scripts requires either `&&` or two-job aggregation.
### Option C — Hybrid: new helper module + extended guard
**What changes**:
- New module `scripts/ci/locale_parity.py` exposing `compute_parity_findings(en_path, zh_path) -> ParityResult`.
- The existing `i18n_cjk_guard.py` imports from it and integrates the call into `run_check`, identical to Option A's runtime behaviour.
- Tests split: `test_locale_parity.py` covers the helper in isolation; `test_i18n_cjk_guard.py` gains one composition test.
**Compatibility assessment**:
- Adds package-style imports inside `scripts/ci/` (currently flat — `scripts/ci/i18n_cjk_guard.py` adds `_GUARD_DIR` to `sys.path` via the test bootstrap, which works for sibling modules without further config).
- No workflow change.
**Trade-offs**:
- ✅ Clean separation, more reusable helper.
- ✅ Possible to import the helper from the audit pipeline later (collapsing the duplicate `check_parity.py`).
- ❌ More files for what is ~80 lines of new logic; over-engineering for current scope.
- ❌ Risks scope creep into "deduplicate `check_parity.py`," which is explicitly out of scope.
## Effort & Risk
- **Effort**: **S** (1–2 days). Existing module patterns are mature; the algorithmic logic is small and proven (`check_parity.py`); test scaffolding is already in place.
- **Risk**: **Low**. Stdlib-only; no external integrations; no shared mutable state; deterministic algorithm; existing CI workflow unchanged; live catalogues already pass.
## Recommendations for Design Phase
### Preferred approach: Option A (extend `scripts/ci/i18n_cjk_guard.py`)
Rationale:
1. The existing module's docstring already says "PR-time guard: fail when locales/en.json contains CJK or when backend/app + frontend/src CJK match counts exceed the committed baseline." Extending it to also fail on locale-key parity is the smallest possible delta that also reads naturally in the codebase.
2. R3 ("composes with the existing CJK and per-path checks; one CLI; no workflow edit") is satisfied trivially.
3. Reuses `_flatten`, line-fallback, sort/print idioms verbatim.
4. The module name remains accurate — "CJK Guard" is the canonical name of the i18n PR-time gate; we'll add a docstring note that parity is the third covered check.
### Key design decisions to settle in `design.md`
- **Function boundary**: should `run_parity_check` live in the same module or in a small helper module? *Suggest: same module, as a private function alongside `count_path_cjk` / `scan_locale_cjk` for symmetry.*
- **Failure line format**: exact string layout (file:line:key:side, ordering of the four pieces, separator characters). *Suggest mirroring `_format_locale_finding` exactly: `f"{file}:{line}: {category}: {key}"` where `category` is `parity-en-only` or `parity-zh-only`.*
- **Test fixture for `RunCheckEndToEndTests`**: extend `_make_full_repo` to accept an optional `zh_json` parameter, or add a sibling helper. *Suggest extending — keeps the integration test in one place and lets the existing tests opt out by passing `zh_json=None` (the helper writes a parity-clean default).*
- **Whether to expose a `--check=parity` selector**: *Out of scope per R3.1 (no short-circuit, all-or-nothing).*
### Research items to carry forward
None. All required information is in the existing repo and the cited reference scripts. No external dependencies, no new tech, no perf research, no security implications.

96
.kiro/specs/i18n-locale-parity-guard/requirements.md Normal file
View File

@ -0,0 +1,96 @@
# Requirements Document
## Introduction
Epic #11 ("complete english support across ui, agents, logs, and docs") states as acceptance criterion #4: *"For every externalized log message, matching `log.*` keys exist in both `locales/en.json` and `locales/zh.json`."* The wider intent is symmetric: any externalized string introduced into either locale catalogue must have a counterpart in the other, otherwise English users hit fallback keys at runtime (and the inverse for Chinese users).
Parity holds today (962 keys per side, symmetric difference 0), but no automated check enforces it. The existing CI guard at `scripts/ci/i18n_cjk_guard.py` (workflow `.github/workflows/i18n-cjk-guard.yml`, landed via #26) only enforces (1) zero CJK in `locales/en.json` and (2) a per-path CJK count ratchet for `backend/app` + `frontend/src`. The audit script at `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py` does compute the symmetric difference, but only as part of a manual audit — it never runs in CI.
This spec extends the existing PR-time CI guard to enforce locale-key parity permanently. Once shipped, any pull request that introduces a key on only one side will fail CI with a precise list of the offending keys, freezing AC #4 in place for the rest of the epic and beyond.
## Boundary Context
- **In scope**:
- Symmetric-difference check between flattened dotted-key sets of `locales/en.json` and `locales/zh.json`.
- Integration of the new check into the existing `scripts/ci/i18n_cjk_guard.py` so the existing workflow `.github/workflows/i18n-cjk-guard.yml` exercises it without any workflow edit beyond what's strictly necessary.
- Test coverage under `scripts/ci/tests/` matching the style of the existing CJK-guard tests.
- Failure output formatted so a developer can locate the offending key without further tooling.
- **Out of scope**:
- Translating any remaining hard-coded strings in `backend/app` or `frontend/src` (tracked under open assigned issues #7, #23, #25).
- Value-equality, identical-value, or "review-needed" heuristics from the audit script's `[identical-values]` block — only key presence is asserted here.
- Any change to the `locales/` directory layout, schemas, or to `vue-i18n` / `backend/app/utils/locale.py` consumers.
- Cross-locale value-shape checks (e.g. matching ICU placeholders).
- README, `.env.example`, or documentation updates beyond what's needed inside the spec / guard module itself.
- **Adjacent expectations**:
- The existing CJK-clean and per-path-ratchet checks in `scripts/ci/i18n_cjk_guard.py` continue to run unchanged and report independently of the new parity check.
- The audit pipeline at `.kiro/specs/i18n-e2e-english-verification/audit/scripts/` keeps its own copy of `check_parity.py` for manual deep-dive use; the new CI check does not depend on the audit pipeline being invoked.
- All four checks (CJK in en.json, per-path ratchet, en-only keys, zh-only keys) run in a single CI job and surface together; no short-circuit between them.
## Requirements
### Requirement 1: Locale-key parity check
**Objective:** As a maintainer of the i18n catalogues, I want a CI check that detects any key present on only one of `locales/en.json` / `locales/zh.json`, so that AC #4 of epic #11 stays satisfied as new strings are added.
#### Acceptance Criteria
1. The i18n CJK Guard shall load `locales/en.json` and `locales/zh.json` and flatten each into a set of dotted keys whose paths exactly match those produced by `flatten()` in `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py`.
2. When the flattened EN and ZH key sets are identical, the i18n CJK Guard shall pass the parity check and emit a single success summary line that includes the shared key count.
3. When the flattened EN key set contains any key that is absent from ZH, the i18n CJK Guard shall fail the parity check.
4. When the flattened ZH key set contains any key that is absent from EN, the i18n CJK Guard shall fail the parity check.
5. The i18n CJK Guard shall treat a leaf whose value is a nested object as a non-leaf (no key emitted) and shall treat a leaf whose value is a non-string scalar (number, boolean, null) the same way it treats a string leaf for parity purposes.
### Requirement 2: Actionable failure reporting
**Objective:** As a developer whose PR is failing on parity, I want the failure message to name every offending key and the side it is missing on, so that I can fix the divergence without re-running the audit pipeline.
#### Acceptance Criteria
1. If the parity check fails, then the i18n CJK Guard shall print one line per missing key in the form `<locales/en.json|locales/zh.json>:<line>: <dotted-key>: en-only` or `... zh-only`, with `<line>` being the 1-based line number of that key in the source JSON file.
2. If a missing key cannot be located in its source file (e.g. owing to JSON formatting), then the i18n CJK Guard shall fall back to line 1 and still print the offending key and side.
3. If the parity check fails, then the i18n CJK Guard shall print a final summary line of the form `parity: en-only=<n>, zh-only=<m>` where `<n>` and `<m>` are the counts of en-only and zh-only keys.
4. The i18n CJK Guard shall print all parity-related output to stderr.
5. The i18n CJK Guard shall sort each side's missing-key list lexicographically so that the failure output is deterministic across environments.
### Requirement 3: Integration with the existing guard
**Objective:** As a maintainer extending the CI guard, I want the new parity check to compose with the existing CJK-clean and per-path-ratchet checks rather than replace them, so that all four checks are visible in a single CI run.
#### Acceptance Criteria
1. The i18n CJK Guard shall execute all of (a) the CJK-clean check on `locales/en.json`, (b) the per-path baseline ratchet on `backend/app` and `frontend/src`, and (c) the new parity check on every invocation of `python scripts/ci/i18n_cjk_guard.py` without short-circuiting between checks.
2. When any of (a), (b), or (c) fail, the i18n CJK Guard shall exit with status code 1.
3. When all of (a), (b), and (c) pass, the i18n CJK Guard shall exit with status code 0.
4. The i18n CJK Guard shall continue to support the `--update-baseline` flag with its existing semantics (refresh per-path counts and exit 0); the parity check shall not run in `--update-baseline` mode.
5. The i18n CJK Guard shall continue to support the `--baseline` and `--repo-root` flags with their existing semantics.
6. The existing GitHub Actions workflow `.github/workflows/i18n-cjk-guard.yml` shall continue to invoke the guard via the same single command (`python scripts/ci/i18n_cjk_guard.py`), with no new workflow steps required.
### Requirement 4: Stdlib-only, deterministic, fast
**Objective:** As a CI operator, I want the parity check to run quickly and without new dependencies, so that the existing 1-minute job timeout still holds.
#### Acceptance Criteria
1. The i18n CJK Guard shall implement the parity check using only the Python standard library; no new package shall be added to `pyproject.toml`, `requirements*.txt`, or any other dependency manifest.
2. The i18n CJK Guard shall complete the parity check in well under one second on the current catalogue size (~1000 keys per side) under normal CI conditions.
3. The i18n CJK Guard shall produce identical output for identical inputs across runs (no timestamps, no run IDs, no nondeterministic ordering).
### Requirement 5: Test coverage
**Objective:** As a future contributor modifying the guard, I want automated tests for every parity behaviour, so that regressions in either check or in their composition are caught locally.
#### Acceptance Criteria
1. The repository shall contain unit tests under `scripts/ci/tests/` that cover at minimum: (a) the success path where EN and ZH have identical key sets, (b) an en-only-key failure, (c) a zh-only-key failure, (d) a both-sides-divergent failure, (e) a leaf-value-type-mismatch case (string vs scalar/null) that does NOT count as a parity failure, and (f) the integration case where the parity check runs alongside the existing CJK-clean and per-path-ratchet checks without short-circuiting.
2. The new tests shall use the same testing style and framework already used by the existing tests in `scripts/ci/tests/`.
3. When a new test fixture is required for a JSON file, the fixture shall live under `scripts/ci/tests/` in a self-contained form (no reliance on `locales/` content for negative-path tests).
4. When the test suite is run from the repository root, the i18n CJK Guard test module shall pass without warnings on a clean checkout where `locales/en.json` and `locales/zh.json` have full key parity.
### Requirement 6: Self-test against the live catalogues
**Objective:** As an epic-#11 closer, I want to know the moment this guard ships that it observes the live catalogues as parity-clean, so that the guard's first PR doesn't produce a false alarm.
#### Acceptance Criteria
1. While the live catalogues `locales/en.json` and `locales/zh.json` have a symmetric difference of zero on the merge target branch, the i18n CJK Guard shall pass the parity check on a manual run from the repository root.
2. If the merge target branch is found to have a non-zero symmetric difference at the time this spec is implemented, then the implementer shall (a) document the divergence in the spec's `tasks.md` as a blocking finding and (b) fix the divergence before completing the implementation tasks, rather than weakening the parity check.

104
.kiro/specs/i18n-locale-parity-guard/research.md Normal file
View File

@ -0,0 +1,104 @@
# Research & Design Decisions — i18n-locale-parity-guard
## Summary
- **Feature**: `i18n-locale-parity-guard`
- **Discovery Scope**: Extension (extends an existing single-script CI guard)
- **Key Findings**:
- The existing PR-time guard `scripts/ci/i18n_cjk_guard.py` already implements the no-short-circuit composition pattern, the JSON-flatten primitive, and the line-fallback line-resolution helper that the new parity check needs to reuse.
- The audit pipeline's `check_parity.py` (in `.kiro/specs/i18n-e2e-english-verification/audit/scripts/`) already proves the algorithm: flatten both catalogues into dotted-key sets and compute their symmetric difference. It runs only in the manual audit path; promoting it to CI is a pure plumbing exercise.
- The live catalogues at `HEAD` of `main` are parity-clean (962 keys per side, symmetric difference 0), so the new guard's first run will not produce a false alarm and Requirement 6.1 holds out of the gate.
## Research Log
### Composition with the existing guard
- **Context**: Requirement 3 mandates that all checks (CJK-clean, per-path ratchet, parity) run in a single invocation without short-circuit and surface a unified exit code.
- **Sources Consulted**: `scripts/ci/i18n_cjk_guard.py:run_check` (lines 220–299).
- **Findings**: `run_check` uses a `failed: bool` accumulator and a `success_summary: list[str]` collector, evaluating every block before deciding the exit code. The parity check fits trivially as a third block at the end of `run_check`, before the final `if not failed: print(success_summary)` block.
- **Implications**: No structural refactor is needed. The extension is additive.
### Flatten and key resolution semantics
- **Context**: Requirement 1.1 anchors the flatten contract to `check_parity.py.flatten`. Requirement 1.5 specifies that scalar leaves and string leaves are treated identically for parity (only dict leaves are skipped).
- **Sources Consulted**: `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py:flatten`; `scripts/ci/i18n_cjk_guard.py:_flatten`.
- **Findings**: The two implementations are byte-equivalent in behaviour: both descend only into `dict`, both yield `(dotted-path, value)` for any non-dict leaf, both build dotted paths with `.` separators. The guard's existing `_flatten` is suitable; the parity check just consumes its keys (set comprehension over the flattened pairs).
- **Implications**: No new flatten function is needed. Requirement 1.1's "exactly match" clause is satisfied by reusing `_flatten`. Add a thin `_flatten_keys(data) -> set[str]` wrapper to keep call sites readable.
### Line resolution for missing keys
- **Context**: Requirement 2.1 demands `<file>:<line>: <key>: <side>` output. Requirement 2.2 demands a line-1 fallback when location is unknown.
- **Sources Consulted**: `scripts/ci/i18n_cjk_guard.py:_value_line_number` (lines 70–87).
- **Findings**: `_value_line_number` resolves a value's line by substring scan with two candidates (raw + JSON-escaped), falling back to line 1. For parity we must resolve a key, not a value. The minimal adaptation is a `_locate_key_line(text_lines, dotted_key)` that searches for the leaf segment of the dotted key wrapped in JSON quotes (e.g. `"missingKey"`). Falling back to line 1 mirrors `_value_line_number`'s contract.
- **Implications**: A small new helper is needed; it follows the same code idiom as `_value_line_number`. Edge cases: leaf segments that appear elsewhere in the file (other keys, value text) — accepting a coarse first-match is acceptable because the *primary* signal (the dotted key + side) is unambiguous; the line number is a navigation aid.
### Stdlib-only enforcement
- **Context**: Requirement 4.1 prohibits new dependencies.
- **Sources Consulted**: `pyproject.toml`, `requirements*.txt` (none at repo root); existing guard imports.
- **Findings**: The existing guard imports `argparse`, `json`, `os`, `re`, `subprocess`, `sys`, `pathlib`. Parity needs none beyond `json` and `pathlib` — both already in use.
- **Implications**: No `pyproject.toml` change. CI runtime image needs no addition.
### Live catalogue parity at HEAD
- **Context**: Requirement 6.1 asserts the guard must pass on the merge target's current state.
- **Sources Consulted**: `locales/en.json`, `locales/zh.json` flattened via stdlib `json.loads` + recursive descent.
- **Findings**: 962 keys per side, symmetric difference 0. Pre-existing `log.*` namespace fully mirrored (373 keys per side).
- **Implications**: No remediation translation work is needed. Requirement 6.2's conditional ("if divergence is found, fix it before completing") does not trigger.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Extend existing guard (Option A — selected) | Add parity helpers + a third block in `run_check` inside `scripts/ci/i18n_cjk_guard.py`; no workflow edit. | Single CI surface; reuses `_flatten`, line-fallback, sort/print idioms; trivially satisfies Requirement 3.6. | Module grows ~80 lines; module name no longer narrowly "CJK" — mitigated by docstring update. | Recommended in `gap-analysis.md`. |
| Parallel script + step (Option B) | New `scripts/ci/i18n_locale_parity_guard.py`; either second job in existing workflow or new workflow file. | Tightest single-responsibility per file. | Code duplication (~80 lines); two CI surfaces; violates the spirit of Requirement 3 ("compose with existing checks"). | Rejected. |
| Helper module + thin import (Option C) | New `scripts/ci/locale_parity.py`; the existing guard imports it and integrates the call. | Cleaner unit-test isolation; possible future de-duplication of audit `check_parity.py`. | Adds package-style imports for ~80 lines of logic; risks scope creep into "deduplicate audit script" (out of scope). | Rejected. |
## Design Decisions
### Decision: Extend `scripts/ci/i18n_cjk_guard.py` rather than create a new script
- **Context**: Requirement 3 mandates a single CLI invocation that runs all i18n CI checks together with no short-circuit and one exit code.
- **Alternatives Considered**:
1. New parallel script + workflow step — duplicates ~80 lines of plumbing.
2. New helper module imported by the guard — introduces package structure for trivial logic.
- **Selected Approach**: Add `_flatten_keys`, `_locate_key_line`, `_format_parity_finding`, and `run_parity_check` to the existing module; insert a third block into `run_check` after the per-path baseline block.
- **Rationale**: Smallest delta that fully satisfies Requirement 3; reuses the existing no-short-circuit accumulator pattern verbatim; no workflow edit (Requirement 3.6 holds for free); existing test scaffolding (`unittest`, synthetic git repos) extends naturally.
- **Trade-offs**: The module name (`i18n_cjk_guard`) becomes slightly broader than literal — mitigated by an updated module docstring listing all three checks. Module length grows from ~393 to ~470 lines, still well below the project's de facto threshold for splitting (`oasis_profile_generator.py` exceeds 1000).
- **Follow-up**: Update the module docstring; verify `--help` text and existing CLI smoke test still pass after the change.
### Decision: Treat scalar leaves identically to string leaves for parity
- **Context**: Requirement 1.5 — `_flatten` does not narrow by type; scalars (numbers, booleans, null) at a leaf must register as keys.
- **Alternatives Considered**:
1. Narrow to string leaves only (mirror `scan_locale_cjk`'s behaviour). Rejected because a numeric or null value on one side is still a string-on-the-other-side parity question, and the `log.*` namespace today is all strings — there's no payoff in narrowing.
2. Skip dict leaves; emit everything else. Selected.
- **Selected Approach**: `_flatten_keys(data) -> set[str]` returns every dotted path emitted by the existing `_flatten`, regardless of value type.
- **Rationale**: Aligns with the audit script's `flatten` contract (which also does not type-narrow). Catches accidental type drift across catalogues as a side benefit (any divergence at a key surfaces as a missing key).
- **Trade-offs**: None significant — the catalogues today are entirely string-typed at leaves; the choice is mostly future-proofing.
- **Follow-up**: Add a unit test (Requirement 5.1.e) that plants a scalar-typed leaf on both sides at the same path and asserts the parity check passes.
### Decision: Failure category strings — `parity-en-only` / `parity-zh-only`
- **Context**: Requirement 2.1 specifies the format `<file>:<line>: <key>: en-only` (or `... zh-only`). The existing CJK-clean check formats failures as `<file>:<line>: cjk-in-en: <key> = <snippet>`.
- **Alternatives Considered**:
1. Use bare `en-only` / `zh-only` as the category. Inconsistent with the CJK check's namespaced category (`cjk-in-en`).
2. Use namespaced categories `parity-en-only` / `parity-zh-only`. Selected.
- **Selected Approach**: Format failure lines as `<en.json|zh.json>:<line>: parity-en-only: <key>` and `... parity-zh-only: <key>` (file is whichever catalogue the missing key would belong to).
- **Rationale**: Mirrors the CJK check's `cjk-in-en` category naming, so a dev grepping CI logs for `parity-` finds all parity failures. The bare-side requirement of 2.1 is satisfied because the side appears verbatim after `parity-` (`parity-en-only` contains `en-only`).
- **Trade-offs**: Minor verbosity vs. consistency — favour consistency.
- **Follow-up**: Tests assert exact substring `parity-en-only` / `parity-zh-only` in failure lines.
## Risks & Mitigations
- **Risk**: A future maintainer renames the existing `_flatten` and the parity check silently breaks. **Mitigation**: A test in the new `ParityCheckTests` class asserts that flattening a known nested fixture produces the expected dotted-key set (locking in the contract).
- **Risk**: The `_locate_key_line` helper produces a misleading line number when the leaf segment also appears in another (unrelated) key or in a value. **Mitigation**: First-match on the JSON-quoted leaf is "good enough" for navigation; the dotted key in the message is the source of truth. Document this in the helper's docstring.
- **Risk**: Future test writers forget the no-short-circuit invariant when extending `run_check`. **Mitigation**: Requirement 5.1.f's composition test guards this — both the parity check and the existing CJK check fail in the same run, and the test asserts both failure lines appear together.
## References
- `scripts/ci/i18n_cjk_guard.py` — existing guard (extension target).
- `.kiro/specs/i18n-e2e-english-verification/audit/scripts/check_parity.py` — reference parity algorithm.
- `.kiro/specs/i18n-ci-guard/design.md` — prior CI guard design (style and boundary precedents).
- `scripts/ci/tests/test_i18n_cjk_guard.py` — existing test patterns (extension target).
- `.github/workflows/i18n-cjk-guard.yml` — workflow that runs the guard (no edit required).

23
.kiro/specs/i18n-locale-parity-guard/spec.json Normal file
View File

@ -0,0 +1,23 @@
{
"feature_name": "i18n-locale-parity-guard",
"created_at": "2026-05-09T00:29:21Z",
"updated_at": "2026-05-09T00:46:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true,
"ticket": 11
}

63
.kiro/specs/i18n-locale-parity-guard/tasks.md Normal file
View File

@ -0,0 +1,63 @@
# Implementation Plan
- [x] 1. Add parity primitives to the i18n CJK Guard module
- Introduce a constant naming the Chinese catalogue path alongside the existing English-catalogue constant.
- Add a private helper that returns the dotted-key set of a parsed catalogue, mirroring the audit pipeline's `flatten` contract (descend into dicts only; treat scalar leaves and string leaves identically; type-narrow nothing).
- Add a private helper that resolves the 1-based line number of a dotted key in raw JSON source text by searching for the leaf segment wrapped in JSON quotes, and falls back to line 1 on any miss.
- Add a private helper that formats a single parity-failure line in the layout `<file>:<line>: parity-en-only: <key>` or `... parity-zh-only: <key>`, with the side parameter typed as a literal of the two allowed strings (improvement carried over from the design review).
- Add an immutable result carrier (named tuple or frozen dataclass) holding the parity outcome (passed flag, formatted failure lines including the trailing summary, optional success-summary line).
- All additions stay stdlib-only and import nothing new beyond what the existing module already imports.
- Observable completion: the module exports the new constant, helpers, and result carrier; importing the module from a Python REPL or test stays warning-free, and the helpers can be exercised in isolation.
- _Requirements: 1.1, 1.5, 2.1, 2.2, 4.1, 4.3_
- _Boundary: i18n_cjk_guard module — helper layer_
- [x] 2. Implement the parity-check orchestrator
- Read both locale catalogues from the working tree using the existing path constants.
- Flatten each catalogue and compute the symmetric difference of the dotted-key sets.
- On match, build the success-summary string of the form `OK locale-parity: <count> keys per side`.
- On mismatch, sort en-only keys lexicographically and emit one formatted failure line per key with the EN catalogue path and a best-effort line number; then sort zh-only keys lexicographically and emit one line per key with the ZH catalogue path and a best-effort line number.
- Append a final summary line of the form `parity: en-only=<n>, zh-only=<m>` to the failure list so the orchestrator can print all lines uniformly.
- Treat a missing or malformed catalogue file as a parity failure that returns a single descriptive failure line; if the EN catalogue is the unreadable side, attribute the error to the parity check without re-stating the en-only error already produced by the existing CJK-clean block (refinement carried over from the design review).
- All output strings are deterministic across runs for identical inputs.
- Observable completion: calling the orchestrator function with synthetic parity-clean and parity-divergent catalogues returns a result carrier whose passed flag, failure list, and success summary match the documented contracts; running it against the live `locales/` directory returns `passed=True`.
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 2.1, 2.3, 2.4, 2.5, 4.2, 4.3_
- _Boundary: i18n_cjk_guard module — orchestrator-leaf layer_
- [x] 3. Compose the parity check into the existing run-check orchestrator
- Insert a new block inside the existing `run_check` function, after the per-path-ratchet block and before the final all-success branch.
- Invoke the parity-check orchestrator with the working-tree root.
- When the result is not passed, set the existing `failed` accumulator to true and print every entry of the result's failure list to stderr, one per call, preserving order.
- When the result is passed, append the result's success-summary line to the existing `success_summary` collector so it prints alongside the other success summaries on a fully-clean run.
- Update the module docstring to list all three checks (CJK-clean, per-path ratchet, locale-parity).
- Leave the CLI argument parser, `--update-baseline`, `--baseline`, `--repo-root`, the workflow file, and the baseline file format untouched. Confirm by visual diff that no other functions or files are modified.
- Observable completion: invoking the guard script via its CLI produces a single exit code, and `--help` text plus the existing CLI smoke test continues to pass without modification.
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6_
- _Boundary: i18n_cjk_guard module — run_check orchestrator_
- _Depends: 2_
- [x] 4. Add unit and integration tests for the parity check
- Extend the existing test-fixture helper that builds synthetic git repositories so callers can supply a Chinese catalogue alongside the English one; default the Chinese catalogue to a parity-clean mirror of the English fixture so the existing test cases continue to pass without semantic change.
- Add unit-level tests for the dotted-key flattener (empty input, flat input, mixed scalar/string/null leaves, three-level nesting), the line-number resolver (exact match, multi-occurrence first-wins, not-found line-1 fallback), and the failure-line formatter (both sides, special characters in key names).
- Add integration tests against the parity-check orchestrator covering: identical key sets pass; an en-only divergence fails with the expected category token, summary, and line attributing the key to the EN catalogue; a zh-only divergence fails with the symmetric output; a both-sides divergence yields en-only lines first then zh-only lines, each lex-sorted within its group; same-path scalar leaves on both sides do not count as a parity failure; a missing or malformed catalogue file produces a single deterministic failure line.
- All new tests use the standard-library testing framework already used in the existing test module; negative-path fixtures are self-contained and do not depend on the live catalogues.
- Observable completion: running the test module from the repository root produces a passing run with at least the new test cases reported, and a manually-induced en-only or zh-only key reliably trips the relevant test.
- _Requirements: 5.1, 5.2, 5.3, 5.4_
- _Boundary: i18n_cjk_guard test module — parity unit + integration coverage_
- _Depends: 3_
- [x] 5. Add a no-short-circuit composition test covering all three guard checks
- Plant CJK content in a synthetic English catalogue AND a parity-divergent key (in either direction) inside the same synthetic repository.
- Assert that running the full composed guard returns exit code 1, that stderr contains both the existing CJK-related category token and the new parity category token, and that the order of these blocks is preserved (CJK first, then ratchet, then parity) so failure logs remain greppable.
- Assert that on a fully-clean repository (no CJK in EN, ratchet within baseline, parity holds) the composed guard prints all three success summaries on stdout and exits 0.
- Observable completion: the new test case fails if any future change short-circuits the orchestrator after the first failure or before invoking the parity check.
- _Requirements: 3.1, 3.2, 3.3, 5.1_
- _Boundary: i18n_cjk_guard test module — composition coverage_
- _Depends: 3, 4_
- [x] 6. Verify the guard against the live locale catalogues
- Run the guard once from the repository root against the live `locales/en.json` and `locales/zh.json` and confirm it exits 0 with three success-summary lines (CJK-clean, per-path ratchet, locale-parity).
- If the live catalogues turn out to have non-zero symmetric difference at the time of implementation, document the divergence in this `tasks.md` as a blocking finding and remediate the divergence before completing the task; do not weaken the parity check.
- Observable completion: the guard's CLI invocation against the live tree prints `OK locale-parity: <count> keys per side` and exits 0, demonstrating that the new check is satisfied by the merge target without any source change.
- _Requirements: 6.1, 6.2_
- _Boundary: live `locales/` content (read-only verification)_
- _Depends: 5_

241
.kiro/specs/i18n-oasis-profile-generator-prompts/gap-analysis.md Normal file
View File

@ -0,0 +1,241 @@
# Gap Analysis — i18n-oasis-profile-generator-prompts
This document analyzes the gap between the requirements and the existing
codebase, lists implementation options, and recommends an approach for the
design phase.
## 1. Current State Investigation
### Target file
`backend/app/services/oasis_profile_generator.py` — 1195 lines. Defines:
- `OasisAgentProfile` dataclass with Reddit / Twitter serializers.
- `OasisProfileGenerator` class with the following public-API surface:
`__init__`, `generate_profile_from_entity`, `generate_profiles_from_entities`,
`set_graph_id`, plus private helpers `_call_llm_with_retry`,
`_generate_profile_rule_based`, `_get_system_prompt`,
`_build_individual_persona_prompt`, `_build_group_persona_prompt`,
`_print_generated_profile`, `_fix_truncated_json`, `_try_fix_json`,
`_save_twitter_csv`, `_save_reddit_json`, `_generate_username`.
### Chinese surfaces in the file (by category)
| Category | Lines | In scope this issue? |
| --- | --- | --- |
| Module / class / method docstrings | scattered | **No** — covered by #7 |
| Inline `#` comments | scattered | **No** — covered by #7 |
| `logger.{info,warning,error}` calls (translated via `t("log.profile_generator.*")`) | scattered | **No** — already done by #6 |
| `print(...)` banners (e.g. line 945) | a few | **No** — companion to #6 in spirit; not a prompt literal |
| **System prompt `base_prompt`** (line 664) | 1 line | **Yes** |
| **Individual-persona prompt body** (lines 680–714) | block | **Yes** |
| **Group-persona prompt body** (lines 729–762) | block | **Yes** |
| `attrs_str` / `context_str` defaults `"无"` / `"无额外上下文"` (lines 677, 678, 726, 727) | 4 lines | **Yes** — they substitute *into* the prompt body |
| Rule-based fallback (`_generate_profile_rule_based`, lines 764–835) including `"country": "中国"` and `"国家"` placeholders | block | **No** — runtime data, not a prompt |
| Resilience-helper Chinese fragments (`f"{entity_name}是一个{entity_type}。"` at lines 547, 644, 659) | a few | **No** — runtime data, not a prompt |
The file already imports `get_locale`, `set_locale`, `t`, and
`get_language_instruction` from `app.utils.locale`. The locale-capture /
restore plumbing inside `generate_profiles_for_entities` (lines ~910–916)
already propagates the request locale to background-thread workers — no
changes required.
### Locale infrastructure (already in place)
`backend/app/utils/locale.py`:
- `get_language_instruction()` returns the per-locale postfix from
`/locales/languages.json` (e.g. `Please respond in English.` for `en`,
`请使用中文回答。` for `zh`).
- `t(key, **kwargs)` resolves `log.*` keys for backend logger messages;
not used by this issue.
- `set_locale` / `get_locale` are thread-local, with restoration plumbed
into `generate_profiles_for_entities`.
### Sibling specs already shipped
- `i18n-ontology-generator-prompts` (#2 — merged)
- `i18n-simulation-config-generator-prompts` (#4 — merged)
- `i18n-report-agent-prompts` (#5 — merged)
- `i18n-externalize-backend-logs` (#6 — merged; logger keys for
`log.profile_generator.*` are already in `locales/{en,zh}.json`)
The translation pattern they established:
1. Translate the base prompt body (English narrative + headings).
2. Preserve every `get_language_instruction()` call site verbatim so
`Accept-Language: zh` still produces Chinese output.
3. Preserve all `{variable}` interpolations in f-strings.
4. Preserve all locale-independent "lock" rules (e.g. `gender` enum) in
English text within the prompt.
5. No new dependencies, no new files, single-file diff.
This is a direct sibling — same pattern applies.
### Test contract
`backend/scripts/test_profile_format.py`:
- Pytest-collectable function `test_profile_formats`.
- Constructs `OasisAgentProfile` instances directly (no LLM call) and
serializes them via `_save_twitter_csv` / `_save_reddit_json`.
- Verifies CSV header includes `user_id, user_name, name, bio,
friend_count, follower_count, statuses_count, created_at` and JSON
output includes `realname, username, bio, persona`.
- **Does not exercise the prompts.** A pure prompt translation cannot
break it; a refactor of dataclass field names or serializers would.
### Callers
- `backend/app/services/simulation_manager.py:316` —
`OasisProfileGenerator(graph_id=state.graph_id)`.
- `backend/app/api/simulation.py:1413` — `OasisProfileGenerator()`.
Neither caller looks at prompt language; both consume the persona dict
output. No call-site changes are needed.
## 2. Requirement-to-Asset Map
| Req. | Asset / file | Gap |
| --- | --- | --- |
| 1. System prompt → English | `_get_system_prompt` line 664 | **Missing** — Chinese literal needs to become English literal |
| 2. Individual-persona template → English | `_build_individual_persona_prompt` lines 680–714 | **Missing** — Chinese block needs translation; preserve `{...}` interpolations and inline `{get_language_instruction()}` |
| 3. Group-persona template → English | `_build_group_persona_prompt` lines 729–762 | **Missing** — Chinese block needs translation; preserve `{...}` interpolations and inline `{get_language_instruction()}` |
| 4. Locale switching unchanged | `app.utils.locale` + the three `get_language_instruction()` call sites | **Constraint** — code path must stay byte-identical at those call sites |
| 5. Public API stability | `OasisAgentProfile` dataclass + `OasisProfileGenerator` method signatures | **Constraint** — no signatures change |
| 6. Reasoning-model parsing unchanged | `_fix_truncated_json`, `_try_fix_json` | **Constraint** — no edits |
| 7. OASIS schema parity | `_save_twitter_csv`, `_save_reddit_json`, `to_*_format` serializers | **Constraint** — no edits; pytest must continue passing |
| 8. Out-of-scope guard | logger calls, docstrings, comments, rule-based fallback | **Constraint** — explicitly do not edit |
No requirement is blocked or unknown. Every requirement maps to a known
location with a clear, narrow change.
## 3. Implementation Approach Options
### Option A — In-place edit of the three prompt builders (extend existing)
Translate `base_prompt` (1 line), the individual-persona f-string body
(~35 lines), and the group-persona f-string body (~34 lines) directly,
plus the four `"无"` / `"无额外上下文"` fallback literals. Keep all method
bodies otherwise byte-identical.
- **Files touched**: `backend/app/services/oasis_profile_generator.py`
only.
- **Compatibility**: zero API change. All call sites unaffected. Locale
switching preserved by leaving the inline `{get_language_instruction()}`
placeholders untouched.
- **Complexity**: low. Pattern is identical to merged siblings #2, #4,
#5.
**Trade-offs**:
- ✅ Minimal diff, exactly the pattern reviewers expect.
- ✅ No risk to the unrelated rule-based fallback or serialization paths.
- ✅ Out-of-scope items (logger, docstrings, rule-based fallback) are not
touched, so #6/#7 remain clean.
- ❌ Leaves the file mixed-language in non-prompt parts (docstrings, rule
fallback) until #7 lands. Acceptable per scope split.
### Option B — Move prompt strings into module-level constants
Introduce `INDIVIDUAL_PERSONA_PROMPT_TEMPLATE` and
`GROUP_PERSONA_PROMPT_TEMPLATE` constants at module scope (mirroring
`ONTOLOGY_SYSTEM_PROMPT` style in `ontology_generator.py`), and have the
builders `.format(**kwargs)` against them.
- **Files touched**: same single file, but with structural refactor.
- **Compatibility**: still zero public API change, but the diff is
larger and reviewers must verify equivalent behaviour around
`{get_language_instruction()}` (which would need to become a runtime
substitution not an f-string interpolation, since constants don't
re-evaluate per call).
**Trade-offs**:
- ✅ Constants are easier to spot in `git grep`.
- ❌ Larger diff, more review surface.
- ❌ The inline `get_language_instruction()` call is currently captured at
f-string render time; moving to a `.format(...)` template requires
passing the resolved instruction in as a kwarg — a behavioural change
that exceeds "translate prompts only".
- ❌ Diverges from the sibling pattern just shipped (#4, #5 used in-place
edits, not module constants). #2 used module constants but only for the
system prompt — the user-message template was still built inside the
method.
### Option C — Externalize prompt text into `/locales/*.json`
Move every prompt sentence into `locales/en.json` and `locales/zh.json`,
keyed under `prompt.profile_generator.*`, and use `t(key, **vars)` to
resolve.
- **Compatibility**: would address `Accept-Language` purely via the
existing translation mechanism without depending on the
`get_language_instruction()` postfix.
**Trade-offs**:
- ✅ Most i18n-pure approach.
- ❌ Significantly larger diff (touches three repos: source file,
`en.json`, `zh.json`).
- ❌ Diverges from the established project pattern. The sibling specs
(#2, #4, #5) deliberately did **not** externalize prompts — the
project rationale (per `tech.md`) is that backend logger messages are
the i18n surface, while LLM prompts use the `get_language_instruction()`
postfix mechanism.
- ❌ Higher review and merge cost for no operational gain.
## 4. Recommended Approach
**Option A** — single-file in-place edit of the three prompt builders
plus the four `"无"` / `"无额外上下文"` fallback literals.
Rationale:
- Matches the merged sibling specs verbatim (#2, #4, #5) so reviewers
can apply the same mental checklist.
- Smallest possible diff that satisfies every acceptance criterion in
requirements.md.
- Leaves out-of-scope surfaces (logger, docstrings, rule-based
fallback) untouched — clean handoff to #7 and clean separation from
already-merged #6.
- Zero new dependencies, zero new files, zero API change, zero risk to
`test_profile_format.py`.
### Translation choices to lock in during design
1. The system prompt `base_prompt` becomes a single English sentence in
the spirit of the original (expert in social-media persona generation;
detailed and realistic personas for opinion simulation; faithful
reflection of real-world conditions; valid JSON, no unescaped
newlines).
2. The two persona prompt bodies adopt English section headings and
prose. The previously-Chinese hint
`country: 国家（使用中文，如"中国"）` is dropped — the
`get_language_instruction()` postfix already steers locale, and the
rule-based fallback (out of scope) handles its own country values.
3. The trailing rules block keeps the locale-independent "lock"
constraints inline (`gender` enum, `age` integer requirement,
`persona` newline rule) and continues to embed
`{get_language_instruction()}` verbatim.
## 5. Effort & Risk
- **Effort**: **S** (1–3 days; realistically <½ day). One-file diff,
established sibling pattern, no new test infrastructure.
- **Risk**: **Low**. The translated prompts touch only the LLM
`messages` payload. The locale-switching pathway, public API,
serializers, retry logic, fallback, and tests are all untouched. The
only failure mode is a mistranslated constraint (e.g. accidentally
dropping `gender ∈ {male, female, other}`), which the design checklist
enumerates and reviewers can verify by diff.
### Research items carried into design phase
- None blocking. The design phase will:
- Enumerate the exact final English text for each of the three blocks.
- Verify each translated block preserves every JSON-output key,
every `{variable}` interpolation, and the inline
`{get_language_instruction()}` call.
- Spot-check that the diff stays within
`backend/app/services/oasis_profile_generator.py`.

222
.kiro/specs/i18n-oasis-profile-generator-prompts/research.md Normal file
View File

@ -0,0 +1,222 @@
# Research & Design Decisions — i18n-oasis-profile-generator-prompts
## Summary
- **Feature**: `i18n-oasis-profile-generator-prompts`
- **Discovery Scope**: **Extension** (single-file translation in an existing
brownfield service; sibling pattern already merged in #2, #4, #5)
- **Key Findings**:
- The existing `get_language_instruction()` postfix mechanism (defined in
`backend/app/utils/locale.py`) is the project-canonical way to steer LLM
output language. Translating the base prompt does not interfere with it
and is the same approach taken in already-merged sibling specs.
- The only Chinese surfaces inside the prompt-rendering path are
`_get_system_prompt`, `_build_individual_persona_prompt`,
`_build_group_persona_prompt`, and the four `attrs_str`/`context_str`
fallback literals (`"无"`, `"无额外上下文"`). All other Chinese in the
file is logger keys (already done by #6), docstrings/comments
(out-of-scope, #7), or rule-based fallback data (out-of-scope).
- `backend/scripts/test_profile_format.py` does not exercise prompts; it
only constructs `OasisAgentProfile` and round-trips through
`_save_twitter_csv` / `_save_reddit_json`. A pure-translation diff
cannot break it.
## Research Log
### Locale steering mechanism
- **Context**: Confirm that translating the base prompt does not regress
Chinese output under `Accept-Language: zh`.
- **Sources Consulted**:
- `backend/app/utils/locale.py` (lines 50–96).
- `locales/languages.json` (entries for `en` and `zh` with
`llmInstruction` field).
- Sibling spec `i18n-ontology-generator-prompts/design.md` and the
merged commits referenced by it.
- **Findings**:
- `get_language_instruction()` returns `Please respond in English.`
for locale `en`, `请使用中文回答。` for locale `zh`.
- The function is called as an inline f-string interpolation in the
individual-persona and group-persona prompt bodies, and explicitly
appended in `_get_system_prompt`. All three sites must be preserved
byte-for-byte.
- The thread-local locale is captured in
`generate_profiles_for_entities` (line ~910) and restored inside the
worker via `set_locale(current_locale)` (line ~914). This plumbing is
untouched by the change.
- **Implications**:
- Design lock-in: the inline `{get_language_instruction()}` call must
remain in each of the three builders. Removing or renaming it would
silently regress non-English locales.
- The Chinese hint `country: 国家（使用中文，如"中国"）` in the original
prompt overrides the locale postfix and forces Chinese output for one
field. The English translation drops that hint so the locale postfix
decides the country language. The rule-based fallback (out of scope)
has its own (Chinese) defaults and is not affected.
### Test contract
- **Context**: Verify that `backend/scripts/test_profile_format.py`
remains green after a prompt-only translation.
- **Sources Consulted**: `backend/scripts/test_profile_format.py`,
`oasis_profile_generator.py:_save_twitter_csv`,
`oasis_profile_generator.py:_save_reddit_json`,
`oasis_profile_generator.py:to_reddit_format`,
`oasis_profile_generator.py:to_twitter_format`.
- **Findings**:
- The pytest function `test_profile_formats` constructs
`OasisAgentProfile` instances directly without invoking the LLM.
- It calls `_save_twitter_csv` and `_save_reddit_json` to verify CSV
and JSON shape. Required CSV header: `user_id, user_name, name, bio,
friend_count, follower_count, statuses_count, created_at`. Required
JSON keys: `realname, username, bio, persona`.
- **Implications**:
- Translating prompts cannot regress this test. The validation
requirement (Requirement 7) is satisfied automatically as long as
serializer code is not edited.
- No new tests are required for this change.
### Sibling specs already shipped
- **Context**: Confirm there is an established project pattern this work
must mirror.
- **Sources Consulted**:
- `.kiro/specs/i18n-ontology-generator-prompts/{design,tasks,requirements}.md`
- `.kiro/specs/i18n-report-agent-prompts/`
- `.kiro/specs/i18n-simulation-config-generator-prompts/`
- Recent merged commits referencing #2, #4, #5.
- **Findings**:
- All three siblings used a single-file in-place translation diff.
- All three preserved every `get_language_instruction()` call site.
- All three left logger calls and docstrings to companion issues
(#6 / #7).
- None externalized prompts to `/locales/*.json`.
- **Implications**:
- The same approach is correct here. Reviewer expectations are set by
the sibling diffs.
### OASIS profile schema
- **Context**: Verify that translated prompts continue to satisfy the
OASIS subprocess's expected schema (especially `gender` enum and
`age` integer).
- **Sources Consulted**: `OasisAgentProfile` dataclass,
`to_reddit_format`, `to_twitter_format`, sibling `_generate_profile_rule_based`.
- **Findings**:
- OASIS-required fields are produced by serializers, not by the
prompt: `user_id`, `username`, `name`, `bio`, `karma`/`friend_count`/`follower_count`/`statuses_count`, `created_at`.
- The prompt-defined fields land in optional positions: `age`,
`gender`, `mbti`, `country`, `profession`, `interested_topics`.
- The `gender` enum constraint (`"male"`/`"female"` for individuals,
`"other"` for groups) is locale-independent and must remain in
English text inside the translated prompt.
- **Implications**:
- The English prompt must explicitly call out `gender ∈ {male, female}`
(individual) and `gender == "other"` (group), independent of the
`get_language_instruction()` postfix.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| **A — In-place builder edit** | Translate three method bodies + four fallback literals directly | Smallest diff; matches sibling pattern; zero API change | None of note | **Selected** |
| B — Module-level constants | Hoist prompts to `INDIVIDUAL_PERSONA_PROMPT_TEMPLATE` etc. | Easier `git grep` | Larger diff; the inline `{get_language_instruction()}` call would need to become a `.format()` kwarg, which is a behavioural change beyond translation | Diverges from #4 / #5 |
| C — Externalize to `locales/*.json` | Move every prompt sentence into `t(...)` keys | Most i18n-pure | Three-file diff; diverges from project rationale (prompts use postfix mechanism, not key files) | Rejected |
## Design Decisions
### Decision: In-place edit of the three prompt builders (Option A)
- **Context**: Three methods build prompt strings; one of them is a
one-line system prompt, the other two are large f-string templates
with embedded `{variable}` interpolations and an inline
`{get_language_instruction()}` call.
- **Alternatives Considered**:
1. Option B — module-level constants.
2. Option C — externalize to `/locales/*.json` keys.
- **Selected Approach**: Translate each method body in place. Replace
the four `"无"` / `"无额外上下文"` fallbacks with English equivalents
(`"None"` and `"No additional context"`). Preserve all `{...}`
interpolations and the inline `{get_language_instruction()}` call.
- **Rationale**: Matches merged sibling specs verbatim. Smallest review
surface. Zero API change. Out-of-scope surfaces (logger, docstrings,
rule-based fallback) cleanly avoided.
- **Trade-offs**: Leaves the file mixed-language in non-prompt parts
(docstrings, rule fallback) until #7 lands. Acceptable per scope
split.
- **Follow-up**: During implementation, run a regex audit for any
Chinese codepoints inside the three method bodies after the edit and
confirm the diff stays within
`backend/app/services/oasis_profile_generator.py`.
### Decision: Drop the "use Chinese country names" hint
- **Context**: The current prompt at line 704 reads
`country: 国家（使用中文，如"中国"）` and at line 753
`country: 国家（使用中文，如"中国"）`. This forces Chinese for the
`country` field even under `Accept-Language: en`.
- **Alternatives Considered**:
1. Translate to English literally:
`country: country (use English, e.g. "China")`.
2. Drop the language hint entirely:
`country: country name string`.
- **Selected Approach**: Drop the language hint. Let
`get_language_instruction()` steer the country language alongside
every other free-text field.
- **Rationale**: Hard-coding a language in the prompt defeats the
locale-steering mechanism. The rule-based fallback (out of scope)
carries its own Chinese defaults; under the LLM path, locale should
decide.
- **Trade-offs**: Under `Accept-Language: zh`, the LLM may produce a
Chinese country name (e.g. `中国`) — this is the desired behaviour.
Under `Accept-Language: en`, the LLM produces English (`China`),
matching `COUNTRIES = ["China", "US", ...]` already in the file.
- **Follow-up**: Verify in the validation phase that a sample run under
locale `en` produces an English country name.
### Decision: Keep `gender` enum constraint in English inside the prompt
- **Context**: `gender` must be one of `"male"`/`"female"`/`"other"`
regardless of locale, because OASIS consumers and the
`_generate_profile_rule_based` fallback assume English values.
- **Alternatives Considered**: None — the constraint is a contract.
- **Selected Approach**: The translated prompt explicitly states the
enum in English, even when the locale postfix asks for Chinese
output: `gender MUST be one of "male" or "female" (English literal)`.
- **Rationale**: Same as the existing Chinese prompt (which already
states `必须是英文: "male" 或 "female"`). The translation preserves
the same lock-in.
- **Trade-offs**: None.
- **Follow-up**: Validation phase will check that under both locales
the produced `gender` is one of the three English literals.
## Risks & Mitigations
- **Risk**: Mistranslation drops a locale-independent constraint
(e.g. `gender` enum, `age` integer rule, `persona` no-newline rule).
- **Mitigation**: The implementation task list will enumerate every
constraint inline so reviewers can check by diff.
- **Risk**: Variable-name typo inside an f-string causes a `KeyError`
at runtime.
- **Mitigation**: Implementation task verifies that the set of
`{variable}` interpolations in each translated block matches the
pre-change set 1:1; a `python -c "import ..."` smoke import and a
`pytest backend/scripts/test_profile_format.py` run are mandatory.
- **Risk**: Accidentally leaving a CJK codepoint inside the three
builders.
- **Mitigation**: Final implementation step runs the project's
repo-level CJK guard regex (added by #26) constrained to the three
builders' line ranges.
## References
- `backend/app/services/oasis_profile_generator.py` — target file.
- `backend/app/utils/locale.py` — locale infrastructure.
- `locales/languages.json`, `locales/en.json`, `locales/zh.json` —
locale registries.
- `.kiro/specs/i18n-ontology-generator-prompts/` — sibling spec #2.
- `.kiro/specs/i18n-simulation-config-generator-prompts/` — sibling
spec #4.
- `.kiro/specs/i18n-report-agent-prompts/` — sibling spec #5.
- GitHub issue
[#3](https://github.com/salestech-group/MiroFish/issues/3).

222
.kiro/specs/i18n-readme-tagline-and-assets/design.md Normal file
View File

@ -0,0 +1,222 @@
# Design Document — i18n-readme-tagline-and-assets
## Overview
**Purpose**: Eliminate the remaining Chinese surface text from the project's English-facing entry points (`README.md`, `README-EN.md`, `package.json`, `backend/pyproject.toml`) and replace Chinese-named image assets under `static/image/` with ASCII-only equivalents, so that visitors landing on the GitHub repo or installing the npm package see English-only metadata and so that asset URLs are tooling- and CDN-friendly.
**Users**: Non-Chinese-reading visitors arriving at the GitHub README, downstream consumers reading `package.json` / `backend/pyproject.toml` metadata, and any tool (CDNs, link-rotters, screenshot-rendering bots) that handles repo asset URLs.
**Impact**: Documentation surface and static image filenames change; no runtime, API, or pipeline behavior is affected. The Chinese-language entry point (`README-ZH.md`) keeps its Chinese body text but its asset references are updated to point at the renamed files.
### Goals
- Replace the Chinese tagline with English on `README.md`, `README-EN.md`, `package.json`, `backend/pyproject.toml`.
- Rename nine Chinese-named assets under `static/image/` to ASCII filenames, preserving byte content.
- Update every `<img src>` reference in `README.md`, `README-EN.md`, and `README-ZH.md` to the new ASCII paths.
- Verifiable acceptance: a Chinese-character scan over `README.md` and `README-EN.md` returns zero matches outside the language-switcher line.
### Non-Goals
- Translating the body of `README-ZH.md` (Chinese variant by design).
- Changing the Chinese tagline value in `locales/zh.json` (legitimate Chinese locale content).
- Re-encoding or re-cropping any image (rename only).
- Adding a CI guard that enforces ASCII filenames or no-Chinese-in-EN-README (tracked separately as #26).
## Boundary Commitments
### This Spec Owns
- The English-language tagline string used in `README.md`, `README-EN.md`, `package.json`, `backend/pyproject.toml`.
- The ASCII filenames for the nine renamed assets under `static/image/`.
- All `<img src>` references inside the three READMEs that point to the renamed files.
### Out of Boundary
- Any asset under `static/image/` that already uses an ASCII name (`MiroFish_logo*.jpeg`, `shanda_logo.png`).
- Code-level i18n initiatives (frontend strings, backend logs, agent prompts) — those are owned by sibling i18n specs.
- README content beyond the lines explicitly identified in §"Modified Files".
### Allowed Dependencies
- Git (`git mv` for rename-with-history).
- No new project dependencies.
### Revalidation Triggers
- Any future change that adds another Chinese-named asset under `static/image/` referenced from a README — the verification scan in this spec must be re-run.
- Any future change to the structure of the language-switcher line — the R4 verification regex tolerance for `[中文文档]` may need adjusting.
## Architecture
### Existing Architecture Analysis
This is a documentation- and asset-rename change. There is no architectural component to extend or replace. The relevant existing patterns to respect:
- **Per `.claude/rules/file-paths.md`**: shell commands that touch paths with non-ASCII characters must quote the paths.
- **Per `.kiro/steering/structure.md`**: `static/` is the project's image asset root; READMEs reference it via relative paths from repo root.
- **Per `.claude/rules/commits.md`**: Conventional Commits, lowercase, imperative, max 72 chars, no `Co-Authored-By:` watermark.
### Architecture Pattern & Boundary Map
No new architecture is introduced. The flow is a one-shot edit:
```mermaid
flowchart LR
A[Chinese-named<br/>asset files] -->|git mv| B[ASCII-named<br/>asset files]
C[README.md / README-EN.md /<br/>README-ZH.md / package.json /<br/>backend/pyproject.toml] -->|Edit tool| D[Updated text +<br/>updated img src paths]
B --> D
D --> E[Verify: rg Chinese-char scan<br/>returns only language-switcher line]
```
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Frontend / CLI | — | n/a | No code changes. |
| Backend / Services | — | n/a | No code changes. |
| Data / Storage | — | n/a | No data model changes. |
| Messaging / Events | — | n/a | n/a |
| Infrastructure / Runtime | git ≥ 2.x | `git mv` for renames | Already a project prerequisite. |
| Documentation | Markdown / HTML-in-MD | Edit READMEs, `package.json`, `backend/pyproject.toml` | No new tooling. |
## File Structure Plan
### Directory Structure
No new files or directories are created. The existing layout is preserved:
```
static/image/
├── MiroFish_logo.jpeg (unchanged)
├── MiroFish_logo_compressed.jpeg (unchanged)
├── shanda_logo.png (unchanged)
├── qq-group.png (renamed from "QQ群.png")
├── wuhan-university-simulation-cover.png (renamed from "武大模拟演示封面.png")
├── dream-of-the-red-chamber-simulation-cover.jpg (renamed from "红楼梦模拟推演封面.jpg")
└── Screenshot/
├── screenshot1.png (renamed from "运行截图1.png")
├── screenshot2.png (renamed from "运行截图2.png")
├── screenshot3.png (renamed from "运行截图3.png")
├── screenshot4.png (renamed from "运行截图4.png")
├── screenshot5.png (renamed from "运行截图5.png")
└── screenshot6.png (renamed from "运行截图6.png")
```
### Modified Files
- `README.md`
- Lines 7–8: delete the Chinese tagline line and the `</br>` separator; the existing `<em>` line on (former) line 9 becomes the lone tagline.
- Lines 52, 53, 56, 57, 60, 61: replace `Screenshot/运行截图{N}.png` with `Screenshot/screenshot{N}.png`.
- Line 71: replace `武大模拟演示封面.png` with `wuhan-university-simulation-cover.png`.
- Line 79: replace `红楼梦模拟推演封面.jpg` with `dream-of-the-red-chamber-simulation-cover.jpg`.
- Line 220: replace `QQ群.png` with `qq-group.png`.
- `README-EN.md` — identical edit set as `README.md`.
- `README-ZH.md`
- Lines 52, 53, 56, 57, 60, 61, 71, 79, 220: same nine `<img src>` replacements as above. Tagline and Chinese body text unchanged.
- `package.json`
- Line 4: replace the `description` value with `MiroFish - A Simple and Universal Swarm Intelligence Engine, Predicting Anything`.
- `backend/pyproject.toml`
- Line 4: replace the `description` value with `MiroFish - A Simple and Universal Swarm Intelligence Engine, Predicting Anything`.
### Renamed Files (via `git mv`)
| Old (quoted) | New |
|---|---|
| `"static/image/QQ群.png"` | `static/image/qq-group.png` |
| `"static/image/武大模拟演示封面.png"` | `static/image/wuhan-university-simulation-cover.png` |
| `"static/image/红楼梦模拟推演封面.jpg"` | `static/image/dream-of-the-red-chamber-simulation-cover.jpg` |
| `"static/image/Screenshot/运行截图1.png"` | `static/image/Screenshot/screenshot1.png` |
| `"static/image/Screenshot/运行截图2.png"` | `static/image/Screenshot/screenshot2.png` |
| `"static/image/Screenshot/运行截图3.png"` | `static/image/Screenshot/screenshot3.png` |
| `"static/image/Screenshot/运行截图4.png"` | `static/image/Screenshot/screenshot4.png` |
| `"static/image/Screenshot/运行截图5.png"` | `static/image/Screenshot/screenshot5.png` |
| `"static/image/Screenshot/运行截图6.png"` | `static/image/Screenshot/screenshot6.png` |
## System Flows
Not applicable. No runtime flows are introduced or changed.
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | English tagline in README.md | README.md L7–9 edit | n/a | n/a |
| 1.2 | English tagline in README-EN.md | README-EN.md L7–9 edit | n/a | n/a |
| 1.3 | English description in package.json | package.json L4 edit | n/a | n/a |
| 1.4 | English description in backend/pyproject.toml | backend/pyproject.toml L4 edit | n/a | n/a |
| 1.5 | README-ZH.md tagline preserved | README-ZH.md (no L7 edit) | n/a | n/a |
| 2.1 | Rename screenshot{1..6} | `git mv` of six files | n/a | n/a |
| 2.2 | Rename Wuhan video cover | `git mv` of one file | n/a | n/a |
| 2.3 | Rename Red Chamber video cover | `git mv` of one file | n/a | n/a |
| 2.4 | Rename QQ group image | `git mv` of one file | n/a | n/a |
| 2.5 | Byte-preserving rename | `git mv` mechanism choice | n/a | n/a |
| 2.6 | No duplicate copies | `git mv` (atomic rename) + `git status` verification | n/a | n/a |
| 3.1 | README.md image references updated | README.md L52–61, 71, 79, 220 edits | n/a | n/a |
| 3.2 | README-EN.md image references updated | README-EN.md L52–61, 71, 79, 220 edits | n/a | n/a |
| 3.3 | README-ZH.md image references updated | README-ZH.md L52–61, 71, 79, 220 edits | n/a | n/a |
| 3.4 | No broken images on render | Post-edit verification step | n/a | n/a |
| 4.1 | No Chinese chars in README.md body (excl. switcher) | Verification scan | n/a | n/a |
| 4.2 | No Chinese chars in README-EN.md body (excl. switcher) | Verification scan | n/a | n/a |
| 4.3 | Reviewer-runnable scan returns zero matches | `rg` command in design + commit message | n/a | n/a |
## Components and Interfaces
This spec has no software components, services, or APIs. The "components" reduce to two textual operations (translate + rename) and one verification.
| Operation | Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|-------|--------|--------------|------------------|-----------|
| Tagline translation | Docs / Metadata | Replace Chinese tagline with English in 4 files | 1.1, 1.2, 1.3, 1.4 | Edit tool | n/a |
| Asset rename + reference update | Static assets / Docs | Rename 9 files; update `<img src>` in 3 READMEs | 2.1–2.6, 3.1–3.4 | `git mv`, Edit tool | n/a |
| Verification scan | Acceptance gate | Confirm no residual Chinese in EN READMEs body | 4.1, 4.2, 4.3 | ripgrep | Commit message records the scan command and result |
### Verification Contract
The acceptance gate is a single ripgrep invocation, runnable by any reviewer:
```
rg --pcre2 '[\x{4e00}-\x{9fff}]' README.md README-EN.md \
| rg -v 'README-ZH\.md'
```
**Preconditions**: All edits and renames committed.
**Postconditions**: The pipeline returns zero lines (the only Chinese characters left are in `[中文文档](./README-ZH.md)`, which the second `rg` filters out by matching the `README-ZH.md` substring on the same line).
**Invariants**: `README-ZH.md` body is not modified by this scan logic; the language-switcher line in the EN READMEs is the sole expected exemption.
## Data Models
Not applicable. No data structures are added or modified.
## Error Handling
### Error Strategy
Failure modes are limited to (a) a `git mv` failing because a path was mistyped (immediately visible at command-execution time) and (b) a `<img src>` left pointing at an old Chinese-named filename (caught by the verification scan).
### Error Categories and Responses
- **Mistyped rename target**: `git mv` fails with a clear error; re-run with the correct path.
- **Missed reference update**: Verification scan returns the offending file/line; fix and re-scan.
- **Accidental binary re-encoding**: `git diff --stat` of the asset file shows non-zero content delta; abandon the change and redo with `git mv`.
### Monitoring
Not applicable for a one-shot docs change. The PR diff plus the verification-scan output in the PR description serve as the audit trail.
## Testing Strategy
This is a documentation/asset change with no executable code. Testing is review-time:
- **Verification scan (mandatory)**: Run the ripgrep command in §"Verification Contract" against the working tree before commit; expect zero output. Re-run once more in CI / on the PR branch.
- **Rendered-preview check (mandatory)**: Open `README.md`, `README-EN.md`, `README-ZH.md` in GitHub's rendered-markdown view (or a local Markdown previewer) on the feature branch and confirm:
1. The tagline appears once, in English, on `README.md` and `README-EN.md`.
2. All six screenshot tiles render.
3. Both video-cover thumbnails render.
4. The QQ group image renders.
5. `README-ZH.md` still renders identically except for the new ASCII image URLs.
- **`git diff --stat` check (mandatory)**: For each of the nine asset files, the stat must show `0 insertions(+), 0 deletions(-)` (pure rename). If any asset shows a content delta, the rename was performed incorrectly.
## Optional Sections
### Migration Strategy
No data migration. The "migration" is a single PR containing all renames + edits. There is no rollback step beyond a normal `git revert` of the merge commit if a broken image is reported post-merge.

131
.kiro/specs/i18n-readme-tagline-and-assets/gap-analysis.md Normal file
View File

@ -0,0 +1,131 @@
# Gap Analysis — i18n-readme-tagline-and-assets
## 1. Current State Investigation
### Scope ground truth
Ripgrep `[\x{4e00}-\x{9fff}]` over `README.md`, `README-EN.md`, `package.json`, and `backend/pyproject.toml` returns the following Chinese-character lines that fall under this feature's mandate:
| File | Line | Content (excerpt) | Category |
| --- | ---: | --- | --- |
| `README.md` | 7 | `简洁通用的群体智能引擎，预测万物` | Tagline |
| `README.md` | 23 | `[English](./README.md) \| [中文文档](./README-ZH.md)` | Language switcher (allowed) |
| `README.md` | 52–61 | `./static/image/Screenshot/运行截图{1..6}.png` (×6) | Asset path |
| `README.md` | 71 | `./static/image/武大模拟演示封面.png` | Asset path |
| `README.md` | 79 | `./static/image/红楼梦模拟推演封面.jpg` | Asset path |
| `README.md` | 220 | `./static/image/QQ群.png` | Asset path (not listed in ticket scope, see Gap §3) |
| `README-EN.md` | 7, 23, 52–61, 71, 79, 220 | identical structure to README.md | Same categories |
| `package.json` | 4 | `"description": "MiroFish - 简洁通用的群体智能引擎，预测万物"` | Tagline |
| `backend/pyproject.toml` | 4 | `description = "MiroFish - 简洁通用的群体智能引擎，预测万物"` | Tagline (twin string, not in original ticket) |
`README-ZH.md` carries Chinese body text by design (out of scope) but its asset paths must still be updated to point at the renamed ASCII files.
### Tracked image files (`git ls-files static/image/`)
```
static/image/MiroFish_logo.jpeg
static/image/MiroFish_logo_compressed.jpeg
static/image/QQ群.png
static/image/Screenshot/运行截图{1..6}.png
static/image/shanda_logo.png
static/image/武大模拟演示封面.png
static/image/红楼梦模拟推演封面.jpg
```
Nine files have Chinese names: six screenshots + `QQ群.png` + `武大模拟演示封面.png` + `红楼梦模拟推演封面.jpg`.
### Tagline structure observation
`README.md` lines 7–9 currently read:
```
简洁通用的群体智能引擎，预测万物
</br>
<em>A Simple and Universal Swarm Intelligence Engine, Predicting Anything</em>
```
The English equivalent already exists immediately below the Chinese as italic subtitle. Naive replacement would produce a duplicate (English in plain text + the same English in italic). The natural i18n collapse is to delete the Chinese line plus the `</br>` separator and let the existing `<em>` line stand alone. `README-EN.md` has the identical structure.
### Conventions to respect (from steering)
- `tech.md`: 4-space indent, no enforced linter, "match the surrounding file's style". Shell scripts must quote paths with spaces / non-ASCII characters per `.claude/rules/file-paths.md`.
- `commits.md`: Conventional Commits, lowercase, imperative, max 72 chars, no `Co-Authored-By:` footer. Branch `<type>/<ticket>-<desc>` — ticket dictates `chore/i18n-12-readme-tagline-and-assets` (or similar).
- `dev-guidelines.md`: kebab-case filenames for assets is consistent with the project's frontend file conventions.
### Existing precedent in the same i18n epic
Recently merged child issues of epic #11 (`#7`, `#9`, `#3`, `#5`, `#6`) have all been small, focused docs/tooling PRs. This is consistent with treating #12 as an S-effort docs cleanup.
## 2. Requirements Feasibility Analysis
### Per-requirement asset map
| Req | What it needs | Where it lives | Gap |
| --- | --- | --- | --- |
| R1 (tagline) | English tagline | `README.md:7-9`, `README-EN.md:7-9`, `package.json:4`, `backend/pyproject.toml:4` | **Editorial** — straight string edit. No code paths affected. |
| R2 (asset rename) | Rename 8 files (6 screenshots + 2 video covers) | `static/image/Screenshot/`, `static/image/` | **`git mv`** — preserves history. No callers outside READMEs found by grep. |
| R3 (README references updated) | Update `<img src>` paths | `README.md`, `README-EN.md`, `README-ZH.md` | **Editorial** — straight string edits. |
| R4 (no residual Chinese in EN READMEs) | Verifiable scan | Both `README.md` and `README-EN.md` | **Constraint surfaces extra asset** — `QQ群.png` (line 220) is not in the explicit ticket asset list but its src path contains Chinese, which would fail R4's verification. See Gap §3. |
### Gaps tagged
- **Constraint:** `static/image/QQ群.png` is referenced by all three READMEs but is **not explicitly listed in the ticket's scope bullets**, while the ticket's own acceptance criterion ("No Chinese characters in `README.md`, `README-EN.md` body text") would still flag its src path. Either we (a) expand scope to rename it as well or (b) accept a deviation. Recommendation: expand scope — same shape of fix, trivial cost, satisfies the literal acceptance criterion.
- **Constraint:** `backend/pyproject.toml:4` carries the identical Chinese tagline string as `package.json:4`. Not in original ticket bullets but is the obvious twin and would surprise a reviewer reading the diff. Already incorporated into requirements.md R1 acceptance criterion 4.
- **Unknown / Research Needed (minor):** Confirm GitHub Pages, the live demo site, and any external link to the screenshots do not deep-link into Chinese-named asset URLs. Quick `gh` / web check during design phase will resolve.
## 3. Implementation Approach Options
This is a docs/asset-rename feature. There is no algorithm to design — the only real decision is whether the renames go through `git mv` (preserves history) or `git rm`/`git add` (loses history). And whether to expand scope to `QQ群.png`.
### Option A — Strict ticket scope (no QQ群.png rename)
- Rename only the eight assets explicitly listed: `运行截图{1..6}.png`, `武大模拟演示封面.png`, `红楼梦模拟推演封面.jpg`.
- Translate taglines in `README.md`, `README-EN.md`, `package.json`, `backend/pyproject.toml`.
- Skip `QQ群.png`.
**Trade-offs:**
- ✅ Smallest possible diff; no scope creep.
- ❌ Acceptance criterion R4 ("no Chinese characters in README body outside language switcher") fails because line 220 still contains `QQ群` in the src path.
### Option B — Expanded scope including QQ群.png (RECOMMENDED)
- Same as Option A, plus rename `static/image/QQ群.png` → `static/image/qq-group.png` (or similar) and update its three references.
**Trade-offs:**
- ✅ Satisfies the ticket's own R4 acceptance criterion literally.
- ✅ One additional `git mv` + 3 string edits — negligible cost.
- ❌ Slightly broader than the ticket bullets (but explicitly justified by the ticket's own acceptance criteria).
### Option C — Hybrid (rename listed + leave QQ群 + edit alt-only)
Not viable: there is no way to leave the file in place and still satisfy R4 without renaming.
### Decision direction
Recommend Option B. Update requirements R2/R3 to include `QQ群.png` explicitly so the spec is internally consistent with R4.
## 4. Out-of-Scope for Gap Analysis
- Choice of exact ASCII filename slugs (decided in design phase).
- Whether to re-encode any image (No — bytes-preserving rename only, per R2.4).
## 5. Implementation Complexity & Risk
- **Effort:** **S (≈ half-day).** All work is text edits + `git mv` of 9 files + 3 README string-substitution passes + 2 description-field edits. No code changes, no tests.
- **Risk:** **Low.** Single failure mode is broken image links; mitigated by a simple grep + rendered-preview check before commit. No runtime, dependency, or pipeline impact. `git mv` preserves history.
## 6. Recommendations for Design Phase
- Adopt **Option B** (expanded scope including `QQ群.png`).
- Use `git mv` for all renames so history follows.
- Pick deterministic ASCII slugs; propose:
- `Screenshot/screenshot{1..6}.png`
- `wuhan-university-simulation-cover.png`
- `dream-of-the-red-chamber-simulation-cover.jpg`
- `qq-group.png`
- Collapse the duplicated tagline lines in `README.md` / `README-EN.md`: delete the Chinese line + `</br>` separator and let the existing `<em>` English subtitle become the lone tagline (avoids a verbatim-duplicate line).
- Verification step: re-run `rg '[\x{4e00}-\x{9fff}]' README.md README-EN.md package.json backend/pyproject.toml` after edits and confirm only the language-switcher line on each README returns a hit.
## Research items to carry forward
- (Light) confirm no off-repo deep-link into the renamed assets (live demo site, social cards). If a deep link is found, decide whether to leave a redirect / note in the PR.

65
.kiro/specs/i18n-readme-tagline-and-assets/requirements.md Normal file
View File

@ -0,0 +1,65 @@
# Requirements Document
## Project Description (Input)
Translate the Chinese tagline in README.md, README-EN.md, and package.json to English, and rename Chinese-named image asset files in static/image/Screenshot/ to ASCII filenames (Option A from the ticket), updating all references in README.md and README-ZH.md. Acceptance: no Chinese characters in README.md or README-EN.md body text (except the language switcher link to README-ZH.md); package.json description in English; all image links work. Source: GitHub issue #12 (.ticket/12.md).
## Introduction
This feature removes the remaining Chinese surface text from the English documentation entry points (`README.md`, `README-EN.md`) and from the npm package metadata (`package.json`), and replaces Chinese-named image asset filenames under `static/image/` with ASCII equivalents so that asset URLs are CDN- and tooling-friendly. References to those assets are updated in all three READMEs (`README.md`, `README-EN.md`, `README-ZH.md`) so that the Chinese-language entry point continues to render correctly. The Chinese-language README (`README-ZH.md`) keeps its Chinese body text by design.
## Boundary Context
- **In scope**:
- English tagline replacing Chinese tagline in `README.md`, `README-EN.md`, and `package.json` `description`.
- Renaming `static/image/Screenshot/运行截图{1..6}.png` to ASCII filenames.
- Renaming `static/image/武大模拟演示封面.png` and `static/image/红楼梦模拟推演封面.jpg` to ASCII filenames.
- Renaming `static/image/QQ群.png` to an ASCII filename (added per gap-analysis: required by R4 because the existing src path on README.md:220 / README-EN.md:220 contains Chinese characters and would fail the "no Chinese characters in body text" check).
- Updating all `<img src="...">` references to those renamed files in `README.md`, `README-EN.md`, and `README-ZH.md`.
- Updating `backend/pyproject.toml` `description` field, which carries an identical Chinese tagline string (adjacent twin of `package.json`).
- **Out of scope**:
- Translating the body of `README-ZH.md` (Chinese variant by design).
- Translating the language switcher link label `[中文文档]` (allowed by acceptance criteria).
- Touching `locales/zh.json` Chinese tagline value (legitimate Chinese locale content).
- **Adjacent expectations**:
- The ticket recommends Option A (rename to ASCII). This spec adopts Option A.
- This work is a child of the i18n epic (#11) and follows the project's existing `i18n-*` spec naming.
## Requirements
### Requirement 1: English tagline in English-facing documentation
**Objective:** As a non-Chinese-reading visitor landing on the GitHub repo or installing the npm package, I want the tagline in the English README files and the npm package metadata to be in English, so that I am not surprised by untranslated Chinese strings on the entry surface.
#### Acceptance Criteria
1. The README.md file shall contain the English tagline `A Simple and Universal Swarm Intelligence Engine, Predicting Anything` in place of the Chinese tagline `简洁通用的群体智能引擎，预测万物` on the same line.
2. The README-EN.md file shall contain the same English tagline replacement on the corresponding line.
3. The package.json `description` field shall contain an English description (no Chinese characters).
4. The backend/pyproject.toml `description` field shall contain the same English description used in package.json.
5. The README-ZH.md file shall keep its Chinese tagline unchanged.
### Requirement 2: ASCII filenames for screenshot and video-cover assets
**Objective:** As a developer cloning the repo or a CDN serving these assets, I want all image filenames under `static/image/` referenced from the READMEs to be ASCII, so that paths are URL-safe, copy-pasteable, and friendly to tools that mishandle non-ASCII filenames.
#### Acceptance Criteria
1. The `static/image/Screenshot/运行截图{N}.png` files (for N from 1 to 6) shall be renamed to `static/image/Screenshot/screenshot{N}.png`.
2. The `static/image/武大模拟演示封面.png` file shall be renamed to `static/image/wuhan-university-simulation-cover.png`.
3. The `static/image/红楼梦模拟推演封面.jpg` file shall be renamed to `static/image/dream-of-the-red-chamber-simulation-cover.jpg`.
4. The `static/image/QQ群.png` file shall be renamed to `static/image/qq-group.png`.
5. The renamed asset files shall preserve the original byte content (rename only, no re-encoding).
6. The static/image/ directory shall not contain duplicate copies of the renamed files (the original Chinese-named files are removed, not kept alongside).
### Requirement 3: All README references updated to the ASCII filenames
**Objective:** As a reader of any README variant, I want the screenshot and video-cover images to render correctly, so that the documentation remains visually intact after the rename.
#### Acceptance Criteria
1. The README.md file shall reference each renamed image at its new ASCII path; no `<img src="...">` in the file shall point to a Chinese-named file under `static/image/`.
2. The README-EN.md file shall reference each renamed image at its new ASCII path; no `<img src="...">` in the file shall point to a Chinese-named file under `static/image/`.
3. The README-ZH.md file shall reference each renamed image at its new ASCII path; no `<img src="...">` in the file shall point to a Chinese-named file under `static/image/`.
4. When a reader views the rendered README on GitHub after the change, the system shall display every screenshot and video-cover image without a broken-image placeholder.
### Requirement 4: No residual Chinese in English README body text
**Objective:** As a reviewer verifying acceptance, I want a single objective check that confirms `README.md` and `README-EN.md` body text contains no Chinese characters (apart from the explicit allowance for the language-switcher link), so that the acceptance criteria from the ticket are unambiguously satisfied.
#### Acceptance Criteria
1. The README.md file shall contain no Chinese characters (Unicode CJK Unified Ideographs blocks U+4E00–U+9FFF and adjacent CJK punctuation) outside of the language-switcher link `[中文文档](./README-ZH.md)`.
2. The README-EN.md file shall contain no Chinese characters outside of the same language-switcher link.
3. If a reviewer runs a Chinese-character scan over `README.md` and `README-EN.md` excluding the language-switcher line, the scan shall report zero matches.

107
.kiro/specs/i18n-readme-tagline-and-assets/research.md Normal file
View File

@ -0,0 +1,107 @@
# Research & Design Decisions — i18n-readme-tagline-and-assets
## Summary
- **Feature**: `i18n-readme-tagline-and-assets`
- **Discovery Scope**: Simple Addition (docs cleanup + asset rename, no runtime code paths)
- **Key Findings**:
- The duplicate Chinese-tagline / English-`<em>` structure on lines 7–9 of `README.md` and `README-EN.md` means a verbatim translation produces a duplicate; a structural collapse is preferable.
- `git ls-files` shows nine Chinese-named assets under `static/image/`; only the eight visible in READMEs need renaming for this spec (the `MiroFish_logo` files and `shanda_logo.png` already use ASCII names).
- `backend/pyproject.toml:4` is a twin of `package.json:4` (identical Chinese tagline string); leaving it untranslated would visibly contradict the spec's intent.
## Research Log
### Topic — Inventory of Chinese-named assets and references
- **Context**: Confirm the full set of files and references the spec must touch so no broken-image regression slips in.
- **Sources Consulted**: `git ls-files static/image/`, `rg '[\x{4e00}-\x{9fff}]'` over `README.md`, `README-EN.md`, `README-ZH.md`, `package.json`, `backend/pyproject.toml`.
- **Findings**:
- Tracked Chinese-named files (9): `QQ群.png`, six `Screenshot/运行截图{N}.png`, `武大模拟演示封面.png`, `红楼梦模拟推演封面.jpg`.
- Each Chinese-named asset is referenced exactly three times — once in each README. No code path or test references them.
- `locales/zh.json:36` contains the tagline as a Chinese-locale value (legitimate, out of scope).
- **Implications**: The rename is a closed set: 9 file moves + (3 README × N references) edits. No runtime impact.
### Topic — Tagline structure on lines 7–9
- **Context**: Decide the cleanest replacement for the Chinese tagline on the English-facing READMEs.
- **Sources Consulted**: `README.md:7-9`, `README-EN.md:7-9`.
- **Findings**: The current structure is `<chinese tagline>\n</br>\n<em>English equivalent</em>`. The English subtitle already exists. Naive replacement (substitute Chinese with English on line 7) produces `<english>\n</br>\n<em>English</em>` — visible duplicate.
- **Implications**: Collapse to the single existing `<em>` line by deleting the Chinese tagline line and the `</br>` separator on both files.
### Topic — `git mv` vs. `rm`/`add` for renames
- **Context**: Choose a rename mechanism that preserves blame/history on the assets.
- **Sources Consulted**: Project commit history shows `git mv` usage for prior renames (no formal rule, but consistent practice).
- **Findings**: `git mv "old" "new"` records a rename in the index. Git's heuristic file-move detection also picks up `rm + add` of identical bytes, but `git mv` is unambiguous and preserves rename detection across thresholds.
- **Implications**: Use `git mv` for all nine renames. Quote source paths (rule from `.claude/rules/file-paths.md`) since they contain non-ASCII characters.
### Topic — Off-repo deep links to renamed assets (light check)
- **Context**: The ticket's gap analysis flagged a research item: confirm no external pages deep-link the Chinese-named files.
- **Sources Consulted**: `git grep` of repo (no off-repo references). The bilibili links in the READMEs point to videos, not to the cover images. The `mirofish-live-demo` site and `Trendshift` badge are independent assets hosted elsewhere.
- **Findings**: No in-repo references outside the READMEs. Out-of-repo deep links are not enumerable from inside the repo; the cost of a broken external deep link is low (a missing image on someone else's page) and accepted. If a deep link surfaces post-merge, a same-day re-add of a redirect symlink resolves it.
- **Implications**: Proceed with hard renames; no redirect/copy-on-rename needed.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Strict ticket scope | Rename only the 8 explicitly listed assets; leave `QQ群.png` | Smallest diff | Fails the ticket's own R4 acceptance criterion | Rejected |
| Expanded scope (selected) | Also rename `QQ群.png` and update `backend/pyproject.toml` | Internally consistent with R4; trivial cost | Slightly broader than ticket bullets | Selected |
| Hybrid (allow exception in R4) | Rename the 8 listed, exempt `QQ群` in the verification scan | Preserves the ticket bullets exactly | Adds an explicit ad-hoc exception that future readers must decode | Rejected |
## Design Decisions
### Decision: Rename `static/image/QQ群.png` to ASCII despite not being in the ticket's bullet list
- **Context**: Acceptance criterion R4 ("no Chinese characters in `README.md` / `README-EN.md` body") would fail because `QQ群` appears in the `<img src>` path on line 220 of both files.
- **Alternatives Considered**:
1. Strict scope — leave `QQ群.png` and accept R4 fail.
2. Expand scope — rename and update.
3. Exempt `QQ群.png` in R4's verification scope with explicit allow-list.
- **Selected Approach**: Expand scope. Rename `static/image/QQ群.png` → `static/image/qq-group.png`, update three references.
- **Rationale**: Trivial cost; same fix shape as the listed assets; the ticket's own acceptance criterion is the source of truth.
- **Trade-offs**: One extra file move. None material.
- **Follow-up**: None.
### Decision: Translate `backend/pyproject.toml:4` description in the same PR
- **Context**: `backend/pyproject.toml` carries the identical Chinese tagline as `package.json`. Leaving it untranslated produces a half-finished diff.
- **Alternatives Considered**:
1. Leave it for a follow-up ticket.
2. Translate it now alongside `package.json`.
- **Selected Approach**: Translate now.
- **Rationale**: Identical string, identical fix, same review surface. Splitting would create needless coordination.
- **Trade-offs**: One additional one-line diff. None material.
- **Follow-up**: None.
### Decision: Collapse duplicate tagline structure rather than substitute in place
- **Context**: Lines 7–9 of `README.md` and `README-EN.md` would yield a verbatim duplicate after a one-for-one Chinese-to-English substitution.
- **Alternatives Considered**:
1. Substitute Chinese line in place (produces duplicate).
2. Delete Chinese line + `</br>` separator; let the existing `<em>` line stand alone.
3. Delete the existing `<em>` line; keep a single non-italic English tagline on line 7.
- **Selected Approach**: Option 2 — delete lines 7 and 8, keep line 9 (`<em>` English tagline).
- **Rationale**: Preserves the existing visual treatment (italic subtitle below the Trendshift badge). Avoids style drift on a docs-only PR.
- **Trade-offs**: Slightly different visual weight (italic only) vs. the prior bilingual stack (plain Chinese + italic English). Acceptable for an English-facing doc.
- **Follow-up**: None.
### Decision: Use `git mv` for all renames
- **Context**: Need to preserve rename detection.
- **Alternatives Considered**: `git mv` vs. shell `mv` + `git rm` / `git add`.
- **Selected Approach**: `git mv "old" "new"` with quoted paths.
- **Rationale**: Unambiguous record in the index; matches existing project practice.
- **Trade-offs**: None.
- **Follow-up**: None.
## Risks & Mitigations
- **Risk:** Broken images on rendered GitHub README after merge. **Mitigation:** Post-edit grep to confirm zero remaining Chinese-named asset references in any README; preview rendered markdown locally or on a branch before merge.
- **Risk:** Off-repo deep links to old asset URLs (Trendshift cards, social previews). **Mitigation:** Accepted; cost is a single missing image on an external page.
- **Risk:** Diff churn from accidentally re-encoding a binary on macOS or Windows checkout. **Mitigation:** Use `git mv` (no content transform); verify `git diff --stat` shows only renames for the asset files (no content delta).
## References
- Ticket source: `.ticket/12.md` / GitHub issue #12.
- Project rule on quoting paths: `.claude/rules/file-paths.md`.
- Project commit conventions: `.claude/rules/commits.md` and `.kiro/steering/structure.md`.

23
.kiro/specs/i18n-readme-tagline-and-assets/spec.json Normal file
View File

@ -0,0 +1,23 @@
{
"feature_name": "i18n-readme-tagline-and-assets",
"created_at": "2026-05-07T19:24:24Z",
"updated_at": "2026-05-07T19:32:00Z",
"language": "en",
"phase": "tasks-generated",
"ticket": "12",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": false
}
},
"ready_for_implementation": true
}

47
.kiro/specs/i18n-readme-tagline-and-assets/tasks.md Normal file
View File

@ -0,0 +1,47 @@
# Implementation Plan
- [x] 1. Translate Chinese taglines to English in the project's English-facing metadata
- In `README.md`, delete the Chinese tagline line and the immediately following `</br>` line so the existing italic English subtitle on the next line stands as the lone tagline; verify the result still renders with one tagline visible above the Shanda badge
- Apply the identical edit to `README-EN.md`
- In `package.json`, set the `description` value to `MiroFish - A Simple and Universal Swarm Intelligence Engine, Predicting Anything`
- In `backend/pyproject.toml`, set the `description` value to the same English string used in `package.json`
- Leave `README-ZH.md` line 7 (the Chinese tagline) untouched
- Observable completion: a ripgrep scan for `[\x{4e00}-\x{9fff}]` over `README.md`, `README-EN.md`, `package.json`, and `backend/pyproject.toml` returns hits **only** on the language-switcher line of the two READMEs
- _Requirements: 1.1, 1.2, 1.3, 1.4, 1.5_
- [x] 2. (P) Rename Chinese-named static image assets to ASCII filenames using git mv
- Move the six screenshot files `static/image/Screenshot/运行截图{1..6}.png` to `static/image/Screenshot/screenshot{1..6}.png`
- Move `static/image/武大模拟演示封面.png` to `static/image/wuhan-university-simulation-cover.png`
- Move `static/image/红楼梦模拟推演封面.jpg` to `static/image/dream-of-the-red-chamber-simulation-cover.jpg`
- Move `static/image/QQ群.png` to `static/image/qq-group.png`
- Quote source paths in shell invocations because they contain non-ASCII characters
- Use `git mv` (not shell `mv` + `git add`) so rename detection is recorded directly in the index
- Observable completion: `git status` reports nine `renamed:` entries with no other file modifications; `git diff --stat -M` shows zero content-line delta for each asset
- _Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6_
- _Boundary: static/image/_
- [x] 3. Update README image references to point at the renamed ASCII asset paths
- In `README.md`, rewrite the nine `<img src="...">` paths on lines 52–61, 71, 79, and 220 so each points at the corresponding ASCII filename from task 2
- Apply the identical nine edits to `README-EN.md`
- Apply the identical nine edits to `README-ZH.md` (asset path updates only — Chinese body text and Chinese alt attributes preserved)
- Observable completion: a ripgrep search for `运行截图|武大模拟演示封面|红楼梦模拟推演封面|QQ群` in `README.md`, `README-EN.md`, and `README-ZH.md` returns zero matches
- _Requirements: 3.1, 3.2, 3.3_
- _Depends: 2_
- [x] 4. Verify acceptance gates before commit
- [x] 4.1 Run the Chinese-character verification scan and confirm zero residual hits in the EN READMEs body
- Execute `rg --pcre2 '[\x{4e00}-\x{9fff}]' README.md README-EN.md | rg -v 'README-ZH\.md'` from the repo root
- Observable completion: the pipeline produces zero output lines, confirming the only Chinese characters left in the EN READMEs are inside the language-switcher link to `README-ZH.md`
- _Requirements: 4.1, 4.2, 4.3_
- [x] 4.2 Confirm asset renames are byte-preserving and unambiguous
- Run `git diff --stat -M` and verify each of the nine asset files appears as a pure rename (no `+` or `-` line counts)
- Run `git status` and confirm there are no untracked Chinese-named files left behind in `static/image/` or `static/image/Screenshot/`
- Observable completion: nine `renamed:` entries in `git status`; zero untracked Chinese-named asset files; zero content delta on the asset rows of `git diff --stat`
- _Requirements: 2.5, 2.6, 3.4_
- [x] 4.3 Confirm rendered images by spot-checking the README in a Markdown previewer
- Open `README.md`, `README-EN.md`, and `README-ZH.md` in a Markdown preview (GitHub preview on the feature branch or local previewer) and inspect the screenshot grid, the two video-cover thumbnails, and the QQ group image on each file
- Observable completion: every `<img>` element renders an actual image (no broken-image placeholder) on all three READMEs
- _Requirements: 3.4_
- **Note**: This task ran in an autonomous environment where no Markdown previewer was available; instead, every `<img src>` path in all three READMEs was cross-checked against the working tree and all 33 references resolved to existing files (zero broken paths). A reviewer should still spot-check on the GitHub-rendered PR preview.

78
.kiro/specs/i18n-translate-backend-comments/HANDOFF.md Normal file
View File

@ -0,0 +1,78 @@
# Handoff — `i18n-translate-backend-comments` (Issue #7)
## Status
**Complete.** All in-scope Chinese docstrings and `#` comments under `backend/` have been translated to English.
This second installment of the ticket-#7 cleanup builds on the first installment (PR #20) and finishes the remaining 12 files. Together, the two installments cover the full 35-file in-scope set.
## Completed across both installments (35 files)
### First installment (PR #20 — landed on `feat/i18n-6-externalize-backend-logs`, then merged here via `merge main` into this branch)
- **Root**: `backend/app/__init__.py`, `backend/app/config.py`, `backend/run.py`
- **API package init**: `backend/app/api/__init__.py`
- **Models** (full package): `backend/app/models/__init__.py`, `project.py`, `task.py`
- **Utils** (full package): `backend/app/utils/__init__.py`, `file_parser.py`, `llm_client.py`, `locale.py`, `logger.py`, `retry.py`, `zep_paging.py`
- **Services** (partial): `backend/app/services/__init__.py`, `graph_builder.py`, `ontology_generator.py`, `simulation_ipc.py`, `simulation_manager.py`, `text_processor.py`, `zep_entity_reader.py`
- **Scripts** (partial): `backend/scripts/action_logger.py`, `backend/scripts/test_profile_format.py`
### Second installment (this PR — finishes the ticket)
| File | Starting in-scope hits | Comment-the-obvious deletions |
| --- | --- | --- |
| `backend/app/api/graph.py` | 70 | 25 |
| `backend/app/api/report.py` | 104 | 11 |
| `backend/app/api/simulation.py` | 351 | ~25 |
| `backend/app/services/oasis_profile_generator.py` | 185 | ~14 |
| `backend/app/services/report_agent.py` | 335 | 8 |
| `backend/app/services/simulation_config_generator.py` | 148 | 0 |
| `backend/app/services/simulation_runner.py` | 277 | ~31 |
| `backend/app/services/zep_graph_memory_updater.py` | 97 | 5 |
| `backend/app/services/zep_tools.py` | 269 | 6 |
| `backend/scripts/run_parallel_simulation.py` | 227 | ~7 |
| `backend/scripts/run_reddit_simulation.py` | 75 | 12 |
| `backend/scripts/run_twitter_simulation.py` | 97 | 21 |
| **Total** | **2,235** | **~165** |
After the pass, every file in the table reports zero in-scope hits from the AST scanner.
## Remaining residuals (out of scope — owned by sibling tickets)
After this PR, the only files under `backend/` that still contain CJK characters do so exclusively inside string literals. These are owned by sibling tickets and are intentional residuals for this spec:
- LLM prompt template strings: `oasis_profile_generator.py`, `ontology_generator.py`, `simulation_config_generator.py`, `report_agent.py` — owned by tickets #2 / #3 / #4 / #5.
- Runtime log strings, API response messages, exception arguments, CLI prints: distributed across `api/`, `services/`, `scripts/`, `utils/retry.py`, `utils/locale.py`, `run.py`, `app/config.py` — owned by ticket #6 (with follow-up tickets #18, #24 for residuals).
- Sample-data values returned to clients: `services/zep_tools.py`, `services/zep_graph_memory_updater.py`, `services/zep_entity_reader.py`, etc.
The CJK CI guard (`scripts/ci/i18n_cjk_guard.py`) enforces that this set never grows; the per-path baseline at `.kiro/specs/i18n-ci-guard/baseline.txt` is updated as part of this PR to reflect the new (lower) count.
## Verification methodology
The AST-aware scanner at `.kiro/specs/i18n-translate-backend-comments/scan_chinese.py` (committed in this branch) classifies every CJK-bearing line into one of three buckets:
- `DOCSTRING` — line lies inside a module/class/function docstring (in scope).
- `COMMENT` — line contains a `#` and is not inside a docstring or string-literal span (in scope).
- `STRING` — line is part of a string-literal value (out of scope, owned by sibling tickets).
For every translated file in this installment:
1. `python3 -m py_compile <file>` succeeds.
2. The scanner reports `0` in-scope hits.
3. `git diff <file>` shows only docstring lines and `#` comment lines changed; no signature, import, decorator, expression, or string-literal byte changes.
For two of the largest files (`api/simulation.py`, `report_agent.py`), the implementing agent additionally ran an AST-equivalence check (parsing both before and after, stripping docstrings, and confirming structural equality) to validate that no executable surface changed.
## Test environment caveat
The repo's `uv sync` builds `tiktoken` from source, which requires a Rust toolchain. The sandbox running this implementation pass does not have Rust, so `cd backend && uv run python -m pytest scripts/test_profile_format.py` cannot be executed end-to-end here. Because the change set is comments-and-docstrings-only, runtime behavior cannot be affected; the syntactic-validity check (`py_compile` across all 12 files) stands in for the test run in this environment.
A developer with the project's normal dev environment (Rust toolchain installed, full `uv sync` succeeded) should re-run `cd backend && uv run python -m pytest scripts/test_profile_format.py` against this branch before merging to confirm.
## What is NOT changed
- No string literal anywhere in the touched files (verified by AST classification).
- No executable Python statement.
- No symbol renamed; `zep_*` legacy filenames preserved per steering rule.
- No file added or removed (other than the AST scanner inside `.kiro/specs/i18n-translate-backend-comments/`).
- No dependency added or version-bumped.
## Branch & PR
- Branch: `docs/i18n-7-translate-backend-comments` (re-used from PR #20; that PR was merged into `feat/i18n-6-externalize-backend-logs` after `feat/i18n-6` had already merged into `main`, which orphaned PR #20's content from `main`).
- This PR re-targets the branch at `main`, including: the four prior commits from PR #20, a `Merge branch 'main'` commit (one conflict resolved in `services/ontology_generator.py` to combine PR #20's translated comment with main's English prompt-string), and the new commits for the 12 files completed here.
- Commits follow Conventional Commits in the form `docs(i18n): translate chinese docstrings/comments in backend/<area>`.
- The PR description references issue #7 with `Closes #7`.
- No `Co-Authored-By:` watermarks.

316
.kiro/specs/i18n-translate-backend-comments/design.md Normal file
View File

@ -0,0 +1,316 @@
# Design Document — `i18n-translate-backend-comments`
## Overview
**Purpose**: Translate Chinese-language docstrings and `#` comments across `backend/` Python files into English, so that English-speaking maintainers can read and review the codebase without translation overhead.
**Users**: Backend maintainers and code reviewers who do not read Chinese.
**Impact**: Improves developer ergonomics and review throughput. No runtime, behavior, or interface change. Adjacent i18n tickets (#2/#3/#4/#5/#6), which own the string-literal Chinese, remain unaffected.
### Goals
- Eliminate Chinese characters from docstrings and `#` comments under the in-scope paths.
- Preserve Google-style docstring shape and project formatting rules (4-space indent, ≤120 chars/line, double-quoted strings).
- Keep the diff comments-and-docstrings-only — no executable, string-literal, or symbol changes.
### Non-Goals
- Translating Chinese inside string literals (prompt templates, `logger.{info,warning,error}` arguments, API responses, error messages). These are owned by issues #2/#3/#4/#5/#6.
- Refactoring code, reformatting style, or renaming symbols.
- Introducing new tooling, linters, or CI rules.
- Translating `backend/tests/test_locale*.py` (Chinese there is intentional test data inside string literals; outside ticket scope).
## Boundary Commitments
### This Spec Owns
- Comment and docstring text under: `backend/app/__init__.py`, `backend/app/config.py`, `backend/app/api/`, `backend/app/models/`, `backend/app/services/`, `backend/app/utils/`, `backend/run.py`, `backend/scripts/`.
- The decision rule for distinguishing docstrings from value strings (first-statement rule).
- The Chinese→English Google-style docstring key map.
- The verification workflow (residual `grep`, `pytest`, diff sanity check).
### Out of Boundary
- All string-literal content, including triple-quoted strings used as values.
- Files under `backend/tests/`, `backend/.venv/`, and any non-Python file.
- Refactors, renames, formatting changes, or new dependencies.
- Front-end localization, locale JSON files, or i18n runtime behavior.
### Allowed Dependencies
- The repository's Python source (read + write for in-scope files only).
- The existing test suite (`backend/scripts/test_profile_format.py`) for verification.
- The existing `grep`-based residual scan for verification.
### Revalidation Triggers
- A new in-scope file added under the listed paths (would expand the file list).
- A change to `dev-guidelines.md` regarding docstring style (would change the key map or quote/indent rule).
- A merge of any adjacent i18n ticket (#2/#3/#4/#5/#6) that turns a string literal into a docstring or vice versa.
## Architecture
### Existing Architecture Analysis
This change touches only commentary; no architectural element of the backend is modified. The work spans the following packages:
- `backend/app/__init__.py`, `backend/app/config.py` (Flask app and configuration entrypoint).
- `backend/app/api/` (Flask blueprints).
- `backend/app/models/` (`Project`, `Task` models).
- `backend/app/services/` (graph builder, simulation runner, report agent, etc.).
- `backend/app/utils/` (LLM client, file parser, retry, logger, locale, paging).
- `backend/run.py` (process entrypoint).
- `backend/scripts/` (simulation runners, profile-format test).
### Architecture Pattern & Boundary Map
```mermaid
graph TB
Discovery[Residual Grep Scan]
Plan[Per-Package Plan]
Translator[Translation Pass]
Verify[Verification Gate]
Commit[Per-Package Commit]
PR[Single PR to main]
Discovery --> Plan
Plan --> Translator
Translator --> Verify
Verify -->|all checks pass| Commit
Verify -->|any check fails| Translator
Commit --> Plan
Commit -->|all packages done| PR
```
**Architecture Integration**:
- Selected pattern: **Iterative pass per package** with a verification gate after each pass. Linear, deterministic, low-coordination.
- Domain/feature boundaries: One pass per backend package; commits are package-scoped to keep review chunks small.
- Existing patterns preserved: 4-space indent, double-quoted strings, Google-style docstrings, `snake_case`, project file layout.
- New components rationale: None — no new code, no new files.
- Steering compliance: Conforms to repo-level coding rules and the commits ruleset.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Backend / Services | Python ≥3.11 | Source language whose docstrings/comments are being translated | No version change; no dependency change |
| Tooling | `git`, `grep`, `pytest` (existing) | Discovery, verification, regression check | No new tools |
No frontend, data, messaging, or infrastructure layer is touched.
## File Structure Plan
### Directory Structure (no additions, no deletions)
```
backend/
├── app/
│ ├── __init__.py # docstrings/comments only
│ ├── config.py # docstrings/comments only
│ ├── api/ # all *.py: docstrings/comments only
│ ├── models/ # all *.py: docstrings/comments only
│ ├── services/ # all *.py: docstrings/comments only
│ └── utils/ # all *.py: docstrings/comments only
├── run.py # docstrings/comments only
└── scripts/ # all *.py: docstrings/comments only
```
### Modified Files
The 37 in-scope files identified in `gap-analysis.md` are modified — comment and docstring lines only. No other paths are touched.
## Translation Rules
These rules drive the translation pass and the verification gate. They are normative; the implementation must follow them exactly.
### Rule 1 — Docstring vs Value String Disambiguation
A triple-quoted string is treated as a **docstring** (in scope) iff it is the first statement of a module, class, or function body. All other triple-quoted strings are **values** (out of scope) and must not be modified.
### Rule 2 — Translate Docstrings to English Google-style
- Translate Chinese narrative text to faithful English.
- Convert the following Chinese section keys to canonical English Google-style keys when present:
| Chinese key | English key |
| --- | --- |
| `参数:` | `Args:` |
| `返回:` | `Returns:` |
| `异常:` | `Raises:` |
| `产生:` / `生成:` | `Yields:` |
| `示例:` | `Examples:` |
| `注意:` / `备注:` | `Note:` |
- Preserve double-quoted triple-quoted form (`"""..."""`).
- Preserve indentation matching the surrounding scope.
### Rule 3 — Translate Inline `#` Comments to English
- Translate the comment text to English.
- If the translated comment would merely restate the immediately following executable line (a redundant verb-phrase paraphrase), delete the comment.
- Preserve `TODO:` / `FIXME:` markers and any embedded ticket reference verbatim.
- Preserve trailing in-line comments on the same line as code (e.g. `PENDING = "pending" # waiting`).
### Rule 4 — Style Compliance
- Keep every translated line ≤120 characters.
- Do not introduce trailing whitespace.
- Preserve the original indentation of each comment/docstring.
- Use double quotes for any docstring rewritten.
### Rule 5 — Preservation
- Do not modify any executable Python statement.
- Do not modify any string literal (single-, double-, triple-quoted, f-string, raw, byte) that is not a docstring under Rule 1. The single exception is the docstring being rewritten under Rule 2: quote-style normalization to triple double-quoted form (`"""..."""`) is permitted on the docstring only, since it is the artifact under translation.
- Do not rename any symbol.
## System Flows
### Per-package iteration
```mermaid
sequenceDiagram
participant Dev as Translator
participant Repo as Repo
participant Tests as Test Suite
Dev->>Repo: git checkout docs/i18n-7-translate-backend-comments
loop For each package in [models, utils, services, api, scripts, root]
Dev->>Repo: Translate docstrings/comments
Dev->>Repo: git diff --stat (sanity check)
Dev->>Tests: cd backend then uv run python -m pytest scripts/test_profile_format.py
Tests-->>Dev: pass / fail
Dev->>Repo: Re-run residual grep
Repo-->>Dev: residual hits (string-literal only)
Dev->>Repo: git commit -m "docs(i18n): translate chinese docstrings/comments in backend/<area>"
end
Dev->>Repo: gh pr create -> single PR closing #7
```
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | No Chinese in docstrings under in-scope paths | Translation Pass | Rule 1, Rule 2 | Per-package iteration |
| 1.2 | No Chinese in `#` comments under in-scope paths | Translation Pass | Rule 3 | Per-package iteration |
| 1.3 | Residual grep returns only string-literal Chinese | Verification Gate | Residual grep workflow | Per-package iteration |
| 1.4 | Google-style docstring shape preserved | Translation Pass | Rule 2 (key map) | — |
| 2.1 | No executable statement modified | Verification Gate | Rule 5 | Per-package iteration |
| 2.2 | No string literal modified | Verification Gate | Rule 1 (first-statement rule), Rule 5 | Per-package iteration |
| 2.3 | No symbol renamed | Verification Gate | Rule 5 | Per-package iteration |
| 2.4 | `pytest` passes | Verification Gate | Test suite invocation | Per-package iteration |
| 2.5 | Hunks touching code rejected | Verification Gate | `git diff --stat` review | Per-package iteration |
| 3.1 | Drop redundant comments | Translation Pass | Rule 3 | — |
| 3.2 | Translate the *why* faithfully | Translation Pass | Rule 3 | — |
| 3.3 | Preserve `TODO:`/`FIXME:` and ticket refs | Translation Pass | Rule 3 | — |
| 3.4 | No new comments introduced | Translation Pass | Rule 3 | — |
| 4.1 | ≤120 chars/line | Verification Gate | Rule 4 | — |
| 4.2 | No trailing whitespace | Verification Gate | Rule 4 | — |
| 4.3 | Preserve indentation | Translation Pass | Rule 4 | — |
| 4.4 | Double quotes on rewritten docstrings | Translation Pass | Rule 4 | — |
| 4.5 | Preserve 4-space indentation | Translation Pass | Rule 4 | — |
| 5.1 | Use grep for discovery | Verification Gate | Discovery scan | — |
| 5.2 | Re-run grep after each batch | Verification Gate | Residual grep workflow | Per-package iteration |
| 5.3 | Continue until non-string-literal residual cleared | Verification Gate | Rule 1 disambiguation | Per-package iteration |
| 5.4 | `git diff --stat` only in-scope paths | Verification Gate | Diff sanity check | Per-package iteration |
| 6.1 | Branch `docs/i18n-7-translate-backend-comments` | Tracking & Branching | `/done` skill | — |
| 6.2 | Reference issue #7 | Tracking & Branching | Commit/PR template | — |
| 6.3 | Conventional Commits `docs(i18n)` | Tracking & Branching | `.claude/rules/commits.md` | — |
| 6.4 | No unrelated changes | Verification Gate | Diff sanity check | — |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|--------------|--------------------------|-----------|
| Translation Pass | Process | Apply Rules 1–5 to one package's `*.py` | 1.1, 1.2, 1.4, 3.1, 3.2, 3.3, 3.4, 4.3, 4.4, 4.5 | None (manual + AI-assisted) | Process |
| Verification Gate | Process | Run residual grep, `pytest`, and diff sanity check after each package | 1.3, 2.1, 2.2, 2.3, 2.4, 2.5, 4.1, 4.2, 5.1, 5.2, 5.3, 5.4, 6.4 | `git`, `grep`, `pytest` (P0) | Process |
| Tracking & Branching | Process | Branching, commit messages, PR | 6.1, 6.2, 6.3 | `/done` skill, `gh` CLI (P0) | Process |
### Process
#### Translation Pass
| Field | Detail |
|-------|--------|
| Intent | Translate docstrings and `#` comments in one package without touching code or string literals |
| Requirements | 1.1, 1.2, 1.4, 3.1, 3.2, 3.3, 3.4, 4.3, 4.4, 4.5 |
**Responsibilities & Constraints**
- Apply Rule 1 (first-statement disambiguation) before editing any triple-quoted string.
- Apply Rule 2 (key map) for any Chinese Google-style key encountered.
- Apply Rule 3 to inline comments; delete redundant ones.
- Operate on one package at a time; do not interleave packages.
**Dependencies**
- Inbound: Verification Gate (provides feedback if a previous batch failed).
- Outbound: Verification Gate (hands off post-pass).
- External: None.
**Contracts**: Process [x] / Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]
**Implementation Notes**
- Integration: Operates directly on the working tree on branch `docs/i18n-7-translate-backend-comments`.
- Validation: After each file is rewritten, sanity-check that the diff for that file shows changes only on comment/docstring lines.
- Risks: Accidental edit to a string-literal triple-quoted value — mitigated by Rule 1 + diff review.
#### Verification Gate
| Field | Detail |
|-------|--------|
| Intent | Confirm a package's translation pass left runtime behavior intact |
| Requirements | 1.3, 2.1, 2.2, 2.3, 2.4, 2.5, 4.1, 4.2, 5.1, 5.2, 5.3, 5.4, 6.4 |
**Responsibilities & Constraints**
- Re-run `grep -rln '[一-鿿]' backend/ --include='*.py'` after each package and confirm residual hits are limited to string-literal Chinese owned by adjacent tickets.
- Run `uv run python -m pytest backend/scripts/test_profile_format.py` and confirm exit 0.
- Run `git diff --stat` and confirm only in-scope file paths are listed.
- Spot-check a sample of changed files to confirm only comment/docstring lines changed.
**Dependencies**
- Inbound: Translation Pass.
- Outbound: Tracking & Branching (commits) when all checks pass; loops back to Translation Pass otherwise.
- External: `git`, `grep`, `pytest` (P0 — required for verification).
**Contracts**: Process [x] / Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]
**Implementation Notes**
- Integration: Run from the repo root; no environment variables required beyond what `uv run` already provides.
- Validation: All four checks (grep / pytest / diff scope / spot diff) must pass before committing.
- Risks: A flaky `pytest` run unrelated to this change would block progress — mitigated by reading the failure and re-running once.
#### Tracking & Branching
| Field | Detail |
|-------|--------|
| Intent | Branch, commit, push, and open PR per project conventions |
| Requirements | 6.1, 6.2, 6.3 |
**Responsibilities & Constraints**
- Branch name: `docs/i18n-7-translate-backend-comments`.
- Commit messages follow Conventional Commits with `docs(i18n)` scope (e.g. `docs(i18n): translate chinese docstrings/comments in backend/services`).
- PR closes #7 and references the spec.
**Dependencies**
- Inbound: Verification Gate (only commits when all checks pass).
- External: `gh` CLI (P0), `/done` skill (P0).
**Contracts**: Process [x] / Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [ ]
**Implementation Notes**
- Integration: Use `/done` skill at the end to handle branch/push/PR uniformly.
- Validation: Confirm PR body references issue #7 with `Closes #7` and lists each commit.
- Risks: None.
## Error Handling
### Error Strategy
This is a build-time / source-edit task — there is no runtime error path. Errors are caught by the Verification Gate.
### Error Categories and Responses
- **Translation slipped into a string literal**: caught by `git diff --stat` + spot diff. Response: revert that hunk, re-apply translation against the docstring/comment only.
- **Test suite fails after a pass**: caught by `pytest`. Response: read failure, identify which line was incorrectly modified (likely a string the translator misclassified as a docstring), revert that hunk, re-apply.
- **Residual grep returns non-string-literal Chinese**: caught by post-pass grep. Response: classify those hits as in-scope and translate them in the next sub-pass.
- **Line exceeds 120 chars after translation**: caught by spot diff. Response: reflow the comment/docstring without changing executable code.
### Monitoring
None — this is a one-shot change. No production observability required.
## Testing Strategy
The repository's existing tests are the safety net. No new tests are added.
### Default sections
- **Unit Tests**: Not applicable; nothing executable changes.
- **Integration Tests**: `uv run python -m pytest backend/scripts/test_profile_format.py` must continue to pass after each commit.
- **E2E/UI Tests**: Not applicable.
- **Verification checks (per package commit)**:
1. Residual `grep -rln '[一-鿿]' backend/ --include='*.py'` (run from repo root) returns only files whose remaining Chinese is in string literals owned by adjacent tickets.
2. `cd backend && uv run python -m pytest scripts/test_profile_format.py` exits 0.
3. `git diff --stat HEAD~..HEAD` shows only in-scope file paths.
4. Spot diff on three random changed files confirms only comment/docstring lines changed.
## Supporting References (Optional)
- `gap-analysis.md` — full file enumeration and pattern survey.
- `research.md` — discovery log, alternatives, and decisions.

92
.kiro/specs/i18n-translate-backend-comments/gap-analysis.md Normal file
View File

@ -0,0 +1,92 @@
# Gap Analysis — `i18n-translate-backend-comments`
## Scope Recap
- **Ticket**: salestech-group/MiroFish#7
- **Goal**: Translate Chinese docstrings and `#` comments in `backend/` to English without behavior changes.
- **Blast radius**: Comments and docstrings only; runtime semantics preserved.
## Current State Investigation
### Discovered files
A scan with the regex `[一-鿿]` across `backend/**/*.py` (excluding `.venv`) returns **37 in-app files** plus 2 test files:
| Area | Count | Files |
| --- | --- | --- |
| `backend/app/__init__.py` | 1 | `__init__.py` |
| `backend/app/config.py` | 1 | `config.py` |
| `backend/app/api/` | 4 | `__init__.py`, `graph.py`, `report.py`, `simulation.py` |
| `backend/app/models/` | 3 | `__init__.py`, `project.py`, `task.py` |
| `backend/app/services/` | 12 | `__init__.py`, `graph_builder.py`, `oasis_profile_generator.py`, `ontology_generator.py`, `report_agent.py`, `simulation_config_generator.py`, `simulation_ipc.py`, `simulation_manager.py`, `simulation_runner.py`, `text_processor.py`, `zep_entity_reader.py`, `zep_graph_memory_updater.py`, `zep_tools.py` |
| `backend/app/utils/` | 7 | `__init__.py`, `file_parser.py`, `llm_client.py`, `locale.py`, `logger.py`, `retry.py`, `zep_paging.py` |
| `backend/run.py` | 1 | `run.py` |
| `backend/scripts/` | 5 | `action_logger.py`, `run_parallel_simulation.py`, `run_reddit_simulation.py`, `run_twitter_simulation.py`, `test_profile_format.py` |
| `backend/tests/` (extra, not in ticket file list) | 2 | `test_locale.py`, `test_locale_request_resolution.py` |
Spot checks (`models/task.py`, `models/project.py`, `services/text_processor.py`, `utils/locale.py`):
- Module-level docstrings in Chinese (e.g. `"""任务状态管理"""`).
- Class/method docstrings in Chinese, often Google-shaped (`Args:` translated as `参数:`).
- Inline `#` comments tagging fields, sections, or restating obvious code (e.g. `# 标准化换行` above an `\n` normalization call).
- Status-enum trailing comments (e.g. `PENDING = "pending" # 等待中`).
### Conventions to preserve
- Project guideline: 4-space indent, max 120 char/line, double-quoted strings (Python).
- Docstring style: Google-style per `dev-guidelines.md`. Existing files mix English-shape `Args:`/`Returns:` keys with Chinese descriptions, or use Chinese keys (`参数:`, `返回:`). Translate both to canonical Google-style English.
- File-level convention: `snake_case` filenames, Python `__init__.py` modules typically have a one-line module docstring.
### Integration surfaces
None. This work touches only commentary; no API contracts, schemas, or imports change.
## Requirements Feasibility
| Requirement | Status | Notes |
| --- | --- | --- |
| R1 (coverage) | Feasible — straightforward | Files identified by `grep` rule. |
| R2 (behavior preservation) | Feasible | Achieved by limiting diffs to comment/docstring lines. Need to be careful with multi-line triple-quoted docstrings vs string literals (they are syntactically identical to strings — disambiguation: docstring is the *first* statement of a module/class/function body). |
| R3 (comment hygiene) | Feasible | Some judgment required; will adopt heuristic: drop comments whose translated form would be a single verb-phrase paraphrase of the next executable line. |
| R4 (style compliance) | Feasible | Watch line-length when translating dense Chinese to English (English is typically longer); rewrap as needed without changing executable code. |
| R5 (verification) | Feasible | The `grep -rln '[一-鿿]'` rule is reliable. Residual hits should land only in: prompt template strings (#2/#3/#4/#5), logger/API string literals (#6), and the `tests/test_locale*` files (intentional Chinese test data). |
| R6 (tracking/branching) | Feasible | Branch + commit conventions are standard for this repo; `/done` skill enforces them. |
### Gaps and constraints
- **Constraint**: Triple-quoted strings used as values (not as docstrings) must NOT be edited if their content is in scope of issues #2–#6 (prompts/log messages/error messages). Disambiguation matters.
- **Constraint**: Chinese characters appearing inside f-string literal segments must remain. They are out of scope.
- **Unknown / Research Needed**: None — task is mechanical and well-bounded.
### Adjacent specs / overlap with other tickets
- `i18n-externalize-backend-logs` (#6) owns translating `logger.{info,warning,error}` Chinese arguments and API response strings.
- `i18n-report-agent-prompts` (#5), and tickets #2/#3/#4 own prompt template strings.
- We must NOT touch any string literal that those tickets own. After this PR, residual `grep` hits should reduce by exactly the count of comments and docstrings translated and nothing else.
- The two `backend/tests/test_locale*.py` files are **not in the ticket's listed file scope**, and inspection shows their Chinese is exclusively in string literals (test data and a Unicode range check). They are out of scope by R1's enumerated paths and remain untouched.
## Implementation Approach Options
### Option A — Single-pass file-by-file translation (recommended)
- Walk the 37 in-scope files in a deterministic order (alphabetical), translating docstrings/comments per file, running the residual grep after each batch.
- Group commit by area (models, utils, services, api, scripts, root) to keep PR diff readable.
- ✅ Simple, low risk, easy to revert per-area.
- ✅ Maps directly to the requirements; easy to verify.
- ❌ Larger PR than option B, but ticket explicitly allows a single PR.
### Option B — Multi-PR per package
- Split into one PR per package (`models/`, `utils/`, …). The ticket allows this.
- ✅ Smaller diffs to review.
- ❌ More overhead (multiple branches/PRs); not necessary for a mechanical change of this size.
### Option C — Tooling-assisted bulk script
- Build a one-shot translation script (LLM-driven) that rewrites docstrings/comments.
- ✅ Could scale to other repos.
- ❌ Out of proportion for a single-ticket task; risk of errant edits to string literals; tooling itself becomes a deliverable to test and maintain.
## Effort and Risk
- **Effort**: **M (3–7 days of focused work)** — 37 files, hundreds of comments. In an interactive AI-assisted run, this collapses to a few hours.
- **Risk**: **Low** — comments-only diff; covered by mechanical verification (grep + pytest); easy to rollback per file/area.
## Recommendations for Design Phase
- **Preferred approach**: Option A (single-pass file-by-file, package-grouped commits, single PR).
- **Key decisions to capture in design**:
- Order of traversal (proposed: `models/` → `utils/` → `services/` → `api/` → `scripts/` → root files `__init__.py`, `config.py`, `run.py`).
- Heuristic for "drops the obvious comment" (one-line rule).
- How to handle Google-style docstring keys: always translate `参数:` → `Args:`, `返回:` → `Returns:`, `异常:` → `Raises:`.
- Verification cadence: re-run the grep after each package batch.
- **Research items to carry forward**: None.

67
.kiro/specs/i18n-translate-backend-comments/requirements.md Normal file
View File

@ -0,0 +1,67 @@
# Requirements Document
## Introduction
This specification covers the developer-facing internationalization of `backend/` Python source: translating Chinese docstrings and inline comments to English so that English-speaking maintainers can read and review the code without translation overhead. The change is mechanical — no behavior, no public strings, no symbol names are modified. It is one of several i18n tickets (#2, #3, #4, #5, #6, #7); this spec covers ticket #7 only.
## Boundary Context
- **In scope**: Translation of Chinese-language characters that appear in Python docstrings (module/class/function) and inline `#` comments under `backend/`. Removal of comments that merely restate the code. Preservation of `TODO:` / `FIXME:` markers and embedded ticket references.
- **Out of scope**: Chinese characters inside string literals (prompt templates, `logger.{info,warning,error}` arguments, API response bodies, error messages returned to clients) — these are tracked separately by issues #2/#3/#4/#5/#6. No refactoring, reformatting, renaming, or behavior changes.
- **Adjacent expectations**: Spec `i18n-externalize-backend-logs` (issue #6) and the prompt-translation specs handle string-literal Chinese; this spec must leave those untouched so the other tickets remain mergeable.
## Requirements
### Requirement 1: Translation Coverage of In-Scope Files
**Objective:** As a maintainer, I want every Chinese docstring and inline comment in the in-scope backend files translated to English, so that I can read and review the code without translation tools.
#### Acceptance Criteria
1. The Backend Codebase shall contain no Chinese characters (Unicode range U+4E00–U+9FFF) inside Python docstrings under `backend/app/__init__.py`, `backend/app/config.py`, `backend/app/models/`, `backend/app/services/`, `backend/app/api/`, `backend/app/utils/`, `backend/run.py`, and `backend/scripts/`.
2. The Backend Codebase shall contain no Chinese characters inside Python `#` inline comments under the same paths.
3. When `grep -rln '[一-鿿]' backend/ --include='*.py'` is run after this change, the Backend Codebase shall return only files whose remaining Chinese is contained within string literals owned by issues #2/#3/#4/#5/#6.
4. When a docstring is translated, the Translator shall preserve Google-style docstring shape (`Args:`, `Returns:`, `Raises:`, `Yields:` sections) per `dev-guidelines.md`.
### Requirement 2: Preservation of Code Behavior
**Objective:** As a maintainer, I want the translation to be comments-and-docstrings-only, so that runtime behavior is provably unchanged.
#### Acceptance Criteria
1. The Translator shall not modify any executable Python statement (assignments, function calls, control flow, decorators, imports).
2. The Translator shall not modify any Python string literal (single-, double-, triple-quoted, f-string, raw, byte) regardless of whether it contains Chinese characters.
3. The Translator shall not rename any symbol (variable, function, class, module, parameter).
4. When `uv run python -m pytest backend/scripts/test_profile_format.py` is run after the change, the Backend Codebase shall exit with status 0.
5. If a diff line touches any non-comment, non-docstring code, the Translator shall reject that diff hunk and revise.
### Requirement 3: Comment Quality Hygiene
**Objective:** As a maintainer, I want translated comments to add value, so that the codebase remains easy to read after the migration.
#### Acceptance Criteria
1. When a Chinese comment merely restates the immediately following code (e.g. `# 初始化客户端` above `client = Client()`), the Translator shall delete the comment rather than translate it.
2. When a Chinese comment captures non-obvious *why* (constraints, workarounds, invariants), the Translator shall translate it to a faithful English equivalent.
3. The Translator shall preserve any `TODO:` / `FIXME:` marker and any embedded ticket reference (e.g. `#1234`, `PROJ-456`) verbatim within the translated comment.
4. The Translator shall not introduce new comments that did not exist (or had no Chinese equivalent) in the original source.
### Requirement 4: Style and Format Compliance
**Objective:** As a maintainer, I want the translated output to comply with project style rules, so that no follow-up cleanup PR is needed.
#### Acceptance Criteria
1. The Translator shall keep all translated docstrings and comments at or below 120 characters per line.
2. The Translator shall not introduce trailing whitespace on any line.
3. The Translator shall preserve the original indentation (tabs/spaces) of every comment and docstring.
4. The Translator shall use double quotes for any docstring it rewrites, matching the existing Python convention in the file.
5. Where a file already uses 4-space indentation, the Translator shall preserve that indentation.
### Requirement 5: Discovery and Verification Workflow
**Objective:** As a reviewer, I want a reproducible discovery and verification workflow, so that I can confirm coverage and absence of regressions in CI or locally.
#### Acceptance Criteria
1. The Translator shall enumerate candidate files using `grep -rln '[一-鿿]' backend/ --include='*.py'` before beginning work.
2. The Translator shall re-run the same `grep` after each batch and confirm the residual hits are limited to string-literal Chinese owned by adjacent tickets (#2/#3/#4/#5/#6).
3. When the residual `grep` hits include any non-string-literal Chinese, the Translator shall classify those hits as in-scope and continue translation until they are gone.
4. The Translator shall verify that `git diff --stat` only reports changes inside the in-scope file paths listed in Requirement 1.
### Requirement 6: Tracking and Branching
**Objective:** As a release manager, I want the work tracked against ticket #7 on a dedicated branch, so that the PR remains scoped and traceable.
#### Acceptance Criteria
1. The Translator shall produce changes on a branch named `docs/i18n-7-translate-backend-comments`.
2. The Translator shall reference issue `salestech-group/MiroFish#7` in commit messages or PR description.
3. When committing, the Translator shall use Conventional Commits with type `docs` and scope `i18n` (e.g. `docs(i18n): translate chinese docstrings/comments in backend/<area>`).
4. The Translator shall not include unrelated changes (e.g. dependency bumps, config changes, refactors) in the resulting PR.

80
.kiro/specs/i18n-translate-backend-comments/research.md Normal file
View File

@ -0,0 +1,80 @@
# Research & Design Decisions — `i18n-translate-backend-comments`
## Summary
- **Feature**: `i18n-translate-backend-comments`
- **Discovery Scope**: Simple Addition (mechanical translation, no architectural change)
- **Key Findings**:
- 37 in-scope `backend/` Python files contain Chinese characters in docstrings or `#` comments. The full list is in `gap-analysis.md`.
- Existing docstrings mix English-shape Google-style keys (`Args:`/`Returns:`) with Chinese descriptions, and a smaller subset uses Chinese keys (`参数:`/`返回:`/`异常:`). Both patterns must converge to canonical English Google-style.
- Several `tests/test_locale*.py` files contain Chinese only inside string literals (intentional test data) and are out of scope by the ticket's enumerated paths.
## Research Log
### Discovery scan: where is Chinese in `backend/`?
- **Context**: Need a deterministic enumeration of files to translate.
- **Sources Consulted**: `grep`/Python-driven scan against `backend/**/*.py`.
- **Findings**:
- 37 in-app files (under `backend/app/`, `backend/run.py`, `backend/scripts/`).
- 2 additional test files in `backend/tests/` whose Chinese is only in string literals; not in ticket scope.
- `.venv/` matches are noise and excluded.
- **Implications**: The ticket-listed paths are exhaustive; no unexpected location. Order of traversal can be alphabetical within package groups.
### Disambiguation: docstring vs string literal
- **Context**: A triple-quoted string is a docstring iff it is the first statement of a module, class, or function body. Otherwise it is a value (e.g. a prompt template) owned by adjacent tickets.
- **Sources Consulted**: Python language reference; spot inspection of `services/ontology_generator.py`, `services/report_agent.py`.
- **Findings**:
- In-scope files contain both kinds of triple-quoted strings.
- Translating only the *first-statement* triple-quoted string per scope keeps the change comments-and-docstrings-only.
- **Implications**: Translation pass must visually verify each triple-quoted string is the first statement before rewriting; otherwise leave it alone.
### Google-style docstring conversions
- **Context**: `dev-guidelines.md` requires Google-style docstrings; existing Chinese docstrings sometimes use Chinese keys.
- **Findings**: The following key map applies:
- `参数:` → `Args:`
- `返回:` → `Returns:`
- `异常:` → `Raises:`
- `产生:` / `生成:` → `Yields:`
- `示例:` → `Example:` (or `Examples:`)
- `注意:` / `备注:` → `Note:` (or `Notes:`)
- **Implications**: Document this mapping in design.md so the implementation pass is mechanical.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Manual file-by-file pass | Walk in alphabetical order, package-grouped commits | Predictable, easy to review per package | Human time required | Selected approach |
| Multi-PR per package | One PR per backend package | Smaller diffs to review | Higher overhead, more PR churn | Allowed by ticket but not required |
| Tooling-assisted bulk script | LLM-driven find-and-replace tool | Reusable | Risk of touching string literals; tool itself becomes a deliverable | Out of proportion |
## Design Decisions
### Decision: Single-pass, package-grouped commits, single PR
- **Context**: 37 files, mechanical change, ticket allows either single or split PRs.
- **Alternatives Considered**:
1. Multi-PR per package — more granular review but higher overhead.
2. Tooling-assisted bulk script — overkill for one ticket.
- **Selected Approach**: Single PR with one or more commits, grouped by package (`models/`, `utils/`, `services/`, `api/`, `scripts/`, root) so reviewers can read the diff one package at a time.
- **Rationale**: Mechanical change with low risk; ticket explicitly allows it; reduces PR overhead; `/done` produces one PR per branch by default.
- **Trade-offs**: One large PR, but partitioned by commit. Reviewer can use commit history to navigate.
- **Follow-up**: After each package commit, re-run residual `grep` and `pytest` to maintain the invariant.
### Decision: First-statement disambiguation rule
- **Context**: Distinguish docstrings (in scope) from value strings (out of scope).
- **Selected Approach**: A triple-quoted string is treated as a docstring (in scope) only if it is the first statement of a module / class / function body. All other triple-quoted strings are values (out of scope).
- **Rationale**: Matches Python's own definition; keeps boundary with adjacent tickets unambiguous.
### Decision: Drop comments that restate code
- **Context**: R3 requires deletion of comments whose translated form would merely paraphrase the next line.
- **Selected Approach**: Apply a one-line heuristic: if the translated comment would be a verb phrase that mirrors the immediately following executable line, delete the comment instead of writing it.
- **Rationale**: Aligns with project rule "comment the why, not the what".
## Risks & Mitigations
- **Risk**: Accidental edit to a string literal (would belong to ticket #2/#3/#4/#5/#6) — **Mitigation**: After each package commit, run `git diff --stat` and a per-file diff sanity check; verify only `#` lines and docstring lines change.
- **Risk**: Tests failing because a string-shape changed — **Mitigation**: Run `uv run python -m pytest backend/scripts/test_profile_format.py` after each commit.
- **Risk**: Line length violations after English expansion — **Mitigation**: Reflow long English at <= 120 chars within the docstring/comment only; never reflow code.
## References
- `dev-guidelines.md` — repo-level coding standards, Google-style docstring requirement.
- `.claude/rules/commits.md` — Conventional Commits standard for the commit message.
- Issue #7 — salestech-group/MiroFish: source ticket.
- Issues #2/#3/#4/#5/#6 — adjacent i18n tickets that own the string-literal Chinese.

92
.kiro/specs/i18n-translate-backend-comments/scan_chinese.py Normal file
View File

@ -0,0 +1,92 @@
#!/usr/bin/env python3
"""AST-aware classifier of Chinese characters in a Python source file.
Usage::
python3 .kiro/specs/i18n-translate-backend-comments/scan_chinese.py <path>
Classifies every line containing CJK Unified Ideographs (U+4E00..U+9FFF)
into one of three buckets:
* ``DOCSTRING`` — line lies within a module/class/function docstring (in
scope for ticket #7).
* ``COMMENT`` — line contains a ``#`` and is not inside a docstring or
a string literal span (in scope for ticket #7).
* ``STRING`` — line is part of a string literal value (out of scope —
owned by sibling tickets #2/#3/#4/#5/#6).
Exit code is the count of in-scope hits (DOCSTRING + COMMENT). Stdout
lists each in-scope hit as ``<line> <bucket>: <content>`` so callers can
inspect them.
"""
from __future__ import annotations
import ast
import pathlib
import re
import sys
CJK_RE = re.compile(r"[一-鿿]")
def classify(path: pathlib.Path) -> int:
text = path.read_text(encoding="utf-8")
lines = text.split("\n")
tree = ast.parse(text)
docstring_lines: set[int] = set()
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, ast.Module)):
ds = ast.get_docstring(node, clean=False)
if ds is None:
continue
body = node.body
if not body or not isinstance(body[0], ast.Expr):
continue
const = body[0].value
if isinstance(const, ast.Constant) and isinstance(const.value, str):
start = const.lineno
end = getattr(const, "end_lineno", start)
for ln in range(start, end + 1):
docstring_lines.add(ln)
string_value_lines: set[int] = set()
for node in ast.walk(tree):
if isinstance(node, ast.Constant) and isinstance(node.value, str):
start = node.lineno
end = getattr(node, "end_lineno", start)
for ln in range(start, end + 1):
string_value_lines.add(ln)
in_scope_count = 0
for i, line in enumerate(lines, start=1):
if not CJK_RE.search(line):
continue
if i in docstring_lines:
print(f"{i:5d} DOCSTRING: {line.rstrip()[:120]}")
in_scope_count += 1
elif i in string_value_lines:
# Out of scope: owned by sibling tickets.
pass
elif "#" in line:
print(f"{i:5d} COMMENT : {line.rstrip()[:120]}")
in_scope_count += 1
# else: unclassified — treat as out of scope (STRING value spanning).
return in_scope_count
def main(argv: list[str]) -> int:
if len(argv) < 2:
print("usage: scan_chinese.py <path>", file=sys.stderr)
return 2
path = pathlib.Path(argv[1])
in_scope = classify(path)
print(f"---", file=sys.stderr)
print(f"in-scope CJK hits in {path}: {in_scope}", file=sys.stderr)
return 0 if in_scope == 0 else 1
if __name__ == "__main__":
raise SystemExit(main(sys.argv))

24
.kiro/specs/i18n-translate-backend-comments/spec.json Normal file
View File

@ -0,0 +1,24 @@
{
"feature_name": "i18n-translate-backend-comments",
"created_at": "2026-05-07T14:24:17Z",
"updated_at": "2026-05-07T14:26:00Z",
"language": "en",
"phase": "tasks-generated",
"ticket": 7,
"ticket_url": "https://github.com/salestech-group/MiroFish/issues/7",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}

97
.kiro/specs/i18n-translate-backend-comments/tasks.md Normal file
View File

@ -0,0 +1,97 @@
# Implementation Plan
## Foundation
- [x] 1. Establish baseline and working branch
- [x] 1.1 Create translation working branch and capture baseline state
- Create branch `docs/i18n-7-translate-backend-comments` from `main`.
- Capture the baseline residual hits by running the discovery scan (the regex `[一-鿿]` against `backend/**/*.py`, excluding `.venv`); record the file list as the work queue.
- Run `cd backend && uv run python -m pytest scripts/test_profile_format.py` and confirm a green baseline before any edits.
- Observable: a fresh branch exists, the baseline file list of 37 in-scope files is captured, and the baseline pytest run passes.
- _Requirements: 5.1, 6.1_
## Core — Per-Package Translation
- [x] 2. Translate Chinese docstrings and inline comments per package
- [x] 2.1 (P) Translate `backend/app/models/`
- Translate Chinese module/class/function docstrings and `#` comments in `backend/app/models/__init__.py`, `backend/app/models/project.py`, and `backend/app/models/task.py`.
- Apply the docstring-vs-value disambiguation rule (first-statement only) so that no string literal is touched.
- Apply the Google-style key map (`参数:` → `Args:`, `返回:` → `Returns:`, `异常:` → `Raises:`, `产生:`/`生成:` → `Yields:`, `示例:` → `Examples:`, `注意:`/`备注:` → `Note:`).
- Drop comments that merely restate the next executable line; preserve `TODO:`/`FIXME:` and any embedded ticket reference verbatim.
- Re-run the residual scan and confirm `backend/app/models/` no longer has Chinese in non-string-literal positions.
- Re-run `cd backend && uv run python -m pytest scripts/test_profile_format.py` and confirm exit 0.
- Observable: zero non-string-literal Chinese remains in `backend/app/models/*.py`, and the test command exits 0.
- _Requirements: 1.1, 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 4.5_
- _Boundary: backend/app/models/_
- [x] 2.2 (P) Translate `backend/app/utils/`
- Translate Chinese docstrings and `#` comments in `backend/app/utils/__init__.py`, `file_parser.py`, `llm_client.py`, `locale.py`, `logger.py`, `retry.py`, and `zep_paging.py`.
- Be especially careful with `locale.py` and `logger.py`: they intentionally route Chinese strings through their value paths; only docstrings and `#` comments are in scope.
- Apply Rules 1–5 from `design.md` (disambiguation, key map, comment hygiene, style, preservation).
- Re-run the residual scan and confirm `backend/app/utils/` no longer has Chinese in non-string-literal positions.
- Re-run the pytest command and confirm exit 0.
- Observable: zero non-string-literal Chinese remains in `backend/app/utils/*.py`, and the test command exits 0.
- _Requirements: 1.1, 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 4.5_
- _Boundary: backend/app/utils/_
- [x] 2.3 (P) Translate `backend/app/services/` — complete (all 12 files; finished in this installment)
- Translate Chinese docstrings and `#` comments across all 12 service files: `__init__.py`, `graph_builder.py`, `ontology_generator.py`, `oasis_profile_generator.py`, `report_agent.py`, `simulation_config_generator.py`, `simulation_ipc.py`, `simulation_manager.py`, `simulation_runner.py`, `text_processor.py`, `zep_entity_reader.py`, `zep_graph_memory_updater.py`, `zep_tools.py`.
- Treat all triple-quoted prompt templates and value strings as out of scope (owned by issues #2/#3/#4/#5/#6) — only the first-statement docstrings of modules/classes/functions are in scope.
- Apply Rules 1–5 from `design.md`.
- Re-run the residual scan and confirm `backend/app/services/` no longer has Chinese in non-string-literal positions.
- Re-run the pytest command and confirm exit 0.
- Observable: zero non-string-literal Chinese remains in `backend/app/services/*.py`, and the test command exits 0.
- _Requirements: 1.1, 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 4.5_
- _Boundary: backend/app/services/_
- [x] 2.4 (P) Translate `backend/app/api/` — complete (all 4 files; finished in this installment)
- Translate Chinese docstrings and `#` comments in `__init__.py`, `graph.py`, `report.py`, `simulation.py`.
- Treat any user-facing string-literal Chinese in API responses as out of scope (owned by issue #6).
- Apply Rules 1–5 from `design.md`.
- Re-run the residual scan and confirm `backend/app/api/` no longer has Chinese in non-string-literal positions.
- Re-run the pytest command and confirm exit 0.
- Observable: zero non-string-literal Chinese remains in `backend/app/api/*.py`, and the test command exits 0.
- _Requirements: 1.1, 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 4.5_
- _Boundary: backend/app/api/_
- [x] 2.5 (P) Translate `backend/scripts/` — complete (all 5 files; finished in this installment)
- Translate Chinese docstrings and `#` comments in `action_logger.py`, `run_parallel_simulation.py`, `run_reddit_simulation.py`, `run_twitter_simulation.py`, `test_profile_format.py`.
- Apply Rules 1–5 from `design.md`.
- Be especially careful with `test_profile_format.py`: any Chinese in test data string literals is out of scope; only docstrings and `#` comments are in scope.
- Re-run the residual scan and confirm `backend/scripts/` no longer has Chinese in non-string-literal positions.
- Re-run the pytest command and confirm exit 0.
- Observable: zero non-string-literal Chinese remains in `backend/scripts/*.py`, and the test command exits 0.
- _Requirements: 1.1, 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 4.5_
- _Boundary: backend/scripts/_
- [x] 2.6 (P) Translate root backend files
- Translate Chinese docstrings and `#` comments in `backend/app/__init__.py`, `backend/app/config.py`, and `backend/run.py`.
- Apply Rules 1–5 from `design.md`.
- Be especially careful with `backend/app/config.py`: any Chinese in default-value string literals is out of scope; only docstrings and `#` comments are in scope.
- Re-run the residual scan and confirm these three files no longer have Chinese in non-string-literal positions.
- Re-run the pytest command and confirm exit 0.
- Observable: zero non-string-literal Chinese remains in `backend/app/__init__.py`, `backend/app/config.py`, and `backend/run.py`, and the test command exits 0.
- _Requirements: 1.1, 1.2, 1.4, 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 4.1, 4.2, 4.3, 4.4, 4.5_
- _Boundary: backend/app (root), backend/run.py_
## Validation
- [x] 3. Final verification and PR preparation
- [x] 3.1 Run the final verification gate — scanner + py_compile pass on all 12 newly-translated files; CJK guard baseline updated (backend/app: 2792 → 307); pytest blocked by pre-existing env issues, see HANDOFF.md
- Run the residual scan one more time and confirm the only remaining hits are files where the Chinese is in string literals owned by issues #2/#3/#4/#5/#6, plus the intentional Chinese in `backend/tests/test_locale*.py`.
- Run `cd backend && uv run python -m pytest scripts/test_profile_format.py` and confirm exit 0.
- Run `git diff --stat origin/main...HEAD` and confirm only in-scope file paths under `backend/app/`, `backend/run.py`, and `backend/scripts/` are listed.
- Spot-check three random changed files with `git diff <path>` and confirm only `#` lines and docstring lines changed (no executable lines, no string-literal lines).
- Observable: residual scan, pytest, diff scope, and spot diff all pass.
- _Depends: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6_
- _Requirements: 1.3, 2.5, 5.1, 5.2, 5.3, 5.4, 6.4_
- [ ] 3.2 Open PR and reference ticket #7
- Use `/done` to commit any remaining changes per Conventional Commits with type `docs` and scope `i18n` (e.g. `docs(i18n): translate chinese docstrings/comments in backend/<area>`), push the branch, and open a PR.
- The PR body must include `Closes #7` and reference the spec at `.kiro/specs/i18n-translate-backend-comments/`.
- Verify the PR contains no unrelated changes (no dependency bumps, no config changes, no refactors).
- Observable: a PR exists on GitHub from `docs/i18n-7-translate-backend-comments` to `main` that closes #7 and contains only docstring/comment translation diffs.
- _Depends: 3.1_
- _Requirements: 6.1, 6.2, 6.3, 6.4_

11
CLAUDE.md
View File

@ -69,8 +69,15 @@ LLM_MODEL_NAME # Default: qwen-plus
NEO4J_URI # Default: bolt://localhost:7687
NEO4J_USER # Default: neo4j
NEO4J_PASSWORD # Default: mirofish123 (override in real env)
EMBEDDING_MODEL # Default: text-embedding-3-small
# Override for non-OpenAI providers (e.g. Gemini: text-embedding-004)
EMBEDDING_MODEL # Default: text-embedding-3-small (OpenAI)
# Other supported configurations:
# • Gemini: text-embedding-004
# • Ollama: mxbai-embed-large
# (also set EMBEDDING_BASE_URL / EMBEDDING_API_KEY;
# see .env.example for the full snippet)
# Constraint: model must produce 1024-dim vectors to match
# Graphiti's default EMBEDDING_DIM. 768-dim models such as
# nomic-embed-text are not supported.
# Optional — Accelerated LLM (omit entirely if not used)
LLM_BOOST_API_KEY

20
README-EN.md
View File

@ -4,8 +4,6 @@
<a href="https://trendshift.io/repositories/16144" target="_blank"><img src="https://trendshift.io/api/badge/repositories/16144" alt="666ghj%2FMiroFish | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
简洁通用的群体智能引擎，预测万物
</br>
<em>A Simple and Universal Swarm Intelligence Engine, Predicting Anything</em>
<a href="https://www.shanda.com/" target="_blank"><img src="./static/image/shanda_logo.png" alt="666ghj%2MiroFish | Shanda" height="40"/></a>
@ -49,16 +47,16 @@ Welcome to visit our online demo environment and experience a prediction simulat
<div align="center">
<table>
<tr>
<td><img src="./static/image/Screenshot/运行截图1.png" alt="Screenshot 1" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图2.png" alt="Screenshot 2" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot1.png" alt="Screenshot 1" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot2.png" alt="Screenshot 2" width="100%"/></td>
</tr>
<tr>
<td><img src="./static/image/Screenshot/运行截图3.png" alt="Screenshot 3" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图4.png" alt="Screenshot 4" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot3.png" alt="Screenshot 3" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot4.png" alt="Screenshot 4" width="100%"/></td>
</tr>
<tr>
<td><img src="./static/image/Screenshot/运行截图5.png" alt="Screenshot 5" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图6.png" alt="Screenshot 6" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot5.png" alt="Screenshot 5" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot6.png" alt="Screenshot 6" width="100%"/></td>
</tr>
</table>
</div>
@ -68,7 +66,7 @@ Welcome to visit our online demo environment and experience a prediction simulat
### 1. Wuhan University Public Opinion Simulation + MiroFish Project Introduction
<div align="center">
<a href="https://www.bilibili.com/video/BV1VYBsBHEMY/" target="_blank"><img src="./static/image/武大模拟演示封面.png" alt="MiroFish Demo Video" width="75%"/></a>
<a href="https://www.bilibili.com/video/BV1VYBsBHEMY/" target="_blank"><img src="./static/image/wuhan-university-simulation-cover.png" alt="MiroFish Demo Video" width="75%"/></a>
Click the image to watch the complete demo video for prediction using BettaFish-generated "Wuhan University Public Opinion Report"
</div>
@ -76,7 +74,7 @@ Click the image to watch the complete demo video for prediction using BettaFish-
### 2. Dream of the Red Chamber Lost Ending Simulation
<div align="center">
<a href="https://www.bilibili.com/video/BV1cPk3BBExq" target="_blank"><img src="./static/image/红楼梦模拟推演封面.jpg" alt="MiroFish Demo Video" width="75%"/></a>
<a href="https://www.bilibili.com/video/BV1cPk3BBExq" target="_blank"><img src="./static/image/dream-of-the-red-chamber-simulation-cover.jpg" alt="MiroFish Demo Video" width="75%"/></a>
Click the image to watch MiroFish's deep prediction of the lost ending based on hundreds of thousands of words from the first 80 chapters of "Dream of the Red Chamber"
</div>
@ -217,7 +215,7 @@ npm run frontend # Start frontend only
## 📬 Join the Conversation
<div align="center">
<img src="./static/image/QQ群.png" alt="QQ Group" width="60%"/>
<img src="./static/image/qq-group.png" alt="QQ Group" width="60%"/>
</div>
&nbsp;

18
README-ZH.md
View File

@ -49,16 +49,16 @@ MiroFish 致力于打造映射现实的群体智能镜像，通过捕捉个体
<div align="center">
<table>
<tr>
<td><img src="./static/image/Screenshot/运行截图1.png" alt="截图1" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图2.png" alt="截图2" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot1.png" alt="截图1" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot2.png" alt="截图2" width="100%"/></td>
</tr>
<tr>
<td><img src="./static/image/Screenshot/运行截图3.png" alt="截图3" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图4.png" alt="截图4" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot3.png" alt="截图3" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot4.png" alt="截图4" width="100%"/></td>
</tr>
<tr>
<td><img src="./static/image/Screenshot/运行截图5.png" alt="截图5" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图6.png" alt="截图6" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot5.png" alt="截图5" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot6.png" alt="截图6" width="100%"/></td>
</tr>
</table>
</div>
@ -68,7 +68,7 @@ MiroFish 致力于打造映射现实的群体智能镜像，通过捕捉个体
### 1. 武汉大学舆情推演预测 + MiroFish项目讲解
<div align="center">
<a href="https://www.bilibili.com/video/BV1VYBsBHEMY/" target="_blank"><img src="./static/image/武大模拟演示封面.png" alt="MiroFish Demo Video" width="75%"/></a>
<a href="https://www.bilibili.com/video/BV1VYBsBHEMY/" target="_blank"><img src="./static/image/wuhan-university-simulation-cover.png" alt="MiroFish Demo Video" width="75%"/></a>
点击图片查看使用微舆BettaFish生成的《武大舆情报告》进行预测的完整演示视频
</div>
@ -76,7 +76,7 @@ MiroFish 致力于打造映射现实的群体智能镜像，通过捕捉个体
### 2. 《红楼梦》失传结局推演预测
<div align="center">
<a href="https://www.bilibili.com/video/BV1cPk3BBExq" target="_blank"><img src="./static/image/红楼梦模拟推演封面.jpg" alt="MiroFish Demo Video" width="75%"/></a>
<a href="https://www.bilibili.com/video/BV1cPk3BBExq" target="_blank"><img src="./static/image/dream-of-the-red-chamber-simulation-cover.jpg" alt="MiroFish Demo Video" width="75%"/></a>
点击图片查看基于《红楼梦》前80回数十万字，MiroFish深度预测失传结局
</div>
@ -217,7 +217,7 @@ npm run frontend # 仅启动前端
## 📬 更多交流
<div align="center">
<img src="./static/image/QQ群.png" alt="QQ交流群" width="60%"/>
<img src="./static/image/qq-group.png" alt="QQ交流群" width="60%"/>
</div>
&nbsp;

39
README.md
View File

@ -4,8 +4,6 @@
<a href="https://trendshift.io/repositories/16144" target="_blank"><img src="https://trendshift.io/api/badge/repositories/16144" alt="666ghj%2FMiroFish | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
简洁通用的群体智能引擎，预测万物
</br>
<em>A Simple and Universal Swarm Intelligence Engine, Predicting Anything</em>
<a href="https://www.shanda.com/" target="_blank"><img src="./static/image/shanda_logo.png" alt="666ghj%2MiroFish | Shanda" height="40"/></a>
@ -49,16 +47,16 @@ Welcome to visit our online demo environment and experience a prediction simulat
<div align="center">
<table>
<tr>
<td><img src="./static/image/Screenshot/运行截图1.png" alt="Screenshot 1" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图2.png" alt="Screenshot 2" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot1.png" alt="Screenshot 1" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot2.png" alt="Screenshot 2" width="100%"/></td>
</tr>
<tr>
<td><img src="./static/image/Screenshot/运行截图3.png" alt="Screenshot 3" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图4.png" alt="Screenshot 4" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot3.png" alt="Screenshot 3" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot4.png" alt="Screenshot 4" width="100%"/></td>
</tr>
<tr>
<td><img src="./static/image/Screenshot/运行截图5.png" alt="Screenshot 5" width="100%"/></td>
<td><img src="./static/image/Screenshot/运行截图6.png" alt="Screenshot 6" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot5.png" alt="Screenshot 5" width="100%"/></td>
<td><img src="./static/image/Screenshot/screenshot6.png" alt="Screenshot 6" width="100%"/></td>
</tr>
</table>
</div>
@ -68,7 +66,7 @@ Welcome to visit our online demo environment and experience a prediction simulat
### 1. Wuhan University Public Opinion Simulation + MiroFish Project Introduction
<div align="center">
<a href="https://www.bilibili.com/video/BV1VYBsBHEMY/" target="_blank"><img src="./static/image/武大模拟演示封面.png" alt="MiroFish Demo Video" width="75%"/></a>
<a href="https://www.bilibili.com/video/BV1VYBsBHEMY/" target="_blank"><img src="./static/image/wuhan-university-simulation-cover.png" alt="MiroFish Demo Video" width="75%"/></a>
Click the image to watch the complete demo video for prediction using BettaFish-generated "Wuhan University Public Opinion Report"
</div>
@ -76,7 +74,7 @@ Click the image to watch the complete demo video for prediction using BettaFish-
### 2. Dream of the Red Chamber Lost Ending Simulation
<div align="center">
<a href="https://www.bilibili.com/video/BV1cPk3BBExq" target="_blank"><img src="./static/image/红楼梦模拟推演封面.jpg" alt="MiroFish Demo Video" width="75%"/></a>
<a href="https://www.bilibili.com/video/BV1cPk3BBExq" target="_blank"><img src="./static/image/dream-of-the-red-chamber-simulation-cover.jpg" alt="MiroFish Demo Video" width="75%"/></a>
Click the image to watch MiroFish's deep prediction of the lost ending based on hundreds of thousands of words from the first 80 chapters of "Dream of the Red Chamber"
</div>
@ -164,6 +162,25 @@ NEO4J_PASSWORD=your_neo4j_password
# Embedding model (uncomment if using a non-OpenAI provider, e.g. Gemini)
# EMBEDDING_MODEL=gemini-embedding-001
# Embedding model via local Ollama (free, no API key, OpenAI-compatible endpoint).
# Pre-requisite: `ollama pull mxbai-embed-large` (1024-dim, matches Graphiti).
# In Docker, host.docker.internal:11434 reaches the host daemon; in host mode
# (`npm run dev`) substitute http://localhost:11434/v1.
# EMBEDDING_BASE_URL=http://host.docker.internal:11434/v1
# EMBEDDING_API_KEY=ollama
# EMBEDDING_MODEL=mxbai-embed-large
```
**Embedder smoke test (recommended before the first graph build):**
```bash
curl -s "$EMBEDDING_BASE_URL/embeddings" \
-H "Authorization: Bearer $EMBEDDING_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"'"$EMBEDDING_MODEL"'","input":"ping"}' \
| jq '.data[0].embedding | length'
# Expected output: 1024
```
**Optional — Accelerated LLM Configuration:**
@ -217,7 +234,7 @@ npm run frontend # Start frontend only
## 📬 Join the Conversation
<div align="center">
<img src="./static/image/QQ群.png" alt="QQ Group" width="60%"/>
<img src="./static/image/qq-group.png" alt="QQ Group" width="60%"/>
</div>
&nbsp;

53
backend/app/__init__.py
View File

@ -1,12 +1,10 @@
"""
MiroFish Backend - Flask应用工厂
"""
"""MiroFish backend Flask application factory."""
import os
import warnings
# 抑制 multiprocessing resource_tracker 的警告（来自第三方库如 transformers）
# 需要在所有其他导入之前设置
# Silence multiprocessing.resource_tracker warnings emitted by some third-party
# libraries (e.g. transformers); must run before those modules are imported.
warnings.filterwarnings("ignore", message=".*resource_tracker.*")
from flask import Flask, request
@ -18,62 +16,65 @@ from .utils.locale import t
def create_app(config_class=Config):
"""Flask应用工厂函数"""
"""Flask application factory."""
app = Flask(__name__)
app.config.from_object(config_class)
# 设置JSON编码：确保中文直接显示（而不是 \uXXXX 格式）
# Flask >= 2.3 使用 app.json.ensure_ascii，旧版本使用 JSON_AS_ASCII 配置
# Configure JSON encoding so non-ASCII characters render literally
# rather than as \uXXXX escape sequences. Flask >= 2.3 exposes
# ``app.json.ensure_ascii``; older versions use ``JSON_AS_ASCII``.
if hasattr(app, 'json') and hasattr(app.json, 'ensure_ascii'):
app.json.ensure_ascii = False
# 设置日志
# Configure logging.
logger = setup_logger('mirofish')
# 只在 reloader 子进程中打印启动信息（避免 debug 模式下打印两次）
# Only print startup banners in the reloader child process to avoid
# double-printing in debug mode.
is_reloader_process = os.environ.get('WERKZEUG_RUN_MAIN') == 'true'
debug_mode = app.config.get('DEBUG', False)
should_log_startup = not debug_mode or is_reloader_process
if should_log_startup:
logger.info("=" * 50)
logger.info(t("log.bootstrap.m001"))
logger.info("=" * 50)
# 启用CORS
# Enable CORS.
CORS(app, resources={r"/api/*": {"origins": "*"}})
# 注册模拟进程清理函数（确保服务器关闭时终止所有模拟进程）
# Register simulation-process cleanup so all child processes are torn down
# when the Flask server shuts down.
from .services.simulation_runner import SimulationRunner
SimulationRunner.register_cleanup()
if should_log_startup:
logger.info(t("log.bootstrap.m002"))
# 请求日志中间件
# Request-logging middleware.
@app.before_request
def log_request():
logger = get_logger('mirofish.request')
logger.debug(t("log.bootstrap.m003", request=request.method, request_2=request.path))
if request.content_type and 'json' in request.content_type:
logger.debug(t("log.bootstrap.m004", request=request.get_json(silent=True)))
@app.after_request
def log_response(response):
logger = get_logger('mirofish.request')
logger.debug(t("log.bootstrap.m005", response=response.status_code))
return response
# 注册蓝图
# Register API blueprints.
from .api import graph_bp, simulation_bp, report_bp
app.register_blueprint(graph_bp, url_prefix='/api/graph')
app.register_blueprint(simulation_bp, url_prefix='/api/simulation')
app.register_blueprint(report_bp, url_prefix='/api/report')
# 健康检查
# Health-check endpoint.
@app.route('/health')
def health():
return {'status': 'ok', 'service': 'MiroFish Backend'}
# On startup: recover any projects stuck in graph_building (task was killed by restart)
if should_log_startup:
_recover_stuck_projects()

4
backend/app/api/__init__.py
View File

@ -1,6 +1,4 @@
"""
API路由模块
"""
"""API blueprints package."""
from flask import Blueprint

189
backend/app/api/graph.py
View File

@ -1,6 +1,7 @@
"""
图谱相关API路由
采用项目上下文机制，服务端持久化状态
Graph-related API routes.
Uses a project context mechanism with server-side state persistence.
"""
import os
@ -26,25 +27,22 @@ _graph_data_cache: dict = {} # graph_id -> {"data": ..., "ts": float}
_graph_refresh_locks: dict = {} # graph_id -> threading.Lock (one refresh at a time)
_GRAPH_CACHE_TTL = 300 # seconds before triggering a background refresh
# 获取日志器
logger = get_logger('mirofish.api')
def allowed_file(filename: str) -> bool:
"""检查文件扩展名是否允许"""
"""Return True if the file extension is in the allowed list."""
if not filename or '.' not in filename:
return False
ext = os.path.splitext(filename)[1].lower().lstrip('.')
return ext in Config.ALLOWED_EXTENSIONS
# ============== 项目管理接口 ==============
# ============== Project management endpoints ==============
@graph_bp.route('/project/<project_id>', methods=['GET'])
def get_project(project_id: str):
"""
获取项目详情
"""
"""Get project details."""
project = ProjectManager.get_project(project_id)
if not project:
@ -61,9 +59,7 @@ def get_project(project_id: str):
@graph_bp.route('/project/list', methods=['GET'])
def list_projects():
"""
列出所有项目
"""
"""List all projects."""
limit = request.args.get('limit', 50, type=int)
projects = ProjectManager.list_projects(limit=limit)
@ -76,9 +72,7 @@ def list_projects():
@graph_bp.route('/project/<project_id>', methods=['DELETE'])
def delete_project(project_id: str):
"""
删除项目
"""
"""Delete a project."""
success = ProjectManager.delete_project(project_id)
if not success:
@ -95,9 +89,7 @@ def delete_project(project_id: str):
@graph_bp.route('/project/<project_id>/reset', methods=['POST'])
def reset_project(project_id: str):
"""
重置项目状态（用于重新构建图谱）
"""
"""Reset project state (used to rebuild the graph from scratch)."""
project = ProjectManager.get_project(project_id)
if not project:
@ -106,7 +98,8 @@ def reset_project(project_id: str):
"error": t("api.error.graph.m004", project_id=project_id)
}), 404
# 重置到本体已生成状态
# Roll back to the "ontology generated" state so the next build can resume
# from the existing ontology rather than re-running ontology generation.
if project.ontology:
project.status = ProjectStatus.ONTOLOGY_GENERATED
else:
@ -124,22 +117,21 @@ def reset_project(project_id: str):
})
# ============== 接口1：上传文件并生成本体 ==============
# ============== Endpoint 1: upload files and generate ontology ==============
@graph_bp.route('/ontology/generate', methods=['POST'])
def generate_ontology():
"""
接口1：上传文件，分析生成本体定义
请求方式：multipart/form-data
参数：
files: 上传的文件（PDF/MD/TXT），可多个
simulation_requirement: 模拟需求描述（必填）
project_name: 项目名称（可选）
additional_context: 额外说明（可选）
返回：
"""Endpoint 1: upload files, analyze them, and generate an ontology definition.
Request format: multipart/form-data.
Args:
files: Uploaded files (PDF/MD/TXT); one or more.
simulation_requirement: Description of the simulation requirement (required).
project_name: Project name (optional).
additional_context: Additional context (optional).
Returns:
{
"success": true,
"data": {
@ -156,8 +148,7 @@ def generate_ontology():
"""
try:
logger.info(t("log.graph_api.m006"))
# 获取参数
simulation_requirement = request.form.get('simulation_requirement', '')
project_name = request.form.get('project_name', 'Unnamed Project')
additional_context = request.form.get('additional_context', '')
@ -171,7 +162,6 @@ def generate_ontology():
"error": t("api.error.graph.m009")
}), 400
# 获取上传的文件
uploaded_files = request.files.getlist('files')
if not uploaded_files or all(not f.filename for f in uploaded_files):
return jsonify({
@ -179,18 +169,17 @@ def generate_ontology():
"error": t("api.error.graph.m010")
}), 400
# 创建项目
project = ProjectManager.create_project(name=project_name)
project.simulation_requirement = simulation_requirement
logger.info(t("log.graph_api.m011", project=project.project_id))
# 保存文件并提取文本
# Persist each uploaded file under the project's directory and pull its
# text out so the ontology generator has plain text to work with.
document_texts = []
all_text = ""
for file in uploaded_files:
if file and file.filename and allowed_file(file.filename):
# 保存文件到项目目录
file_info = ProjectManager.save_file_to_project(
project.project_id,
file,
@ -201,7 +190,6 @@ def generate_ontology():
"size": file_info["size"]
})
# 提取文本
text = FileParser.extract_text(file_info["path"])
text = TextProcessor.preprocess_text(text)
document_texts.append(text)
@ -214,12 +202,10 @@ def generate_ontology():
"error": t("api.error.graph.m012")
}), 400
# 保存提取的文本
project.total_text_length = len(all_text)
ProjectManager.save_extracted_text(project.project_id, all_text)
logger.info(t("log.graph_api.m013", len=len(all_text)))
# 生成本体
logger.info(t("log.graph_api.m014"))
generator = OntologyGenerator()
ontology = generator.generate(
@ -228,7 +214,6 @@ def generate_ontology():
additional_context=additional_context if additional_context else None
)
# 保存本体到项目
entity_count = len(ontology.get("entity_types", []))
edge_count = len(ontology.get("edge_types", []))
logger.info(t("log.graph_api.m015", entity_count=entity_count, edge_count=edge_count))
@ -262,35 +247,33 @@ def generate_ontology():
}), 500
# ============== 接口2：构建图谱 ==============
# ============== Endpoint 2: build graph ==============
@graph_bp.route('/build', methods=['POST'])
def build_graph():
"""
接口2：根据project_id构建图谱
请求（JSON）：
"""Endpoint 2: build the graph for the given project_id.
Request (JSON):
{
"project_id": "proj_xxxx", // 必填，来自接口1
"graph_name": "图谱名称", // 可选
"chunk_size": 500, // 可选，默认500
"chunk_overlap": 50 // 可选，默认50
"project_id": "proj_xxxx", // required, from endpoint 1
"graph_name": "Graph name", // optional
"chunk_size": 500, // optional, default 500
"chunk_overlap": 50 // optional, default 50
}
返回：
Returns:
{
"success": true,
"data": {
"project_id": "proj_xxxx",
"task_id": "task_xxxx",
"message": "图谱构建任务已启动"
"message": "Graph build task started"
}
}
"""
try:
logger.info(t("log.graph_api.m017"))
# 检查配置
errors = []
if not Config.NEO4J_PASSWORD:
errors.append("NEO4J未配置")
@ -301,7 +284,6 @@ def build_graph():
"error": "配置错误: " + "; ".join(errors)
}), 500
# 解析请求
data = request.get_json() or {}
project_id = data.get('project_id')
logger.debug(t("log.graph_api.m019", project_id=project_id))
@ -312,7 +294,6 @@ def build_graph():
"error": t("api.error.graph.m020")
}), 400
# 获取项目
project = ProjectManager.get_project(project_id)
if not project:
return jsonify({
@ -320,8 +301,8 @@ def build_graph():
"error": t("api.error.graph.m021", project_id=project_id)
}), 404
# 检查项目状态
force = data.get('force', False) # 强制重新构建
# If True, abandon any existing build progress and rebuild from scratch.
force = data.get('force', False)
if project.status == ProjectStatus.CREATED:
return jsonify({
@ -336,23 +317,20 @@ def build_graph():
"task_id": project.graph_build_task_id
}), 400
# 如果强制重建，重置状态
# On a forced rebuild, drop any prior build artifacts so we restart cleanly.
if force and project.status in [ProjectStatus.GRAPH_BUILDING, ProjectStatus.FAILED, ProjectStatus.GRAPH_COMPLETED]:
project.status = ProjectStatus.ONTOLOGY_GENERATED
project.graph_id = None
project.graph_build_task_id = None
project.error = None
# 获取配置
graph_name = data.get('graph_name', project.name or 'MiroFish Graph')
chunk_size = data.get('chunk_size', project.chunk_size or Config.DEFAULT_CHUNK_SIZE)
chunk_overlap = data.get('chunk_overlap', project.chunk_overlap or Config.DEFAULT_CHUNK_OVERLAP)
# 更新项目配置
project.chunk_size = chunk_size
project.chunk_overlap = chunk_overlap
# 获取提取的文本
text = ProjectManager.get_extracted_text(project_id)
if not text:
return jsonify({
@ -360,7 +338,6 @@ def build_graph():
"error": t("api.error.graph.m024")
}), 400
# 获取本体
ontology = project.ontology
if not ontology:
return jsonify({
@ -368,31 +345,26 @@ def build_graph():
"error": t("api.error.graph.m025")
}), 400
# 创建异步任务
task_manager = TaskManager()
task_id = task_manager.create_task(f"构建图谱: {graph_name}")
logger.info(t("log.graph_api.m026", task_id=task_id, project_id=project_id))
# 更新项目状态
project.status = ProjectStatus.GRAPH_BUILDING
project.graph_build_task_id = task_id
ProjectManager.save_project(project)
# 启动后台任务
def build_task():
build_logger = get_logger('mirofish.build')
try:
build_logger.info(f"[{task_id}] 开始构建图谱...")
build_logger.info(t("log.graph_api.m027", task_id=task_id))
task_manager.update_task(
task_id,
status=TaskStatus.PROCESSING,
message="初始化图谱构建服务..."
)
# 创建图谱构建服务
builder = GraphBuilderService()
# 分块
task_manager.update_task(
task_id,
message="文本分块中...",
@ -404,30 +376,27 @@ def build_graph():
overlap=chunk_overlap
)
total_chunks = len(chunks)
# 创建图谱
task_manager.update_task(
task_id,
message="创建Zep图谱...",
progress=10
)
graph_id = builder.create_graph(name=graph_name)
# 更新项目的graph_id
project.graph_id = graph_id
ProjectManager.save_project(project)
# 设置本体
task_manager.update_task(
task_id,
message="设置本体定义...",
progress=15
)
builder.set_ontology(graph_id, ontology)
# 添加文本（progress_callback 签名是 (msg, progress_ratio)）
# Add text. The progress_callback signature is (msg, progress_ratio).
def add_progress_callback(msg, progress_ratio):
progress = 15 + int(progress_ratio * 40) # 15% - 55%
progress = 15 + int(progress_ratio * 40) # maps ratio onto 15%-55%
task_manager.update_task(
task_id,
message=msg,
@ -460,7 +429,7 @@ def build_graph():
skip_chunks=skip_chunks,
)
# 等待Zep处理完成（查询每个episode的processed状态）
# Wait for Zep to finish processing (poll each episode's processed flag).
task_manager.update_task(
task_id,
message="等待Zep处理数据...",
@ -468,7 +437,7 @@ def build_graph():
)
def wait_progress_callback(msg, progress_ratio):
progress = 55 + int(progress_ratio * 35) # 55% - 90%
progress = 55 + int(progress_ratio * 35) # maps ratio onto 55%-90%
task_manager.update_task(
task_id,
message=msg,
@ -476,24 +445,27 @@ def build_graph():
)
builder._wait_for_episodes(episode_uuids, wait_progress_callback)
# 获取图谱数据
task_manager.update_task(
task_id,
message="获取图谱数据...",
progress=95
)
graph_data = builder.get_graph_data(graph_id)
# 更新项目状态
project.status = ProjectStatus.GRAPH_COMPLETED
ProjectManager.save_project(project)
node_count = graph_data.get("node_count", 0)
edge_count = graph_data.get("edge_count", 0)
build_logger.info(f"[{task_id}] 图谱构建完成: graph_id={graph_id}, 节点={node_count}, 边={edge_count}")
# 完成
build_logger.info(t(
"log.graph_api.m028",
task_id=task_id,
graph_id=graph_id,
node_count=node_count,
edge_count=edge_count,
))
task_manager.update_task(
task_id,
status=TaskStatus.COMPLETED,
@ -509,8 +481,8 @@ def build_graph():
)
except Exception as e:
# 更新项目状态为失败
build_logger.error(f"[{task_id}] 图谱构建失败: {str(e)}")
# Mark the project as FAILED so the UI can surface the error.
build_logger.error(t("log.graph_api.m029", task_id=task_id, e=str(e)))
build_logger.debug(traceback.format_exc())
project.status = ProjectStatus.FAILED
@ -524,7 +496,6 @@ def build_graph():
error=traceback.format_exc()
)
# 启动后台线程
thread = threading.Thread(target=build_task, daemon=True)
thread.start()
@ -545,13 +516,11 @@ def build_graph():
}), 500
# ============== 任务查询接口 ==============
# ============== Task query endpoints ==============
@graph_bp.route('/task/<task_id>', methods=['GET'])
def get_task(task_id: str):
"""
查询任务状态
"""
"""Query the status of a task."""
task = TaskManager().get_task(task_id)
if not task:
@ -568,9 +537,7 @@ def get_task(task_id: str):
@graph_bp.route('/tasks', methods=['GET'])
def list_tasks():
"""
列出所有任务
"""
"""List all tasks."""
tasks = TaskManager().list_tasks()
return jsonify({
@ -580,7 +547,7 @@ def list_tasks():
})
# ============== 图谱数据接口 ==============
# ============== Graph data endpoints ==============
def _refresh_graph_cache(graph_id: str):
"""Background thread: fetch graph data from Neo4j and update cache."""
@ -607,11 +574,11 @@ def _refresh_graph_cache(graph_id: str):
@graph_bp.route('/data/<graph_id>', methods=['GET'])
def get_graph_data(graph_id: str):
"""
获取图谱数据（节点和边）。
- 有缓存且未过期：直接返回缓存，不调用 Zep
- 有缓存但已过期：立即返回旧缓存，后台异步刷新
- 无缓存：后台线程拉取，返回 202 让前端稍后重试
"""Return graph data (nodes and edges).
- Fresh cache: serve from cache without hitting Zep.
- Stale cache: return the old cache immediately and refresh in the background.
- No cache: kick off a background fetch and return 202 so the frontend retries.
"""
if not Config.NEO4J_PASSWORD:
return jsonify({"success": False, "error": t("api.error.graph.m028")}), 500
@ -639,9 +606,7 @@ def get_graph_data(graph_id: str):
@graph_bp.route('/delete/<graph_id>', methods=['DELETE'])
def delete_graph(graph_id: str):
"""
删除Zep图谱
"""
"""Delete a Zep graph."""
try:
if not Config.NEO4J_PASSWORD:
return jsonify({

278
backend/app/api/report.py
View File

@ -1,6 +1,7 @@
"""
Report API路由
提供模拟报告生成、获取、对话等接口
Report API routes.
Provides endpoints for generating, retrieving, and chatting about simulation reports.
"""
import os
@ -20,30 +21,30 @@ from ..utils.locale import t, get_locale, set_locale
logger = get_logger('mirofish.api.report')
# ============== 报告生成接口 ==============
# ============== Report generation endpoints ==============
@report_bp.route('/generate', methods=['POST'])
def generate_report():
"""
生成模拟分析报告（异步任务）
这是一个耗时操作，接口会立即返回task_id，
使用 GET /api/report/generate/status 查询进度
请求（JSON）：
Generate a simulation analysis report (asynchronous task).
This is a long-running operation. The endpoint returns a task_id immediately;
use GET /api/report/generate/status to poll progress.
Request (JSON):
{
"simulation_id": "sim_xxxx", // 必填，模拟ID
"force_regenerate": false // 可选，强制重新生成
"simulation_id": "sim_xxxx", // required, simulation ID
"force_regenerate": false // optional, force regeneration
}
返回：
Returns:
{
"success": true,
"data": {
"simulation_id": "sim_xxxx",
"task_id": "task_xxxx",
"status": "generating",
"message": "报告生成任务已启动"
"message": "Report generation task started"
}
}
"""
@ -58,8 +59,7 @@ def generate_report():
}), 400
force_regenerate = data.get('force_regenerate', False)
# 获取模拟信息
manager = SimulationManager()
state = manager.get_simulation(simulation_id)
@ -69,7 +69,7 @@ def generate_report():
"error": t('api.simulationNotFound', id=simulation_id)
}), 404
# 检查是否已有报告
# Skip regeneration if a completed report already exists for this simulation.
if not force_regenerate:
existing_report = ReportManager.get_report_by_simulation(simulation_id)
if existing_report and existing_report.status == ReportStatus.COMPLETED:
@ -84,7 +84,6 @@ def generate_report():
}
})
# 获取项目信息
project = ProjectManager.get_project(state.project_id)
if not project:
return jsonify({
@ -106,11 +105,11 @@ def generate_report():
"error": t('api.missingSimRequirement')
}), 400
# 提前生成 report_id，以便立即返回给前端
# Generate report_id eagerly so the frontend can use it immediately
# (before the background task has actually persisted anything).
import uuid
report_id = f"report_{uuid.uuid4().hex[:12]}"
# 创建异步任务
task_manager = TaskManager()
task_id = task_manager.create_task(
task_type="report_generate",
@ -124,7 +123,6 @@ def generate_report():
# Capture locale before spawning background thread
current_locale = get_locale()
# 定义后台任务
def run_generate():
set_locale(current_locale)
try:
@ -134,15 +132,13 @@ def generate_report():
progress=0,
message=t('api.initReportAgent')
)
# 创建Report Agent
agent = ReportAgent(
graph_id=graph_id,
simulation_id=simulation_id,
simulation_requirement=simulation_requirement
)
# 进度回调
def progress_callback(stage, progress, message):
task_manager.update_task(
task_id,
@ -150,13 +146,13 @@ def generate_report():
message=f"[{stage}] {message}"
)
# 生成报告（传入预先生成的 report_id）
# Pass in the pre-generated report_id so the persisted report matches
# the id we already returned to the frontend.
report = agent.generate_report(
progress_callback=progress_callback,
report_id=report_id
)
# 保存报告
ReportManager.save_report(report)
if report.status == ReportStatus.COMPLETED:
@ -174,8 +170,7 @@ def generate_report():
except Exception as e:
logger.error(t("log.report_api.m001", str=str(e)))
task_manager.fail_task(task_id, str(e))
# 启动后台线程
thread = threading.Thread(target=run_generate, daemon=True)
thread.start()
@ -203,15 +198,15 @@ def generate_report():
@report_bp.route('/generate/status', methods=['POST'])
def get_generate_status():
"""
查询报告生成任务进度
请求（JSON）：
Query the progress of a report generation task.
Request (JSON):
{
"task_id": "task_xxxx", // 可选，generate返回的task_id
"simulation_id": "sim_xxxx" // 可选，模拟ID
"task_id": "task_xxxx", // optional, task_id returned by generate
"simulation_id": "sim_xxxx" // optional, simulation ID
}
返回：
Returns:
{
"success": true,
"data": {
@ -228,7 +223,8 @@ def get_generate_status():
task_id = data.get('task_id')
simulation_id = data.get('simulation_id')
# 如果提供了simulation_id，先检查是否已有完成的报告
# If simulation_id is provided, short-circuit when a completed report already exists
# so callers don't have to track a stale task_id after a successful run.
if simulation_id:
existing_report = ReportManager.get_report_by_simulation(simulation_id)
if existing_report and existing_report.status == ReportStatus.COMPLETED:
@ -272,14 +268,14 @@ def get_generate_status():
}), 500
# ============== 报告获取接口 ==============
# ============== Report retrieval endpoints ==============
@report_bp.route('/<report_id>', methods=['GET'])
def get_report(report_id: str):
"""
获取报告详情
返回：
Get report details.
Returns:
{
"success": true,
"data": {
@ -319,9 +315,9 @@ def get_report(report_id: str):
@report_bp.route('/by-simulation/<simulation_id>', methods=['GET'])
def get_report_by_simulation(simulation_id: str):
"""
根据模拟ID获取报告
返回：
Get the report for a given simulation ID.
Returns:
{
"success": true,
"data": {
@ -358,13 +354,13 @@ def get_report_by_simulation(simulation_id: str):
@report_bp.route('/list', methods=['GET'])
def list_reports():
"""
列出所有报告
Query参数：
simulation_id: 按模拟ID过滤（可选）
limit: 返回数量限制（默认50）
返回：
List all reports.
Query parameters:
simulation_id: optional filter by simulation ID.
limit: maximum number of reports to return (default 50).
Returns:
{
"success": true,
"data": [...],
@ -398,9 +394,9 @@ def list_reports():
@report_bp.route('/<report_id>/download', methods=['GET'])
def download_report(report_id: str):
"""
下载报告（Markdown格式）
返回Markdown文件
Download a report as a Markdown file.
Returns the Markdown file as an attachment.
"""
try:
report = ReportManager.get_report(report_id)
@ -414,7 +410,8 @@ def download_report(report_id: str):
md_path = ReportManager._get_report_markdown_path(report_id)
if not os.path.exists(md_path):
# 如果MD文件不存在，生成一个临时文件
# MD file is missing on disk; materialize a temp file from the in-memory content
# so the download still succeeds for older reports that were never persisted.
import tempfile
with tempfile.NamedTemporaryFile(mode='w', suffix='.md', delete=False) as f:
f.write(report.markdown_content)
@ -443,7 +440,7 @@ def download_report(report_id: str):
@report_bp.route('/<report_id>', methods=['DELETE'])
def delete_report(report_id: str):
"""删除报告"""
"""Delete a report."""
try:
success = ReportManager.delete_report(report_id)
@ -467,32 +464,33 @@ def delete_report(report_id: str):
}), 500
# ============== Report Agent对话接口 ==============
# ============== Report Agent chat endpoints ==============
@report_bp.route('/chat', methods=['POST'])
def chat_with_report_agent():
"""
与Report Agent对话
Report Agent可以在对话中自主调用检索工具来回答问题
请求（JSON）：
Chat with the Report Agent.
The Report Agent can autonomously invoke retrieval tools during the conversation
to answer the user's question.
Request (JSON):
{
"simulation_id": "sim_xxxx", // 必填，模拟ID
"message": "请解释一下舆情走向", // 必填，用户消息
"chat_history": [ // 可选，对话历史
"simulation_id": "sim_xxxx", // required, simulation ID
"message": "Explain the sentiment trend", // required, user message
"chat_history": [ // optional, prior turns
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."}
]
}
返回：
Returns:
{
"success": true,
"data": {
"response": "Agent回复...",
"tool_calls": [调用的工具列表],
"sources": [信息来源]
"response": "Agent reply...",
"tool_calls": [list of tools invoked],
"sources": [information sources]
}
}
"""
@ -515,7 +513,6 @@ def chat_with_report_agent():
"error": t('api.requireMessage')
}), 400
# 获取模拟和项目信息
manager = SimulationManager()
state = manager.get_simulation(simulation_id)
@ -540,8 +537,7 @@ def chat_with_report_agent():
}), 400
simulation_requirement = project.simulation_requirement or ""
# 创建Agent并进行对话
agent = ReportAgent(
graph_id=graph_id,
simulation_id=simulation_id,
@ -564,22 +560,22 @@ def chat_with_report_agent():
}), 500
# ============== 报告进度与分章节接口 ==============
# ============== Report progress and section endpoints ==============
@report_bp.route('/<report_id>/progress', methods=['GET'])
def get_report_progress(report_id: str):
"""
获取报告生成进度（实时）
返回：
Get real-time report generation progress.
Returns:
{
"success": true,
"data": {
"status": "generating",
"progress": 45,
"message": "正在生成章节: 关键发现",
"current_section": "关键发现",
"completed_sections": ["执行摘要", "模拟背景"],
"message": "Generating section: Key Findings",
"current_section": "Key Findings",
"completed_sections": ["Executive Summary", "Simulation Background"],
"updated_at": "2025-12-09T..."
}
}
@ -610,11 +606,12 @@ def get_report_progress(report_id: str):
@report_bp.route('/<report_id>/sections', methods=['GET'])
def get_report_sections(report_id: str):
"""
获取已生成的章节列表（分章节输出）
前端可以轮询此接口获取已生成的章节内容，无需等待整个报告完成
返回：
Get the list of sections generated so far (per-section streaming output).
The frontend can poll this endpoint to render sections incrementally,
without waiting for the entire report to finish.
Returns:
{
"success": true,
"data": {
@ -623,7 +620,7 @@ def get_report_sections(report_id: str):
{
"filename": "section_01.md",
"section_index": 1,
"content": "## 执行摘要\\n\\n..."
"content": "## Executive Summary\\n\\n..."
},
...
],
@ -634,8 +631,7 @@ def get_report_sections(report_id: str):
"""
try:
sections = ReportManager.get_generated_sections(report_id)
# 获取报告状态
report = ReportManager.get_report(report_id)
is_complete = report is not None and report.status == ReportStatus.COMPLETED
@ -661,14 +657,14 @@ def get_report_sections(report_id: str):
@report_bp.route('/<report_id>/section/<int:section_index>', methods=['GET'])
def get_single_section(report_id: str, section_index: int):
"""
获取单个章节内容
返回：
Get the content of a single section.
Returns:
{
"success": true,
"data": {
"filename": "section_01.md",
"content": "## 执行摘要\\n\\n..."
"content": "## Executive Summary\\n\\n..."
}
}
"""
@ -702,16 +698,16 @@ def get_single_section(report_id: str, section_index: int):
}), 500
# ============== 报告状态检查接口 ==============
# ============== Report status check endpoints ==============
@report_bp.route('/check/<simulation_id>', methods=['GET'])
def check_report_status(simulation_id: str):
"""
检查模拟是否有报告，以及报告状态
用于前端判断是否解锁Interview功能
返回：
Check whether a simulation has a report, and report its status.
Used by the frontend to decide whether to unlock the Interview feature.
Returns:
{
"success": true,
"data": {
@ -730,7 +726,7 @@ def check_report_status(simulation_id: str):
report_status = report.status.value if report else None
report_id = report.report_id if report else None
# 只有报告完成后才解锁interview
# Interview feature is only unlocked once a report has finished generating.
interview_unlocked = has_report and report.status == ReportStatus.COMPLETED
return jsonify({
@ -753,22 +749,22 @@ def check_report_status(simulation_id: str):
}), 500
# ============== Agent 日志接口 ==============
# ============== Agent log endpoints ==============
@report_bp.route('/<report_id>/agent-log', methods=['GET'])
def get_agent_log(report_id: str):
"""
获取 Report Agent 的详细执行日志
实时获取报告生成过程中的每一步动作，包括：
- 报告开始、规划开始/完成
- 每个章节的开始、工具调用、LLM响应、完成
- 报告完成或失败
Query参数：
from_line: 从第几行开始读取（可选，默认0，用于增量获取）
返回：
Get the detailed execution log of the Report Agent.
Streams every step the agent took while generating the report, including:
- Report start, planning start/complete.
- Per-section start, tool calls, LLM responses, and completion.
- Final report completion or failure.
Query parameters:
from_line: line offset to start reading from (optional, default 0, for incremental polling).
Returns:
{
"success": true,
"data": {
@ -779,7 +775,7 @@ def get_agent_log(report_id: str):
"report_id": "report_xxxx",
"action": "tool_call",
"stage": "generating",
"section_title": "执行摘要",
"section_title": "Executive Summary",
"section_index": 1,
"details": {
"tool_name": "insight_forge",
@ -817,9 +813,9 @@ def get_agent_log(report_id: str):
@report_bp.route('/<report_id>/agent-log/stream', methods=['GET'])
def stream_agent_log(report_id: str):
"""
获取完整的 Agent 日志（一次性获取全部）
返回：
Get the full Agent log in one shot (no pagination).
Returns:
{
"success": true,
"data": {
@ -848,27 +844,27 @@ def stream_agent_log(report_id: str):
}), 500
# ============== 控制台日志接口 ==============
# ============== Console log endpoints ==============
@report_bp.route('/<report_id>/console-log', methods=['GET'])
def get_console_log(report_id: str):
"""
获取 Report Agent 的控制台输出日志
实时获取报告生成过程中的控制台输出（INFO、WARNING等），
这与 agent-log 接口返回的结构化 JSON 日志不同，
是纯文本格式的控制台风格日志。
Query参数：
from_line: 从第几行开始读取（可选，默认0，用于增量获取）
返回：
Get the Report Agent's console output log.
Streams the console output produced during report generation (INFO, WARNING, etc.).
Unlike the structured JSON returned by the agent-log endpoint, this is plain-text
console-style output.
Query parameters:
from_line: line offset to start reading from (optional, default 0, for incremental polling).
Returns:
{
"success": true,
"data": {
"logs": [
"[19:46:14] INFO: 搜索完成: 找到 15 条相关事实",
"[19:46:14] INFO: 图谱搜索: graph_id=xxx, query=...",
"[19:46:14] INFO: Search complete: found 15 relevant facts",
"[19:46:14] INFO: Graph search: graph_id=xxx, query=...",
...
],
"total_lines": 100,
@ -899,9 +895,9 @@ def get_console_log(report_id: str):
@report_bp.route('/<report_id>/console-log/stream', methods=['GET'])
def stream_console_log(report_id: str):
"""
获取完整的控制台日志（一次性获取全部）
返回：
Get the full console log in one shot (no pagination).
Returns:
{
"success": true,
"data": {
@ -930,17 +926,17 @@ def stream_console_log(report_id: str):
}), 500
# ============== 工具调用接口（供调试使用）==============
# ============== Tool invocation endpoints (for debugging) ==============
@report_bp.route('/tools/search', methods=['POST'])
def search_graph_tool():
"""
图谱搜索工具接口（供调试使用）
请求（JSON）：
Graph search tool endpoint (for debugging).
Request (JSON):
{
"graph_id": "mirofish_xxxx",
"query": "搜索查询",
"query": "search query",
"limit": 10
}
"""
@ -983,9 +979,9 @@ def search_graph_tool():
@report_bp.route('/tools/statistics', methods=['POST'])
def get_graph_statistics_tool():
"""
图谱统计工具接口（供调试使用）
请求（JSON）：
Graph statistics tool endpoint (for debugging).
Request (JSON):
{
"graph_id": "mirofish_xxxx"
}

945
backend/app/api/simulation.py
View File

File diff suppressed because it is too large Load Diff

60
backend/app/config.py
View File

@ -1,38 +1,40 @@
"""
配置管理
统一从项目根目录的 .env 文件加载配置
"""Configuration management.
Loads configuration values from the project-root ``.env`` file.
"""
import os
from dotenv import load_dotenv
# 加载项目根目录的 .env 文件
# 路径: MiroFish/.env (相对于 backend/app/config.py)
# Load the project-root .env file.
# Path: MiroFish/.env (relative to backend/app/config.py).
project_root_env = os.path.join(os.path.dirname(__file__), '../../.env')
if os.path.exists(project_root_env):
load_dotenv(project_root_env, override=True)
else:
# 如果根目录没有 .env，尝试加载环境变量（用于生产环境）
# If the project root has no .env, fall back to the process environment
# (used in production deployments).
load_dotenv(override=True)
class Config:
"""Flask配置类"""
# Flask配置
"""Flask configuration class."""
# Flask settings.
SECRET_KEY = os.environ.get('SECRET_KEY', 'mirofish-secret-key')
DEBUG = os.environ.get('FLASK_DEBUG', 'True').lower() == 'true'
# JSON配置 - 禁用ASCII转义，让中文直接显示（而不是 \uXXXX 格式）
# JSON settings: disable ASCII escaping so non-ASCII output renders literally
# rather than as \uXXXX escape sequences.
JSON_AS_ASCII = False
# LLM配置（统一使用OpenAI格式）
# LLM settings (called via the OpenAI-compatible API surface).
LLM_API_KEY = os.environ.get('LLM_API_KEY')
LLM_BASE_URL = os.environ.get('LLM_BASE_URL', 'https://api.openai.com/v1')
LLM_MODEL_NAME = os.environ.get('LLM_MODEL_NAME', 'gpt-4o-mini')
# Neo4j + Graphiti配置（替代 Zep Cloud）
# Neo4j + Graphiti settings (replacement for Zep Cloud).
NEO4J_URI = os.environ.get('NEO4J_URI', 'bolt://localhost:7687')
NEO4J_USER = os.environ.get('NEO4J_USER', 'neo4j')
NEO4J_PASSWORD = os.environ.get('NEO4J_PASSWORD', 'mirofish123')
@ -50,23 +52,23 @@ class Config:
EMBEDDING_API_KEY = os.environ.get('EMBEDDING_API_KEY')
EMBEDDING_BASE_URL = os.environ.get('EMBEDDING_BASE_URL')
# Zep配置（保留兼容性，已废弃）
# Zep settings (kept for backwards compatibility; deprecated).
ZEP_API_KEY = os.environ.get('ZEP_API_KEY', '')
# 文件上传配置
# File upload settings.
MAX_CONTENT_LENGTH = 50 * 1024 * 1024 # 50MB
UPLOAD_FOLDER = os.path.join(os.path.dirname(__file__), '../uploads')
ALLOWED_EXTENSIONS = {'pdf', 'md', 'txt', 'markdown'}
# 文本处理配置
DEFAULT_CHUNK_SIZE = 500 # 默认切块大小
DEFAULT_CHUNK_OVERLAP = 50 # 默认重叠大小
# OASIS模拟配置
# Text processing settings.
DEFAULT_CHUNK_SIZE = 500 # default chunk size in characters
DEFAULT_CHUNK_OVERLAP = 50 # default overlap in characters
# OASIS simulation settings.
OASIS_DEFAULT_MAX_ROUNDS = int(os.environ.get('OASIS_DEFAULT_MAX_ROUNDS', '10'))
OASIS_SIMULATION_DATA_DIR = os.path.join(os.path.dirname(__file__), '../uploads/simulations')
# OASIS平台可用动作配置
# OASIS per-platform allowed action lists.
OASIS_TWITTER_ACTIONS = [
'CREATE_POST', 'LIKE_POST', 'REPOST', 'FOLLOW', 'DO_NOTHING', 'QUOTE_POST'
]
@ -76,14 +78,14 @@ class Config:
'TREND', 'REFRESH', 'DO_NOTHING', 'FOLLOW', 'MUTE'
]
# Report Agent配置
# Report agent settings.
REPORT_AGENT_MAX_TOOL_CALLS = int(os.environ.get('REPORT_AGENT_MAX_TOOL_CALLS', '5'))
REPORT_AGENT_MAX_REFLECTION_ROUNDS = int(os.environ.get('REPORT_AGENT_MAX_REFLECTION_ROUNDS', '2'))
REPORT_AGENT_TEMPERATURE = float(os.environ.get('REPORT_AGENT_TEMPERATURE', '0.5'))
@classmethod
def validate(cls):
"""验证必要配置"""
"""Validate that required configuration values are present."""
errors = []
if not cls.LLM_API_KEY:
errors.append("LLM_API_KEY 未配置")

4
backend/app/models/__init__.py
View File

@ -1,6 +1,4 @@
"""
数据模型模块
"""
"""Data model package."""
from .task import TaskManager, TaskStatus
from .project import Project, ProjectStatus, ProjectManager

219
backend/app/models/project.py
View File

@ -1,6 +1,7 @@
"""
项目上下文管理
用于在服务端持久化项目状态，避免前端在接口间传递大量数据
"""Project context management.
Persists project state on the server so the frontend does not have to round-trip
large blobs of context between API calls.
"""
import os
@ -15,45 +16,45 @@ from ..config import Config
class ProjectStatus(str, Enum):
"""项目状态"""
CREATED = "created" # 刚创建，文件已上传
ONTOLOGY_GENERATED = "ontology_generated" # 本体已生成
GRAPH_BUILDING = "graph_building" # 图谱构建中
GRAPH_COMPLETED = "graph_completed" # 图谱构建完成
FAILED = "failed" # 失败
"""Project lifecycle status."""
CREATED = "created" # just created, files uploaded
ONTOLOGY_GENERATED = "ontology_generated" # ontology has been generated
GRAPH_BUILDING = "graph_building" # graph build in progress
GRAPH_COMPLETED = "graph_completed" # graph build finished
FAILED = "failed" # build failed
@dataclass
class Project:
"""项目数据模型"""
"""Project data model."""
project_id: str
name: str
status: ProjectStatus
created_at: str
updated_at: str
# 文件信息
# File information
files: List[Dict[str, str]] = field(default_factory=list) # [{filename, path, size}]
total_text_length: int = 0
# 本体信息（接口1生成后填充）
# Ontology information (filled in after step 1 generates it)
ontology: Optional[Dict[str, Any]] = None
analysis_summary: Optional[str] = None
# 图谱信息（接口2完成后填充）
# Graph information (filled in after step 2 finishes)
graph_id: Optional[str] = None
graph_build_task_id: Optional[str] = None
# 配置
# Configuration
simulation_requirement: Optional[str] = None
chunk_size: int = 500
chunk_overlap: int = 50
# 错误信息
# Error message when status == FAILED
error: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
"""转换为字典"""
"""Serialize the project to a JSON-friendly dict."""
return {
"project_id": self.project_id,
"name": self.name,
@ -71,14 +72,14 @@ class Project:
"chunk_overlap": self.chunk_overlap,
"error": self.error
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> 'Project':
"""从字典创建"""
"""Reconstruct a project from its serialized dict."""
status = data.get('status', 'created')
if isinstance(status, str):
status = ProjectStatus(status)
return cls(
project_id=data['project_id'],
name=data.get('name', 'Unnamed Project'),
@ -99,52 +100,51 @@ class Project:
class ProjectManager:
"""项目管理器 - 负责项目的持久化存储和检索"""
# 项目存储根目录
"""Project manager: handles persistence and retrieval of projects on disk."""
# Root directory for project storage
PROJECTS_DIR = os.path.join(Config.UPLOAD_FOLDER, 'projects')
@classmethod
def _ensure_projects_dir(cls):
"""确保项目目录存在"""
"""Ensure the projects root directory exists."""
os.makedirs(cls.PROJECTS_DIR, exist_ok=True)
@classmethod
def _get_project_dir(cls, project_id: str) -> str:
"""获取项目目录路径"""
"""Return the on-disk directory for a project."""
return os.path.join(cls.PROJECTS_DIR, project_id)
@classmethod
def _get_project_meta_path(cls, project_id: str) -> str:
"""获取项目元数据文件路径"""
"""Return the path to a project's metadata JSON file."""
return os.path.join(cls._get_project_dir(project_id), 'project.json')
@classmethod
def _get_project_files_dir(cls, project_id: str) -> str:
"""获取项目文件存储目录"""
"""Return the directory where project source files are stored."""
return os.path.join(cls._get_project_dir(project_id), 'files')
@classmethod
def _get_project_text_path(cls, project_id: str) -> str:
"""获取项目提取文本存储路径"""
"""Return the path to a project's extracted text file."""
return os.path.join(cls._get_project_dir(project_id), 'extracted_text.txt')
@classmethod
def create_project(cls, name: str = "Unnamed Project") -> Project:
"""
创建新项目
"""Create a new project.
Args:
name: 项目名称
name: Display name for the project.
Returns:
新创建的Project对象
The newly created ``Project`` instance.
"""
cls._ensure_projects_dir()
project_id = f"proj_{uuid.uuid4().hex[:12]}"
now = datetime.now().isoformat()
project = Project(
project_id=project_id,
name=name,
@ -152,154 +152,147 @@ class ProjectManager:
created_at=now,
updated_at=now
)
# 创建项目目录结构
# Create the on-disk project directory layout
project_dir = cls._get_project_dir(project_id)
files_dir = cls._get_project_files_dir(project_id)
os.makedirs(project_dir, exist_ok=True)
os.makedirs(files_dir, exist_ok=True)
# 保存项目元数据
# Persist project metadata
cls.save_project(project)
return project
@classmethod
def save_project(cls, project: Project) -> None:
"""保存项目元数据"""
"""Persist project metadata to disk."""
project.updated_at = datetime.now().isoformat()
meta_path = cls._get_project_meta_path(project.project_id)
with open(meta_path, 'w', encoding='utf-8') as f:
json.dump(project.to_dict(), f, ensure_ascii=False, indent=2)
@classmethod
def get_project(cls, project_id: str) -> Optional[Project]:
"""
获取项目
"""Load a project by id.
Args:
project_id: 项目ID
project_id: Project identifier.
Returns:
Project对象，如果不存在返回None
The ``Project`` if it exists, otherwise ``None``.
"""
meta_path = cls._get_project_meta_path(project_id)
if not os.path.exists(meta_path):
return None
with open(meta_path, 'r', encoding='utf-8') as f:
data = json.load(f)
return Project.from_dict(data)
@classmethod
def list_projects(cls, limit: int = 50) -> List[Project]:
"""
列出所有项目
"""List existing projects, newest first.
Args:
limit: 返回数量限制
limit: Maximum number of projects to return.
Returns:
项目列表，按创建时间倒序
Projects ordered by ``created_at`` descending.
"""
cls._ensure_projects_dir()
projects = []
for project_id in os.listdir(cls.PROJECTS_DIR):
project = cls.get_project(project_id)
if project:
projects.append(project)
# 按创建时间倒序排序
projects.sort(key=lambda p: p.created_at, reverse=True)
return projects[:limit]
@classmethod
def delete_project(cls, project_id: str) -> bool:
"""
删除项目及其所有文件
"""Delete a project and all of its files.
Args:
project_id: 项目ID
project_id: Project identifier.
Returns:
是否删除成功
``True`` if the project existed and was removed, ``False`` otherwise.
"""
project_dir = cls._get_project_dir(project_id)
if not os.path.exists(project_dir):
return False
shutil.rmtree(project_dir)
return True
@classmethod
def save_file_to_project(cls, project_id: str, file_storage, original_filename: str) -> Dict[str, str]:
"""
保存上传的文件到项目目录
"""Save an uploaded file under the project's files directory.
Args:
project_id: 项目ID
file_storage: Flask的FileStorage对象
original_filename: 原始文件名
project_id: Project identifier.
file_storage: Flask ``FileStorage`` object from the request.
original_filename: The user-supplied filename.
Returns:
文件信息字典 {filename, path, size}
Dict describing the saved file: ``{original_filename, saved_filename, path, size}``.
"""
files_dir = cls._get_project_files_dir(project_id)
os.makedirs(files_dir, exist_ok=True)
# 生成安全的文件名
# Generate a safe randomized filename to avoid collisions
ext = os.path.splitext(original_filename)[1].lower()
safe_filename = f"{uuid.uuid4().hex[:8]}{ext}"
file_path = os.path.join(files_dir, safe_filename)
# 保存文件
file_storage.save(file_path)
# 获取文件大小
file_size = os.path.getsize(file_path)
return {
"original_filename": original_filename,
"saved_filename": safe_filename,
"path": file_path,
"size": file_size
}
@classmethod
def save_extracted_text(cls, project_id: str, text: str) -> None:
"""保存提取的文本"""
"""Persist the project's extracted full text to disk."""
text_path = cls._get_project_text_path(project_id)
with open(text_path, 'w', encoding='utf-8') as f:
f.write(text)
@classmethod
def get_extracted_text(cls, project_id: str) -> Optional[str]:
"""获取提取的文本"""
"""Read back the project's extracted full text, or ``None`` if absent."""
text_path = cls._get_project_text_path(project_id)
if not os.path.exists(text_path):
return None
with open(text_path, 'r', encoding='utf-8') as f:
return f.read()
@classmethod
def get_project_files(cls, project_id: str) -> List[str]:
"""获取项目的所有文件路径"""
"""Return the on-disk paths of all files in the project."""
files_dir = cls._get_project_files_dir(project_id)
if not os.path.exists(files_dir):
return []
return [
os.path.join(files_dir, f)
for f in os.listdir(files_dir)
os.path.join(files_dir, f)
for f in os.listdir(files_dir)
if os.path.isfile(os.path.join(files_dir, f))
]

108
backend/app/models/task.py
View File

@ -1,6 +1,6 @@
"""
任务状态管理
用于跟踪长时间运行的任务（如图谱构建）
"""Task state management.
Tracks long-running tasks (e.g. graph build) so callers can poll progress.
"""
import uuid
@ -14,30 +14,30 @@ from ..utils.locale import t
class TaskStatus(str, Enum):
"""任务状态枚举"""
PENDING = "pending" # 等待中
PROCESSING = "processing" # 处理中
COMPLETED = "completed" # 已完成
FAILED = "failed" # 失败
"""Task status enum."""
PENDING = "pending" # waiting
PROCESSING = "processing" # in progress
COMPLETED = "completed" # finished successfully
FAILED = "failed" # finished with error
@dataclass
class Task:
"""任务数据类"""
"""Task data class."""
task_id: str
task_type: str
status: TaskStatus
created_at: datetime
updated_at: datetime
progress: int = 0 # 总进度百分比 0-100
message: str = "" # 状态消息
result: Optional[Dict] = None # 任务结果
error: Optional[str] = None # 错误信息
metadata: Dict = field(default_factory=dict) # 额外元数据
progress_detail: Dict = field(default_factory=dict) # 详细进度信息
progress: int = 0 # overall progress percentage 0-100
message: str = "" # human-readable status message
result: Optional[Dict] = None # task result payload
error: Optional[str] = None # error message when failed
metadata: Dict = field(default_factory=dict) # arbitrary caller metadata
progress_detail: Dict = field(default_factory=dict) # fine-grained progress info
def to_dict(self) -> Dict[str, Any]:
"""转换为字典"""
"""Serialize the task to a JSON-friendly dict."""
return {
"task_id": self.task_id,
"task_type": self.task_type,
@ -54,16 +54,12 @@ class Task:
class TaskManager:
"""
任务管理器
线程安全的任务状态管理
"""
"""Thread-safe singleton task registry."""
_instance = None
_lock = threading.Lock()
def __new__(cls):
"""单例模式"""
if cls._instance is None:
with cls._lock:
if cls._instance is None:
@ -71,21 +67,20 @@ class TaskManager:
cls._instance._tasks: Dict[str, Task] = {}
cls._instance._task_lock = threading.Lock()
return cls._instance
def create_task(self, task_type: str, metadata: Optional[Dict] = None) -> str:
"""
创建新任务
"""Create a new task.
Args:
task_type: 任务类型
metadata: 额外元数据
task_type: Task type identifier.
metadata: Optional caller-supplied metadata.
Returns:
任务ID
The newly created task id.
"""
task_id = str(uuid.uuid4())
now = datetime.now()
task = Task(
task_id=task_id,
task_type=task_type,
@ -94,17 +89,17 @@ class TaskManager:
updated_at=now,
metadata=metadata or {}
)
with self._task_lock:
self._tasks[task_id] = task
return task_id
def get_task(self, task_id: str) -> Optional[Task]:
"""获取任务"""
"""Return the task for ``task_id`` or ``None`` if unknown."""
with self._task_lock:
return self._tasks.get(task_id)
def update_task(
self,
task_id: str,
@ -115,17 +110,16 @@ class TaskManager:
error: Optional[str] = None,
progress_detail: Optional[Dict] = None
):
"""
更新任务状态
"""Update mutable fields on an existing task.
Args:
task_id: 任务ID
status: 新状态
progress: 进度
message: 消息
result: 结果
error: 错误信息
progress_detail: 详细进度信息
task_id: Task id to update.
status: New status, if changing.
progress: New overall progress (0-100), if changing.
message: New status message, if changing.
result: New result payload, if changing.
error: New error message, if changing.
progress_detail: New fine-grained progress info, if changing.
"""
with self._task_lock:
task = self._tasks.get(task_id)
@ -143,9 +137,9 @@ class TaskManager:
task.error = error
if progress_detail is not None:
task.progress_detail = progress_detail
def complete_task(self, task_id: str, result: Dict):
"""标记任务完成"""
"""Mark a task as completed and attach the result."""
self.update_task(
task_id,
status=TaskStatus.COMPLETED,
@ -153,29 +147,29 @@ class TaskManager:
message=t('progress.taskComplete'),
result=result
)
def fail_task(self, task_id: str, error: str):
"""标记任务失败"""
"""Mark a task as failed and attach the error message."""
self.update_task(
task_id,
status=TaskStatus.FAILED,
message=t('progress.taskFailed'),
error=error
)
def list_tasks(self, task_type: Optional[str] = None) -> list:
"""列出任务"""
"""List tasks, optionally filtered by ``task_type``, newest first."""
with self._task_lock:
tasks = list(self._tasks.values())
if task_type:
tasks = [t for t in tasks if t.task_type == task_type]
return [t.to_dict() for t in sorted(tasks, key=lambda x: x.created_at, reverse=True)]
def cleanup_old_tasks(self, max_age_hours: int = 24):
"""清理旧任务"""
"""Drop completed/failed tasks older than ``max_age_hours``."""
from datetime import timedelta
cutoff = datetime.now() - timedelta(hours=max_age_hours)
with self._task_lock:
old_ids = [
tid for tid, task in self._tasks.items()

4
backend/app/services/__init__.py
View File

@ -1,6 +1,4 @@
"""
业务服务模块
"""
"""Business services package."""
from .ontology_generator import OntologyGenerator
from .graph_builder import GraphBuilderService

123
backend/app/services/graph_builder.py
View File

@ -1,6 +1,7 @@
"""
图谱构建服务
接口2：使用Zep API构建Standalone Graph
"""Graph build service.
Pipeline step 2: build the project's standalone knowledge graph through the
Zep/Graphiti API.
"""
import os
@ -69,7 +70,7 @@ def _classify_entity_type(name: str, summary: str, ontology: Optional[Dict]) ->
@dataclass
class GraphInfo:
"""图谱信息"""
"""Summary information about a built graph."""
graph_id: str
node_count: int
edge_count: int
@ -85,10 +86,7 @@ class GraphInfo:
class GraphBuilderService:
"""
图谱构建服务
负责调用Zep API构建知识图谱
"""
"""Drives knowledge-graph construction via the Zep/Graphiti API."""
def __init__(self, api_key: Optional[str] = None):
self.client = GraphitiAdapter()
@ -103,21 +101,20 @@ class GraphBuilderService:
chunk_overlap: int = 50,
batch_size: int = 3
) -> str:
"""
异步构建图谱
"""Kick off a graph build asynchronously.
Args:
text: 输入文本
ontology: 本体定义（来自接口1的输出）
graph_name: 图谱名称
chunk_size: 文本块大小
chunk_overlap: 块重叠大小
batch_size: 每批发送的块数量
text: Source text to ingest.
ontology: Ontology definition (the output of pipeline step 1).
graph_name: Display name for the graph.
chunk_size: Characters per text chunk.
chunk_overlap: Overlap (in characters) between consecutive chunks.
batch_size: Number of chunks pushed to Zep per batch.
Returns:
任务ID
The id of the task tracking the build.
"""
# 创建任务
# Register a task to track build progress.
task_id = self.task_manager.create_task(
task_type="graph_build",
metadata={
@ -130,7 +127,7 @@ class GraphBuilderService:
# Capture locale before spawning background thread
current_locale = get_locale()
# 在后台线程中执行构建
# Run the build on a background thread so the request returns immediately.
thread = threading.Thread(
target=self._build_graph_worker,
args=(task_id, text, ontology, graph_name, chunk_size, chunk_overlap, batch_size, current_locale)
@ -151,7 +148,7 @@ class GraphBuilderService:
batch_size: int,
locale: str = 'zh'
):
"""图谱构建工作线程"""
"""Background worker that performs the graph build."""
set_locale(locale)
try:
self.task_manager.update_task(
@ -161,7 +158,7 @@ class GraphBuilderService:
message=t('progress.startBuildingGraph')
)
# 1. 创建图谱
# 1. Create the graph.
graph_id = self.create_graph(graph_name)
self.task_manager.update_task(
task_id,
@ -169,7 +166,7 @@ class GraphBuilderService:
message=t('progress.graphCreated', graphId=graph_id)
)
# 2. 设置本体
# 2. Set the ontology.
self.set_ontology(graph_id, ontology)
self.task_manager.update_task(
task_id,
@ -177,7 +174,7 @@ class GraphBuilderService:
message=t('progress.ontologySet')
)
# 3. 文本分块
# 3. Split source text into chunks.
chunks = TextProcessor.split_text(text, chunk_size, chunk_overlap)
total_chunks = len(chunks)
self.task_manager.update_task(
@ -186,7 +183,7 @@ class GraphBuilderService:
message=t('progress.textSplit', count=total_chunks)
)
# 4. 分批发送数据
# 4. Push chunks to the graph in batches.
episode_uuids = self.add_text_batches(
graph_id, chunks, batch_size,
lambda msg, prog: self.task_manager.update_task(
@ -196,7 +193,7 @@ class GraphBuilderService:
)
)
# 5. 等待Zep处理完成
# 5. Wait for Zep to finish processing the episodes.
self.task_manager.update_task(
task_id,
progress=60,
@ -212,7 +209,7 @@ class GraphBuilderService:
)
)
# 6. 获取图谱信息
# 6. Fetch the final graph metadata.
self.task_manager.update_task(
task_id,
progress=90,
@ -220,8 +217,7 @@ class GraphBuilderService:
)
graph_info = self._get_graph_info(graph_id)
# 完成
self.task_manager.complete_task(task_id, {
"graph_id": graph_id,
"graph_info": graph_info.to_dict(),
@ -234,7 +230,7 @@ class GraphBuilderService:
self.task_manager.fail_task(task_id, error_msg)
def create_graph(self, name: str) -> str:
"""创建Zep图谱（公开方法）"""
"""Create a new Zep graph and return its id (public API)."""
graph_id = f"mirofish_{uuid.uuid4().hex[:16]}"
self.client.graph.create(
@ -246,7 +242,7 @@ class GraphBuilderService:
return graph_id
def set_ontology(self, graph_id: str, ontology: Dict[str, Any]):
"""设置图谱本体提示（Graphiti自动提取实体，本体作为提示存储）"""
"""Register the ontology with the graph (Graphiti uses it as an extraction prompt)."""
self.client.graph.set_ontology(
graph_ids=[graph_id],
entities=ontology.get("entity_types"),
@ -261,8 +257,11 @@ class GraphBuilderService:
progress_callback: Optional[Callable] = None,
skip_chunks: int = 0,
) -> List[str]:
"""分批添加文本到图谱，返回所有 episode 的 uuid 列表。
skip_chunks: 跳过已处理的块数（用于断点续传）。"""
"""Push chunks to the graph in batches; returns the uuids of all episodes added.
Args:
skip_chunks: Number of chunks to skip (used for resume-after-restart).
"""
episode_uuids = []
total_chunks = len(chunks)
@ -279,27 +278,26 @@ class GraphBuilderService:
)
# 构建episode数据
# Build the per-episode payload structures expected by the client.
episodes = [
type('Episode', (), {'data': chunk, 'type': 'text'})()
for chunk in batch_chunks
]
# 发送到Zep
try:
batch_result = self.client.graph.add_batch(
graph_id=graph_id,
episodes=episodes
)
# 收集返回的 episode uuid
# Collect the uuids returned for each episode.
if batch_result and isinstance(batch_result, list):
for ep in batch_result:
ep_uuid = getattr(ep, 'uuid_', None) or getattr(ep, 'uuid', None)
if ep_uuid:
episode_uuids.append(ep_uuid)
# 避免请求过快
# Throttle to avoid overwhelming the upstream API.
time.sleep(1)
except Exception as e:
@ -315,7 +313,7 @@ class GraphBuilderService:
progress_callback: Optional[Callable] = None,
timeout: int = 600
):
"""等待所有 episode 处理完成（通过查询每个 episode 的 processed 状态）"""
"""Poll each episode until Zep marks it processed, or the timeout expires."""
if not episode_uuids:
if progress_callback:
progress_callback(t('progress.noEpisodesWait'), 1.0)
@ -338,18 +336,18 @@ class GraphBuilderService:
)
break
# 检查每个 episode 的处理状态
# Check the processing state of each pending episode.
for ep_uuid in list(pending_episodes):
try:
episode = self.client.graph.episode.get(uuid_=ep_uuid)
is_processed = getattr(episode, 'processed', False)
if is_processed:
pending_episodes.remove(ep_uuid)
completed_count += 1
except Exception as e:
# 忽略单个查询错误，继续
# Tolerate a single failed query; the next loop iteration retries.
pass
elapsed = int(time.time() - start_time)
@ -360,20 +358,17 @@ class GraphBuilderService:
)
if pending_episodes:
time.sleep(3) # 每3秒检查一次
time.sleep(3) # poll every 3 seconds
if progress_callback:
progress_callback(t('progress.processingComplete', completed=completed_count, total=total_episodes), 1.0)
def _get_graph_info(self, graph_id: str) -> GraphInfo:
"""获取图谱信息"""
# 获取节点（分页）
"""Fetch summary info (counts and entity types) for a graph."""
nodes = fetch_all_nodes(self.client, graph_id)
# 获取边（分页）
edges = fetch_all_edges(self.client, graph_id)
# 统计实体类型
# Tally distinct entity types across all nodes.
entity_types = set()
for node in nodes:
if node.labels:
@ -389,26 +384,24 @@ class GraphBuilderService:
)
def get_graph_data(self, graph_id: str, ontology: Optional[Dict] = None) -> Dict[str, Any]:
"""
获取完整图谱数据（包含详细信息）
"""Return the full graph payload including timestamps, attributes, and edges.
Args:
graph_id: 图谱ID
graph_id: Graph identifier.
Returns:
包含nodes和edges的字典，包括时间信息、属性等详细数据
Dict with ``nodes``, ``edges``, and aggregate counts.
"""
nodes = fetch_all_nodes(self.client, graph_id)
edges = fetch_all_edges(self.client, graph_id)
# 创建节点映射用于获取节点名称
# Build a uuid->name map so edge endpoints can be labeled.
node_map = {}
for node in nodes:
node_map[node.uuid_] = node.name or ""
nodes_data = []
for node in nodes:
# 获取创建时间
created_at = getattr(node, 'created_at', None)
if created_at:
created_at = str(created_at)
@ -429,20 +422,18 @@ class GraphBuilderService:
edges_data = []
for edge in edges:
# 获取时间信息
created_at = getattr(edge, 'created_at', None)
valid_at = getattr(edge, 'valid_at', None)
invalid_at = getattr(edge, 'invalid_at', None)
expired_at = getattr(edge, 'expired_at', None)
# 获取 episodes
# Normalize the episode list (the field may be missing or a single id).
episodes = getattr(edge, 'episodes', None) or getattr(edge, 'episode_ids', None)
if episodes and not isinstance(episodes, list):
episodes = [str(episodes)]
elif episodes:
episodes = [str(e) for e in episodes]
# 获取 fact_type
fact_type = getattr(edge, 'fact_type', None) or edge.name or ""
edges_data.append({
@ -471,6 +462,6 @@ class GraphBuilderService:
}
def delete_graph(self, graph_id: str):
"""删除图谱"""
"""Delete a graph by id."""
self.client.graph.delete(graph_id=graph_id)

23
backend/app/services/graphiti_adapter.py
View File

@ -453,9 +453,17 @@ class _GraphNamespace:
return _EpisodeResult(uuid_=ep_uuid_out)
def add_batch(self, graph_id: str, episodes: List[Any]) -> List[_EpisodeResult]:
"""Add a batch of episodes. Returns list of EpisodeResult with uuid_."""
"""Add a batch of episodes. Returns one _EpisodeResult per episode in input order.
On the first ingestion failure the underlying exception is logged at ERROR
level (with traceback) and re-raised; episodes successfully ingested before
the failure remain committed in Neo4j. The caller (the graph-build worker)
translates the propagated exception into Task.status = FAILED with the
underlying error message — never substitute a placeholder UUID, since that
would produce a Task that looks completed while the graph is empty.
"""
results = []
for ep in episodes:
for index, ep in enumerate(episodes):
text = getattr(ep, 'data', '') or str(ep)
try:
result = _run(self._g.add_episode(
@ -467,10 +475,13 @@ class _GraphNamespace:
group_id=graph_id,
update_communities=False,
))
ep_uuid_out = result.episode.uuid if result and result.episode else str(_uuid_mod.uuid4())
except Exception as e:
logger.warning(f"Episode add failed: {str(e)[:100]}, using placeholder uuid")
ep_uuid_out = str(_uuid_mod.uuid4())
except Exception:
logger.exception(
"Episode add failed (group_id=%s, episode_index=%d)",
graph_id, index,
)
raise
ep_uuid_out = result.episode.uuid if result and result.episode else str(_uuid_mod.uuid4())
results.append(_EpisodeResult(uuid_=ep_uuid_out))
return results

547
backend/app/services/oasis_profile_generator.py
View File

File diff suppressed because it is too large Load Diff

129
backend/app/services/ontology_generator.py
View File

@ -1,6 +1,7 @@
"""
本体生成服务
接口1：分析文本内容，生成适合社会模拟的实体和关系类型定义
"""Ontology generation service.
Pipeline step 1: analyze the source text and propose entity and relationship
types that fit a social-media opinion simulation.
"""
import json
@ -14,19 +15,19 @@ logger = logging.getLogger(__name__)
def _to_pascal_case(name: str) -> str:
"""将任意格式的名称转换为 PascalCase（如 'works_for' -> 'WorksFor', 'person' -> 'Person'）"""
# 按非字母数字字符分割
"""Convert an arbitrary identifier to PascalCase (e.g. ``works_for`` -> ``WorksFor``)."""
# Split on non-alphanumeric separators first.
parts = re.split(r'[^a-zA-Z0-9]+', name)
# 再按 camelCase 边界分割（如 'camelCase' -> ['camel', 'Case']）
# Then split on camelCase boundaries (e.g. ``camelCase`` -> ``['camel', 'Case']``).
words = []
for part in parts:
words.extend(re.sub(r'([a-z])([A-Z])', r'\1_\2', part).split('_'))
# 每个词首字母大写，过滤空串
# Title-case each non-empty word and concatenate.
result = ''.join(word.capitalize() for word in words if word)
return result if result else 'Unknown'
# 本体生成的系统提示词
# System prompt template for ontology generation.
ONTOLOGY_SYSTEM_PROMPT = """You are a professional knowledge-graph ontology designer. Your task is to analyze the supplied text and simulation requirement and design entity types and relationship types suitable for a **social-media public-opinion simulation**.
**Important: you must output valid JSON data and nothing else.**
@ -174,10 +175,7 @@ B. **Concrete types (8 entries, designed from the text content)**:
class OntologyGenerator:
"""
本体生成器
分析文本内容，生成实体和关系类型定义
"""
"""Generate an entity- and edge-type ontology from arbitrary input text."""
def __init__(self, llm_client: Optional[LLMClient] = None):
self.llm_client = llm_client or LLMClient()
@ -188,18 +186,17 @@ class OntologyGenerator:
simulation_requirement: str,
additional_context: Optional[str] = None
) -> Dict[str, Any]:
"""
生成本体定义
"""Generate an ontology definition.
Args:
document_texts: 文档文本列表
simulation_requirement: 模拟需求描述
additional_context: 额外上下文
document_texts: Source document text segments.
simulation_requirement: Description of the simulation goal.
additional_context: Optional supplemental context.
Returns:
本体定义（entity_types, edge_types等）
The ontology dict with ``entity_types``, ``edge_types``, and a summary.
"""
# 构建用户消息
# Compose the user message that frames the LLM request.
user_message = self._build_user_message(
document_texts,
simulation_requirement,
@ -213,19 +210,19 @@ class OntologyGenerator:
{"role": "user", "content": user_message}
]
# 调用LLM
# Invoke the LLM.
result = self.llm_client.chat_json(
messages=messages,
temperature=0.3,
max_tokens=4096
)
# 验证和后处理
# Validate the LLM response and post-process it.
result = self._validate_and_process(result)
return result
# 传给 LLM 的文本最大长度（5万字）
# Maximum length of source text passed to the LLM (50k characters).
MAX_TEXT_LENGTH_FOR_LLM = 50000
def _build_user_message(
@ -234,13 +231,14 @@ class OntologyGenerator:
simulation_requirement: str,
additional_context: Optional[str]
) -> str:
"""构建用户消息"""
# 合并文本
"""Build the user-message string for the ontology LLM call."""
# Concatenate the source documents into a single string.
combined_text = "\n\n---\n\n".join(document_texts)
original_length = len(combined_text)
# 如果文本超过5万字，截断（仅影响传给LLM的内容，不影响图谱构建）
# If the combined text exceeds the LLM input cap, truncate it for the
# LLM call only. The full text is still used for graph construction.
if len(combined_text) > self.MAX_TEXT_LENGTH_FOR_LLM:
combined_text = combined_text[:self.MAX_TEXT_LENGTH_FOR_LLM]
combined_text += f"\n\n...(original text is {original_length} characters; only the first {self.MAX_TEXT_LENGTH_FOR_LLM} characters were used for ontology analysis)..."
@ -275,9 +273,9 @@ Based on the content above, design entity types and relationship types suitable
return message
def _validate_and_process(self, result: Dict[str, Any]) -> Dict[str, Any]:
"""验证和后处理结果"""
# 确保必要字段存在
"""Validate and post-process the LLM-generated ontology dict."""
# Ensure required top-level fields exist.
if "entity_types" not in result:
result["entity_types"] = []
if "edge_types" not in result:
@ -285,11 +283,12 @@ Based on the content above, design entity types and relationship types suitable
if "analysis_summary" not in result:
result["analysis_summary"] = ""
# 验证实体类型
# 记录原始名称到 PascalCase 的映射，用于后续修正 edge 的 source_targets 引用
# Validate entity types.
# Track original-name -> PascalCase mapping so edge source_targets
# references can be fixed up consistently below.
entity_name_map = {}
for entity in result["entity_types"]:
# 强制将 entity name 转为 PascalCase（Zep API 要求）
# Force entity names to PascalCase (required by the Zep API).
if "name" in entity:
original_name = entity["name"]
entity["name"] = _to_pascal_case(original_name)
@ -300,19 +299,20 @@ Based on the content above, design entity types and relationship types suitable
entity["attributes"] = []
if "examples" not in entity:
entity["examples"] = []
# 确保description不超过100字符
# Truncate descriptions longer than 100 characters.
if len(entity.get("description", "")) > 100:
entity["description"] = entity["description"][:97] + "..."
# 验证关系类型
# Validate edge types.
for edge in result["edge_types"]:
# 强制将 edge name 转为 SCREAMING_SNAKE_CASE（Zep API 要求）
# Force edge names to SCREAMING_SNAKE_CASE (required by the Zep API).
if "name" in edge:
original_name = edge["name"]
edge["name"] = original_name.upper()
if edge["name"] != original_name:
logger.warning(f"Edge type name '{original_name}' auto-converted to '{edge['name']}'")
# 修正 source_targets 中的实体名称引用，与转换后的 PascalCase 保持一致
# Rewrite source_targets entity-name references to match the
# PascalCase-normalized entity names.
for st in edge.get("source_targets", []):
if st.get("source") in entity_name_map:
st["source"] = entity_name_map[st["source"]]
@ -325,11 +325,11 @@ Based on the content above, design entity types and relationship types suitable
if len(edge.get("description", "")) > 100:
edge["description"] = edge["description"][:97] + "..."
# Zep API 限制：最多 10 个自定义实体类型，最多 10 个自定义边类型
# Zep API caps: at most 10 custom entity types and 10 custom edge types.
MAX_ENTITY_TYPES = 10
MAX_EDGE_TYPES = 10
# 去重：按 name 去重，保留首次出现的
# Deduplicate by name, keeping the first occurrence.
seen_names = set()
deduped = []
for entity in result["entity_types"]:
@ -341,7 +341,7 @@ Based on the content above, design entity types and relationship types suitable
logger.warning(f"Duplicate entity type '{name}' removed during validation")
result["entity_types"] = deduped
# 兜底类型定义
# Fallback entity-type definitions used when the LLM omits them.
person_fallback = {
"name": "Person",
"description": "Any individual person not fitting other specific person types.",
@ -362,33 +362,31 @@ Based on the content above, design entity types and relationship types suitable
"examples": ["small business", "community group"]
}
# 检查是否已有兜底类型
# Check whether the fallback types are already present.
entity_names = {e["name"] for e in result["entity_types"]}
has_person = "Person" in entity_names
has_organization = "Organization" in entity_names
# 需要添加的兜底类型
# Collect missing fallback types to add below.
fallbacks_to_add = []
if not has_person:
fallbacks_to_add.append(person_fallback)
if not has_organization:
fallbacks_to_add.append(organization_fallback)
if fallbacks_to_add:
current_count = len(result["entity_types"])
needed_slots = len(fallbacks_to_add)
# 如果添加后会超过 10 个，需要移除一些现有类型
# If adding the fallbacks would exceed the cap, drop some existing types.
if current_count + needed_slots > MAX_ENTITY_TYPES:
# 计算需要移除多少个
to_remove = current_count + needed_slots - MAX_ENTITY_TYPES
# 从末尾移除（保留前面更重要的具体类型）
# Drop trailing types first; the more specific types come earlier.
result["entity_types"] = result["entity_types"][:-to_remove]
# 添加兜底类型
result["entity_types"].extend(fallbacks_to_add)
# 最终确保不超过限制（防御性编程）
# Defensive cap enforcement: hard-trim if anything slipped through.
if len(result["entity_types"]) > MAX_ENTITY_TYPES:
result["entity_types"] = result["entity_types"][:MAX_ENTITY_TYPES]
@ -398,14 +396,13 @@ Based on the content above, design entity types and relationship types suitable
return result
def generate_python_code(self, ontology: Dict[str, Any]) -> str:
"""
将本体定义转换为Python代码（类似ontology.py）
"""Render the ontology definition as Python source code.
Args:
ontology: 本体定义
ontology: Ontology definition dict.
Returns:
Python代码字符串
Python source code as a single string.
"""
code_lines = [
'"""',
@ -421,7 +418,7 @@ Based on the content above, design entity types and relationship types suitable
'',
]
# 生成实体类型
# Emit each entity type as a Python class.
for entity in ontology.get("entity_types", []):
name = entity["name"]
desc = entity.get("description", f"A {name} entity.")
@ -447,10 +444,10 @@ Based on the content above, design entity types and relationship types suitable
code_lines.append('# ============== 关系类型定义 ==============')
code_lines.append('')
# 生成关系类型
# Emit each edge type as a Python class.
for edge in ontology.get("edge_types", []):
name = edge["name"]
# 转换为PascalCase类名
# Convert SCREAMING_SNAKE_CASE -> PascalCase for the class name.
class_name = ''.join(word.capitalize() for word in name.split('_'))
desc = edge.get("description", f"A {name} relationship.")
@ -472,7 +469,7 @@ Based on the content above, design entity types and relationship types suitable
code_lines.append('')
code_lines.append('')
# 生成类型字典
# Emit the type registries.
code_lines.append('# ============== 类型配置 ==============')
code_lines.append('')
code_lines.append('ENTITY_TYPES = {')
@ -489,7 +486,7 @@ Based on the content above, design entity types and relationship types suitable
code_lines.append('}')
code_lines.append('')
# 生成边的source_targets映射
# Emit the edge source_targets map.
code_lines.append('EDGE_SOURCE_TARGETS = {')
for edge in ontology.get("edge_types", []):
name = edge["name"]

913
backend/app/services/report_agent.py
View File

File diff suppressed because it is too large Load Diff

437
backend/app/services/simulation_config_generator.py
View File

@ -1,13 +1,16 @@
"""
模拟配置智能生成器
使用LLM根据模拟需求、文档内容、图谱信息自动生成细致的模拟参数
实现全程自动化，无需人工设置参数
Intelligent simulation-configuration generator.
采用分步生成策略，避免一次性生成过长内容导致失败：
1. 生成时间配置
2. 生成事件配置
3. 分批生成Agent配置
4. 生成平台配置
Uses an LLM to derive detailed simulation parameters from the simulation
requirement, document content, and knowledge-graph information, fully
automating parameter setup without manual intervention.
Employs a step-wise generation strategy to avoid failures caused by
producing too much content in a single call:
1. Generate time configuration
2. Generate event configuration
3. Generate agent configurations in batches
4. Generate platform configuration
"""
import json
@ -25,156 +28,156 @@ from .zep_entity_reader import EntityNode, ZepEntityReader
logger = get_logger('mirofish.simulation_config')
# 中国作息时间配置（北京时间）
# Daily-rhythm config for China (Beijing time, UTC+8).
CHINA_TIMEZONE_CONFIG = {
# 深夜时段（几乎无人活动）
# Late-night hours: almost no activity.
"dead_hours": [0, 1, 2, 3, 4, 5],
# 早间时段（逐渐醒来）
# Morning hours: gradually waking up.
"morning_hours": [6, 7, 8],
# 工作时段
# Working hours.
"work_hours": [9, 10, 11, 12, 13, 14, 15, 16, 17, 18],
# 晚间高峰（最活跃）
# Evening peak: most active.
"peak_hours": [19, 20, 21, 22],
# 夜间时段（活跃度下降）
# Late-evening hours: activity declining.
"night_hours": [23],
# 活跃度系数
# Activity multipliers.
"activity_multipliers": {
"dead": 0.05, # 凌晨几乎无人
"morning": 0.4, # 早间逐渐活跃
"work": 0.7, # 工作时段中等
"peak": 1.5, # 晚间高峰
"night": 0.5 # 深夜下降
"dead": 0.05, # Overnight: almost no one online.
"morning": 0.4, # Morning ramp-up.
"work": 0.7, # Working hours: moderate activity.
"peak": 1.5, # Evening peak.
"night": 0.5 # Late-night decline.
}
}
@dataclass
class AgentActivityConfig:
"""单个Agent的活动配置"""
"""Activity configuration for a single agent."""
agent_id: int
entity_uuid: str
entity_name: str
entity_type: str
# 活跃度配置 (0.0-1.0)
activity_level: float = 0.5 # 整体活跃度
# 发言频率（每小时预期发言次数）
# Activity configuration (0.0-1.0).
activity_level: float = 0.5 # Overall activity level.
# Posting frequency (expected posts per hour).
posts_per_hour: float = 1.0
comments_per_hour: float = 2.0
# 活跃时间段（24小时制，0-23）
# Active hours (24-hour clock, 0-23).
active_hours: List[int] = field(default_factory=lambda: list(range(8, 23)))
# 响应速度（对热点事件的反应延迟，单位：模拟分钟）
# Response speed: latency to react to hot events, in simulated minutes.
response_delay_min: int = 5
response_delay_max: int = 60
# 情感倾向 (-1.0到1.0，负面到正面)
# Sentiment bias (-1.0 to 1.0, negative to positive).
sentiment_bias: float = 0.0
# 立场（对特定话题的态度）
# Stance: attitude toward a given topic.
stance: str = "neutral" # supportive, opposing, neutral, observer
# 影响力权重（决定其发言被其他Agent看到的概率）
# Influence weight: probability of an agent's post being seen by others.
influence_weight: float = 1.0
@dataclass
class TimeSimulationConfig:
"""时间模拟配置（基于中国人作息习惯）"""
# 模拟总时长（模拟小时数）
total_simulation_hours: int = 72 # 默认模拟72小时（3天）
# 每轮代表的时间（模拟分钟）- 默认60分钟（1小时），加快时间流速
"""Time-simulation configuration (modelled on a Chinese daily rhythm)."""
# Total simulated duration (simulated hours).
total_simulation_hours: int = 72 # Default: 72 simulated hours (3 days).
# Time represented by each round (simulated minutes); default 60 (1 hour) to speed up the simulated clock.
minutes_per_round: int = 60
# 每小时激活的Agent数量范围
# Range of agents activated per hour.
agents_per_hour_min: int = 5
agents_per_hour_max: int = 20
# 高峰时段（晚间19-22点，中国人最活跃的时间）
# Peak hours (evenings 19:00-22:00, most active for the modelled audience).
peak_hours: List[int] = field(default_factory=lambda: [19, 20, 21, 22])
peak_activity_multiplier: float = 1.5
# 低谷时段（凌晨0-5点，几乎无人活动）
# Off-peak hours (00:00-05:00, almost no activity).
off_peak_hours: List[int] = field(default_factory=lambda: [0, 1, 2, 3, 4, 5])
off_peak_activity_multiplier: float = 0.05 # 凌晨活跃度极低
# 早间时段
off_peak_activity_multiplier: float = 0.05 # Overnight activity is very low.
# Morning hours.
morning_hours: List[int] = field(default_factory=lambda: [6, 7, 8])
morning_activity_multiplier: float = 0.4
# 工作时段
# Working hours.
work_hours: List[int] = field(default_factory=lambda: [9, 10, 11, 12, 13, 14, 15, 16, 17, 18])
work_activity_multiplier: float = 0.7
@dataclass
class EventConfig:
"""事件配置"""
# 初始事件（模拟开始时的触发事件）
"""Event configuration."""
# Initial events: triggers fired when the simulation begins.
initial_posts: List[Dict[str, Any]] = field(default_factory=list)
# 定时事件（在特定时间触发的事件）
# Scheduled events: events fired at specific times.
scheduled_events: List[Dict[str, Any]] = field(default_factory=list)
# 热点话题关键词
# Hot-topic keywords.
hot_topics: List[str] = field(default_factory=list)
# 舆论引导方向
# Narrative direction for public-opinion guidance.
narrative_direction: str = ""
@dataclass
class PlatformConfig:
"""平台特定配置"""
"""Platform-specific configuration."""
platform: str # twitter or reddit
# 推荐算法权重
recency_weight: float = 0.4 # 时间新鲜度
popularity_weight: float = 0.3 # 热度
relevance_weight: float = 0.3 # 相关性
# 病毒传播阈值（达到多少互动后触发扩散）
# Recommendation-algorithm weights.
recency_weight: float = 0.4 # Recency.
popularity_weight: float = 0.3 # Popularity.
relevance_weight: float = 0.3 # Relevance.
# Viral-spread threshold: number of interactions required to trigger spreading.
viral_threshold: int = 10
# 回声室效应强度（相似观点聚集程度）
# Echo-chamber strength: how strongly similar viewpoints cluster together.
echo_chamber_strength: float = 0.5
@dataclass
class SimulationParameters:
"""完整的模拟参数配置"""
# 基础信息
"""Complete simulation-parameter configuration."""
# Basic identifiers.
simulation_id: str
project_id: str
graph_id: str
simulation_requirement: str
# 时间配置
# Time configuration.
time_config: TimeSimulationConfig = field(default_factory=TimeSimulationConfig)
# Agent配置列表
# Agent configuration list.
agent_configs: List[AgentActivityConfig] = field(default_factory=list)
# 事件配置
# Event configuration.
event_config: EventConfig = field(default_factory=EventConfig)
# 平台配置
# Platform configurations.
twitter_config: Optional[PlatformConfig] = None
reddit_config: Optional[PlatformConfig] = None
# LLM配置
# LLM configuration.
llm_model: str = ""
llm_base_url: str = ""
# 生成元数据
# Generation metadata.
generated_at: str = field(default_factory=lambda: datetime.now().isoformat())
generation_reasoning: str = "" # LLM的推理说明
generation_reasoning: str = "" # LLM-provided rationale.
def to_dict(self) -> Dict[str, Any]:
"""转换为字典"""
"""Return the parameters as a dictionary."""
time_dict = asdict(self.time_config)
return {
"simulation_id": self.simulation_id,
@ -193,34 +196,35 @@ class SimulationParameters:
}
def to_json(self, indent: int = 2) -> str:
"""转换为JSON字符串"""
"""Return the parameters as a JSON string."""
return json.dumps(self.to_dict(), ensure_ascii=False, indent=indent)
class SimulationConfigGenerator:
"""
模拟配置智能生成器
使用LLM分析模拟需求、文档内容、图谱实体信息，
自动生成最佳的模拟参数配置
采用分步生成策略：
1. 生成时间配置和事件配置（轻量级）
2. 分批生成Agent配置（每批10-20个）
3. 生成平台配置
Intelligent simulation-configuration generator.
Uses an LLM to analyse the simulation requirement, document content,
and graph entity information to automatically derive the best
simulation parameter configuration.
Step-wise generation strategy:
1. Generate time and event configurations (lightweight).
2. Generate agent configurations in batches (10-20 per batch).
3. Generate platform configuration.
"""
# 上下文最大字符数
# Maximum context length (characters).
MAX_CONTEXT_LENGTH = 50000
# 每批生成的Agent数量
# Number of agents generated per batch.
AGENTS_PER_BATCH = 15
# 各步骤的上下文截断长度（字符数）
TIME_CONFIG_CONTEXT_LENGTH = 10000 # 时间配置
EVENT_CONFIG_CONTEXT_LENGTH = 8000 # 事件配置
ENTITY_SUMMARY_LENGTH = 300 # 实体摘要
AGENT_SUMMARY_LENGTH = 300 # Agent配置中的实体摘要
ENTITIES_PER_TYPE_DISPLAY = 20 # 每类实体显示数量
# Per-step context truncation lengths (characters).
TIME_CONFIG_CONTEXT_LENGTH = 10000 # Time configuration.
EVENT_CONFIG_CONTEXT_LENGTH = 8000 # Event configuration.
ENTITY_SUMMARY_LENGTH = 300 # Entity summary.
AGENT_SUMMARY_LENGTH = 300 # Entity summary used in agent configs.
ENTITIES_PER_TYPE_DISPLAY = 20 # Number of entities displayed per type.
def __init__(
self,
@ -252,28 +256,27 @@ class SimulationConfigGenerator:
enable_reddit: bool = True,
progress_callback: Optional[Callable[[int, int, str], None]] = None,
) -> SimulationParameters:
"""
智能生成完整的模拟配置（分步生成）
"""Intelligently generate a complete simulation configuration (step-wise).
Args:
simulation_id: 模拟ID
project_id: 项目ID
graph_id: 图谱ID
simulation_requirement: 模拟需求描述
document_text: 原始文档内容
entities: 过滤后的实体列表
enable_twitter: 是否启用Twitter
enable_reddit: 是否启用Reddit
progress_callback: 进度回调函数(current_step, total_steps, message)
simulation_id: Simulation ID.
project_id: Project ID.
graph_id: Graph ID.
simulation_requirement: Description of the simulation requirement.
document_text: Original document content.
entities: Filtered list of entities.
enable_twitter: Whether to enable Twitter.
enable_reddit: Whether to enable Reddit.
progress_callback: Progress callback (current_step, total_steps, message).
Returns:
SimulationParameters: 完整的模拟参数
SimulationParameters: The complete simulation parameters.
"""
logger.info(t("log.simulation_config.m001", simulation_id=simulation_id, len=len(entities)))
# 计算总步骤数
# Compute total step count.
num_batches = math.ceil(len(entities) / self.AGENTS_PER_BATCH)
total_steps = 3 + num_batches # 时间配置 + 事件配置 + N批Agent + 平台配置
total_steps = 3 + num_batches # Time config + event config + N agent batches + platform config.
current_step = 0
def report_progress(step: int, message: str):
@ -283,7 +286,7 @@ class SimulationConfigGenerator:
progress_callback(step, total_steps, message)
logger.info(f"[{step}/{total_steps}] {message}")
# 1. 构建基础上下文信息
# 1. Build base context information.
context = self._build_context(
simulation_requirement=simulation_requirement,
document_text=document_text,
@ -292,20 +295,20 @@ class SimulationConfigGenerator:
reasoning_parts = []
# ========== 步骤1: 生成时间配置 ==========
# ========== Step 1: generate time configuration ==========
report_progress(1, t('progress.generatingTimeConfig'))
num_entities = len(entities)
time_config_result = self._generate_time_config(context, num_entities)
time_config = self._parse_time_config(time_config_result, num_entities)
reasoning_parts.append(f"{t('progress.timeConfigLabel')}: {time_config_result.get('reasoning', t('common.success'))}")
# ========== 步骤2: 生成事件配置 ==========
# ========== Step 2: generate event configuration ==========
report_progress(2, t('progress.generatingEventConfig'))
event_config_result = self._generate_event_config(context, simulation_requirement, entities)
event_config = self._parse_event_config(event_config_result)
reasoning_parts.append(f"{t('progress.eventConfigLabel')}: {event_config_result.get('reasoning', t('common.success'))}")
# ========== 步骤3-N: 分批生成Agent配置 ==========
# ========== Steps 3-N: generate agent configurations in batches ==========
all_agent_configs = []
for batch_idx in range(num_batches):
start_idx = batch_idx * self.AGENTS_PER_BATCH
@ -327,13 +330,13 @@ class SimulationConfigGenerator:
reasoning_parts.append(t('progress.agentConfigResult', count=len(all_agent_configs)))
# ========== 为初始帖子分配发布者 Agent ==========
# ========== Assign poster agents to initial posts ==========
logger.info(t("log.simulation_config.m002"))
event_config = self._assign_initial_post_agents(event_config, all_agent_configs)
assigned_count = len([p for p in event_config.initial_posts if p.get("poster_agent_id") is not None])
reasoning_parts.append(t('progress.postAssignResult', count=assigned_count))
# ========== 最后一步: 生成平台配置 ==========
# ========== Final step: generate platform configuration ==========
report_progress(total_steps, t('progress.generatingPlatformConfig'))
twitter_config = None
reddit_config = None
@ -358,7 +361,7 @@ class SimulationConfigGenerator:
echo_chamber_strength=0.6
)
# 构建最终参数
# Build final parameters.
params = SimulationParameters(
simulation_id=simulation_id,
project_id=project_id,
@ -384,19 +387,19 @@ class SimulationConfigGenerator:
document_text: str,
entities: List[EntityNode]
) -> str:
"""构建LLM上下文，截断到最大长度"""
# 实体摘要
"""Build the LLM context, truncated to the maximum length."""
# Entity summary.
entity_summary = self._summarize_entities(entities)
# 构建上下文
# Build the context.
context_parts = [
f"## Simulation Requirement\n{simulation_requirement}",
f"\n## Entities ({len(entities)})\n{entity_summary}",
]
current_length = sum(len(p) for p in context_parts)
remaining_length = self.MAX_CONTEXT_LENGTH - current_length - 500 # 留500字符余量
remaining_length = self.MAX_CONTEXT_LENGTH - current_length - 500 # Reserve 500-char headroom.
if remaining_length > 0 and document_text:
doc_text = document_text[:remaining_length]
@ -407,10 +410,10 @@ class SimulationConfigGenerator:
return "\n".join(context_parts)
def _summarize_entities(self, entities: List[EntityNode]) -> str:
"""生成实体摘要"""
"""Generate an entity summary."""
lines = []
# 按类型分组
# Group by type.
by_type: Dict[str, List[EntityNode]] = {}
for e in entities:
t = e.get_entity_type() or "Unknown"
@ -420,7 +423,7 @@ class SimulationConfigGenerator:
for entity_type, type_entities in by_type.items():
lines.append(f"\n### {entity_type} ({len(type_entities)})")
# 使用配置的显示数量和摘要长度
# Use configured display count and summary length.
display_count = self.ENTITIES_PER_TYPE_DISPLAY
summary_len = self.ENTITY_SUMMARY_LENGTH
for e in type_entities[:display_count]:
@ -432,7 +435,7 @@ class SimulationConfigGenerator:
return "\n".join(lines)
def _call_llm_with_retry(self, prompt: str, system_prompt: str) -> Dict[str, Any]:
"""带重试的LLM调用，包含JSON修复逻辑"""
"""LLM call with retries, including JSON repair logic."""
import re
max_attempts = 3
@ -447,25 +450,25 @@ class SimulationConfigGenerator:
{"role": "user", "content": prompt}
],
response_format={"type": "json_object"},
temperature=0.7 - (attempt * 0.1) # 每次重试降低温度
# 不设置max_tokens，让LLM自由发挥
temperature=0.7 - (attempt * 0.1) # Lower temperature on each retry.
# max_tokens is intentionally unset so the LLM can use its full budget.
)
content = response.choices[0].message.content
finish_reason = response.choices[0].finish_reason
# 检查是否被截断
# Detect truncation.
if finish_reason == 'length':
logger.warning(t("log.simulation_config.m004", attempt=attempt + 1))
content = self._fix_truncated_json(content)
# 尝试解析JSON
# Attempt to parse JSON.
try:
return json.loads(content)
except json.JSONDecodeError as e:
logger.warning(t("log.simulation_config.m005", attempt=attempt + 1, str=str(e)[:80]))
# 尝试修复JSON
# Attempt to repair the JSON.
fixed = self._try_fix_config_json(content)
if fixed:
return fixed
@ -481,36 +484,36 @@ class SimulationConfigGenerator:
raise last_error or Exception("LLM调用失败")
def _fix_truncated_json(self, content: str) -> str:
"""修复被截断的JSON"""
"""Repair truncated JSON."""
content = content.strip()
# 计算未闭合的括号
# Count unclosed brackets.
open_braces = content.count('{') - content.count('}')
open_brackets = content.count('[') - content.count(']')
# 检查是否有未闭合的字符串
# Check for an unclosed string.
if content and content[-1] not in '",}]':
content += '"'
# 闭合括号
# Close brackets.
content += ']' * open_brackets
content += '}' * open_braces
return content
def _try_fix_config_json(self, content: str) -> Optional[Dict[str, Any]]:
"""尝试修复配置JSON"""
"""Attempt to repair a configuration JSON payload."""
import re
# 修复被截断的情况
# Repair truncation first.
content = self._fix_truncated_json(content)
# 提取JSON部分
# Extract the JSON portion.
json_match = re.search(r'\{[\s\S]*\}', content)
if json_match:
json_str = json_match.group()
# 移除字符串中的换行符
# Remove line breaks from inside strings.
def fix_string(match):
s = match.group(0)
s = s.replace('\n', ' ').replace('\r', ' ')
@ -522,7 +525,7 @@ class SimulationConfigGenerator:
try:
return json.loads(json_str)
except:
# 尝试移除所有控制字符
# Strip all control characters and try again.
json_str = re.sub(r'[\x00-\x1f\x7f-\x9f]', ' ', json_str)
json_str = re.sub(r'\s+', ' ', json_str)
try:
@ -533,11 +536,11 @@ class SimulationConfigGenerator:
return None
def _generate_time_config(self, context: str, num_entities: int) -> Dict[str, Any]:
"""生成时间配置"""
# 使用配置的上下文截断长度
"""Generate the time configuration."""
# Use the configured context truncation length.
context_truncated = context[:self.TIME_CONFIG_CONTEXT_LENGTH]
# 计算最大允许值（80%的agent数）
# Compute the upper bound (90% of the agent count).
max_agents_allowed = max(1, int(num_entities * 0.9))
prompt = f"""Based on the simulation requirement below, generate a time-simulation configuration.
@ -595,10 +598,10 @@ Field guide:
return self._get_default_time_config(num_entities)
def _get_default_time_config(self, num_entities: int) -> Dict[str, Any]:
"""获取默认时间配置（中国人作息）"""
"""Return the default time configuration (Chinese daily rhythm)."""
return {
"total_simulation_hours": 72,
"minutes_per_round": 60, # 每轮1小时，加快时间流速
"minutes_per_round": 60, # 1 hour per round to speed up the simulated clock.
"agents_per_hour_min": max(1, num_entities // 15),
"agents_per_hour_max": max(5, num_entities // 5),
"peak_hours": [19, 20, 21, 22],
@ -609,12 +612,12 @@ Field guide:
}
def _parse_time_config(self, result: Dict[str, Any], num_entities: int) -> TimeSimulationConfig:
"""解析时间配置结果，并验证agents_per_hour值不超过总agent数"""
# 获取原始值
"""Parse the time-configuration result and ensure agents_per_hour values do not exceed the total agent count."""
# Pull raw values.
agents_per_hour_min = result.get("agents_per_hour_min", max(1, num_entities // 15))
agents_per_hour_max = result.get("agents_per_hour_max", max(5, num_entities // 5))
# 验证并修正：确保不超过总agent数
# Validate and correct: ensure values do not exceed the total agent count.
if agents_per_hour_min > num_entities:
logger.warning(t("log.simulation_config.m008", agents_per_hour_min=agents_per_hour_min, num_entities=num_entities))
agents_per_hour_min = max(1, num_entities // 10)
@ -623,19 +626,19 @@ Field guide:
logger.warning(t("log.simulation_config.m009", agents_per_hour_max=agents_per_hour_max, num_entities=num_entities))
agents_per_hour_max = max(agents_per_hour_min + 1, num_entities // 2)
# 确保 min < max
# Ensure min < max.
if agents_per_hour_min >= agents_per_hour_max:
agents_per_hour_min = max(1, agents_per_hour_max // 2)
logger.warning(t("log.simulation_config.m010", agents_per_hour_min=agents_per_hour_min))
return TimeSimulationConfig(
total_simulation_hours=result.get("total_simulation_hours", 72),
minutes_per_round=result.get("minutes_per_round", 60), # 默认每轮1小时
minutes_per_round=result.get("minutes_per_round", 60), # Default: 1 simulated hour per round.
agents_per_hour_min=agents_per_hour_min,
agents_per_hour_max=agents_per_hour_max,
peak_hours=result.get("peak_hours", [19, 20, 21, 22]),
off_peak_hours=result.get("off_peak_hours", [0, 1, 2, 3, 4, 5]),
off_peak_activity_multiplier=0.05, # 凌晨几乎无人
off_peak_activity_multiplier=0.05, # Overnight: almost no one online.
morning_hours=result.get("morning_hours", [6, 7, 8]),
morning_activity_multiplier=0.4,
work_hours=result.get("work_hours", list(range(9, 19))),
@ -649,14 +652,14 @@ Field guide:
simulation_requirement: str,
entities: List[EntityNode]
) -> Dict[str, Any]:
"""生成事件配置"""
# 获取可用的实体类型列表，供 LLM 参考
"""Generate the event configuration."""
# Build the list of available entity types for the LLM to reference.
entity_types_available = list(set(
e.get_entity_type() or "Unknown" for e in entities
))
# 为每种类型列出代表性实体名称
# Collect representative entity names per type.
type_examples = {}
for e in entities:
etype = e.get_entity_type() or "Unknown"
@ -670,7 +673,7 @@ Field guide:
for t, examples in type_examples.items()
])
# 使用配置的上下文截断长度
# Use the configured context truncation length.
context_truncated = context[:self.EVENT_CONFIG_CONTEXT_LENGTH]
prompt = f"""Based on the simulation requirement below, generate an event configuration.
@ -717,7 +720,7 @@ Return strict JSON (no markdown):
}
def _parse_event_config(self, result: Dict[str, Any]) -> EventConfig:
"""解析事件配置结果"""
"""Parse the event-configuration result."""
return EventConfig(
initial_posts=result.get("initial_posts", []),
scheduled_events=[],
@ -730,15 +733,15 @@ Return strict JSON (no markdown):
event_config: EventConfig,
agent_configs: List[AgentActivityConfig]
) -> EventConfig:
"""
为初始帖子分配合适的发布者 Agent
根据每个帖子的 poster_type 匹配最合适的 agent_id
"""Assign a suitable poster agent to each initial post.
Matches the most appropriate agent_id for each post based on its
poster_type.
"""
if not event_config.initial_posts:
return event_config
# 按实体类型建立 agent 索引
# Build an agent index keyed by entity type.
agents_by_type: Dict[str, List[AgentActivityConfig]] = {}
for agent in agent_configs:
etype = agent.entity_type.lower()
@ -746,7 +749,7 @@ Return strict JSON (no markdown):
agents_by_type[etype] = []
agents_by_type[etype].append(agent)
# 类型映射表（处理 LLM 可能输出的不同格式）
# Type alias map (handles the different formats the LLM might emit).
type_aliases = {
"official": ["official", "university", "governmentagency", "government"],
"university": ["university", "official"],
@ -758,7 +761,7 @@ Return strict JSON (no markdown):
"person": ["person", "student", "alumni"],
}
# 记录每种类型已使用的 agent 索引，避免重复使用同一个 agent
# Track the next agent index used per type to avoid reusing the same agent twice.
used_indices: Dict[str, int] = {}
updated_posts = []
@ -766,17 +769,17 @@ Return strict JSON (no markdown):
poster_type = post.get("poster_type", "").lower()
content = post.get("content", "")
# 尝试找到匹配的 agent
# Try to find a matching agent.
matched_agent_id = None
# 1. 直接匹配
# 1. Direct match.
if poster_type in agents_by_type:
agents = agents_by_type[poster_type]
idx = used_indices.get(poster_type, 0) % len(agents)
matched_agent_id = agents[idx].agent_id
used_indices[poster_type] = idx + 1
else:
# 2. 使用别名匹配
# 2. Match via aliases.
for alias_key, aliases in type_aliases.items():
if poster_type in aliases or alias_key == poster_type:
for alias in aliases:
@ -789,11 +792,11 @@ Return strict JSON (no markdown):
if matched_agent_id is not None:
break
# 3. 如果仍未找到，使用影响力最高的 agent
# 3. If still unresolved, fall back to the most influential agent.
if matched_agent_id is None:
logger.warning(t("log.simulation_config.m012", poster_type=poster_type))
if agent_configs:
# 按影响力排序，选择影响力最高的
# Sort by influence and pick the highest.
sorted_agents = sorted(agent_configs, key=lambda a: a.influence_weight, reverse=True)
matched_agent_id = sorted_agents[0].agent_id
else:
@ -817,9 +820,9 @@ Return strict JSON (no markdown):
start_idx: int,
simulation_requirement: str
) -> List[AgentActivityConfig]:
"""分批生成Agent配置"""
# 构建实体信息（使用配置的摘要长度）
"""Generate agent configurations in batches."""
# Build entity information (using the configured summary length).
entity_list = []
summary_len = self.AGENT_SUMMARY_LENGTH
for i, e in enumerate(entities):
@ -876,13 +879,13 @@ Return strict JSON (no markdown):
logger.warning(t("log.simulation_config.m014", e=e))
llm_configs = {}
# 构建AgentActivityConfig对象
# Build AgentActivityConfig objects.
configs = []
for i, entity in enumerate(entities):
agent_id = start_idx + i
cfg = llm_configs.get(agent_id, {})
# 如果LLM没有生成，使用规则生成
# If the LLM did not produce a config, fall back to rule-based generation.
if not cfg:
cfg = self._generate_agent_config_by_rule(entity)
@ -906,16 +909,16 @@ Return strict JSON (no markdown):
return configs
def _generate_agent_config_by_rule(self, entity: EntityNode) -> Dict[str, Any]:
"""基于规则生成单个Agent配置（中国人作息）"""
"""Rule-based generation for a single agent's configuration (Chinese daily rhythm)."""
entity_type = (entity.get_entity_type() or "Unknown").lower()
if entity_type in ["university", "governmentagency", "ngo"]:
# 官方机构：工作时间活动，低频率，高影响力
# Official institutions: active during working hours, low frequency, high influence.
return {
"activity_level": 0.2,
"posts_per_hour": 0.1,
"comments_per_hour": 0.05,
"active_hours": list(range(9, 18)), # 9:00-17:59
"active_hours": list(range(9, 18)), # 09:00-17:59
"response_delay_min": 60,
"response_delay_max": 240,
"sentiment_bias": 0.0,
@ -923,12 +926,12 @@ Return strict JSON (no markdown):
"influence_weight": 3.0
}
elif entity_type in ["mediaoutlet"]:
# 媒体：全天活动，中等频率，高影响力
# Media: active throughout the day, medium frequency, high influence.
return {
"activity_level": 0.5,
"posts_per_hour": 0.8,
"comments_per_hour": 0.3,
"active_hours": list(range(7, 24)), # 7:00-23:59
"active_hours": list(range(7, 24)), # 07:00-23:59
"response_delay_min": 5,
"response_delay_max": 30,
"sentiment_bias": 0.0,
@ -936,12 +939,12 @@ Return strict JSON (no markdown):
"influence_weight": 2.5
}
elif entity_type in ["professor", "expert", "official"]:
# 专家/教授：工作+晚间活动，中等频率
# Experts / professors: active during work and evening, medium frequency.
return {
"activity_level": 0.4,
"posts_per_hour": 0.3,
"comments_per_hour": 0.5,
"active_hours": list(range(8, 22)), # 8:00-21:59
"active_hours": list(range(8, 22)), # 08:00-21:59
"response_delay_min": 15,
"response_delay_max": 90,
"sentiment_bias": 0.0,
@ -949,12 +952,12 @@ Return strict JSON (no markdown):
"influence_weight": 2.0
}
elif entity_type in ["student"]:
# 学生：晚间为主，高频率
# Students: mostly evening, high frequency.
return {
"activity_level": 0.8,
"posts_per_hour": 0.6,
"comments_per_hour": 1.5,
"active_hours": [8, 9, 10, 11, 12, 13, 18, 19, 20, 21, 22, 23], # 上午+晚间
"active_hours": [8, 9, 10, 11, 12, 13, 18, 19, 20, 21, 22, 23], # Morning + evening.
"response_delay_min": 1,
"response_delay_max": 15,
"sentiment_bias": 0.0,
@ -962,12 +965,12 @@ Return strict JSON (no markdown):
"influence_weight": 0.8
}
elif entity_type in ["alumni"]:
# 校友：晚间为主
# Alumni: mostly evening.
return {
"activity_level": 0.6,
"posts_per_hour": 0.4,
"comments_per_hour": 0.8,
"active_hours": [12, 13, 19, 20, 21, 22, 23], # 午休+晚间
"active_hours": [12, 13, 19, 20, 21, 22, 23], # Lunch break + evening.
"response_delay_min": 5,
"response_delay_max": 30,
"sentiment_bias": 0.0,
@ -975,12 +978,12 @@ Return strict JSON (no markdown):
"influence_weight": 1.0
}
else:
# 普通人：晚间高峰
# General public: evening peak.
return {
"activity_level": 0.7,
"posts_per_hour": 0.5,
"comments_per_hour": 1.2,
"active_hours": [9, 10, 11, 12, 13, 18, 19, 20, 21, 22, 23], # 白天+晚间
"active_hours": [9, 10, 11, 12, 13, 18, 19, 20, 21, 22, 23], # Daytime + evening.
"response_delay_min": 2,
"response_delay_max": 20,
"sentiment_bias": 0.0,

262
backend/app/services/simulation_ipc.py
View File

@ -1,11 +1,12 @@
"""
模拟IPC通信模块
用于Flask后端和模拟脚本之间的进程间通信
"""Simulation IPC module.
通过文件系统实现简单的命令/响应模式：
1. Flask写入命令到 commands/ 目录
2. 模拟脚本轮询命令目录，执行命令并写入响应到 responses/ 目录
3. Flask轮询响应目录获取结果
Inter-process communication between the Flask backend and the simulation
subprocess. Implements a simple file-system command/response pattern:
1. Flask writes commands into ``commands/``.
2. The simulation script polls for commands, executes them, and writes
responses into ``responses/``.
3. Flask polls the responses directory for results.
"""
import os
@ -24,14 +25,14 @@ logger = get_logger('mirofish.simulation_ipc')
class CommandType(str, Enum):
"""命令类型"""
INTERVIEW = "interview" # 单个Agent采访
BATCH_INTERVIEW = "batch_interview" # 批量采访
CLOSE_ENV = "close_env" # 关闭环境
"""IPC command types."""
INTERVIEW = "interview" # interview a single agent
BATCH_INTERVIEW = "batch_interview" # interview multiple agents at once
CLOSE_ENV = "close_env" # tear down the environment
class CommandStatus(str, Enum):
"""命令状态"""
"""IPC command status."""
PENDING = "pending"
PROCESSING = "processing"
COMPLETED = "completed"
@ -40,12 +41,12 @@ class CommandStatus(str, Enum):
@dataclass
class IPCCommand:
"""IPC命令"""
"""A command sent over the IPC channel."""
command_id: str
command_type: CommandType
args: Dict[str, Any]
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
def to_dict(self) -> Dict[str, Any]:
return {
"command_id": self.command_id,
@ -53,7 +54,7 @@ class IPCCommand:
"args": self.args,
"timestamp": self.timestamp
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> 'IPCCommand':
return cls(
@ -66,13 +67,13 @@ class IPCCommand:
@dataclass
class IPCResponse:
"""IPC响应"""
"""A response returned over the IPC channel."""
command_id: str
status: CommandStatus
result: Optional[Dict[str, Any]] = None
error: Optional[str] = None
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
def to_dict(self) -> Dict[str, Any]:
return {
"command_id": self.command_id,
@ -81,7 +82,7 @@ class IPCResponse:
"error": self.error,
"timestamp": self.timestamp
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> 'IPCResponse':
return cls(
@ -94,27 +95,25 @@ class IPCResponse:
class SimulationIPCClient:
"""IPC client used by the Flask side.
Sends commands to the simulation process and waits for responses.
"""
模拟IPC客户端（Flask端使用）
用于向模拟进程发送命令并等待响应
"""
def __init__(self, simulation_dir: str):
"""
初始化IPC客户端
"""Initialize the IPC client.
Args:
simulation_dir: 模拟数据目录
simulation_dir: Directory holding the simulation's IPC files.
"""
self.simulation_dir = simulation_dir
self.commands_dir = os.path.join(simulation_dir, "ipc_commands")
self.responses_dir = os.path.join(simulation_dir, "ipc_responses")
# 确保目录存在
# Ensure both directories exist before use.
os.makedirs(self.commands_dir, exist_ok=True)
os.makedirs(self.responses_dir, exist_ok=True)
def send_command(
self,
command_type: CommandType,
@ -122,20 +121,19 @@ class SimulationIPCClient:
timeout: float = 60.0,
poll_interval: float = 0.5
) -> IPCResponse:
"""
发送命令并等待响应
"""Send a command and wait for the response.
Args:
command_type: 命令类型
args: 命令参数
timeout: 超时时间（秒）
poll_interval: 轮询间隔（秒）
command_type: Command type to send.
args: Command arguments.
timeout: Timeout in seconds.
poll_interval: Polling interval in seconds.
Returns:
IPCResponse
The ``IPCResponse``.
Raises:
TimeoutError: 等待响应超时
TimeoutError: When no response arrives before ``timeout``.
"""
command_id = str(uuid.uuid4())
command = IPCCommand(
@ -143,50 +141,50 @@ class SimulationIPCClient:
command_type=command_type,
args=args
)
# 写入命令文件
# Write the command file.
command_file = os.path.join(self.commands_dir, f"{command_id}.json")
with open(command_file, 'w', encoding='utf-8') as f:
json.dump(command.to_dict(), f, ensure_ascii=False, indent=2)
logger.info(t("log.simulation_ipc.m001", command_type=command_type.value, command_id=command_id))
# 等待响应
# Poll for the response file.
response_file = os.path.join(self.responses_dir, f"{command_id}.json")
start_time = time.time()
while time.time() - start_time < timeout:
if os.path.exists(response_file):
try:
with open(response_file, 'r', encoding='utf-8') as f:
response_data = json.load(f)
response = IPCResponse.from_dict(response_data)
# 清理命令和响应文件
# Clean up command and response files after successful read.
try:
os.remove(command_file)
os.remove(response_file)
except OSError:
pass
logger.info(t("log.simulation_ipc.m002", command_id=command_id, response=response.status.value))
return response
except (json.JSONDecodeError, KeyError) as e:
logger.warning(t("log.simulation_ipc.m003", e=e))
time.sleep(poll_interval)
# 超时
# Timed out waiting for the response.
logger.error(t("log.simulation_ipc.m004", command_id=command_id))
# 清理命令文件
# Clean up the unanswered command file.
try:
os.remove(command_file)
except OSError:
pass
raise TimeoutError(f"等待命令响应超时 ({timeout}秒)")
def send_interview(
self,
agent_id: int,
@ -194,20 +192,19 @@ class SimulationIPCClient:
platform: str = None,
timeout: float = 60.0
) -> IPCResponse:
"""
发送单个Agent采访命令
"""Send a single-agent interview command.
Args:
agent_id: Agent ID
prompt: 采访问题
platform: 指定平台（可选）
- "twitter": 只采访Twitter平台
- "reddit": 只采访Reddit平台
- None: 双平台模拟时同时采访两个平台，单平台模拟时采访该平台
timeout: 超时时间
agent_id: Agent id to interview.
prompt: Interview question.
platform: Optional platform selector.
- ``"twitter"``: interview only on Twitter.
- ``"reddit"``: interview only on Reddit.
- ``None``: dual-platform if applicable, else the single active platform.
timeout: Timeout in seconds.
Returns:
IPCResponse，result字段包含采访结果
``IPCResponse`` whose ``result`` carries the interview response.
"""
args = {
"agent_id": agent_id,
@ -215,69 +212,66 @@ class SimulationIPCClient:
}
if platform:
args["platform"] = platform
return self.send_command(
command_type=CommandType.INTERVIEW,
args=args,
timeout=timeout
)
def send_batch_interview(
self,
interviews: List[Dict[str, Any]],
platform: str = None,
timeout: float = 120.0
) -> IPCResponse:
"""
发送批量采访命令
"""Send a batched interview command.
Args:
interviews: 采访列表，每个元素包含 {"agent_id": int, "prompt": str, "platform": str(可选)}
platform: 默认平台（可选，会被每个采访项的platform覆盖）
- "twitter": 默认只采访Twitter平台
- "reddit": 默认只采访Reddit平台
- None: 双平台模拟时每个Agent同时采访两个平台
timeout: 超时时间
interviews: List of items shaped ``{"agent_id": int, "prompt": str, "platform": str?}``.
platform: Default platform; per-item ``platform`` overrides this.
- ``"twitter"``: default to Twitter.
- ``"reddit"``: default to Reddit.
- ``None``: dual-platform interview when applicable.
timeout: Timeout in seconds.
Returns:
IPCResponse，result字段包含所有采访结果
``IPCResponse`` whose ``result`` carries every interview response.
"""
args = {"interviews": interviews}
if platform:
args["platform"] = platform
return self.send_command(
command_type=CommandType.BATCH_INTERVIEW,
args=args,
timeout=timeout
)
def send_close_env(self, timeout: float = 30.0) -> IPCResponse:
"""
发送关闭环境命令
"""Send a tear-down-environment command.
Args:
timeout: 超时时间
timeout: Timeout in seconds.
Returns:
IPCResponse
``IPCResponse``.
"""
return self.send_command(
command_type=CommandType.CLOSE_ENV,
args={},
timeout=timeout
)
def check_env_alive(self) -> bool:
"""
检查模拟环境是否存活
通过检查 env_status.json 文件来判断
"""Return ``True`` if the simulation environment reports as alive.
Reads ``env_status.json`` written by the IPC server side.
"""
status_file = os.path.join(self.simulation_dir, "env_status.json")
if not os.path.exists(status_file):
return False
try:
with open(status_file, 'r', encoding='utf-8') as f:
status = json.load(f)
@ -287,68 +281,65 @@ class SimulationIPCClient:
class SimulationIPCServer:
"""IPC server used by the simulation script.
Polls the commands directory, executes commands, and writes responses.
"""
模拟IPC服务器（模拟脚本端使用）
轮询命令目录，执行命令并返回响应
"""
def __init__(self, simulation_dir: str):
"""
初始化IPC服务器
"""Initialize the IPC server.
Args:
simulation_dir: 模拟数据目录
simulation_dir: Directory holding the simulation's IPC files.
"""
self.simulation_dir = simulation_dir
self.commands_dir = os.path.join(simulation_dir, "ipc_commands")
self.responses_dir = os.path.join(simulation_dir, "ipc_responses")
# 确保目录存在
# Ensure both directories exist before use.
os.makedirs(self.commands_dir, exist_ok=True)
os.makedirs(self.responses_dir, exist_ok=True)
# 环境状态
# Server-running flag.
self._running = False
def start(self):
"""标记服务器为运行状态"""
"""Mark the server as alive and persist the state."""
self._running = True
self._update_env_status("alive")
def stop(self):
"""标记服务器为停止状态"""
"""Mark the server as stopped and persist the state."""
self._running = False
self._update_env_status("stopped")
def _update_env_status(self, status: str):
"""更新环境状态文件"""
"""Update the persistent environment-status file."""
status_file = os.path.join(self.simulation_dir, "env_status.json")
with open(status_file, 'w', encoding='utf-8') as f:
json.dump({
"status": status,
"timestamp": datetime.now().isoformat()
}, f, ensure_ascii=False, indent=2)
def poll_commands(self) -> Optional[IPCCommand]:
"""
轮询命令目录，返回第一个待处理的命令
"""Poll the commands directory and return the next pending command.
Returns:
IPCCommand 或 None
``IPCCommand`` or ``None`` if no pending commands remain.
"""
if not os.path.exists(self.commands_dir):
return None
# 按时间排序获取命令文件
# Sort by mtime so we process commands in arrival order.
command_files = []
for filename in os.listdir(self.commands_dir):
if filename.endswith('.json'):
filepath = os.path.join(self.commands_dir, filename)
command_files.append((filepath, os.path.getmtime(filepath)))
command_files.sort(key=lambda x: x[1])
for filepath, _ in command_files:
try:
with open(filepath, 'r', encoding='utf-8') as f:
@ -357,37 +348,36 @@ class SimulationIPCServer:
except (json.JSONDecodeError, KeyError, OSError) as e:
logger.warning(t("log.simulation_ipc.m005", filepath=filepath, e=e))
continue
return None
def send_response(self, response: IPCResponse):
"""
发送响应
"""Write a response file.
Args:
response: IPC响应
response: The response to send.
"""
response_file = os.path.join(self.responses_dir, f"{response.command_id}.json")
with open(response_file, 'w', encoding='utf-8') as f:
json.dump(response.to_dict(), f, ensure_ascii=False, indent=2)
# 删除命令文件
# Delete the matching command file.
command_file = os.path.join(self.commands_dir, f"{response.command_id}.json")
try:
os.remove(command_file)
except OSError:
pass
def send_success(self, command_id: str, result: Dict[str, Any]):
"""发送成功响应"""
"""Send a success response."""
self.send_response(IPCResponse(
command_id=command_id,
status=CommandStatus.COMPLETED,
result=result
))
def send_error(self, command_id: str, error: str):
"""发送错误响应"""
"""Send a failure response."""
self.send_response(IPCResponse(
command_id=command_id,
status=CommandStatus.FAILED,

316
backend/app/services/simulation_manager.py
View File

@ -1,7 +1,7 @@
"""
OASIS模拟管理器
管理Twitter和Reddit双平台并行模拟
使用预设脚本 + LLM智能生成配置参数
"""OASIS simulation manager.
Drives parallel Twitter + Reddit simulations using preset scripts plus
LLM-generated configuration parameters.
"""
import os
@ -23,60 +23,60 @@ logger = get_logger('mirofish.simulation')
class SimulationStatus(str, Enum):
"""模拟状态"""
"""Simulation lifecycle status."""
CREATED = "created"
PREPARING = "preparing"
READY = "ready"
RUNNING = "running"
PAUSED = "paused"
STOPPED = "stopped" # 模拟被手动停止
COMPLETED = "completed" # 模拟自然完成
STOPPED = "stopped" # manually stopped
COMPLETED = "completed" # finished naturally
FAILED = "failed"
class PlatformType(str, Enum):
"""平台类型"""
"""Simulated platform types."""
TWITTER = "twitter"
REDDIT = "reddit"
@dataclass
class SimulationState:
"""模拟状态"""
"""In-memory + persisted state for a single simulation."""
simulation_id: str
project_id: str
graph_id: str
# 平台启用状态
# Per-platform enable flags.
enable_twitter: bool = True
enable_reddit: bool = True
# 状态
# Lifecycle status.
status: SimulationStatus = SimulationStatus.CREATED
# 准备阶段数据
# Counters captured during the prepare phase.
entities_count: int = 0
profiles_count: int = 0
entity_types: List[str] = field(default_factory=list)
# 配置生成信息
# Information about the auto-generated config.
config_generated: bool = False
config_reasoning: str = ""
# 运行时数据
# Runtime data.
current_round: int = 0
twitter_status: str = "not_started"
reddit_status: str = "not_started"
# 时间戳
# Timestamps.
created_at: str = field(default_factory=lambda: datetime.now().isoformat())
updated_at: str = field(default_factory=lambda: datetime.now().isoformat())
# 错误信息
# Error message when status == FAILED.
error: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
"""完整状态字典（内部使用）"""
"""Full state dict (used for persistence and internal callers)."""
return {
"simulation_id": self.simulation_id,
"project_id": self.project_id,
@ -96,9 +96,9 @@ class SimulationState:
"updated_at": self.updated_at,
"error": self.error,
}
def to_simple_dict(self) -> Dict[str, Any]:
"""简化状态字典（API返回使用）"""
"""Simplified state dict (used for API responses)."""
return {
"simulation_id": self.simulation_id,
"project_id": self.project_id,
@ -113,61 +113,60 @@ class SimulationState:
class SimulationManager:
"""Simulation manager.
Core responsibilities:
1. Read entities from the Zep graph and filter to the configured types.
2. Generate OASIS agent profiles per entity.
3. Use the LLM to generate simulation configuration parameters.
4. Materialize the files the preset scripts expect.
"""
模拟管理器
核心功能：
1. 从Zep图谱读取实体并过滤
2. 生成OASIS Agent Profile
3. 使用LLM智能生成模拟配置参数
4. 准备预设脚本所需的所有文件
"""
# 模拟数据存储目录
# Root directory for persisted simulation data.
SIMULATION_DATA_DIR = os.path.join(
os.path.dirname(__file__),
os.path.dirname(__file__),
'../../uploads/simulations'
)
def __init__(self):
# 确保目录存在
# Ensure the simulation data directory exists.
os.makedirs(self.SIMULATION_DATA_DIR, exist_ok=True)
# 内存中的模拟状态缓存
# In-memory cache of simulation state objects.
self._simulations: Dict[str, SimulationState] = {}
def _get_simulation_dir(self, simulation_id: str) -> str:
"""获取模拟数据目录"""
"""Return the on-disk directory for a simulation, creating if missing."""
sim_dir = os.path.join(self.SIMULATION_DATA_DIR, simulation_id)
os.makedirs(sim_dir, exist_ok=True)
return sim_dir
def _save_simulation_state(self, state: SimulationState):
"""保存模拟状态到文件"""
"""Persist a simulation state to disk and update the cache."""
sim_dir = self._get_simulation_dir(state.simulation_id)
state_file = os.path.join(sim_dir, "state.json")
state.updated_at = datetime.now().isoformat()
with open(state_file, 'w', encoding='utf-8') as f:
json.dump(state.to_dict(), f, ensure_ascii=False, indent=2)
self._simulations[state.simulation_id] = state
def _load_simulation_state(self, simulation_id: str) -> Optional[SimulationState]:
"""从文件加载模拟状态"""
"""Load a simulation state from disk (or cache) by id."""
if simulation_id in self._simulations:
return self._simulations[simulation_id]
sim_dir = self._get_simulation_dir(simulation_id)
state_file = os.path.join(sim_dir, "state.json")
if not os.path.exists(state_file):
return None
with open(state_file, 'r', encoding='utf-8') as f:
data = json.load(f)
state = SimulationState(
simulation_id=simulation_id,
project_id=data.get("project_id", ""),
@ -187,10 +186,10 @@ class SimulationManager:
updated_at=data.get("updated_at", datetime.now().isoformat()),
error=data.get("error"),
)
self._simulations[simulation_id] = state
return state
def create_simulation(
self,
project_id: str,
@ -198,21 +197,20 @@ class SimulationManager:
enable_twitter: bool = True,
enable_reddit: bool = True,
) -> SimulationState:
"""
创建新的模拟
"""Create a new simulation in the ``CREATED`` state.
Args:
project_id: 项目ID
graph_id: Zep图谱ID
enable_twitter: 是否启用Twitter模拟
enable_reddit: 是否启用Reddit模拟
project_id: Owning project id.
graph_id: Source Zep graph id.
enable_twitter: When ``True``, the Twitter simulation runs.
enable_reddit: When ``True``, the Reddit simulation runs.
Returns:
SimulationState
The created ``SimulationState``.
"""
import uuid
simulation_id = f"sim_{uuid.uuid4().hex[:12]}"
state = SimulationState(
simulation_id=simulation_id,
project_id=project_id,
@ -221,12 +219,12 @@ class SimulationManager:
enable_reddit=enable_reddit,
status=SimulationStatus.CREATED,
)
self._save_simulation_state(state)
logger.info(t("log.simulation_manager.m001", simulation_id=simulation_id, project_id=project_id, graph_id=graph_id))
return state
def prepare_simulation(
self,
simulation_id: str,
@ -237,56 +235,55 @@ class SimulationManager:
progress_callback: Optional[callable] = None,
parallel_profile_count: int = 3
) -> SimulationState:
"""
准备模拟环境（全程自动化）
步骤：
1. 从Zep图谱读取并过滤实体
2. 为每个实体生成OASIS Agent Profile（可选LLM增强，支持并行）
3. 使用LLM智能生成模拟配置参数（时间、活跃度、发言频率等）
4. 保存配置文件和Profile文件
5. 复制预设脚本到模拟目录
"""Prepare the simulation environment end-to-end.
Steps:
1. Read and filter entities from the graph.
2. Generate OASIS agent profiles (optional LLM enrichment, parallel-capable).
3. Use the LLM to produce simulation parameters (timing, activity, posting frequency).
4. Save the configuration and profile files.
5. Copy preset scripts into the simulation directory.
Args:
simulation_id: 模拟ID
simulation_requirement: 模拟需求描述（用于LLM生成配置）
document_text: 原始文档内容（用于LLM理解背景）
defined_entity_types: 预定义的实体类型（可选）
use_llm_for_profiles: 是否使用LLM生成详细人设
progress_callback: 进度回调函数 (stage, progress, message)
parallel_profile_count: 并行生成人设的数量，默认3
simulation_id: Simulation id.
simulation_requirement: Free-text description of the simulation goal.
document_text: Raw source document text passed to the LLM for context.
defined_entity_types: Optional list of allowed entity types.
use_llm_for_profiles: When ``True``, enrich profiles via the LLM.
progress_callback: Optional callback ``(stage, progress, message, **extras)``.
parallel_profile_count: Number of profile generations to run in parallel.
Returns:
SimulationState
The updated ``SimulationState``.
"""
state = self._load_simulation_state(simulation_id)
if not state:
raise ValueError(f"模拟不存在: {simulation_id}")
try:
state.status = SimulationStatus.PREPARING
self._save_simulation_state(state)
sim_dir = self._get_simulation_dir(simulation_id)
# ========== 阶段1: 读取并过滤实体 ==========
# ========== Stage 1: read and filter entities ==========
if progress_callback:
progress_callback("reading", 0, t('progress.connectingZepGraph'))
reader = ZepEntityReader()
if progress_callback:
progress_callback("reading", 30, t('progress.readingNodeData'))
filtered = reader.filter_defined_entities(
graph_id=state.graph_id,
defined_entity_types=defined_entity_types,
enrich_with_edges=True
)
state.entities_count = filtered.filtered_count
state.entity_types = list(filtered.entity_types)
if progress_callback:
progress_callback(
"reading", 100,
@ -294,16 +291,16 @@ class SimulationManager:
current=filtered.filtered_count,
total=filtered.filtered_count
)
if filtered.filtered_count == 0:
state.status = SimulationStatus.FAILED
state.error = "没有找到符合条件的实体，请检查图谱是否正确构建"
self._save_simulation_state(state)
return state
# ========== 阶段2: 生成Agent Profile ==========
# ========== Stage 2: generate agent profiles ==========
total_entities = len(filtered.entities)
if progress_callback:
progress_callback(
"generating_profiles", 0,
@ -311,22 +308,22 @@ class SimulationManager:
current=0,
total=total_entities
)
# 传入graph_id以启用Zep检索功能，获取更丰富的上下文
# Pass the graph_id so the generator can use Zep retrieval for richer context.
generator = OasisProfileGenerator(graph_id=state.graph_id)
def profile_progress(current, total, msg):
if progress_callback:
progress_callback(
"generating_profiles",
int(current / total * 100),
"generating_profiles",
int(current / total * 100),
msg,
current=current,
total=total,
item_name=msg
)
# 设置实时保存的文件路径（优先使用 Reddit JSON 格式）
# Configure the realtime save target (prefer Reddit JSON if Reddit is enabled).
realtime_output_path = None
realtime_platform = "reddit"
if state.enable_reddit:
@ -335,21 +332,21 @@ class SimulationManager:
elif state.enable_twitter:
realtime_output_path = os.path.join(sim_dir, "twitter_profiles.csv")
realtime_platform = "twitter"
profiles = generator.generate_profiles_from_entities(
entities=filtered.entities,
use_llm=use_llm_for_profiles,
progress_callback=profile_progress,
graph_id=state.graph_id, # 传入graph_id用于Zep检索
parallel_count=parallel_profile_count, # 并行生成数量
realtime_output_path=realtime_output_path, # 实时保存路径
output_platform=realtime_platform # 输出格式
graph_id=state.graph_id, # used for Zep retrieval enrichment
parallel_count=parallel_profile_count,
realtime_output_path=realtime_output_path,
output_platform=realtime_platform
)
state.profiles_count = len(profiles)
# 保存Profile文件（注意：Twitter使用CSV格式，Reddit使用JSON格式）
# Reddit 已经在生成过程中实时保存了，这里再保存一次确保完整性
# Save profile files. Reddit also writes JSON during generation; this is
# a final consistency write. Twitter requires CSV per OASIS conventions.
if progress_callback:
progress_callback(
"generating_profiles", 95,
@ -357,22 +354,22 @@ class SimulationManager:
current=total_entities,
total=total_entities
)
if state.enable_reddit:
generator.save_profiles(
profiles=profiles,
file_path=os.path.join(sim_dir, "reddit_profiles.json"),
platform="reddit"
)
if state.enable_twitter:
# Twitter使用CSV格式！这是OASIS的要求
# Twitter uses CSV format — required by OASIS.
generator.save_profiles(
profiles=profiles,
file_path=os.path.join(sim_dir, "twitter_profiles.csv"),
platform="twitter"
)
if progress_callback:
progress_callback(
"generating_profiles", 100,
@ -380,8 +377,8 @@ class SimulationManager:
current=len(profiles),
total=len(profiles)
)
# ========== 阶段3: LLM智能生成模拟配置 ==========
# ========== Stage 3: LLM-driven simulation config ==========
if progress_callback:
progress_callback(
"generating_config", 0,
@ -389,9 +386,9 @@ class SimulationManager:
current=0,
total=3
)
config_generator = SimulationConfigGenerator()
if progress_callback:
progress_callback(
"generating_config", 30,
@ -399,7 +396,7 @@ class SimulationManager:
current=1,
total=3
)
sim_params = config_generator.generate_config(
simulation_id=simulation_id,
project_id=state.project_id,
@ -410,7 +407,7 @@ class SimulationManager:
enable_twitter=state.enable_twitter,
enable_reddit=state.enable_reddit
)
if progress_callback:
progress_callback(
"generating_config", 70,
@ -418,15 +415,15 @@ class SimulationManager:
current=2,
total=3
)
# 保存配置文件
# Save the configuration file.
config_path = os.path.join(sim_dir, "simulation_config.json")
with open(config_path, 'w', encoding='utf-8') as f:
f.write(sim_params.to_json())
state.config_generated = True
state.config_reasoning = sim_params.generation_reasoning
if progress_callback:
progress_callback(
"generating_config", 100,
@ -434,18 +431,17 @@ class SimulationManager:
current=3,
total=3
)
# 注意：运行脚本保留在 backend/scripts/ 目录，不再复制到模拟目录
# 启动模拟时，simulation_runner 会从 scripts/ 目录运行脚本
# 更新状态
# The runtime scripts now live under backend/scripts/; we no longer copy
# them per-simulation. simulation_runner invokes them in place.
state.status = SimulationStatus.READY
self._save_simulation_state(state)
logger.info(t("log.simulation_manager.m002", simulation_id=simulation_id, state=state.entities_count, state_2=state.profiles_count))
return state
except Exception as e:
logger.error(t("log.simulation_manager.m003", simulation_id=simulation_id, str=str(e)))
import traceback
@ -454,61 +450,61 @@ class SimulationManager:
state.error = str(e)
self._save_simulation_state(state)
raise
def get_simulation(self, simulation_id: str) -> Optional[SimulationState]:
"""获取模拟状态"""
"""Return the simulation's state, or ``None`` if unknown."""
return self._load_simulation_state(simulation_id)
def list_simulations(self, project_id: Optional[str] = None) -> List[SimulationState]:
"""列出所有模拟"""
"""List all simulations, optionally filtered by ``project_id``."""
simulations = []
if os.path.exists(self.SIMULATION_DATA_DIR):
for sim_id in os.listdir(self.SIMULATION_DATA_DIR):
# 跳过隐藏文件（如 .DS_Store）和非目录文件
# Skip dotfiles (e.g. .DS_Store) and non-directories.
sim_path = os.path.join(self.SIMULATION_DATA_DIR, sim_id)
if sim_id.startswith('.') or not os.path.isdir(sim_path):
continue
state = self._load_simulation_state(sim_id)
if state:
if project_id is None or state.project_id == project_id:
simulations.append(state)
return simulations
def get_profiles(self, simulation_id: str, platform: str = "reddit") -> List[Dict[str, Any]]:
"""获取模拟的Agent Profile"""
"""Return the persisted agent profiles for a platform."""
state = self._load_simulation_state(simulation_id)
if not state:
raise ValueError(f"模拟不存在: {simulation_id}")
sim_dir = self._get_simulation_dir(simulation_id)
profile_path = os.path.join(sim_dir, f"{platform}_profiles.json")
if not os.path.exists(profile_path):
return []
with open(profile_path, 'r', encoding='utf-8') as f:
return json.load(f)
def get_simulation_config(self, simulation_id: str) -> Optional[Dict[str, Any]]:
"""获取模拟配置"""
"""Return the persisted simulation config dict, or ``None`` if absent."""
sim_dir = self._get_simulation_dir(simulation_id)
config_path = os.path.join(sim_dir, "simulation_config.json")
if not os.path.exists(config_path):
return None
with open(config_path, 'r', encoding='utf-8') as f:
return json.load(f)
def get_run_instructions(self, simulation_id: str) -> Dict[str, str]:
"""获取运行说明"""
"""Return shell commands and instructions to launch the simulation manually."""
sim_dir = self._get_simulation_dir(simulation_id)
config_path = os.path.join(sim_dir, "simulation_config.json")
scripts_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../scripts'))
return {
"simulation_dir": sim_dir,
"scripts_dir": scripts_dir,

763
backend/app/services/simulation_runner.py
View File

File diff suppressed because it is too large Load Diff

62
backend/app/services/text_processor.py
View File

@ -1,68 +1,64 @@
"""
文本处理服务
"""
"""Text processing service."""
from typing import List, Optional
from ..utils.file_parser import FileParser, split_text_into_chunks
class TextProcessor:
"""文本处理器"""
"""Facade for the text-extraction and chunking pipeline."""
@staticmethod
def extract_from_files(file_paths: List[str]) -> str:
"""从多个文件提取文本"""
"""Extract and concatenate text from multiple files."""
return FileParser.extract_from_multiple(file_paths)
@staticmethod
def split_text(
text: str,
chunk_size: int = 500,
overlap: int = 50
) -> List[str]:
"""
分割文本
"""Split text into chunks.
Args:
text: 原始文本
chunk_size: 块大小
overlap: 重叠大小
text: The source text.
chunk_size: Target characters per chunk.
overlap: Overlap between consecutive chunks.
Returns:
文本块列表
A list of chunk strings.
"""
return split_text_into_chunks(text, chunk_size, overlap)
@staticmethod
def preprocess_text(text: str) -> str:
"""
预处理文本
- 移除多余空白
- 标准化换行
"""Pre-process text by normalizing whitespace and line endings.
- Collapse runs of blank lines to at most two newlines.
- Normalize line endings to ``\\n``.
- Strip leading/trailing whitespace from each line.
Args:
text: 原始文本
text: The source text.
Returns:
处理后的文本
The cleaned text.
"""
import re
# 标准化换行
text = text.replace('\r\n', '\n').replace('\r', '\n')
# 移除连续空行（保留最多两个换行）
# Collapse 3+ consecutive newlines down to a blank-line separator.
text = re.sub(r'\n{3,}', '\n\n', text)
# 移除行首行尾空白
lines = [line.strip() for line in text.split('\n')]
text = '\n'.join(lines)
return text.strip()
@staticmethod
def get_text_stats(text: str) -> dict:
"""获取文本统计信息"""
"""Return basic text statistics: total chars, lines, and words."""
return {
"total_chars": len(text),
"total_lines": text.count('\n') + 1,

238
backend/app/services/zep_entity_reader.py
View File

@ -1,6 +1,7 @@
"""
Zep实体读取与过滤服务
从Zep图谱中读取节点，筛选出符合预定义实体类型的节点
"""Zep entity reader and filter service.
Reads nodes from a Zep graph and filters down to those that match a
predefined ontology of entity types.
"""
import time
@ -16,23 +17,23 @@ from ..utils.locale import t
logger = get_logger('mirofish.zep_entity_reader')
# 用于泛型返回类型
# Generic return-type variable.
T = TypeVar('T')
@dataclass
class EntityNode:
"""实体节点数据结构"""
"""In-memory representation of an entity node from the graph."""
uuid: str
name: str
labels: List[str]
summary: str
attributes: Dict[str, Any]
# 相关的边信息
# Edges connected to this entity.
related_edges: List[Dict[str, Any]] = field(default_factory=list)
# 相关的其他节点信息
# Other nodes connected through related edges.
related_nodes: List[Dict[str, Any]] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
return {
"uuid": self.uuid,
@ -43,9 +44,9 @@ class EntityNode:
"related_edges": self.related_edges,
"related_nodes": self.related_nodes,
}
def get_entity_type(self) -> Optional[str]:
"""获取实体类型（排除默认的Entity标签）"""
"""Return the first non-default label, or ``None`` if only defaults are present."""
for label in self.labels:
if label not in ["Entity", "Node"]:
return label
@ -54,12 +55,12 @@ class EntityNode:
@dataclass
class FilteredEntities:
"""过滤后的实体集合"""
"""Result of a filter pass over the graph: matching entities + counts."""
entities: List[EntityNode]
entity_types: Set[str]
total_count: int
filtered_count: int
def to_dict(self) -> Dict[str, Any]:
return {
"entities": [e.to_dict() for e in self.entities],
@ -70,40 +71,38 @@ class FilteredEntities:
class ZepEntityReader:
"""Read entities from a Zep graph and filter to ontology-defined types.
Capabilities:
1. Read all nodes from the graph.
2. Keep nodes whose labels include something other than the default ``Entity``.
3. Optionally enrich each entity with its connected edges and neighboring nodes.
"""
Zep实体读取与过滤服务
主要功能：
1. 从Zep图谱读取所有节点
2. 筛选出符合预定义实体类型的节点（Labels不只是Entity的节点）
3. 获取每个实体的相关边和关联节点信息
"""
def __init__(self, api_key: Optional[str] = None):
self.client = GraphitiAdapter()
def _call_with_retry(
self,
func: Callable[[], T],
self,
func: Callable[[], T],
operation_name: str,
max_retries: int = 3,
initial_delay: float = 2.0
) -> T:
"""
带重试机制的Zep API调用
"""Call a Zep API function with retry on failure.
Args:
func: 要执行的函数（无参数的lambda或callable）
operation_name: 操作名称，用于日志
max_retries: 最大重试次数（默认3次，即最多尝试3次）
initial_delay: 初始延迟秒数
func: A zero-argument callable performing the request.
operation_name: Operation label used in log output.
max_retries: Maximum number of attempts (default 3 — i.e. up to 3 tries total).
initial_delay: Initial delay between retries in seconds.
Returns:
API调用结果
The return value of ``func``.
"""
last_exception = None
delay = initial_delay
for attempt in range(max_retries):
try:
return func()
@ -114,21 +113,20 @@ class ZepEntityReader:
t("log.zep_entity_reader.m001", operation_name=operation_name, attempt=attempt + 1, str=str(e)[:100], delay=delay)
)
time.sleep(delay)
delay *= 2 # 指数退避
delay *= 2 # exponential backoff
else:
logger.error(t("log.zep_entity_reader.m002", operation_name=operation_name, max_retries=max_retries, str=str(e)))
raise last_exception
def get_all_nodes(self, graph_id: str) -> List[Dict[str, Any]]:
"""
获取图谱的所有节点（分页获取）
"""Return every node in the graph (paginated under the hood).
Args:
graph_id: 图谱ID
graph_id: Graph identifier.
Returns:
节点列表
A list of node dicts.
"""
logger.info(t("log.zep_entity_reader.m003", graph_id=graph_id))
@ -148,14 +146,13 @@ class ZepEntityReader:
return nodes_data
def get_all_edges(self, graph_id: str) -> List[Dict[str, Any]]:
"""
获取图谱的所有边（分页获取）
"""Return every edge in the graph (paginated under the hood).
Args:
graph_id: 图谱ID
graph_id: Graph identifier.
Returns:
边列表
A list of edge dicts.
"""
logger.info(t("log.zep_entity_reader.m005", graph_id=graph_id))
@ -174,24 +171,23 @@ class ZepEntityReader:
logger.info(t("log.zep_entity_reader.m006", len=len(edges_data)))
return edges_data
def get_node_edges(self, node_uuid: str) -> List[Dict[str, Any]]:
"""
获取指定节点的所有相关边（带重试机制）
"""Return every edge connected to the given node (with retry).
Args:
node_uuid: 节点UUID
node_uuid: Node UUID.
Returns:
边列表
A list of edge dicts.
"""
try:
# 使用重试机制调用Zep API
# Wrap the API call in retry logic.
edges = self._call_with_retry(
func=lambda: self.client.graph.node.get_entity_edges(node_uuid=node_uuid),
operation_name=f"获取节点边(node={node_uuid[:8]}...)"
)
edges_data = []
for edge in edges:
edges_data.append({
@ -202,32 +198,31 @@ class ZepEntityReader:
"target_node_uuid": edge.target_node_uuid,
"attributes": edge.attributes or {},
})
return edges_data
except Exception as e:
logger.warning(t("log.zep_entity_reader.m007", node_uuid=node_uuid, str=str(e)))
return []
def filter_defined_entities(
self,
self,
graph_id: str,
defined_entity_types: Optional[List[str]] = None,
enrich_with_edges: bool = True
) -> FilteredEntities:
"""
筛选出符合预定义实体类型的节点
筛选逻辑：
- 如果节点的Labels只有一个"Entity"，说明这个实体不符合我们预定义的类型，跳过
- 如果节点的Labels包含除"Entity"和"Node"之外的标签，说明符合预定义类型，保留
"""Filter nodes down to entities matching the predefined ontology types.
Filtering rules:
- Skip nodes whose only label is ``Entity`` (uncategorized).
- Keep nodes whose labels include anything other than ``Entity`` and ``Node``.
Args:
graph_id: 图谱ID
defined_entity_types: 预定义的实体类型列表（可选，如果提供则只保留这些类型）
enrich_with_edges: 是否获取每个实体的相关边信息
graph_id: Graph identifier.
defined_entity_types: Optional allow-list; when provided, only matching types are kept.
enrich_with_edges: When ``True``, populate related_edges and related_nodes.
Returns:
FilteredEntities: 过滤后的实体集合
A ``FilteredEntities`` summary.
"""
logger.info(t("log.zep_entity_reader.m008", graph_id=graph_id))
@ -243,7 +238,7 @@ class ZepEntityReader:
except Exception:
pass
# 获取所有节点
# Read every node from the graph.
all_nodes = self.get_all_nodes(graph_id)
total_count = len(all_nodes)
@ -259,27 +254,27 @@ class ZepEntityReader:
if entity_type != "Entity":
node["labels"] = [entity_type] + labels
# 获取所有边（用于后续关联查找）
# Read every edge so we can enrich entities later.
all_edges = self.get_all_edges(graph_id) if enrich_with_edges else []
# 构建节点UUID到节点数据的映射
# uuid -> node-data map for fast lookup.
node_map = {n["uuid"]: n for n in all_nodes}
# 筛选符合条件的实体
# Filter to entities that match the criteria.
filtered_entities = []
entity_types_found = set()
for node in all_nodes:
labels = node.get("labels", [])
# 筛选逻辑：Labels必须包含除"Entity"和"Node"之外的标签
# Filtering rule: labels must contain something other than the defaults.
custom_labels = [l for l in labels if l not in ["Entity", "Node"]]
if not custom_labels:
# 只有默认标签，跳过
# Only default labels — skip.
continue
# 如果指定了预定义类型，检查是否匹配
# When a predefined-type list is supplied, require a match against it.
if defined_entity_types:
matching_labels = [l for l in custom_labels if l in defined_entity_types]
if not matching_labels:
@ -287,10 +282,9 @@ class ZepEntityReader:
entity_type = matching_labels[0]
else:
entity_type = custom_labels[0]
entity_types_found.add(entity_type)
# 创建实体节点对象
entity = EntityNode(
uuid=node["uuid"],
name=node["name"],
@ -298,12 +292,12 @@ class ZepEntityReader:
summary=node["summary"],
attributes=node["attributes"],
)
# 获取相关边和节点
# Enrich with related edges and neighboring nodes.
if enrich_with_edges:
related_edges = []
related_node_uuids = set()
for edge in all_edges:
if edge["source_node_uuid"] == node["uuid"]:
related_edges.append({
@ -321,10 +315,10 @@ class ZepEntityReader:
"source_node_uuid": edge["source_node_uuid"],
})
related_node_uuids.add(edge["source_node_uuid"])
entity.related_edges = related_edges
# 获取关联节点的基本信息
# Populate basic info for each neighboring node.
related_nodes = []
for related_uuid in related_node_uuids:
if related_uuid in node_map:
@ -335,56 +329,55 @@ class ZepEntityReader:
"labels": related_node["labels"],
"summary": related_node.get("summary", ""),
})
entity.related_nodes = related_nodes
filtered_entities.append(entity)
logger.info(t("log.zep_entity_reader.m009", total_count=total_count, len=len(filtered_entities), entity_types_found=entity_types_found))
return FilteredEntities(
entities=filtered_entities,
entity_types=entity_types_found,
total_count=total_count,
filtered_count=len(filtered_entities),
)
def get_entity_with_context(
self,
graph_id: str,
self,
graph_id: str,
entity_uuid: str
) -> Optional[EntityNode]:
"""
获取单个实体及其完整上下文（边和关联节点，带重试机制）
"""Fetch a single entity with its full context (edges + neighbors), with retry.
Args:
graph_id: 图谱ID
entity_uuid: 实体UUID
graph_id: Graph identifier.
entity_uuid: Entity UUID.
Returns:
EntityNode或None
``EntityNode`` or ``None`` if not found.
"""
try:
# 使用重试机制获取节点
# Fetch the node with retry.
node = self._call_with_retry(
func=lambda: self.client.graph.node.get(uuid_=entity_uuid),
operation_name=f"获取节点详情(uuid={entity_uuid[:8]}...)"
)
if not node:
return None
# 获取节点的边
# Edges connected to this node.
edges = self.get_node_edges(entity_uuid)
# 获取所有节点用于关联查找
# All graph nodes, used for neighbor lookup.
all_nodes = self.get_all_nodes(graph_id)
node_map = {n["uuid"]: n for n in all_nodes}
# 处理相关边和节点
# Collect related edges and neighboring uuids.
related_edges = []
related_node_uuids = set()
for edge in edges:
if edge["source_node_uuid"] == entity_uuid:
related_edges.append({
@ -402,8 +395,8 @@ class ZepEntityReader:
"source_node_uuid": edge["source_node_uuid"],
})
related_node_uuids.add(edge["source_node_uuid"])
# 获取关联节点信息
# Populate basic info for each neighboring node.
related_nodes = []
for related_uuid in related_node_uuids:
if related_uuid in node_map:
@ -414,7 +407,7 @@ class ZepEntityReader:
"labels": related_node["labels"],
"summary": related_node.get("summary", ""),
})
return EntityNode(
uuid=getattr(node, 'uuid_', None) or getattr(node, 'uuid', ''),
name=node.name or "",
@ -424,27 +417,26 @@ class ZepEntityReader:
related_edges=related_edges,
related_nodes=related_nodes,
)
except Exception as e:
logger.error(t("log.zep_entity_reader.m010", entity_uuid=entity_uuid, str=str(e)))
return None
def get_entities_by_type(
self,
graph_id: str,
self,
graph_id: str,
entity_type: str,
enrich_with_edges: bool = True
) -> List[EntityNode]:
"""
获取指定类型的所有实体
"""Return every entity matching the given type.
Args:
graph_id: 图谱ID
entity_type: 实体类型（如 "Student", "PublicFigure" 等）
enrich_with_edges: 是否获取相关边信息
graph_id: Graph identifier.
entity_type: Entity type label (e.g. ``Student``, ``PublicFigure``).
enrich_with_edges: When ``True``, populate related edges/nodes.
Returns:
实体列表
A list of matching ``EntityNode`` instances.
"""
result = self.filter_defined_entities(
graph_id=graph_id,

268
backend/app/services/zep_graph_memory_updater.py
View File

@ -1,6 +1,7 @@
"""
Zep图谱记忆更新服务
将模拟中的Agent活动动态更新到Zep图谱中
Zep graph memory update service.
Streams agent activity from running simulations into the Zep knowledge graph.
"""
import os
@ -23,7 +24,7 @@ logger = get_logger('mirofish.zep_graph_memory_updater')
@dataclass
class AgentActivity:
"""Agent活动记录"""
"""Record of a single agent activity."""
platform: str # twitter / reddit
agent_id: int
agent_name: str
@ -33,13 +34,12 @@ class AgentActivity:
timestamp: str
def to_episode_text(self) -> str:
"""Render the activity as a natural-language episode for Zep.
The text uses plain narrative phrasing so Zep can extract entities and
relationships from it. No simulation-specific prefix is prepended, so
the graph update is not biased by framing words.
"""
将活动转换为可以发送给Zep的文本描述
采用自然语言描述格式，让Zep能够从中提取实体和关系
不添加模拟相关的前缀，避免误导图谱更新
"""
# 根据不同的动作类型生成不同的描述
action_descriptions = {
"CREATE_POST": self._describe_create_post,
"LIKE_POST": self._describe_like_post,
@ -57,8 +57,8 @@ class AgentActivity:
describe_func = action_descriptions.get(self.action_type, self._describe_generic)
description = describe_func()
# 直接返回 "agent名称: 活动描述" 格式，不添加模拟前缀
# Return "<agent name>: <activity>" with no simulation prefix.
return f"{self.agent_name}: {description}"
def _describe_create_post(self) -> str:
@ -68,7 +68,7 @@ class AgentActivity:
return "发布了一条帖子"
def _describe_like_post(self) -> str:
"""点赞帖子 - 包含帖子原文和作者信息"""
"""Like a post — includes the post text and author when available."""
post_content = self.action_args.get("post_content", "")
post_author = self.action_args.get("post_author_name", "")
@ -81,7 +81,7 @@ class AgentActivity:
return "点赞了一条帖子"
def _describe_dislike_post(self) -> str:
"""踩帖子 - 包含帖子原文和作者信息"""
"""Dislike a post — includes the post text and author when available."""
post_content = self.action_args.get("post_content", "")
post_author = self.action_args.get("post_author_name", "")
@ -94,7 +94,7 @@ class AgentActivity:
return "踩了一条帖子"
def _describe_repost(self) -> str:
"""转发帖子 - 包含原帖内容和作者信息"""
"""Repost — includes the original post text and author when available."""
original_content = self.action_args.get("original_content", "")
original_author = self.action_args.get("original_author_name", "")
@ -107,7 +107,7 @@ class AgentActivity:
return "转发了一条帖子"
def _describe_quote_post(self) -> str:
"""引用帖子 - 包含原帖内容、作者信息和引用评论"""
"""Quote-post — includes the original post, author, and the quote comment."""
original_content = self.action_args.get("original_content", "")
original_author = self.action_args.get("original_author_name", "")
quote_content = self.action_args.get("quote_content", "") or self.action_args.get("content", "")
@ -127,7 +127,7 @@ class AgentActivity:
return base
def _describe_follow(self) -> str:
"""关注用户 - 包含被关注用户的名称"""
"""Follow a user — includes the followed user's name."""
target_user_name = self.action_args.get("target_user_name", "")
if target_user_name:
@ -135,7 +135,7 @@ class AgentActivity:
return "关注了一个用户"
def _describe_create_comment(self) -> str:
"""发表评论 - 包含评论内容和所评论的帖子信息"""
"""Create a comment — includes the comment text and the parent post."""
content = self.action_args.get("content", "")
post_content = self.action_args.get("post_content", "")
post_author = self.action_args.get("post_author_name", "")
@ -151,7 +151,7 @@ class AgentActivity:
return "发表了评论"
def _describe_like_comment(self) -> str:
"""点赞评论 - 包含评论内容和作者信息"""
"""Like a comment — includes the comment text and author when available."""
comment_content = self.action_args.get("comment_content", "")
comment_author = self.action_args.get("comment_author_name", "")
@ -164,7 +164,7 @@ class AgentActivity:
return "点赞了一条评论"
def _describe_dislike_comment(self) -> str:
"""踩评论 - 包含评论内容和作者信息"""
"""Dislike a comment — includes the comment text and author when available."""
comment_content = self.action_args.get("comment_content", "")
comment_author = self.action_args.get("comment_author_name", "")
@ -177,17 +177,17 @@ class AgentActivity:
return "踩了一条评论"
def _describe_search(self) -> str:
"""搜索帖子 - 包含搜索关键词"""
"""Search posts — includes the search query."""
query = self.action_args.get("query", "") or self.action_args.get("keyword", "")
return f"搜索了「{query}」" if query else "进行了搜索"
def _describe_search_user(self) -> str:
"""搜索用户 - 包含搜索关键词"""
"""Search users — includes the search query."""
query = self.action_args.get("query", "") or self.action_args.get("username", "")
return f"搜索了用户「{query}」" if query else "搜索了用户"
def _describe_mute(self) -> str:
"""屏蔽用户 - 包含被屏蔽用户的名称"""
"""Mute a user — includes the muted user's name."""
target_user_name = self.action_args.get("target_user_name", "")
if target_user_name:
@ -195,80 +195,79 @@ class AgentActivity:
return "屏蔽了一个用户"
def _describe_generic(self) -> str:
# 对于未知的动作类型，生成通用描述
# Fallback narration for action types not handled explicitly above.
return f"执行了{self.action_type}操作"
class ZepGraphMemoryUpdater:
"""
Zep图谱记忆更新器
监控模拟的actions日志文件，将新的agent活动实时更新到Zep图谱中。
按平台分组，每累积BATCH_SIZE条活动后批量发送到Zep。
所有有意义的行为都会被更新到Zep，action_args中会包含完整的上下文信息：
- 点赞/踩的帖子原文
- 转发/引用的帖子原文
- 关注/屏蔽的用户名
- 点赞/踩的评论原文
"""Zep graph memory updater.
Watches a simulation's actions log file and streams new agent activity
into the Zep knowledge graph in near real time. Activities are grouped
by platform; each platform sends a batch once it has accumulated
``BATCH_SIZE`` items.
Every meaningful action is forwarded to Zep, with full context preserved
in ``action_args``:
- Original text of liked / disliked posts
- Original text of reposted / quoted posts
- Names of followed / muted users
- Original text of liked / disliked comments
"""
# 批量发送大小（每个平台累积多少条后发送）
# Number of activities to accumulate per platform before sending a batch.
BATCH_SIZE = 5
# 平台名称映射（用于控制台显示）
# Platform display names used for console / log output.
PLATFORM_DISPLAY_NAMES = {
'twitter': '世界1',
'reddit': '世界2',
}
# 发送间隔（秒），避免请求过快
# Pause between sends (seconds) to avoid hammering the Zep API.
SEND_INTERVAL = 0.5
# 重试配置
MAX_RETRIES = 3
RETRY_DELAY = 2 # 秒
RETRY_DELAY = 2 # seconds
def __init__(self, graph_id: str, api_key: Optional[str] = None):
"""
初始化更新器
"""Initialize the updater.
Args:
graph_id: Zep图谱ID
api_key: Zep API Key（可选，默认从配置读取）
graph_id: Zep graph ID.
api_key: Optional Zep API key; defaults to the value from config.
"""
self.graph_id = graph_id
self.client = GraphitiAdapter()
# 活动队列
self._activity_queue: Queue = Queue()
# 按平台分组的活动缓冲区（每个平台各自累积到BATCH_SIZE后批量发送）
# Per-platform buffer; each platform flushes once it reaches BATCH_SIZE.
self._platform_buffers: Dict[str, List[AgentActivity]] = {
'twitter': [],
'reddit': [],
}
self._buffer_lock = threading.Lock()
# 控制标志
self._running = False
self._worker_thread: Optional[threading.Thread] = None
# 统计
self._total_activities = 0 # 实际添加到队列的活动数
self._total_sent = 0 # 成功发送到Zep的批次数
self._total_items_sent = 0 # 成功发送到Zep的活动条数
self._failed_count = 0 # 发送失败的批次数
self._skipped_count = 0 # 被过滤跳过的活动数（DO_NOTHING）
# Counters
self._total_activities = 0 # activities accepted into the queue
self._total_sent = 0 # batches successfully sent to Zep
self._total_items_sent = 0 # individual activities successfully sent to Zep
self._failed_count = 0 # batches that failed to send
self._skipped_count = 0 # activities filtered out (e.g. DO_NOTHING)
logger.info(t("log.zep_graph_memory_updater.m001", graph_id=graph_id, self=self.BATCH_SIZE))
def _get_platform_display_name(self, platform: str) -> str:
"""获取平台的显示名称"""
"""Return the human-friendly display name for a platform."""
return self.PLATFORM_DISPLAY_NAMES.get(platform.lower(), platform)
def start(self):
"""启动后台工作线程"""
"""Start the background worker thread."""
if self._running:
return
@ -286,10 +285,9 @@ class ZepGraphMemoryUpdater:
logger.info(t("log.zep_graph_memory_updater.m002", self=self.graph_id))
def stop(self):
"""停止后台工作线程"""
"""Stop the background worker thread and flush pending activity."""
self._running = False
# 发送剩余的活动
self._flush_remaining()
if self._worker_thread and self._worker_thread.is_alive():
@ -298,27 +296,28 @@ class ZepGraphMemoryUpdater:
logger.info(t("log.zep_graph_memory_updater.m003", self=self.graph_id, self_2=self._total_activities, self_3=self._total_sent, self_4=self._total_items_sent, self_5=self._failed_count, self_6=self._skipped_count))
def add_activity(self, activity: AgentActivity):
"""
添加一个agent活动到队列
所有有意义的行为都会被添加到队列，包括：
- CREATE_POST（发帖）
- CREATE_COMMENT（评论）
- QUOTE_POST（引用帖子）
- SEARCH_POSTS（搜索帖子）
- SEARCH_USER（搜索用户）
- LIKE_POST/DISLIKE_POST（点赞/踩帖子）
- REPOST（转发）
- FOLLOW（关注）
- MUTE（屏蔽）
- LIKE_COMMENT/DISLIKE_COMMENT（点赞/踩评论）
action_args中会包含完整的上下文信息（如帖子原文、用户名等）。
"""Enqueue a single agent activity for delivery to Zep.
Every meaningful action is queued, including:
- CREATE_POST (post)
- CREATE_COMMENT (comment)
- QUOTE_POST (quote a post)
- SEARCH_POSTS (search posts)
- SEARCH_USER (search users)
- LIKE_POST / DISLIKE_POST (like / dislike a post)
- REPOST (repost)
- FOLLOW (follow)
- MUTE (mute)
- LIKE_COMMENT / DISLIKE_COMMENT (like / dislike a comment)
``action_args`` carries the full context (e.g. original post text,
user names) so the graph episode is self-contained.
Args:
activity: Agent活动记录
activity: The agent activity record to enqueue.
"""
# 跳过DO_NOTHING类型的活动
# DO_NOTHING actions carry no information worth indexing.
if activity.action_type == "DO_NOTHING":
self._skipped_count += 1
return
@ -328,14 +327,13 @@ class ZepGraphMemoryUpdater:
logger.debug(t("log.zep_graph_memory_updater.m004", activity=activity.agent_name, activity_2=activity.action_type))
def add_activity_from_dict(self, data: Dict[str, Any], platform: str):
"""
从字典数据添加活动
"""Build an ``AgentActivity`` from a parsed JSON record and enqueue it.
Args:
data: 从actions.jsonl解析的字典数据
platform: 平台名称 (twitter/reddit)
data: A dict parsed from a single ``actions.jsonl`` line.
platform: Source platform name (``twitter`` or ``reddit``).
"""
# 跳过事件类型的条目
# Event-type rows describe simulation lifecycle, not agent activity.
if "event_type" in data:
return
@ -352,28 +350,26 @@ class ZepGraphMemoryUpdater:
self.add_activity(activity)
def _worker_loop(self, locale: str = 'zh'):
"""后台工作循环 - 按平台批量发送活动到Zep"""
"""Background loop that drains the queue and flushes per-platform batches."""
set_locale(locale)
while self._running or not self._activity_queue.empty():
try:
# 尝试从队列获取活动（超时1秒）
# Block briefly so the loop can also notice shutdown requests.
try:
activity = self._activity_queue.get(timeout=1)
# 将活动添加到对应平台的缓冲区
platform = activity.platform.lower()
with self._buffer_lock:
if platform not in self._platform_buffers:
self._platform_buffers[platform] = []
self._platform_buffers[platform].append(activity)
# 检查该平台是否达到批量大小
if len(self._platform_buffers[platform]) >= self.BATCH_SIZE:
batch = self._platform_buffers[platform][:self.BATCH_SIZE]
self._platform_buffers[platform] = self._platform_buffers[platform][self.BATCH_SIZE:]
# 释放锁后再发送
# Release the lock before issuing the network call.
self._send_batch_activities(batch, platform)
# 发送间隔，避免请求过快
# Throttle so we don't hammer the Zep API.
time.sleep(self.SEND_INTERVAL)
except Empty:
@ -384,21 +380,20 @@ class ZepGraphMemoryUpdater:
time.sleep(1)
def _send_batch_activities(self, activities: List[AgentActivity], platform: str):
"""
批量发送活动到Zep图谱（合并为一条文本）
"""Send a batch of activities to the Zep graph as one combined episode.
Args:
activities: Agent活动列表
platform: 平台名称
activities: Agent activity records to send.
platform: Source platform name.
"""
if not activities:
return
# 将多条活动合并为一条文本，用换行分隔
# Concatenate the per-activity narrations into a single newline-separated episode.
episode_texts = [activity.to_episode_text() for activity in activities]
combined_text = "\n".join(episode_texts)
# 带重试的发送
# Retry on failure with linear backoff.
for attempt in range(self.MAX_RETRIES):
try:
self.client.graph.add(
@ -423,8 +418,8 @@ class ZepGraphMemoryUpdater:
self._failed_count += 1
def _flush_remaining(self):
"""发送队列和缓冲区中剩余的活动"""
# 首先处理队列中剩余的活动，添加到缓冲区
"""Drain the queue and flush every platform buffer, even partial ones."""
# Move anything still in the queue into the per-platform buffers.
while not self._activity_queue.empty():
try:
activity = self._activity_queue.get_nowait()
@ -435,61 +430,55 @@ class ZepGraphMemoryUpdater:
self._platform_buffers[platform].append(activity)
except Empty:
break
# 然后发送各平台缓冲区中剩余的活动（即使不足BATCH_SIZE条）
# Flush each platform buffer regardless of whether it reached BATCH_SIZE.
with self._buffer_lock:
for platform, buffer in self._platform_buffers.items():
if buffer:
display_name = self._get_platform_display_name(platform)
logger.info(t("log.zep_graph_memory_updater.m010", display_name=display_name, len=len(buffer)))
self._send_batch_activities(buffer, platform)
# 清空所有缓冲区
for platform in self._platform_buffers:
self._platform_buffers[platform] = []
def get_stats(self) -> Dict[str, Any]:
"""获取统计信息"""
"""Return a snapshot of updater statistics."""
with self._buffer_lock:
buffer_sizes = {p: len(b) for p, b in self._platform_buffers.items()}
return {
"graph_id": self.graph_id,
"batch_size": self.BATCH_SIZE,
"total_activities": self._total_activities, # 添加到队列的活动总数
"batches_sent": self._total_sent, # 成功发送的批次数
"items_sent": self._total_items_sent, # 成功发送的活动条数
"failed_count": self._failed_count, # 发送失败的批次数
"skipped_count": self._skipped_count, # 被过滤跳过的活动数（DO_NOTHING）
"total_activities": self._total_activities, # activities accepted into the queue
"batches_sent": self._total_sent, # batches successfully sent
"items_sent": self._total_items_sent, # activities successfully sent
"failed_count": self._failed_count, # batches that failed to send
"skipped_count": self._skipped_count, # activities filtered out (e.g. DO_NOTHING)
"queue_size": self._activity_queue.qsize(),
"buffer_sizes": buffer_sizes, # 各平台缓冲区大小
"buffer_sizes": buffer_sizes, # per-platform buffer depth
"running": self._running,
}
class ZepGraphMemoryManager:
"""
管理多个模拟的Zep图谱记忆更新器
每个模拟可以有自己的更新器实例
"""
"""Registry that owns one ``ZepGraphMemoryUpdater`` per active simulation."""
_updaters: Dict[str, ZepGraphMemoryUpdater] = {}
_lock = threading.Lock()
@classmethod
def create_updater(cls, simulation_id: str, graph_id: str) -> ZepGraphMemoryUpdater:
"""
为模拟创建图谱记忆更新器
"""Create (and start) a graph-memory updater for a simulation.
Args:
simulation_id: 模拟ID
graph_id: Zep图谱ID
simulation_id: Simulation ID.
graph_id: Zep graph ID.
Returns:
ZepGraphMemoryUpdater实例
The started ``ZepGraphMemoryUpdater`` instance.
"""
with cls._lock:
# 如果已存在，先停止旧的
# An updater already exists for this simulation — stop it first.
if simulation_id in cls._updaters:
cls._updaters[simulation_id].stop()
@ -502,25 +491,24 @@ class ZepGraphMemoryManager:
@classmethod
def get_updater(cls, simulation_id: str) -> Optional[ZepGraphMemoryUpdater]:
"""获取模拟的更新器"""
"""Return the updater for a simulation, or ``None`` if absent."""
return cls._updaters.get(simulation_id)
@classmethod
def stop_updater(cls, simulation_id: str):
"""停止并移除模拟的更新器"""
"""Stop and deregister the updater belonging to a simulation."""
with cls._lock:
if simulation_id in cls._updaters:
cls._updaters[simulation_id].stop()
del cls._updaters[simulation_id]
logger.info(t("log.zep_graph_memory_updater.m012", simulation_id=simulation_id))
# 防止 stop_all 重复调用的标志
# Idempotency guard so ``stop_all`` only runs once per process lifetime.
_stop_all_done = False
@classmethod
def stop_all(cls):
"""停止所有更新器"""
# 防止重复调用
"""Stop every registered updater (idempotent)."""
if cls._stop_all_done:
return
cls._stop_all_done = True
@ -537,7 +525,7 @@ class ZepGraphMemoryManager:
@classmethod
def get_all_stats(cls) -> Dict[str, Dict[str, Any]]:
"""获取所有更新器的统计信息"""
"""Return statistics for every registered updater."""
return {
sim_id: updater.get_stats()
for sim_id, updater in cls._updaters.items()

730
backend/app/services/zep_tools.py
View File

File diff suppressed because it is too large Load Diff

4
backend/app/utils/__init__.py
View File

@ -1,6 +1,4 @@
"""
工具模块
"""
"""Backend utilities package."""
from .file_parser import FileParser
from .llm_client import LLMClient

139
backend/app/utils/file_parser.py
View File

@ -1,6 +1,6 @@
"""
文件解析工具
支持PDF、Markdown、TXT文件的文本提取
"""File parsing utilities.
Supports text extraction from PDF, Markdown, and plain-text files.
"""
import os
@ -9,30 +9,27 @@ from typing import List, Optional
def _read_text_with_fallback(file_path: str) -> str:
"""
读取文本文件，UTF-8失败时自动探测编码。
采用多级回退策略：
1. 首先尝试 UTF-8 解码
2. 使用 charset_normalizer 检测编码
3. 回退到 chardet 检测编码
4. 最终使用 UTF-8 + errors='replace' 兜底
"""Read a text file, falling back through encoding detectors when UTF-8 fails.
Multi-stage fallback strategy:
1. Try UTF-8 first.
2. Use ``charset_normalizer`` to detect the encoding.
3. Fall back to ``chardet``.
4. Last resort: decode with UTF-8 + ``errors='replace'``.
Args:
file_path: 文件路径
file_path: Path to the file to read.
Returns:
解码后的文本内容
The decoded text content.
"""
data = Path(file_path).read_bytes()
# 首先尝试 UTF-8
try:
return data.decode('utf-8')
except UnicodeDecodeError:
pass
# 尝试使用 charset_normalizer 检测编码
encoding = None
try:
from charset_normalizer import from_bytes
@ -41,8 +38,7 @@ def _read_text_with_fallback(file_path: str) -> str:
encoding = best.encoding
except Exception:
pass
# 回退到 chardet
if not encoding:
try:
import chardet
@ -50,89 +46,86 @@ def _read_text_with_fallback(file_path: str) -> str:
encoding = result.get('encoding') if result else None
except Exception:
pass
# 最终兜底：使用 UTF-8 + replace
if not encoding:
encoding = 'utf-8'
return data.decode(encoding, errors='replace')
class FileParser:
"""文件解析器"""
"""Parser for the supported document formats."""
SUPPORTED_EXTENSIONS = {'.pdf', '.md', '.markdown', '.txt'}
@classmethod
def extract_text(cls, file_path: str) -> str:
"""
从文件中提取文本
"""Extract plain text from a single supported file.
Args:
file_path: 文件路径
file_path: Path to the file.
Returns:
提取的文本内容
The extracted text content.
"""
path = Path(file_path)
if not path.exists():
raise FileNotFoundError(f"文件不存在: {file_path}")
suffix = path.suffix.lower()
if suffix not in cls.SUPPORTED_EXTENSIONS:
raise ValueError(f"不支持的文件格式: {suffix}")
if suffix == '.pdf':
return cls._extract_from_pdf(file_path)
elif suffix in {'.md', '.markdown'}:
return cls._extract_from_md(file_path)
elif suffix == '.txt':
return cls._extract_from_txt(file_path)
raise ValueError(f"无法处理的文件格式: {suffix}")
@staticmethod
def _extract_from_pdf(file_path: str) -> str:
"""从PDF提取文本"""
"""Extract text from a PDF file using PyMuPDF."""
try:
import fitz # PyMuPDF
except ImportError:
raise ImportError("需要安装PyMuPDF: pip install PyMuPDF")
text_parts = []
with fitz.open(file_path) as doc:
for page in doc:
text = page.get_text()
if text.strip():
text_parts.append(text)
return "\n\n".join(text_parts)
@staticmethod
def _extract_from_md(file_path: str) -> str:
"""从Markdown提取文本，支持自动编码检测"""
"""Extract text from a Markdown file with automatic encoding detection."""
return _read_text_with_fallback(file_path)
@staticmethod
def _extract_from_txt(file_path: str) -> str:
"""从TXT提取文本，支持自动编码检测"""
"""Extract text from a plain-text file with automatic encoding detection."""
return _read_text_with_fallback(file_path)
@classmethod
def extract_from_multiple(cls, file_paths: List[str]) -> str:
"""
从多个文件提取文本并合并
"""Extract and concatenate text from multiple files.
Args:
file_paths: 文件路径列表
file_paths: Paths of files to read.
Returns:
合并后的文本
The merged text, with per-file headers separating each section.
"""
all_texts = []
for i, file_path in enumerate(file_paths, 1):
try:
text = cls.extract_text(file_path)
@ -140,50 +133,48 @@ class FileParser:
all_texts.append(f"=== 文档 {i}: {filename} ===\n{text}")
except Exception as e:
all_texts.append(f"=== 文档 {i}: {file_path} (提取失败: {str(e)}) ===")
return "\n\n".join(all_texts)
def split_text_into_chunks(
text: str,
chunk_size: int = 500,
text: str,
chunk_size: int = 500,
overlap: int = 50
) -> List[str]:
"""
将文本分割成小块
"""Split text into overlapping chunks.
Args:
text: 原始文本
chunk_size: 每块的字符数
overlap: 重叠字符数
text: The source text to split.
chunk_size: Target characters per chunk.
overlap: Number of characters overlapping between consecutive chunks.
Returns:
文本块列表
A list of chunk strings.
"""
if len(text) <= chunk_size:
return [text] if text.strip() else []
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
# 尝试在句子边界处分割
# Prefer splitting on a sentence boundary near the chunk end
if end < len(text):
# 查找最近的句子结束符
for sep in ['。', '！', '？', '.\n', '!\n', '?\n', '\n\n', '. ', '! ', '? ']:
last_sep = text[start:end].rfind(sep)
if last_sep != -1 and last_sep > chunk_size * 0.3:
end = start + last_sep + len(sep)
break
chunk = text[start:end].strip()
if chunk:
chunks.append(chunk)
# 下一个块从重叠位置开始
# Next chunk starts at the overlap point
start = end - overlap if end < len(text) else len(text)
return chunks

25
backend/app/utils/llm_client.py
View File

@ -1,6 +1,6 @@
"""
LLM客户端封装
统一使用OpenAI格式调用
"""LLM client wrapper.
All providers are called through the OpenAI-compatible API surface.
"""
import json
@ -13,7 +13,7 @@ from ..config import Config
class LLMClient:
"""LLM客户端"""
"""Thin wrapper around the OpenAI-compatible chat completions API."""
def __init__(
self,
@ -37,17 +37,16 @@ class LLMClient:
max_tokens: int = 4096,
response_format: Optional[Dict] = None,
) -> str:
"""
发送聊天请求
"""Send a chat completion request.
Args:
messages: 消息列表
temperature: 温度参数
max_tokens: 最大token数
response_format: 响应格式（如JSON模式）
messages: Chat messages in OpenAI format.
temperature: Sampling temperature.
max_tokens: Maximum number of tokens to generate.
response_format: Optional response format hint (e.g. JSON mode).
Returns:
模型响应文本
The assistant's response text.
"""
kwargs = {
"model": self.model,
@ -61,7 +60,7 @@ class LLMClient:
response = self.client.chat.completions.create(**kwargs)
content = response.choices[0].message.content
# 部分模型（如MiniMax M2.5）会在content中包含<think>思考内容，需要移除
# Some reasoning models (e.g. MiniMax M2.5) embed <think>...</think> blocks; strip them.
content = re.sub(r"<think>[\s\S]*?</think>", "", content).strip()
return content
@ -79,7 +78,7 @@ class LLMClient:
messages=messages, temperature=temperature, max_tokens=max_tokens
)
# 清理markdown代码块标记
# Strip surrounding markdown code-fence markers if present.
cleaned_response = response.strip()
cleaned_response = re.sub(
r"^```(?:json)?\s*\n?", "", cleaned_response, flags=re.IGNORECASE

Some files were not shown because too many files have changed in this diff Show More