When the LLM generates persona quotes (BI economists, ministry officials,
Reddit commenters, etc.) in non-Chinese locales, it can occasionally slip
Chinese characters into otherwise fluent English/Latin prose. The system
prompt language instruction reduces but doesn't eliminate this — the LLM
sometimes reaches back to its Chinese training data for fluent-sounding
speech, producing output like:
"BI economist said: Purbaya过于倾向财政扩张..."
This adds a post-processing step that:
1. Detects runs of CJK Unified Ideographs (U+4E00..U+9FFF) and CJK
Symbols/Punctuation (U+3000..U+303F) ≥ 2 chars in length
2. Batch-translates them via the configured LLM endpoint (reusing the
same LLM_API_KEY/LLM_BASE_URL as the rest of MiroFish)
3. Replaces each run in-place, injecting spaces at ASCII boundaries so
the result reads naturally in surrounding text
4. Iterates up to 3 passes to catch fragments the LLM leaves in pass 1
Behavior:
- Auto-enabled for non-Chinese locales (en, es, fr, pt, ru, de, id)
- Skipped for zh / zh-CN / zh-TW (legitimate CJK content)
- No-op when LLM_API_KEY is not configured (warns and returns original)
- Graceful fallback: any LLM failure returns original text unchanged
- Idempotent: re-running on already-sanitized text is a no-op
Configuration (all optional, set in .env):
CJK_SANITIZE_ENABLED=0 # force off (default: auto for non-zh)
CJK_SANITIZE_LANGS=ja,ko # override target locale set
CJK_SANITIZE_MAX_PASSES=3 # default 3
Files added:
- backend/app/utils/cjk_sanitize.py (~250 lines, the module)
- backend/scripts/test_cjk_sanitize.py (23 unit + integration tests)
Files modified:
- backend/app/services/report_agent.py (wire-in after assemble_full_report)
- README.md (document config env vars)
Tested: 23/23 unit tests pass; live Purbaya/USD-IDR report (24 unique CJK
runs in 14kB markdown) reduced to 0 in 3.4s with real DeepSeek API.
- Add return type annotation (list[str]) to Config.validate()
- Add type annotations (msg: str, -> None) to logger convenience functions
- Add FileParser.is_supported() classmethod for checking file format support
Background threads (graph building, simulation prep, report generation,
profile generation) now inherit the requesting user's locale preference.
Previously these fell back to 'zh' because Flask request context was
unavailable in spawned threads.
Ensure poster_type stays PascalCase English and stance stays English enum
values regardless of language setting. Only natural language fields follow
the user's language preference.
The language instruction was causing LLM to change entity/relation naming
conventions. Now explicitly enforce PascalCase/UPPER_SNAKE_CASE for technical
identifiers while only applying language preference to description fields.
- Implemented `_get_report_id_for_simulation` to find the most recent report ID associated with a simulation ID by scanning the reports directory.
- Updated `get_simulation_history` to include the retrieved report ID in the response, enhancing the simulation data returned to the client.
- Updated simulation history retrieval to read project details directly from the Simulation file.
- Improved simulation configuration handling by reading simulation requirements from JSON.
- Added project file listing to the simulation history, displaying up to three associated files.
- Refined card layout in HistoryDatabase.vue to accommodate new file display features and improved responsiveness.
- Deleted docker-compose.yml, backend Dockerfile, frontend Dockerfile, and nginx configuration to streamline project setup.
- Updated .env.example to reorganize LLM and ZEP API configurations for clarity and ease of use.
- Enhanced README.md to reflect changes in project structure and provide clearer setup instructions.
- Created package-lock.json for dependency management.
- Updated package.json and frontend package.json to version 0.1.0.
- Adjusted backend pyproject.toml to reflect version 0.1.0.
- Introduced uv.lock for Python dependency resolution.
- Modified the backend setup script to clear the virtual environment before installation.
- Improved README.md by restructuring the prerequisites section into a table for better readability.
- Added installation instructions for the `uv` package and clarified terminal requirements post-installation.
- Updated .env.example to reflect new LLM configuration with Aliyun's API.
- Enhanced .gitignore to include additional files and directories for better exclusion of sensitive and build artifacts.
- Added docker-compose.yml for streamlined deployment of backend and frontend services.
- Introduced Dockerfiles for both backend and frontend to facilitate containerized builds.
- Created README.md to provide comprehensive project documentation and setup instructions.
- Established nginx configuration for frontend to support API proxying and static file serving.
- Decreased the maximum tool calls per section from 8 to 5.
- Reduced the maximum iterations in the ReACT loop from 8 to 5, streamlining the report generation process.
- Deleted the backend README.md file as it was no longer needed.
- Changed the favicon from a SVG to a PNG format for better compatibility.
- Updated the page title in index.html to reflect a more concise branding message.
Ardha
BaiFu
lllopic
ghostubborn