When the LLM generates persona quotes (BI economists, ministry officials,
Reddit commenters, etc.) in non-Chinese locales, it can occasionally slip
Chinese characters into otherwise fluent English/Latin prose. The system
prompt language instruction reduces but doesn't eliminate this — the LLM
sometimes reaches back to its Chinese training data for fluent-sounding
speech, producing output like:
"BI economist said: Purbaya过于倾向财政扩张..."
This adds a post-processing step that:
1. Detects runs of CJK Unified Ideographs (U+4E00..U+9FFF) and CJK
Symbols/Punctuation (U+3000..U+303F) ≥ 2 chars in length
2. Batch-translates them via the configured LLM endpoint (reusing the
same LLM_API_KEY/LLM_BASE_URL as the rest of MiroFish)
3. Replaces each run in-place, injecting spaces at ASCII boundaries so
the result reads naturally in surrounding text
4. Iterates up to 3 passes to catch fragments the LLM leaves in pass 1
Behavior:
- Auto-enabled for non-Chinese locales (en, es, fr, pt, ru, de, id)
- Skipped for zh / zh-CN / zh-TW (legitimate CJK content)
- No-op when LLM_API_KEY is not configured (warns and returns original)
- Graceful fallback: any LLM failure returns original text unchanged
- Idempotent: re-running on already-sanitized text is a no-op
Configuration (all optional, set in .env):
CJK_SANITIZE_ENABLED=0 # force off (default: auto for non-zh)
CJK_SANITIZE_LANGS=ja,ko # override target locale set
CJK_SANITIZE_MAX_PASSES=3 # default 3
Files added:
- backend/app/utils/cjk_sanitize.py (~250 lines, the module)
- backend/scripts/test_cjk_sanitize.py (23 unit + integration tests)
Files modified:
- backend/app/services/report_agent.py (wire-in after assemble_full_report)
- README.md (document config env vars)
Tested: 23/23 unit tests pass; live Purbaya/USD-IDR report (24 unique CJK
runs in 14kB markdown) reduced to 0 in 3.4s with real DeepSeek API.
The Shanda sponsor logo's alt text was `666ghj%2MiroFish | Shanda`,
missing the `F` from the URL-encoded `/`. Every other badge in both
READMEs uses the correct `666ghj%2FMiroFish`. Bring this one in line
with the rest.
- Changed Python version specification to indicate compatibility with versions ≥3.11 and ≤3.12.
- Renamed "Contact Us" section to "Join the Conversation" for improved engagement.
- Added a centered image of the QQ group to the contact section for better visibility.
- Implemented a modal to display detailed information about selected projects, including simulation requirements and associated files.
- Added navigation buttons within the modal for quick access to project, simulation, and report pages.
- Enhanced user experience by allowing users to view project details without leaving the main interface.
- Introduced a new README-EN.md file to provide English documentation for the MiroFish project.
- Included an overview of the project, vision, demo videos, quick start guide, and environment variable configuration.
- Updated the existing README.md to link to the new English documentation.
- Deleted docker-compose.yml, backend Dockerfile, frontend Dockerfile, and nginx configuration to streamline project setup.
- Updated .env.example to reorganize LLM and ZEP API configurations for clarity and ease of use.
- Enhanced README.md to reflect changes in project structure and provide clearer setup instructions.
- Modified the backend setup script to clear the virtual environment before installation.
- Improved README.md by restructuring the prerequisites section into a table for better readability.
- Added installation instructions for the `uv` package and clarified terminal requirements post-installation.
- Updated .env.example to reflect new LLM configuration with Aliyun's API.
- Enhanced .gitignore to include additional files and directories for better exclusion of sensitive and build artifacts.
- Added docker-compose.yml for streamlined deployment of backend and frontend services.
- Introduced Dockerfiles for both backend and frontend to facilitate containerized builds.
- Created README.md to provide comprehensive project documentation and setup instructions.
- Established nginx configuration for frontend to support API proxying and static file serving.
Ardha
YunyueLi
666ghj