When the LLM generates persona quotes (BI economists, ministry officials,
Reddit commenters, etc.) in non-Chinese locales, it can occasionally slip
Chinese characters into otherwise fluent English/Latin prose. The system
prompt language instruction reduces but doesn't eliminate this — the LLM
sometimes reaches back to its Chinese training data for fluent-sounding
speech, producing output like:
"BI economist said: Purbaya过于倾向财政扩张..."
This adds a post-processing step that:
1. Detects runs of CJK Unified Ideographs (U+4E00..U+9FFF) and CJK
Symbols/Punctuation (U+3000..U+303F) ≥ 2 chars in length
2. Batch-translates them via the configured LLM endpoint (reusing the
same LLM_API_KEY/LLM_BASE_URL as the rest of MiroFish)
3. Replaces each run in-place, injecting spaces at ASCII boundaries so
the result reads naturally in surrounding text
4. Iterates up to 3 passes to catch fragments the LLM leaves in pass 1
Behavior:
- Auto-enabled for non-Chinese locales (en, es, fr, pt, ru, de, id)
- Skipped for zh / zh-CN / zh-TW (legitimate CJK content)
- No-op when LLM_API_KEY is not configured (warns and returns original)
- Graceful fallback: any LLM failure returns original text unchanged
- Idempotent: re-running on already-sanitized text is a no-op
Configuration (all optional, set in .env):
CJK_SANITIZE_ENABLED=0 # force off (default: auto for non-zh)
CJK_SANITIZE_LANGS=ja,ko # override target locale set
CJK_SANITIZE_MAX_PASSES=3 # default 3
Files added:
- backend/app/utils/cjk_sanitize.py (~250 lines, the module)
- backend/scripts/test_cjk_sanitize.py (23 unit + integration tests)
Files modified:
- backend/app/services/report_agent.py (wire-in after assemble_full_report)
- README.md (document config env vars)
Tested: 23/23 unit tests pass; live Purbaya/USD-IDR report (24 unique CJK
runs in 14kB markdown) reduced to 0 in 3.4s with real DeepSeek API.
423f3437da