When the LLM generates persona quotes (BI economists, ministry officials,
Reddit commenters, etc.) in non-Chinese locales, it can occasionally slip
Chinese characters into otherwise fluent English/Latin prose. The system
prompt language instruction reduces but doesn't eliminate this — the LLM
sometimes reaches back to its Chinese training data for fluent-sounding
speech, producing output like:
"BI economist said: Purbaya过于倾向财政扩张..."
This adds a post-processing step that:
1. Detects runs of CJK Unified Ideographs (U+4E00..U+9FFF) and CJK
Symbols/Punctuation (U+3000..U+303F) ≥ 2 chars in length
2. Batch-translates them via the configured LLM endpoint (reusing the
same LLM_API_KEY/LLM_BASE_URL as the rest of MiroFish)
3. Replaces each run in-place, injecting spaces at ASCII boundaries so
the result reads naturally in surrounding text
4. Iterates up to 3 passes to catch fragments the LLM leaves in pass 1
Behavior:
- Auto-enabled for non-Chinese locales (en, es, fr, pt, ru, de, id)
- Skipped for zh / zh-CN / zh-TW (legitimate CJK content)
- No-op when LLM_API_KEY is not configured (warns and returns original)
- Graceful fallback: any LLM failure returns original text unchanged
- Idempotent: re-running on already-sanitized text is a no-op
Configuration (all optional, set in .env):
CJK_SANITIZE_ENABLED=0 # force off (default: auto for non-zh)
CJK_SANITIZE_LANGS=ja,ko # override target locale set
CJK_SANITIZE_MAX_PASSES=3 # default 3
Files added:
- backend/app/utils/cjk_sanitize.py (~250 lines, the module)
- backend/scripts/test_cjk_sanitize.py (23 unit + integration tests)
Files modified:
- backend/app/services/report_agent.py (wire-in after assemble_full_report)
- README.md (document config env vars)
Tested: 23/23 unit tests pass; live Purbaya/USD-IDR report (24 unique CJK
runs in 14kB markdown) reduced to 0 in 3.4s with real DeepSeek API.
The Shanda sponsor logo's alt text was `666ghj%2MiroFish | Shanda`,
missing the `F` from the URL-encoded `/`. Every other badge in both
READMEs uses the correct `666ghj%2FMiroFish`. Bring this one in line
with the rest.
- Add return type annotation (list[str]) to Config.validate()
- Add type annotations (msg: str, -> None) to logger convenience functions
- Add FileParser.is_supported() classmethod for checking file format support
- Add sans-serif font for English left-pane (status, workflow sections)
- Shorten English workflow step descriptions
- Reduce English report title font-size from 36px to 28px
- Use sans-serif font for English titles, descriptions and navbar
- Shorten English hero text to avoid overflow
- Fix :global() scoped CSS issue that was setting root font-size to 3.5rem
- Use separate unscoped style block for html[lang] selectors
Background threads (graph building, simulation prep, report generation,
profile generation) now inherit the requesting user's locale preference.
Previously these fell back to 'zh' because Flask request context was
unavailable in spawned threads.
Ensure poster_type stays PascalCase English and stance stays English enum
values regardless of language setting. Only natural language fields follow
the user's language preference.
The language instruction was causing LLM to change entity/relation naming
conventions. Now explicitly enforce PascalCase/UPPER_SNAKE_CASE for technical
identifiers while only applying language preference to description fields.
Ardha
Ardha
BaiFu
BaiFu
YunyueLi
lllopic
ghostubborn