When the LLM generates persona quotes (BI economists, ministry officials,
Reddit commenters, etc.) in non-Chinese locales, it can occasionally slip
Chinese characters into otherwise fluent English/Latin prose. The system
prompt language instruction reduces but doesn't eliminate this — the LLM
sometimes reaches back to its Chinese training data for fluent-sounding
speech, producing output like:
"BI economist said: Purbaya过于倾向财政扩张..."
This adds a post-processing step that:
1. Detects runs of CJK Unified Ideographs (U+4E00..U+9FFF) and CJK
Symbols/Punctuation (U+3000..U+303F) ≥ 2 chars in length
2. Batch-translates them via the configured LLM endpoint (reusing the
same LLM_API_KEY/LLM_BASE_URL as the rest of MiroFish)
3. Replaces each run in-place, injecting spaces at ASCII boundaries so
the result reads naturally in surrounding text
4. Iterates up to 3 passes to catch fragments the LLM leaves in pass 1
Behavior:
- Auto-enabled for non-Chinese locales (en, es, fr, pt, ru, de, id)
- Skipped for zh / zh-CN / zh-TW (legitimate CJK content)
- No-op when LLM_API_KEY is not configured (warns and returns original)
- Graceful fallback: any LLM failure returns original text unchanged
- Idempotent: re-running on already-sanitized text is a no-op
Configuration (all optional, set in .env):
CJK_SANITIZE_ENABLED=0 # force off (default: auto for non-zh)
CJK_SANITIZE_LANGS=ja,ko # override target locale set
CJK_SANITIZE_MAX_PASSES=3 # default 3
Files added:
- backend/app/utils/cjk_sanitize.py (~250 lines, the module)
- backend/scripts/test_cjk_sanitize.py (23 unit + integration tests)
Files modified:
- backend/app/services/report_agent.py (wire-in after assemble_full_report)
- README.md (document config env vars)
Tested: 23/23 unit tests pass; live Purbaya/USD-IDR report (24 unique CJK
runs in 14kB markdown) reduced to 0 in 3.4s with real DeepSeek API.
- Eliminated unnecessary warnings suppression related to multiprocessing resource tracking in `run.py` and `run_parallel_simulation.py`, streamlining the code and reducing potential confusion.
- This change enhances clarity in logging by allowing relevant warnings to be displayed during execution, improving traceability and debugging capabilities.
- Added signal handling to gracefully manage shutdown events across simulation scripts, ensuring proper cleanup of resources.
- Introduced a global shutdown event to allow simulations to respond to termination signals, improving robustness.
- Suppressed warnings related to multiprocessing resource tracking to avoid unnecessary log clutter during execution.
- Updated cleanup logic in `SimulationRunner` and `ZepGraphMemoryManager` to prevent redundant calls and ensure efficient resource management.
- Enhanced logging to provide clearer feedback during shutdown processes, improving traceability.
- Added context enrichment for actions in `fetch_new_actions_from_db`, providing complete information for posts, comments, and user interactions.
- Introduced a new `_enrich_action_context` function to supplement action arguments with relevant details such as post content, author names, and comment information.
- Updated the `ZepGraphMemoryUpdater` to batch send activities by platform, improving efficiency in processing and logging.
- Enhanced logging to include detailed statistics on sent activities and skipped actions, ensuring better traceability and monitoring of the activity flow.
- Added a new Interview module to facilitate interactions with agents post-simulation, allowing for single and batch interviews.
- Introduced IPC communication mechanism for command and response handling between the Flask backend and simulation scripts.
- Updated README.md to include detailed instructions on the new Interview functionality, including API endpoints and usage examples.
- Enhanced simulation scripts to support waiting for commands after completion, improving user control over the simulation environment.
- Implemented error handling and logging for interview processes, ensuring robust operation and traceability.
- Removed content truncation logic from the AgentActivity class, allowing full content to be logged for posts, comments, and quotes.
- Updated the `fetch_new_actions_from_db` function to retain complete content in action arguments, enhancing data accuracy.
- Adjusted simulation scripts to ensure that full content is sent during action creation, improving the representation of agent activities.
- Added logging for the start and end of round 0 in both Twitter and Reddit simulations, improving traceability of initial events.
- Updated the logging mechanism to record round end even when no active agents are present, ensuring comprehensive action tracking.
- Introduced initial action count tracking to provide insights into the number of actions taken during the initial phase of simulations.
- Added support for a `max_rounds` parameter in simulation API, allowing users to limit the number of simulation rounds, improving control over simulation duration.
- Updated README.md to reflect the new `max_rounds` parameter and its usage in simulation requests.
- Enhanced error handling for `max_rounds` input validation to ensure it is a positive integer.
- Modified simulation runner and related scripts to incorporate `max_rounds` functionality, ensuring consistent application across Twitter and Reddit simulations.
- Improved logging to indicate when the number of rounds is truncated due to the `max_rounds` setting, enhancing traceability during simulation execution.
- Introduced `get_agent_names_from_config` function to map agent IDs to their entity names from the simulation configuration, enhancing clarity in action representation.
- Updated simulation scripts to utilize this new function for fetching agent names, ensuring that real entity names are displayed instead of default identifiers.
- Improved handling of agent names by falling back to default names only if not specified in the configuration, maintaining consistency across simulations.
- Updated .env.example to include new keys for dual LLM configuration, allowing for both general and boost settings.
- Modified create_model function to support an optional use_boost parameter, enabling the selection of either general or boost LLM configurations based on availability.
- Improved logging to indicate which LLM configuration is being used during model creation, enhancing clarity for users.
- Introduced MaxTokensWarningFilter to suppress specific warnings related to max_tokens in the logging output across simulation scripts.
- Added a semaphore parameter to limit the maximum concurrent LLM requests in Twitter and Reddit simulation functions, preventing API overload.
- Ensured the filter is applied immediately upon module loading for effective logging management.
- Registered a cleanup function for simulation processes to ensure proper termination on server shutdown.
- Improved logging during application startup to confirm the registration of the cleanup function.
- Updated simulation preparation checks to clarify the conditions for considering a simulation ready, enhancing error handling and user feedback.
- Added detailed logging for simulation status changes, improving traceability during the simulation lifecycle.
- Introduced new files for simulation configuration and profile data, supporting enhanced testing and visualization capabilities.
- Updated simulation preparation checks to exclude script files from the required files list, improving clarity on file management.
- Implemented a robust retry mechanism for Zep API calls in the ZepEntityReader service, enhancing reliability.
- Enhanced logging in simulation scripts to provide clearer insights into the simulation process and errors.
- Updated simulation runner to manage stdout and stderr logs more effectively, ensuring better error tracking.
- Improved profile generation to standardize gender fields and ensure all required fields are populated correctly.
- Updated README.md to include new simulation scripts and configuration details for OASIS, including API retry mechanisms and environment variable settings.
- Added simulation management and configuration generation services to streamline the simulation process across Twitter and Reddit platforms.
- Introduced new API routes for simulation-related operations, including entity retrieval and simulation status management.
- Implemented a robust retry mechanism for external API calls to improve system stability.
- Enhanced task management model to include detailed progress tracking.
- Added logging capabilities for action tracking during simulations.
- Included new scripts for running parallel simulations and testing profile formats.
Ardha
666ghj