MicroFish/backend/app/utils
rqd6f4g6zn-bit c2d533d933 fix: robust LLM JSON parsing + sanitize report sections + chat history dedup
Closes #624, #622, #601, #599, #577

## LLM JSON parsing (#624 / #622 / #601)
- New `_parse_llm_json()` in llm_client.py with 5-stage fallback:
  1. Strip markdown fences (existing)
  2. Strict json.loads (fast path)
  3. json.JSONDecoder.raw_decode (handles trailing prose after JSON)
  4. Balanced-brace extraction (leading prose + embedded JSON)
  5. Strip control chars + retry
- Replaces strict json.loads in chat_json() that was failing on any LLM
  appending text after the JSON (common with qwen-plus, ollama, gemma even
  with response_format=json_object).
- Logs which fallback was used so problematic LLMs are visible.
- 8 unit-test cases covering each strategy.

## Report section tool_call leak (#599)
- New `_sanitize_section_content()` in report_agent.py detects when a
  section's "final answer" is actually an unexecuted tool_call JSON
  (e.g. `{"name":"quick_search","parameters":{...}}`) and replaces it
  with a clear fallback message instead of writing the raw artifact to
  the report.
- Applied at all 3 places where final_answer is returned in
  write_section(): the Final Answer path, the no-prefix fallback, and
  the force-finalize path.

## Chat history duplicate user message (#577)
- In report_agent.py chat(), defensively dedupe chat_history:
  - Only keep {role, content} from history items
  - Skip entries that match the current message exactly
- This prevents LLM from seeing a duplicate trailing user message and
  echoing back the previous answer.
- Added debug log of constructed messages array for diagnostics.
2026-05-17 08:23:16 +02:00
..
__init__.py fix(i18n): pass locale to background threads via thread-local storage 2026-04-01 16:55:51 +08:00
file_parser.py fix(file_parser): handle non-UTF-8 encoded text files with automatic encoding detection 2026-01-22 18:28:37 +08:00
llm_client.py fix: robust LLM JSON parsing + sanitize report sections + chat history dedup 2026-05-17 08:23:16 +02:00
locale.py fix(i18n): validate Accept-Language header against registered locales 2026-04-02 14:20:15 +08:00
logger.py Add UTF-8 encoding support for Windows console in run.py and logger.py to prevent character encoding issues 2025-12-26 17:58:48 +08:00
retry.py Enhance backend functionality with OASIS simulation features 2025-12-01 15:03:44 +08:00
zep_paging.py feat(graph): implement pagination for fetching nodes and edges; add utility functions for streamlined data retrieval 2026-02-27 15:53:29 +08:00