RomanNum3ral
/
MicroFish
mirror of https://github.com/666ghj/MiroFish.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
275 Commits 1 Branch 3 Tags 74 MiB
Commit Graph

97 Commits

Author SHA1 Message Date
liyizhouAI f1f2ea4ea5 fix: v0.3.1 - 7 bug fixes unblock end-to-end production flow 
First successful E2E test uncovered 7 blocking bugs. All fixed and deployed
to production server. User confirmed full pipeline now runs through.

## Bug Fixes

### #1 LLMClient exponential backoff retry
- recognize RateLimitError / 429 / 5xx / 1302 / timeout / connection errors
- 5 retries with 1→2→4→8→16s backoff + random jitter
- file: backend/app/utils/llm_client.py

### #2 GLM → Qwen 32B dual-LLM fallback (ontology generation)
- primary LLM (智谱 GLM-4-Flash) retries exhausted → single attempt fallback to
  SiliconFlow Qwen 2.5-32B via GRAPHITI_LLM_* config
- solves GLM low RPM quota intermittent throttling
- fallback does NOT retry (avoid cascade)
- file: backend/app/services/ontology_generator.py

### #3 Qwen 32K context overflow
- MAX_TEXT_LENGTH_FOR_LLM 50000 → 28000 chars
- ensures prompt+response stays under 32768 tokens
- truncated docs get marker line "[文档已截断以适应 LLM 上下文窗口]"
- file: backend/app/services/ontology_generator.py

### #4 Neo4j entity summary flatten via monkey-patch
- Qwen occasionally returns {summary: {value, type, title, description}}
  which Neo4j rejects (properties must be primitives)
- monkey-patch Neo4jEntityNodeOperations.save / save_bulk
- recursive _flatten_entity_property extracts .value from nested dicts,
  json.dumps as string fallback
- file: backend/app/services/graphiti_client.py

### #5 Frontend stuck at /process/new with no pending state
- pendingUpload is in-memory reactive, lost on refresh / direct URL
- MainView.handleNewProject early-returned on empty state without navigating,
  leaving UI permanently in "waiting for ontology" limbo
- fix: detect empty state, log redirect msg, router.replace to Home after 800ms
- file: frontend/src/views/MainView.vue

### #6 Semaphore 100 → 30 (OASIS simulation)
- high concurrency triggered GLM rate limit cascade during profile/action LLM calls
- drop to 30 eliminates 1302 retries, total runtime impact < 10%
- file: backend/scripts/run_parallel_simulation.py

### #7 SimulationReplayView redesigned to Manus cinematic style
- previous 3-column analyst panel felt like a dashboard, not a replay
- new single-column immersive layout matching Manus's "observation window":
  - top breadcrumb: "Foresight is running Reddit simulation · Round 8/15"
  - main stage: browser chrome + native-style platform post card
    (Reddit subreddit header / Twitter tweet header)
  - action type badges, agent avatars with gradient palettes
  - stage meta bar: per-round action counts by type
- bottom scrubber row: timestamp chip + Jump to live button + live indicator
  (pulsing green dot when sim is running) + step/play/step buttons + 5 speed levels
- bottom task bar: current pipeline step icon/label/progress
- dark theme (#0A0A0B base + #FF5722 accent)
- analyst mode toggle (◫/▦) preserves original 3-column view
- auto-polling every 10s while sim is running, auto-tracks live position
- file: frontend/src/views/SimulationReplayView.vue

## Docs
- PRD.md bumped to v0.3.1 with full hotfix changelog + decision log entries
 2026-04-15 13:25:05 +08:00
liyizhouAI 1fef01d979 feat: v0.3 - Manus replay + token tracking + SIGTERM fix + accelerate button 
This is the v0.3 milestone commit before the v0.4 big version push.
Major themes: process replay, runtime stability, cost observability.

## New Features

- **Manus-style process replay** (frontend + backend)
  - `GET /api/simulation/<id>/replay` returns full workflow + agents + rounds + aggregate
  - `frontend/src/views/SimulationReplayView.vue` 3-column layout (workflow / actions / stats)
  - bottom scrubber with play/pause/step + 5 speed levels (0.5x-10x)
  - filters out stale actions from previous runs via latest simulation_start timestamp

- **Token usage tracking** (`backend/app/utils/token_tracker.py`)
  - process-wide stage→model→tokens counter
  - LLMClient auto-records prompt/completion tokens after each call
  - stages tagged at API entry: step1_ontology, step2_graph_build, step3_prepare, step5_report
  - `GET /api/usage/summary` for live stats + CNY cost estimate
  - `GET /api/usage/estimate-simulation` for OASIS subprocess estimation
  - pricing table for GLM/SiliconFlow/MiniMax/OpenAI/Anthropic models
  - documented as internal-use, removed from customer-facing builds

- **Step 2 "Skip & Continue" button**
  - lets user stop profile generation early and proceed with what's already generated
  - `simulation_manager.request_accelerate()` + cancel_check in oasis_profile_generator
  - new endpoint `POST /api/simulation/prepare/accelerate`

## Critical Bug Fix

- **SIGTERM no longer kills running simulation subprocess**
  - root cause: `SimulationRunner.register_cleanup()` registered SIGTERM/SIGINT/SIGHUP handlers
    that called `os.killpg` on every tracked sim child, even though spawn already used
    `start_new_session=True` to give children isolated sessions
  - fix: neutered `register_cleanup` to a no-op; `cleanup_all_simulations` itself preserved
    for explicit stop_simulation paths
  - validated: killed Flask backend twice, simulation subprocess kept running
  - impact: hot-reload backend code without interrupting in-flight simulations

## Performance & Tuning

- semaphore 30 → 100 (twitter + reddit) for higher LLM concurrency
- discovered 200 agents as memory/cost/statistical sweet spot for 8G server
  (503 agents OOMs both platforms; 200 agents fits cleanly with 95% confidence margin)

## Documentation

- **PRD.md** rewritten as v0.3 baseline (10 chapters + 2 appendices, 639 lines)
  - product positioning across 3 usage modes (one-shot / model-reuse / SaaS)
  - v0.4 roadmap: domestic platforms (douyin/wechat/xiaohongshu/weibo), fork sim, multi-tenant
  - operational lessons: HF mirror, Tencent PyPI mirror, GLM-4-Flash choice, agent count
  - decision log with dates
  - per-stage token/cost breakdown for typical 200-agent run
- **README.md / README-ZH.md** updated with replay step + Graphiti+Neo4j

## Files Touched

19 files changed, 1105 insertions(+), 246 deletions(-)
 2026-04-15 09:37:22 +08:00
liyizhouAI 8088b927f0 fix: accept all Entity nodes from Graphiti (no custom labels) 
Graphiti doesn't assign custom labels like Zep does - all nodes get
generic 'Entity' label. Updated filter to accept all named Entity
nodes instead of requiring custom labels like 'Person', 'Organization'.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
 2026-04-14 19:13:50 +08:00
liyizhouAI 67f33a2336 perf: 4x speedup for graph building 
- Increase chunk size 500→2000 (reduces chunks from ~125 to ~32 for 60K docs)
- Use Graphiti add_episode_bulk for parallel processing (5 episodes per batch)
- Fallback to sequential if bulk fails
- Target: 60K doc graph build in ~10-15 min instead of 45 min

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
 2026-04-14 14:09:41 +08:00
liyizhouAI 339f2f2e50 fix: Neo4j DateTime serialization + match all edge types 
- Convert Neo4j DateTime objects to strings via _safe_str() helper
- Query all edge types (RELATES_TO + MENTIONS), not just RELATES_TO
- Fix get_node_edges to match any relationship type

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
 2026-04-14 13:38:11 +08:00
liyizhouAI d5a908077a perf: upgrade Graphiti LLM to Qwen 32B + improve progress feedback 
- Upgrade from Qwen2.5-7B to Qwen2.5-32B-Instruct (faster, better quality)
- Add real-time progress messages: "正在处理第 X/Y 个文本块"
- Log per-episode processing time
- Expand progress range from 15-55% to 15-90% (no more episode polling step)
- Add GRAPHITI_LLM_* separate config for graph building LLM

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
 2026-04-13 15:31:16 +08:00
liyizhouAI a35ba347f3 fix: use SiliconFlow free LLM+embedding for Graphiti 
- Graphiti LLM: SiliconFlow Qwen/Qwen2.5-7B-Instruct (free, supports structured output)
- Graphiti Embedding: SiliconFlow BAAI/bge-m3 (free, OpenAI-compatible)
- MiniMax Coding Plan doesn't support structured output needed by Graphiti
- Separate GRAPHITI_LLM_* config from main LLM_* config
- Remove _wait_for_episodes call (Graphiti processes synchronously)

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
 2026-04-13 14:09:26 +08:00
liyizhouAI 05a5ba0775 fix: add MiniMax embedder and reranker for Graphiti 
- MiniMaxEmbedder: custom EmbedderClient for MiniMax's non-OpenAI-compatible
  embedding API (uses 'texts' field instead of 'input')
- Pass LLM config to OpenAIRerankerClient to avoid OPENAI_API_KEY requirement
- Both embedder and reranker now use MiniMax API credentials

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
 2026-04-13 13:12:30 +08:00
liyizhouAI fff7edce2a feat: migrate from Zep Cloud to self-hosted Graphiti + Neo4j 
Replace Zep Cloud ($25/mo) with open-source Graphiti + Neo4j:
- New graphiti_client.py: unified wrapper with sync bridge for async Graphiti
- Modified graph_builder.py: use Graphiti add_episode instead of Zep batch API
- Modified zep_entity_reader.py: Neo4j Cypher queries replace Zep pagination
- Modified zep_tools.py: Graphiti search replaces Zep Cloud search
- Modified zep_graph_memory_updater.py: Graphiti add_episode replaces Zep add
- Modified oasis_profile_generator.py: GraphitiClient replaces Zep client
- Updated config.py: NEO4J_URI/USER/PASSWORD replace ZEP_API_KEY
- Updated requirements.txt: graphiti-core + neo4j replace zep-cloud
- Added PRD.md: product requirements document with token analysis

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
 2026-04-13 12:07:32 +08:00
liyizhouAI 36e616101a chore: remove accidentally committed venv, add to .gitignore 2026-04-11 16:40:27 +08:00
liyizhouAI ded715feb2 feat: merge upstream i18n + rebrand MiroFish → Foresight 先见之明 
- Merge 41 upstream commits (i18n for 7 languages, security fixes, new features)
- Rebrand all MiroFish references to Foresight/先见之明 across 37 files
- Re-apply dark theme CSS overhaul (pure black/gray, no blue tints)
- Re-apply Teleport-based theme toggle (inline with brand, 20px gap)
- Restore Foresight logo and favicon
- Update GitHub links to liyizhouAI/foresight
- Update locale files, README, package.json

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
 2026-04-11 16:40:13 +08:00
BaiFu af71244974
Merge pull request #428 from Ghostubborn/feat/i18n 
feat(i18n): 添加多语言切换功能，支持中英文
 2026-04-02 14:27:04 +08:00
ghostubborn f2404903d6 fix(i18n): validate Accept-Language header against registered locales 2026-04-02 14:20:15 +08:00
ghostubborn 24e9bee5be feat(i18n): replace all user-visible Chinese logger messages in zep_tools.py 
These are shown to users via ConsoleLogger in the report page.
 2026-04-01 17:46:39 +08:00
ghostubborn e79569ab4f feat(i18n): replace all user-visible Chinese in report_agent.py 
Covers ReportLogger message fields and logger messages shown via ConsoleLogger.
 2026-04-01 17:44:52 +08:00
666ghj e3350a919d fix(graph): enforce PascalCase for entity names and SCREAMING_SNAKE_CASE for edge names in ontology validation 2026-04-01 17:42:27 +08:00
ghostubborn 380e456d41 fix(i18n): replace hardcoded Chinese stage names in simulation prepare SSE 2026-04-01 17:31:00 +08:00
ghostubborn 0e55e4cf6b feat(i18n): replace remaining Chinese in config generator and profile generator 
Also update simulation prompts to be locale-neutral for timezone/schedule.
 2026-04-01 17:19:12 +08:00
ghostubborn 7c07237544 fix(i18n): pass locale to background threads via thread-local storage 
Background threads (graph building, simulation prep, report generation,
profile generation) now inherit the requesting user's locale preference.
Previously these fell back to 'zh' because Flask request context was
unavailable in spawned threads.
 2026-04-01 16:55:51 +08:00
ghostubborn 592ee52f59 feat(i18n): replace remaining hardcoded Chinese in progress callbacks 2026-04-01 16:53:29 +08:00
ghostubborn da2490ec31 fix(i18n): protect JSON field values from language instruction in config generator 
Ensure poster_type stays PascalCase English and stance stays English enum
values regardless of language setting. Only natural language fields follow
the user's language preference.
 2026-04-01 16:41:22 +08:00
ghostubborn 97aa58384e fix(i18n): ensure ontology names stay PascalCase regardless of language setting 
The language instruction was causing LLM to change entity/relation naming
conventions. Now explicitly enforce PascalCase/UPPER_SNAKE_CASE for technical
identifiers while only applying language preference to description fields.
 2026-04-01 16:40:17 +08:00
ghostubborn 9d43b77511 feat(i18n): replace hardcoded Chinese in backend SSE progress messages 2026-04-01 16:32:10 +08:00
ghostubborn f75c6487b3 fix(i18n): replace remaining hardcoded language directives in LLM prompts 
- oasis_profile_generator: replace hardcoded "使用中文" with dynamic get_language_instruction()
- ontology_generator: remove hardcoded "（中文）" from schema annotation
- report_agent: replace Chinese-specific language consistency rules with language-neutral ones
- zep_tools: dynamically select quote style based on locale
 2026-04-01 15:55:04 +08:00
ghostubborn 74f673a238 feat(i18n): replace hardcoded Chinese in backend API responses with t() calls 2026-04-01 15:32:24 +08:00
ghostubborn 8f6110df0f feat(i18n): inject language instruction into LLM system prompts 2026-04-01 15:24:12 +08:00
ghostubborn 0c18e1aeca feat(i18n): add backend translation utility with shared locale files 2026-04-01 15:22:14 +08:00
666ghj 985f89f49a fix: resolve 500 error caused by <think> tags and markdown code fences in content field from reasoning models like MiniMax/GLM 2026-03-06 00:30:31 +08:00
666ghj da6548e96f feat(graph): implement pagination for fetching nodes and edges; add utility functions for streamlined data retrieval 2026-02-27 15:53:29 +08:00
666ghj 25aa4f75d2 fix(report_agent): refine tool call handling and response validation; enforce strict separation between tool calls and final answers 2026-02-24 17:47:44 +08:00
666ghj 08ec856a58 fix(report_agent): update max_agents parameter description and enforce maximum limit of 10 agents 2026-02-14 18:35:05 +08:00
666ghj ddd9ff2479 feat(report_agent): update report language consistency guidelines; ensure all quoted content is translated to the report language for clarity 2026-02-14 18:24:03 +08:00
666ghj 7601d78fd4 feat(report_agent): enhance interview text processing and response handling; improve quote extraction and formatting for better clarity 2026-02-14 16:56:48 +08:00
666ghj dc0a9261d1 feat(report_agent): add detailed tool descriptions and prompts for future prediction report generation 2026-02-14 15:16:17 +08:00
666ghj d2041f6fb8 fix(report_agent): update description of insight_forge tool to remove "最强大" and enhance clarity 2026-02-14 14:48:23 +08:00
666ghj 0a59bace92 fix(report_agent): increase minimum tool call requirement from 2 to 3 per chapter and enhance user prompts to encourage diverse tool usage 2026-02-06 19:37:52 +08:00
666ghj e004fe8f14 fix(report_agent): update tool call requirements in content generation to allow up to 5 tool calls per chapter and clarify user prompts for insufficient data 2026-02-06 18:34:19 +08:00
666ghj f9abaf8e9f refactor(report_agent, Step4Report): simplify logging and remove subsection handling; update UI to reflect changes in section content generation 2026-02-06 18:13:30 +08:00
666ghj 54f1291967 fix(report_agent): handle None responses from LLM during content generation and enforce fallback behavior 2026-01-29 17:08:39 +08:00
666ghj 40f7035947 fix(config): enable overriding of environment variables when loading .env file 2026-01-23 16:20:24 +08:00
666ghj 390c120fef fix(file_parser): handle non-UTF-8 encoded text files with automatic encoding detection 2026-01-22 18:28:37 +08:00
666ghj 56b8babf17 feat(ZepGraphMemoryUpdater): add platform display name mapping and logging enhancements. 2026-01-16 09:00:10 +08:00
666ghj e25d2e38f4 feat(SimulationAPI): add function to retrieve latest report ID for a given simulation 
- Implemented `_get_report_id_for_simulation` to find the most recent report ID associated with a simulation ID by scanning the reports directory.
- Updated `get_simulation_history` to include the retrieved report ID in the response, enhancing the simulation data returned to the client.
 2026-01-09 16:04:35 +08:00
666ghj 992f7d13c3 refactor(simulation): enhance simulation data retrieval and project file handling 
- Updated simulation history retrieval to read project details directly from the Simulation file.
- Improved simulation configuration handling by reading simulation requirements from JSON.
- Added project file listing to the simulation history, displaying up to three associated files.
- Refined card layout in HistoryDatabase.vue to accommodate new file display features and improved responsiveness.
 2026-01-07 18:54:33 +08:00
666ghj e6da45ee63 feat(history): 添加首页历史项目展示组件 
- 新增 HistoryDatabase.vue 组件，实现扇形堆叠到网格展开的动画效果
- 后端 simulation.py 添加历史模拟数据 API 支持
- 修复 SimulationManager 过滤隐藏文件问题
- 前端 simulation.js 添加获取历史模拟数据的 API 方法
- Home.vue 集成历史项目展示组件
- 实现正方形网格背景装饰效果
 2025-12-31 17:54:39 +08:00
666ghj 4be144c3f2 Refactor process termination in SimulationRunner to support cross-platform handling and improve code clarity. Update development script to ensure concurrent processes are terminated correctly. 2025-12-30 17:45:27 +08:00
666ghj 8bd768718e Add SIGHUP signal handling in SimulationRunner for Unix systems 2025-12-30 15:28:26 +08:00
666ghj 067855f7b5 Add UTF-8 encoding support for Windows in simulation_runner.py and run_parallel_simulation.py to resolve character encoding issues with third-party libraries. 2025-12-26 18:14:57 +08:00
666ghj f46c1a9ec7 Add UTF-8 encoding support for Windows console in run.py and logger.py to prevent character encoding issues 2025-12-26 17:58:48 +08:00
666ghj 4e96019e0d Change project license from Apache-2.0 to AGPL-3.0 in package.json, package-lock.json, and backend/pyproject.toml. 2025-12-22 16:41:45 +08:00