Commit Graph

275 Commits

Author SHA1 Message Date
liyizhouAI f1f2ea4ea5 fix: v0.3.1 - 7 bug fixes unblock end-to-end production flow
First successful E2E test uncovered 7 blocking bugs. All fixed and deployed
to production server. User confirmed full pipeline now runs through.

## Bug Fixes

### #1 LLMClient exponential backoff retry
- recognize RateLimitError / 429 / 5xx / 1302 / timeout / connection errors
- 5 retries with 1→2→4→8→16s backoff + random jitter
- file: backend/app/utils/llm_client.py

### #2 GLM → Qwen 32B dual-LLM fallback (ontology generation)
- primary LLM (智谱 GLM-4-Flash) retries exhausted → single attempt fallback to
  SiliconFlow Qwen 2.5-32B via GRAPHITI_LLM_* config
- solves GLM low RPM quota intermittent throttling
- fallback does NOT retry (avoid cascade)
- file: backend/app/services/ontology_generator.py

### #3 Qwen 32K context overflow
- MAX_TEXT_LENGTH_FOR_LLM 50000 → 28000 chars
- ensures prompt+response stays under 32768 tokens
- truncated docs get marker line "[文档已截断以适应 LLM 上下文窗口]"
- file: backend/app/services/ontology_generator.py

### #4 Neo4j entity summary flatten via monkey-patch
- Qwen occasionally returns {summary: {value, type, title, description}}
  which Neo4j rejects (properties must be primitives)
- monkey-patch Neo4jEntityNodeOperations.save / save_bulk
- recursive _flatten_entity_property extracts .value from nested dicts,
  json.dumps as string fallback
- file: backend/app/services/graphiti_client.py

### #5 Frontend stuck at /process/new with no pending state
- pendingUpload is in-memory reactive, lost on refresh / direct URL
- MainView.handleNewProject early-returned on empty state without navigating,
  leaving UI permanently in "waiting for ontology" limbo
- fix: detect empty state, log redirect msg, router.replace to Home after 800ms
- file: frontend/src/views/MainView.vue

### #6 Semaphore 100 → 30 (OASIS simulation)
- high concurrency triggered GLM rate limit cascade during profile/action LLM calls
- drop to 30 eliminates 1302 retries, total runtime impact < 10%
- file: backend/scripts/run_parallel_simulation.py

### #7 SimulationReplayView redesigned to Manus cinematic style
- previous 3-column analyst panel felt like a dashboard, not a replay
- new single-column immersive layout matching Manus's "observation window":
  - top breadcrumb: "Foresight is running Reddit simulation · Round 8/15"
  - main stage: browser chrome + native-style platform post card
    (Reddit subreddit header / Twitter tweet header)
  - action type badges, agent avatars with gradient palettes
  - stage meta bar: per-round action counts by type
- bottom scrubber row: timestamp chip + Jump to live button + live indicator
  (pulsing green dot when sim is running) + step/play/step buttons + 5 speed levels
- bottom task bar: current pipeline step icon/label/progress
- dark theme (#0A0A0B base + #FF5722 accent)
- analyst mode toggle (◫/▦) preserves original 3-column view
- auto-polling every 10s while sim is running, auto-tracks live position
- file: frontend/src/views/SimulationReplayView.vue

## Docs
- PRD.md bumped to v0.3.1 with full hotfix changelog + decision log entries
2026-04-15 13:25:05 +08:00
liyizhouAI 1fef01d979 feat: v0.3 - Manus replay + token tracking + SIGTERM fix + accelerate button
This is the v0.3 milestone commit before the v0.4 big version push.
Major themes: process replay, runtime stability, cost observability.

## New Features

- **Manus-style process replay** (frontend + backend)
  - `GET /api/simulation/<id>/replay` returns full workflow + agents + rounds + aggregate
  - `frontend/src/views/SimulationReplayView.vue` 3-column layout (workflow / actions / stats)
  - bottom scrubber with play/pause/step + 5 speed levels (0.5x-10x)
  - filters out stale actions from previous runs via latest simulation_start timestamp

- **Token usage tracking** (`backend/app/utils/token_tracker.py`)
  - process-wide stage→model→tokens counter
  - LLMClient auto-records prompt/completion tokens after each call
  - stages tagged at API entry: step1_ontology, step2_graph_build, step3_prepare, step5_report
  - `GET /api/usage/summary` for live stats + CNY cost estimate
  - `GET /api/usage/estimate-simulation` for OASIS subprocess estimation
  - pricing table for GLM/SiliconFlow/MiniMax/OpenAI/Anthropic models
  - documented as internal-use, removed from customer-facing builds

- **Step 2 "Skip & Continue" button**
  - lets user stop profile generation early and proceed with what's already generated
  - `simulation_manager.request_accelerate()` + cancel_check in oasis_profile_generator
  - new endpoint `POST /api/simulation/prepare/accelerate`

## Critical Bug Fix

- **SIGTERM no longer kills running simulation subprocess**
  - root cause: `SimulationRunner.register_cleanup()` registered SIGTERM/SIGINT/SIGHUP handlers
    that called `os.killpg` on every tracked sim child, even though spawn already used
    `start_new_session=True` to give children isolated sessions
  - fix: neutered `register_cleanup` to a no-op; `cleanup_all_simulations` itself preserved
    for explicit stop_simulation paths
  - validated: killed Flask backend twice, simulation subprocess kept running
  - impact: hot-reload backend code without interrupting in-flight simulations

## Performance & Tuning

- semaphore 30 → 100 (twitter + reddit) for higher LLM concurrency
- discovered 200 agents as memory/cost/statistical sweet spot for 8G server
  (503 agents OOMs both platforms; 200 agents fits cleanly with 95% confidence margin)

## Documentation

- **PRD.md** rewritten as v0.3 baseline (10 chapters + 2 appendices, 639 lines)
  - product positioning across 3 usage modes (one-shot / model-reuse / SaaS)
  - v0.4 roadmap: domestic platforms (douyin/wechat/xiaohongshu/weibo), fork sim, multi-tenant
  - operational lessons: HF mirror, Tencent PyPI mirror, GLM-4-Flash choice, agent count
  - decision log with dates
  - per-stage token/cost breakdown for typical 200-agent run
- **README.md / README-ZH.md** updated with replay step + Graphiti+Neo4j

## Files Touched

19 files changed, 1105 insertions(+), 246 deletions(-)
2026-04-15 09:37:22 +08:00
liyizhouAI 8088b927f0 fix: accept all Entity nodes from Graphiti (no custom labels)
Graphiti doesn't assign custom labels like Zep does - all nodes get
generic 'Entity' label. Updated filter to accept all named Entity
nodes instead of requiring custom labels like 'Person', 'Organization'.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-14 19:13:50 +08:00
liyizhouAI 67f33a2336 perf: 4x speedup for graph building
- Increase chunk size 500→2000 (reduces chunks from ~125 to ~32 for 60K docs)
- Use Graphiti add_episode_bulk for parallel processing (5 episodes per batch)
- Fallback to sequential if bulk fails
- Target: 60K doc graph build in ~10-15 min instead of 45 min

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-14 14:09:41 +08:00
liyizhouAI c46a166cf5 fix: CORS duplicate header + Neo4j query + DateTime serialization
- Remove CORS headers from nginx (let Flask-CORS handle it alone)
- Fix Neo4j edge query to match all relationship types (RELATES_TO + MENTIONS)
- Convert Neo4j DateTime objects to strings for JSON serialization
- Add .serena/, .vercel/, .venv_direct/ to gitignore

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-14 13:53:47 +08:00
liyizhouAI 339f2f2e50 fix: Neo4j DateTime serialization + match all edge types
- Convert Neo4j DateTime objects to strings via _safe_str() helper
- Query all edge types (RELATES_TO + MENTIONS), not just RELATES_TO
- Fix get_node_edges to match any relationship type

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-14 13:38:11 +08:00
liyizhouAI c8eb8711cc chore: ignore playwright-mcp cache files
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-14 09:25:37 +08:00
liyizhouAI a245106514 feat: add error recovery UI + HTTPS API endpoint
- Add retry/home buttons when errors occur in graph building
- Improve error messages: Network Error, timeout, missing files
- Clear guidance for users when page refresh loses upload state
- API endpoint switched from HTTP to HTTPS (api.foresight.yizhou.chat)

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-13 23:00:15 +08:00
liyizhouAI d5a908077a perf: upgrade Graphiti LLM to Qwen 32B + improve progress feedback
- Upgrade from Qwen2.5-7B to Qwen2.5-32B-Instruct (faster, better quality)
- Add real-time progress messages: "正在处理第 X/Y 个文本块"
- Log per-episode processing time
- Expand progress range from 15-55% to 15-90% (no more episode polling step)
- Add GRAPHITI_LLM_* separate config for graph building LLM

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-13 15:31:16 +08:00
liyizhouAI a35ba347f3 fix: use SiliconFlow free LLM+embedding for Graphiti
- Graphiti LLM: SiliconFlow Qwen/Qwen2.5-7B-Instruct (free, supports structured output)
- Graphiti Embedding: SiliconFlow BAAI/bge-m3 (free, OpenAI-compatible)
- MiniMax Coding Plan doesn't support structured output needed by Graphiti
- Separate GRAPHITI_LLM_* config from main LLM_* config
- Remove _wait_for_episodes call (Graphiti processes synchronously)

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-13 14:09:26 +08:00
liyizhouAI 05a5ba0775 fix: add MiniMax embedder and reranker for Graphiti
- MiniMaxEmbedder: custom EmbedderClient for MiniMax's non-OpenAI-compatible
  embedding API (uses 'texts' field instead of 'input')
- Pass LLM config to OpenAIRerankerClient to avoid OPENAI_API_KEY requirement
- Both embedder and reranker now use MiniMax API credentials

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-13 13:12:30 +08:00
liyizhouAI fff7edce2a feat: migrate from Zep Cloud to self-hosted Graphiti + Neo4j
Replace Zep Cloud ($25/mo) with open-source Graphiti + Neo4j:
- New graphiti_client.py: unified wrapper with sync bridge for async Graphiti
- Modified graph_builder.py: use Graphiti add_episode instead of Zep batch API
- Modified zep_entity_reader.py: Neo4j Cypher queries replace Zep pagination
- Modified zep_tools.py: Graphiti search replaces Zep Cloud search
- Modified zep_graph_memory_updater.py: Graphiti add_episode replaces Zep add
- Modified oasis_profile_generator.py: GraphitiClient replaces Zep client
- Updated config.py: NEO4J_URI/USER/PASSWORD replace ZEP_API_KEY
- Updated requirements.txt: graphiti-core + neo4j replace zep-cloud
- Added PRD.md: product requirements document with token analysis

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-13 12:07:32 +08:00
liyizhouAI 4b682f24f4 feat: dark mode full-page responsive layout overrides
Override inner container max-width constraints (.main-content, .dashboard-section,
.content-area, .main-content-area, .panel-wrapper) so dark mode content fills
widescreen displays like light mode does natively.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-11 17:32:52 +08:00
liyizhouAI 36e616101a chore: remove accidentally committed venv, add to .gitignore 2026-04-11 16:40:27 +08:00
liyizhouAI ded715feb2 feat: merge upstream i18n + rebrand MiroFish → Foresight 先见之明
- Merge 41 upstream commits (i18n for 7 languages, security fixes, new features)
- Rebrand all MiroFish references to Foresight/先见之明 across 37 files
- Re-apply dark theme CSS overhaul (pure black/gray, no blue tints)
- Re-apply Teleport-based theme toggle (inline with brand, 20px gap)
- Restore Foresight logo and favicon
- Update GitHub links to liyizhouAI/foresight
- Update locale files, README, package.json

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2026-04-11 16:40:13 +08:00
666ghj fa0f6519b1 docs: rename README-EN.md to README.md as default English documentation 2026-04-02 16:52:29 +08:00
666ghj 0e9420e0f8 docs: rename README.md to README-ZH.md for Chinese documentation 2026-04-02 16:52:29 +08:00
BaiFu 7d07fb7f03
Merge pull request #440 from Ghostubborn/fix/security-deps
fix(security): 修复前端 3 个高危依赖漏洞
2026-04-02 15:17:46 +08:00
ghostubborn 223b283da7 fix(security): upgrade axios, rollup, picomatch to fix 3 high severity vulnerabilities 2026-04-02 15:00:33 +08:00
BaiFu af71244974
Merge pull request #428 from Ghostubborn/feat/i18n
feat(i18n): 添加多语言切换功能,支持中英文
2026-04-02 14:27:04 +08:00
ghostubborn ed465908db fix(i18n): set HTML lang attribute before Vue mounts via inline script 2026-04-02 14:21:09 +08:00
ghostubborn 65df257e19 chore(deps): upgrade vue-i18n from v9 to v11 2026-04-02 14:20:50 +08:00
ghostubborn f2404903d6 fix(i18n): validate Accept-Language header against registered locales 2026-04-02 14:20:15 +08:00
ghostubborn 2421010fe1 fix(i18n): fix English workflow desc font size with correct CSS selectors 2026-04-01 19:11:22 +08:00
ghostubborn 3929c3ade2 fix(i18n): further shorten English metrics and improve workflow layout 2026-04-01 19:07:19 +08:00
ghostubborn 21922da6cc fix(i18n): improve English layout for homepage left-pane and report title
- Add sans-serif font for English left-pane (status, workflow sections)
- Shorten English workflow step descriptions
- Reduce English report title font-size from 36px to 28px
2026-04-01 19:04:38 +08:00
ghostubborn c6cafdd532 fix(i18n): translate world1/world2 platform labels in interview tool display 2026-04-01 18:38:22 +08:00
ghostubborn 5072a2eaa8 feat(i18n): replace Chinese UI text in Step4Report.vue render functions
Only UI display text is replaced. Regex parsing patterns are kept as-is
since they match the backend output format.
2026-04-01 18:35:18 +08:00
ghostubborn 6db3f98a48 fix(i18n): fix English homepage layout with proper font and shorter copy
- Use sans-serif font for English titles, descriptions and navbar
- Shorten English hero text to avoid overflow
- Fix :global() scoped CSS issue that was setting root font-size to 3.5rem
- Use separate unscoped style block for html[lang] selectors
2026-04-01 18:04:05 +08:00
ghostubborn 24e9bee5be feat(i18n): replace all user-visible Chinese logger messages in zep_tools.py
These are shown to users via ConsoleLogger in the report page.
2026-04-01 17:46:39 +08:00
ghostubborn e79569ab4f feat(i18n): replace all user-visible Chinese in report_agent.py
Covers ReportLogger message fields and logger messages shown via ConsoleLogger.
2026-04-01 17:44:52 +08:00
ghostubborn 1d358fc492 feat(i18n): replace expand/collapse Chinese text in Step4Report.vue 2026-04-01 17:44:45 +08:00
666ghj e3350a919d fix(graph): enforce PascalCase for entity names and SCREAMING_SNAKE_CASE for edge names in ontology validation 2026-04-01 17:42:27 +08:00
ghostubborn 380e456d41 fix(i18n): replace hardcoded Chinese stage names in simulation prepare SSE 2026-04-01 17:31:00 +08:00
ghostubborn 3a8451c119 feat(i18n): replace remaining hardcoded Chinese in frontend addLog calls 2026-04-01 17:21:55 +08:00
ghostubborn 0e55e4cf6b feat(i18n): replace remaining Chinese in config generator and profile generator
Also update simulation prompts to be locale-neutral for timezone/schedule.
2026-04-01 17:19:12 +08:00
ghostubborn 7c07237544 fix(i18n): pass locale to background threads via thread-local storage
Background threads (graph building, simulation prep, report generation,
profile generation) now inherit the requesting user's locale preference.
Previously these fell back to 'zh' because Flask request context was
unavailable in spawned threads.
2026-04-01 16:55:51 +08:00
ghostubborn 592ee52f59 feat(i18n): replace remaining hardcoded Chinese in progress callbacks 2026-04-01 16:53:29 +08:00
ghostubborn da2490ec31 fix(i18n): protect JSON field values from language instruction in config generator
Ensure poster_type stays PascalCase English and stance stays English enum
values regardless of language setting. Only natural language fields follow
the user's language preference.
2026-04-01 16:41:22 +08:00
ghostubborn 97aa58384e fix(i18n): ensure ontology names stay PascalCase regardless of language setting
The language instruction was causing LLM to change entity/relation naming
conventions. Now explicitly enforce PascalCase/UPPER_SNAKE_CASE for technical
identifiers while only applying language preference to description fields.
2026-04-01 16:40:17 +08:00
ghostubborn e1db8bacc2 feat(i18n): replace hardcoded Chinese in frontend addLog() messages 2026-04-01 16:35:35 +08:00
ghostubborn 9d43b77511 feat(i18n): replace hardcoded Chinese in backend SSE progress messages 2026-04-01 16:32:10 +08:00
ghostubborn ffe6369c52 fix(i18n): fix curly quotes in Home.vue and remove unused dark theme from LanguageSwitcher 2026-04-01 16:23:52 +08:00
ghostubborn f75c6487b3 fix(i18n): replace remaining hardcoded language directives in LLM prompts
- oasis_profile_generator: replace hardcoded "使用中文" with dynamic get_language_instruction()
- ontology_generator: remove hardcoded "(中文)" from schema annotation
- report_agent: replace Chinese-specific language consistency rules with language-neutral ones
- zep_tools: dynamically select quote style based on locale
2026-04-01 15:55:04 +08:00
ghostubborn fbaa3dbbeb fix(i18n): fix curly quotes and missing @ alias in vite config 2026-04-01 15:45:43 +08:00
ghostubborn fc47ae81b5 feat(i18n): replace hardcoded Chinese in frontend components with i18n calls
Replace all user-visible hardcoded Chinese strings in 7 component files
with $t() / t() calls using vue-i18n:

- Step1GraphBuild: ontology generation, graph build, status badges
- Step2EnvSetup: simulation setup, agent personas, platform config,
  time config, initial activation, modal profile details
- Step3Simulation: report generation button
- Step4Report: section loading text, interaction button
- Step5Interaction: chat interface, survey UI, tool descriptions,
  error messages, agent selection
- GraphPanel: graph status hints, loading states, tooltips
- HistoryDatabase: history cards, modal, replay buttons

Added missing translation keys to both zh.json and en.json locale files.
Added useI18n imports to components that need script-level translations.
2026-04-01 15:43:11 +08:00
ghostubborn 70833821a2 feat(i18n): replace hardcoded Chinese in frontend views with i18n calls 2026-04-01 15:36:29 +08:00
ghostubborn 74f673a238 feat(i18n): replace hardcoded Chinese in backend API responses with t() calls 2026-04-01 15:32:24 +08:00
ghostubborn 3d5e5d024d feat(i18n): add language switcher component to navigation 2026-04-01 15:24:58 +08:00
ghostubborn 8f6110df0f feat(i18n): inject language instruction into LLM system prompts 2026-04-01 15:24:12 +08:00