Commit Graph

18 Commits

Author SHA1 Message Date
liyizhouAI d2b5b17b98 perf: simulation acceleration - filter non-social entities + configurable semaphore
- Filter out non-social entity types (API, SDK, Database, etc.) from agent configs
  reduces agent count ~20%, fewer LLM calls per round
- Make semaphore configurable via SIMULATION_SEMAPHORE env var (default 50, up from 30)
  allows higher concurrent LLM requests for faster round execution
- Reddit semaphore also reads from env var
- Estimated speedup: ~2x (25min → ~12min for 40 rounds)
2026-04-16 23:11:55 +08:00
liyizhouAI b7df4880c8 fix: v0.3.4 - production stability fixes for simulation pipeline
- LLM JSON truncation: add _repair_truncated_json() to handle max_tokens cutoff
- Ontology generation: increase max_tokens 4096→8192 to prevent truncation
- Config generation: parallel batch processing (3 threads × 30 agents/batch) for ~3x speedup
- Rate limit: reduce semaphore 30→8 to avoid GLM API 429 storms
- HuggingFace offline: set HF_HUB_OFFLINE=1 to skip unreachable hf-mirror.com
- Simulation recovery: auto-reset stuck "preparing" states on Flask restart
- Prepare endpoint: handle concurrent prepare requests gracefully
- Frontend: handle page-refresh-during-prepare edge case
- API: return 429 instead of 500 for LLM rate limit errors
2026-04-16 22:44:40 +08:00
liyizhouAI f1f2ea4ea5 fix: v0.3.1 - 7 bug fixes unblock end-to-end production flow
First successful E2E test uncovered 7 blocking bugs. All fixed and deployed
to production server. User confirmed full pipeline now runs through.

## Bug Fixes

### #1 LLMClient exponential backoff retry
- recognize RateLimitError / 429 / 5xx / 1302 / timeout / connection errors
- 5 retries with 1→2→4→8→16s backoff + random jitter
- file: backend/app/utils/llm_client.py

### #2 GLM → Qwen 32B dual-LLM fallback (ontology generation)
- primary LLM (智谱 GLM-4-Flash) retries exhausted → single attempt fallback to
  SiliconFlow Qwen 2.5-32B via GRAPHITI_LLM_* config
- solves GLM low RPM quota intermittent throttling
- fallback does NOT retry (avoid cascade)
- file: backend/app/services/ontology_generator.py

### #3 Qwen 32K context overflow
- MAX_TEXT_LENGTH_FOR_LLM 50000 → 28000 chars
- ensures prompt+response stays under 32768 tokens
- truncated docs get marker line "[文档已截断以适应 LLM 上下文窗口]"
- file: backend/app/services/ontology_generator.py

### #4 Neo4j entity summary flatten via monkey-patch
- Qwen occasionally returns {summary: {value, type, title, description}}
  which Neo4j rejects (properties must be primitives)
- monkey-patch Neo4jEntityNodeOperations.save / save_bulk
- recursive _flatten_entity_property extracts .value from nested dicts,
  json.dumps as string fallback
- file: backend/app/services/graphiti_client.py

### #5 Frontend stuck at /process/new with no pending state
- pendingUpload is in-memory reactive, lost on refresh / direct URL
- MainView.handleNewProject early-returned on empty state without navigating,
  leaving UI permanently in "waiting for ontology" limbo
- fix: detect empty state, log redirect msg, router.replace to Home after 800ms
- file: frontend/src/views/MainView.vue

### #6 Semaphore 100 → 30 (OASIS simulation)
- high concurrency triggered GLM rate limit cascade during profile/action LLM calls
- drop to 30 eliminates 1302 retries, total runtime impact < 10%
- file: backend/scripts/run_parallel_simulation.py

### #7 SimulationReplayView redesigned to Manus cinematic style
- previous 3-column analyst panel felt like a dashboard, not a replay
- new single-column immersive layout matching Manus's "observation window":
  - top breadcrumb: "Foresight is running Reddit simulation · Round 8/15"
  - main stage: browser chrome + native-style platform post card
    (Reddit subreddit header / Twitter tweet header)
  - action type badges, agent avatars with gradient palettes
  - stage meta bar: per-round action counts by type
- bottom scrubber row: timestamp chip + Jump to live button + live indicator
  (pulsing green dot when sim is running) + step/play/step buttons + 5 speed levels
- bottom task bar: current pipeline step icon/label/progress
- dark theme (#0A0A0B base + #FF5722 accent)
- analyst mode toggle (◫/▦) preserves original 3-column view
- auto-polling every 10s while sim is running, auto-tracks live position
- file: frontend/src/views/SimulationReplayView.vue

## Docs
- PRD.md bumped to v0.3.1 with full hotfix changelog + decision log entries
2026-04-15 13:25:05 +08:00
liyizhouAI 1fef01d979 feat: v0.3 - Manus replay + token tracking + SIGTERM fix + accelerate button
This is the v0.3 milestone commit before the v0.4 big version push.
Major themes: process replay, runtime stability, cost observability.

## New Features

- **Manus-style process replay** (frontend + backend)
  - `GET /api/simulation/<id>/replay` returns full workflow + agents + rounds + aggregate
  - `frontend/src/views/SimulationReplayView.vue` 3-column layout (workflow / actions / stats)
  - bottom scrubber with play/pause/step + 5 speed levels (0.5x-10x)
  - filters out stale actions from previous runs via latest simulation_start timestamp

- **Token usage tracking** (`backend/app/utils/token_tracker.py`)
  - process-wide stage→model→tokens counter
  - LLMClient auto-records prompt/completion tokens after each call
  - stages tagged at API entry: step1_ontology, step2_graph_build, step3_prepare, step5_report
  - `GET /api/usage/summary` for live stats + CNY cost estimate
  - `GET /api/usage/estimate-simulation` for OASIS subprocess estimation
  - pricing table for GLM/SiliconFlow/MiniMax/OpenAI/Anthropic models
  - documented as internal-use, removed from customer-facing builds

- **Step 2 "Skip & Continue" button**
  - lets user stop profile generation early and proceed with what's already generated
  - `simulation_manager.request_accelerate()` + cancel_check in oasis_profile_generator
  - new endpoint `POST /api/simulation/prepare/accelerate`

## Critical Bug Fix

- **SIGTERM no longer kills running simulation subprocess**
  - root cause: `SimulationRunner.register_cleanup()` registered SIGTERM/SIGINT/SIGHUP handlers
    that called `os.killpg` on every tracked sim child, even though spawn already used
    `start_new_session=True` to give children isolated sessions
  - fix: neutered `register_cleanup` to a no-op; `cleanup_all_simulations` itself preserved
    for explicit stop_simulation paths
  - validated: killed Flask backend twice, simulation subprocess kept running
  - impact: hot-reload backend code without interrupting in-flight simulations

## Performance & Tuning

- semaphore 30 → 100 (twitter + reddit) for higher LLM concurrency
- discovered 200 agents as memory/cost/statistical sweet spot for 8G server
  (503 agents OOMs both platforms; 200 agents fits cleanly with 95% confidence margin)

## Documentation

- **PRD.md** rewritten as v0.3 baseline (10 chapters + 2 appendices, 639 lines)
  - product positioning across 3 usage modes (one-shot / model-reuse / SaaS)
  - v0.4 roadmap: domestic platforms (douyin/wechat/xiaohongshu/weibo), fork sim, multi-tenant
  - operational lessons: HF mirror, Tencent PyPI mirror, GLM-4-Flash choice, agent count
  - decision log with dates
  - per-stage token/cost breakdown for typical 200-agent run
- **README.md / README-ZH.md** updated with replay step + Graphiti+Neo4j

## Files Touched

19 files changed, 1105 insertions(+), 246 deletions(-)
2026-04-15 09:37:22 +08:00
666ghj 067855f7b5 Add UTF-8 encoding support for Windows in simulation_runner.py and run_parallel_simulation.py to resolve character encoding issues with third-party libraries. 2025-12-26 18:14:57 +08:00
666ghj 78230b7b38 Remove warnings suppression for multiprocessing in run scripts
- Eliminated unnecessary warnings suppression related to multiprocessing resource tracking in `run.py` and `run_parallel_simulation.py`, streamlining the code and reducing potential confusion.
- This change enhances clarity in logging by allowing relevant warnings to be displayed during execution, improving traceability and debugging capabilities.
2025-12-09 00:41:59 +08:00
666ghj 91eb73ae44 Enhance signal handling and suppress warnings in simulation scripts
- Added signal handling to gracefully manage shutdown events across simulation scripts, ensuring proper cleanup of resources.
- Introduced a global shutdown event to allow simulations to respond to termination signals, improving robustness.
- Suppressed warnings related to multiprocessing resource tracking to avoid unnecessary log clutter during execution.
- Updated cleanup logic in `SimulationRunner` and `ZepGraphMemoryManager` to prevent redundant calls and ensure efficient resource management.
- Enhanced logging to provide clearer feedback during shutdown processes, improving traceability.
2025-12-09 00:37:12 +08:00
666ghj 3f750ffda2 Enhance action context enrichment and update activity logging
- Added context enrichment for actions in `fetch_new_actions_from_db`, providing complete information for posts, comments, and user interactions.
- Introduced a new `_enrich_action_context` function to supplement action arguments with relevant details such as post content, author names, and comment information.
- Updated the `ZepGraphMemoryUpdater` to batch send activities by platform, improving efficiency in processing and logging.
- Enhanced logging to include detailed statistics on sent activities and skipped actions, ensuring better traceability and monitoring of the activity flow.
2025-12-08 18:32:23 +08:00
666ghj 1042d50306 Implement Interview feature for agent interactions in simulations
- Added a new Interview module to facilitate interactions with agents post-simulation, allowing for single and batch interviews.
- Introduced IPC communication mechanism for command and response handling between the Flask backend and simulation scripts.
- Updated README.md to include detailed instructions on the new Interview functionality, including API endpoints and usage examples.
- Enhanced simulation scripts to support waiting for commands after completion, improving user control over the simulation environment.
- Implemented error handling and logging for interview processes, ensuring robust operation and traceability.
2025-12-08 15:55:39 +08:00
666ghj 29bff9ca27 Refactor content handling in activity logging and simulation scripts
- Removed content truncation logic from the AgentActivity class, allowing full content to be logged for posts, comments, and quotes.
- Updated the `fetch_new_actions_from_db` function to retain complete content in action arguments, enhancing data accuracy.
- Adjusted simulation scripts to ensure that full content is sent during action creation, improving the representation of agent activities.
2025-12-05 18:30:40 +08:00
666ghj e4761dab06 Enhance action logging in simulation scripts
- Added logging for the start and end of round 0 in both Twitter and Reddit simulations, improving traceability of initial events.
- Updated the logging mechanism to record round end even when no active agents are present, ensuring comprehensive action tracking.
- Introduced initial action count tracking to provide insights into the number of actions taken during the initial phase of simulations.
2025-12-05 16:26:04 +08:00
666ghj 5b4f02f421 Enhance simulation configuration and management features
- Added support for a `max_rounds` parameter in simulation API, allowing users to limit the number of simulation rounds, improving control over simulation duration.
- Updated README.md to reflect the new `max_rounds` parameter and its usage in simulation requests.
- Enhanced error handling for `max_rounds` input validation to ensure it is a positive integer.
- Modified simulation runner and related scripts to incorporate `max_rounds` functionality, ensuring consistent application across Twitter and Reddit simulations.
- Improved logging to indicate when the number of rounds is truncated due to the `max_rounds` setting, enhancing traceability during simulation execution.
2025-12-05 15:50:54 +08:00
666ghj 3c1d554152 Add function to retrieve agent names from configuration
- Introduced `get_agent_names_from_config` function to map agent IDs to their entity names from the simulation configuration, enhancing clarity in action representation.
- Updated simulation scripts to utilize this new function for fetching agent names, ensuring that real entity names are displayed instead of default identifiers.
- Improved handling of agent names by falling back to default names only if not specified in the configuration, maintaining consistency across simulations.
2025-12-04 19:19:16 +08:00
666ghj 39253b3213 Update .env.example and enhance create_model function for dual LLM configuration
- Updated .env.example to include new keys for dual LLM configuration, allowing for both general and boost settings.
- Modified create_model function to support an optional use_boost parameter, enabling the selection of either general or boost LLM configurations based on availability.
- Improved logging to indicate which LLM configuration is being used during model creation, enhancing clarity for users.
2025-12-04 15:51:25 +08:00
666ghj 0302b8fd70 Add MaxTokensWarningFilter to logging and set semaphore for LLM requests
- Introduced MaxTokensWarningFilter to suppress specific warnings related to max_tokens in the logging output across simulation scripts.
- Added a semaphore parameter to limit the maximum concurrent LLM requests in Twitter and Reddit simulation functions, preventing API overload.
- Ensured the filter is applied immediately upon module loading for effective logging management.
2025-12-04 15:00:08 +08:00
666ghj d4fac63eb4 Enhance simulation management and logging features
- Registered a cleanup function for simulation processes to ensure proper termination on server shutdown.
- Improved logging during application startup to confirm the registration of the cleanup function.
- Updated simulation preparation checks to clarify the conditions for considering a simulation ready, enhancing error handling and user feedback.
- Added detailed logging for simulation status changes, improving traceability during the simulation lifecycle.
- Introduced new files for simulation configuration and profile data, supporting enhanced testing and visualization capabilities.
2025-12-02 17:11:47 +08:00
666ghj 3cc5e3f479 Refactor simulation management and enhance logging capabilities
- Updated simulation preparation checks to exclude script files from the required files list, improving clarity on file management.
- Implemented a robust retry mechanism for Zep API calls in the ZepEntityReader service, enhancing reliability.
- Enhanced logging in simulation scripts to provide clearer insights into the simulation process and errors.
- Updated simulation runner to manage stdout and stderr logs more effectively, ensuring better error tracking.
- Improved profile generation to standardize gender fields and ensure all required fields are populated correctly.
2025-12-02 14:25:53 +08:00
666ghj 5f159f6d88 Enhance backend functionality with OASIS simulation features
- Updated README.md to include new simulation scripts and configuration details for OASIS, including API retry mechanisms and environment variable settings.
- Added simulation management and configuration generation services to streamline the simulation process across Twitter and Reddit platforms.
- Introduced new API routes for simulation-related operations, including entity retrieval and simulation status management.
- Implemented a robust retry mechanism for external API calls to improve system stability.
- Enhanced task management model to include detailed progress tracking.
- Added logging capabilities for action tracking during simulations.
- Included new scripts for running parallel simulations and testing profile formats.
2025-12-01 15:03:44 +08:00