Report generation now supports REPORT_LLM_* env vars to route report
LLM calls through a separate provider (SiliconFlow Qwen-32B) instead
of sharing the GLM rate limit with simulation. Also passes the report
LLM client to ZepToolsService so tool calls (PanoramaSearch, QuickSearch,
InsightForge, InterviewAgents) use the same provider.
- Filter out non-social entity types (API, SDK, Database, etc.) from agent configs
reduces agent count ~20%, fewer LLM calls per round
- Make semaphore configurable via SIMULATION_SEMAPHORE env var (default 50, up from 30)
allows higher concurrent LLM requests for faster round execution
- Reddit semaphore also reads from env var
- Estimated speedup: ~2x (25min → ~12min for 40 rounds)
scripts/deploy.sh unifies backend/frontend deployment:
- --backend (default): rsync app+scripts → ubuntu@124.223.92.72:/opt/foresight/backend
+ kill/restart Flask + health check
- --frontend: vite build + coscmd upload to COS bucket foresight-1317962478
+ tccli CDN purge
- --full: both
- --no-restart: skip Flask restart
- --dry-run: preview only
Safety:
- Python syntax check before rsync (aborts on error)
- health check loop after restart (aborts if /health fails 5x)
- rsync --delete excludes __pycache__ / *.pyc / .pytest_cache
- rsync --rsync-path="sudo rsync" for remote permissions
- set -euo pipefail throughout
- "$@" not eval (handles spaces in project path)
Solves:
- Manual deploys of backend bits scattered across sessions missed v0.3
accelerate methods on server until this was caught in production
- No consistent way to push frontend + purge CDN in one step
End of a long debugging rabbit hole. After ~10 attempts patching Graphiti's
compatibility with non-OpenAI LLMs (GLM, Qwen via SiliconFlow), every fix
uncovered another bug. Made the strategic call to abandon Graphiti entirely
for the graph-build path and write a minimal custom builder.
## Why Graphiti had to go
- Qwen 32B 32K context overflow on cumulative episode retrieval (60K-82K tokens)
- SiliconFlow Qwen 72B also 32K (not 128K as docs implied)
- GLM 4 Flash gives 128K context but triggers "20015 parameter invalid" on
Graphiti's structured output calls (logprobs in reranker, empty input in
embedder, nested dict in Neo4j writes)
- Each patch target was 1-2 levels deep in Graphiti internals
- 4 separate monkey-patches (EntityNode.save, bulk_utils, driver layer,
reranker, embedder) still couldn't cover the extract_nodes path
## New architecture
backend/app/services/custom_graph_builder.py (NEW, 348 lines):
- Uses Foresight's own LLMClient (retry + GLM→Qwen fallback + token tracker)
- 1 LLM call per chunk (Graphiti needed 4-5)
- Single prompt extracts entities + relationships as JSON
- Writes directly to Neo4j via cypher MERGE (no Graphiti dependency)
- Schema identical to what Graphiti produced — downstream get_all_nodes /
get_all_edges / zep_entity_reader all work unchanged
- ThreadPoolExecutor (10 workers) for concurrent chunk extraction
- Sequential Neo4j writes via shared session (avoids name conflicts in dedup)
- DDL wrapped in try/except (tolerates pre-existing Graphiti indexes)
- Entity name dedup via in-memory map → single uuid per canonical name
- Regex-sanitized label / relation type to prevent cypher injection
backend/app/api/graph.py:
- /api/graph/build endpoint now calls CustomGraphBuilder instead of
builder.add_text_batches() → client.add_episodes_batch() (the old Graphiti path)
backend/app/services/graph_builder.py:
- _build_graph_worker also updated to use CustomGraphBuilder (dead code path
but kept in sync)
backend/app/models/project.py:
- Default chunk_size 500 → 250 (to keep individual LLM prompts small)
backend/app/services/graphiti_client.py:
- Kept all monkey-patches for backward compat — they now only affect legacy
Graphiti code paths that CustomGraphBuilder bypasses entirely:
* _patch_neo4j_driver (AsyncSession.run nested-dict sanitize)
* _patch_reranker_for_non_openai (GLM logprobs workaround)
* _patch_embedder_empty_input (SiliconFlow empty-input guard)
* _patch_entity_node_ops (EntityNode.save sanitize)
* _patch_add_nodes_and_edges_bulk_tx (bulk-episode sanitize)
- These are kept for safety; Graphiti code path is no longer invoked in the
production build flow but methods like add_episode still exist on the class
## Performance
- Sequential: ~194 chunks × 2-5s = 10-15 min
- Concurrent (10 workers): ~194 / 10 × 3-5s = **1-2 min** (5-8x speedup)
- Rate limiting handled by LLMClient retry/backoff, not raw thread contention
## Downstream compatibility
Verified:
- get_all_nodes: MATCH (n:Entity) WHERE n.group_id = $gid RETURN n, labels(n)
- get_all_edges: MATCH (a)-[r]->(b) WHERE r.group_id = $gid RETURN r, type(r)
- get_node / get_node_edges: MATCH by uuid
CustomGraphBuilder writes:
- (n:Entity [optional second label]) with uuid, name, summary, group_id, created_at
- [r:RELATION_TYPE] with uuid, name, fact, group_id, created_at
Schema matches exactly.
## Pipeline wiring
Step 1 ontology generation → Step 2 graph build (CustomGraphBuilder) →
Step 3 profile generation (ZepEntityReader queries Neo4j) → Step 4 simulation
No changes needed downstream of Step 2.
Graphiti doesn't assign custom labels like Zep does - all nodes get
generic 'Entity' label. Updated filter to accept all named Entity
nodes instead of requiring custom labels like 'Person', 'Organization'.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- Increase chunk size 500→2000 (reduces chunks from ~125 to ~32 for 60K docs)
- Use Graphiti add_episode_bulk for parallel processing (5 episodes per batch)
- Fallback to sequential if bulk fails
- Target: 60K doc graph build in ~10-15 min instead of 45 min
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- Remove CORS headers from nginx (let Flask-CORS handle it alone)
- Fix Neo4j edge query to match all relationship types (RELATES_TO + MENTIONS)
- Convert Neo4j DateTime objects to strings for JSON serialization
- Add .serena/, .vercel/, .venv_direct/ to gitignore
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- Convert Neo4j DateTime objects to strings via _safe_str() helper
- Query all edge types (RELATES_TO + MENTIONS), not just RELATES_TO
- Fix get_node_edges to match any relationship type
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- Add retry/home buttons when errors occur in graph building
- Improve error messages: Network Error, timeout, missing files
- Clear guidance for users when page refresh loses upload state
- API endpoint switched from HTTP to HTTPS (api.foresight.yizhou.chat)
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- Upgrade from Qwen2.5-7B to Qwen2.5-32B-Instruct (faster, better quality)
- Add real-time progress messages: "正在处理第 X/Y 个文本块"
- Log per-episode processing time
- Expand progress range from 15-55% to 15-90% (no more episode polling step)
- Add GRAPHITI_LLM_* separate config for graph building LLM
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- MiniMaxEmbedder: custom EmbedderClient for MiniMax's non-OpenAI-compatible
embedding API (uses 'texts' field instead of 'input')
- Pass LLM config to OpenAIRerankerClient to avoid OPENAI_API_KEY requirement
- Both embedder and reranker now use MiniMax API credentials
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Override inner container max-width constraints (.main-content, .dashboard-section,
.content-area, .main-content-area, .panel-wrapper) so dark mode content fills
widescreen displays like light mode does natively.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- Merge 41 upstream commits (i18n for 7 languages, security fixes, new features)
- Rebrand all MiroFish references to Foresight/先见之明 across 37 files
- Re-apply dark theme CSS overhaul (pure black/gray, no blue tints)
- Re-apply Teleport-based theme toggle (inline with brand, 20px gap)
- Restore Foresight logo and favicon
- Update GitHub links to liyizhouAI/foresight
- Update locale files, README, package.json
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
- Add sans-serif font for English left-pane (status, workflow sections)
- Shorten English workflow step descriptions
- Reduce English report title font-size from 36px to 28px
- Use sans-serif font for English titles, descriptions and navbar
- Shorten English hero text to avoid overflow
- Fix :global() scoped CSS issue that was setting root font-size to 3.5rem
- Use separate unscoped style block for html[lang] selectors
Background threads (graph building, simulation prep, report generation,
profile generation) now inherit the requesting user's locale preference.
Previously these fell back to 'zh' because Flask request context was
unavailable in spawned threads.
Ensure poster_type stays PascalCase English and stance stays English enum
values regardless of language setting. Only natural language fields follow
the user's language preference.
The language instruction was causing LLM to change entity/relation naming
conventions. Now explicitly enforce PascalCase/UPPER_SNAKE_CASE for technical
identifiers while only applying language preference to description fields.
liyizhouAI
666ghj
ghostubborn