fix: v0.3.1 - 7 bug fixes unblock end-to-end production flow

First successful E2E test uncovered 7 blocking bugs. All fixed and deployed
to production server. User confirmed full pipeline now runs through.

## Bug Fixes

### #1 LLMClient exponential backoff retry
- recognize RateLimitError / 429 / 5xx / 1302 / timeout / connection errors
- 5 retries with 1→2→4→8→16s backoff + random jitter
- file: backend/app/utils/llm_client.py

### #2 GLM → Qwen 32B dual-LLM fallback (ontology generation)
- primary LLM (智谱 GLM-4-Flash) retries exhausted → single attempt fallback to
  SiliconFlow Qwen 2.5-32B via GRAPHITI_LLM_* config
- solves GLM low RPM quota intermittent throttling
- fallback does NOT retry (avoid cascade)
- file: backend/app/services/ontology_generator.py

### #3 Qwen 32K context overflow
- MAX_TEXT_LENGTH_FOR_LLM 50000 → 28000 chars
- ensures prompt+response stays under 32768 tokens
- truncated docs get marker line "[文档已截断以适应 LLM 上下文窗口]"
- file: backend/app/services/ontology_generator.py

### #4 Neo4j entity summary flatten via monkey-patch
- Qwen occasionally returns {summary: {value, type, title, description}}
  which Neo4j rejects (properties must be primitives)
- monkey-patch Neo4jEntityNodeOperations.save / save_bulk
- recursive _flatten_entity_property extracts .value from nested dicts,
  json.dumps as string fallback
- file: backend/app/services/graphiti_client.py

### #5 Frontend stuck at /process/new with no pending state
- pendingUpload is in-memory reactive, lost on refresh / direct URL
- MainView.handleNewProject early-returned on empty state without navigating,
  leaving UI permanently in "waiting for ontology" limbo
- fix: detect empty state, log redirect msg, router.replace to Home after 800ms
- file: frontend/src/views/MainView.vue

### #6 Semaphore 100 → 30 (OASIS simulation)
- high concurrency triggered GLM rate limit cascade during profile/action LLM calls
- drop to 30 eliminates 1302 retries, total runtime impact < 10%
- file: backend/scripts/run_parallel_simulation.py

### #7 SimulationReplayView redesigned to Manus cinematic style
- previous 3-column analyst panel felt like a dashboard, not a replay
- new single-column immersive layout matching Manus's "observation window":
  - top breadcrumb: "Foresight is running Reddit simulation · Round 8/15"
  - main stage: browser chrome + native-style platform post card
    (Reddit subreddit header / Twitter tweet header)
  - action type badges, agent avatars with gradient palettes
  - stage meta bar: per-round action counts by type
- bottom scrubber row: timestamp chip + Jump to live button + live indicator
  (pulsing green dot when sim is running) + step/play/step buttons + 5 speed levels
- bottom task bar: current pipeline step icon/label/progress
- dark theme (#0A0A0B base + #FF5722 accent)
- analyst mode toggle (◫/▦) preserves original 3-column view
- auto-polling every 10s while sim is running, auto-tracks live position
- file: frontend/src/views/SimulationReplayView.vue

## Docs
- PRD.md bumped to v0.3.1 with full hotfix changelog + decision log entries
This commit is contained in:
liyizhouAI 2026-04-15 13:25:05 +08:00
parent 1fef01d979
commit f1f2ea4ea5
7 changed files with 869 additions and 592 deletions

36
PRD.md
View File

@ -1,6 +1,6 @@
# Foresight 先见之明 — 产品需求文档 (PRD)
> **版本**v0.3 — 2026-04-15 大版本前最后一次校准
> **版本**v0.3.1 — 2026-04-15 生产链路打通
> **基线**:基于 [MiroFish](https://github.com/666ghj/MiroFish) v0.1.2 二次开发,已大量重构。
---
@ -449,7 +449,35 @@ FLASK_DEBUG=False
## 8. v0.3 当前状态与已交付
### v0.3 已完成(本次大版本前的全部更新)
### v0.3.1 hotfix2026-04-15 生产链路打通)
经过一次完整的端到端测试,修复了 7 个阻塞生产跑通的 bug
- [x] **Manus 式沉浸视图重构** — 回放界面从三栏分析面板 → Manus 风格单栏 cinematic 视图
- 顶部 breadcrumb、浏览器外壳 + 平台原生帖子卡片、Jump to live 按钮、live 指示灯
- 右上角 ◫/▦ 可切回三栏分析视图
- 自动轮询sim running 时每 10s 拉数据
- 文件:`frontend/src/views/SimulationReplayView.vue`
- [x] **LLM 客户端指数退避重试** — 识别 `RateLimitError / 429 / 5xx / 1302 / 超时`5 次重试 1→2→4→8→16s 带随机抖动
- 文件:`backend/app/utils/llm_client.py`
- [x] **GLM → Qwen 32B 双 LLM 降级 fallback** — 主 LLM 重试耗尽后自动切 SiliconFlow Qwen 32B 单次兜底,解决 GLM 低 RPM 配额的间歇性限流
- 文件:`backend/app/services/ontology_generator.py`
- [x] **Qwen 32B 上下文爆修复**`MAX_TEXT_LENGTH_FOR_LLM` 50000 → 28000 chars保证 prompt+response < 32768 tokens
- 文件:`backend/app/services/ontology_generator.py`
- [x] **Neo4j entity summary flatten** — monkey-patch `Neo4jEntityNodeOperations.save/save_bulk`LLM 返回嵌套 dict 时自动取 `.value` 扁平化,防 Neo4j TypeError
- 文件:`backend/app/services/graphiti_client.py`
- [x] **Frontend 无 pending state 路由修复**`/process/new` 访问时无上传状态,自动 `router.replace({name:'Home'})` 不再死循环
- 文件:`frontend/src/views/MainView.vue`
- [x] **semaphore 100 → 30** — OASIS 模拟并发峰值降低,避免 GLM 限流雪崩
- 文件:`backend/scripts/run_parallel_simulation.py`
### v0.3 完成(本次大版本前的全部更新)
#### 基础设施迁移
- [x] 从 Zep Cloud → 自托管 Graphiti + Neo4j脱离外部 SaaS 依赖)
@ -578,3 +606,7 @@ FLASK_DEBUG=False
| 2026-04-15 | 200 agents 设为甜点 | 8G 容量 + 95% 置信度足够 |
| 2026-04-15 | 上线 Manus 式 replay UI | 给客户演示 + 复盘工具 |
| 2026-04-15 | v0.4 路线图:国内平台 + SaaS | 用户战略需求 |
| 2026-04-15 | v0.3.1 hotfix7 个 bug 修复 | 第一次完整 E2E 跑通压力测试 |
| 2026-04-15 | LLM client 加指数退避重试 + 双 LLM fallback | GLM RPM 配额低retry 不够兜底 |
| 2026-04-15 | Replay UI 重构为 Manus cinematic 风格 | 三栏分析面板对客户演示不够"沉浸" |
| 2026-04-15 | semaphore 100 → 30 | 高并发触发 GLM 限流雪崩 |

View File

@ -27,6 +27,34 @@ def _safe_str(val):
return str(val)
def _flatten_entity_property(value):
"""
Flatten a potentially nested dict/list value into a Neo4j-safe primitive.
Priority:
1. Primitive (str/int/float/bool/None) return as-is
2. dict with "value" key recurse into value (handles LLM JSON-schema pollution)
3. dict json.dumps
4. list flatten each element recursively
5. Other str()
"""
if value is None or isinstance(value, (str, int, float, bool)):
return value
if isinstance(value, dict):
if "value" in value:
return _flatten_entity_property(value["value"])
import json
return json.dumps(value, ensure_ascii=False)
if isinstance(value, list):
return [_flatten_entity_property(v) for v in value]
return str(value)
def _sanitize_entity_data(entity_data: dict) -> dict:
"""Sanitize all values in entity_data dict to Neo4j-safe primitives."""
return {k: _flatten_entity_property(v) for k, v in entity_data.items()}
from graphiti_core.embedder.client import EmbedderClient
@ -201,11 +229,79 @@ class GraphitiClient:
embedder=embedder,
cross_encoder=reranker,
)
# Monkey-patch Neo4j entity node ops to sanitize LLM-polluted summary fields
# before they reach Neo4j (which rejects non-primitive property values).
self._patch_entity_node_ops()
# Build indices
_run_async(self._graphiti.build_indices_and_constraints())
self._initialized = True
logger.info("Graphiti core initialized with indices")
def _patch_entity_node_ops(self):
"""
Patch Neo4jEntityNodeOperations.save and save_bulk to sanitize entity_data
before writing to Neo4j. Handles the case where Qwen/other LLMs return
nested dicts (JSON-schema metadata) in fields like 'summary'.
"""
from graphiti_core.driver.neo4j.operations.entity_node_ops import Neo4jEntityNodeOperations
from graphiti_core.driver.driver import GraphProvider
from graphiti_core.driver.query_executor import QueryExecutor, Transaction
from graphiti_core.nodes import EntityNode
from graphiti_core.models.nodes.node_db_queries import (
get_entity_node_save_query,
get_entity_node_save_bulk_query,
)
from typing import Any
_flatten = _sanitize_entity_data # closure over module-level helper
original_save = Neo4jEntityNodeOperations.save
original_save_bulk = Neo4jEntityNodeOperations.save_bulk
async def patched_save(self_ops, executor, node: EntityNode, tx=None):
entity_data: dict[str, Any] = {
'uuid': node.uuid,
'name': node.name,
'name_embedding': node.name_embedding,
'group_id': node.group_id,
'summary': node.summary,
'created_at': node.created_at,
}
entity_data.update(node.attributes or {})
entity_data = _flatten(entity_data)
labels = ':'.join(list(set(node.labels + ['Entity'])))
query = get_entity_node_save_query(GraphProvider.NEO4J, labels)
if tx is not None:
await tx.run(query, entity_data=entity_data)
else:
await executor.execute_query(query, entity_data=entity_data)
async def patched_save_bulk(self_ops, executor, nodes: list, tx=None, batch_size: int = 100):
prepared: list[dict[str, Any]] = []
for node in nodes:
entity_data: dict[str, Any] = {
'uuid': node.uuid,
'name': node.name,
'group_id': node.group_id,
'summary': node.summary,
'created_at': node.created_at,
'name_embedding': node.name_embedding,
'labels': list(set(node.labels + ['Entity'])),
}
entity_data.update(node.attributes or {})
prepared.append(_flatten(entity_data))
query = get_entity_node_save_bulk_query(GraphProvider.NEO4J, prepared)
if tx is not None:
await tx.run(query, nodes=prepared)
else:
await executor.execute_query(query, nodes=prepared)
Neo4jEntityNodeOperations.save = patched_save
Neo4jEntityNodeOperations.save_bulk = patched_save_bulk
logger.info("Patched Neo4jEntityNodeOperations to sanitize LLM-polluted entity fields")
@classmethod
def get_instance(cls) -> 'GraphitiClient':
"""Singleton accessor."""

View File

@ -9,6 +9,7 @@ import re
from typing import Dict, Any, List, Optional
from ..utils.llm_client import LLMClient
from ..utils.locale import get_language_instruction
from ..config import Config
logger = logging.getLogger(__name__)
@ -181,6 +182,17 @@ class OntologyGenerator:
def __init__(self, llm_client: Optional[LLMClient] = None):
self.llm_client = llm_client or LLMClient()
# Fallback LLM: SiliconFlow Qwen 32B高 RPMGLM 限流时兜底)
self._fallback_llm: Optional[LLMClient] = None
try:
if Config.GRAPHITI_LLM_API_KEY:
self._fallback_llm = LLMClient(
api_key=Config.GRAPHITI_LLM_API_KEY,
base_url=Config.GRAPHITI_LLM_BASE_URL,
model=Config.GRAPHITI_LLM_MODEL,
)
except Exception as e:
logger.warning(f"Fallback LLM 初始化失败: {e}")
def generate(
self,
@ -213,20 +225,35 @@ class OntologyGenerator:
{"role": "user", "content": user_message}
]
# 调用LLM
result = self.llm_client.chat_json(
messages=messages,
temperature=0.3,
max_tokens=4096
)
# 调用 LLM主 LLM 重试 5 次全失败后,自动降级到 fallback
try:
result = self.llm_client.chat_json(
messages=messages,
temperature=0.3,
max_tokens=4096
)
except Exception as primary_err:
if not self._fallback_llm:
raise
logger.warning(
f"主 LLM (GLM) 生成本体失败,降级到 Qwen 32B fallback: "
f"{type(primary_err).__name__}: {str(primary_err)[:200]}"
)
result = self._fallback_llm.chat_json(
messages=messages,
temperature=0.3,
max_tokens=4096
)
# 验证和后处理
result = self._validate_and_process(result)
return result
# 传给 LLM 的文本最大长度5万字
MAX_TEXT_LENGTH_FOR_LLM = 50000
# 传给 LLM 的文本最大长度2.8万字)
# Qwen 32768 token 窗口system prompt ~2K tokens + 响应 ~4K tokens
# 留给文档的安全预算 ≈ 26K tokens保守按 1 char ≈ 1 token 估算取 28000 chars
MAX_TEXT_LENGTH_FOR_LLM = 28000
def _build_user_message(
self,
@ -243,7 +270,7 @@ class OntologyGenerator:
# 如果文本超过5万字截断仅影响传给LLM的内容不影响图谱构建
if len(combined_text) > self.MAX_TEXT_LENGTH_FOR_LLM:
combined_text = combined_text[:self.MAX_TEXT_LENGTH_FOR_LLM]
combined_text += f"\n\n...(原文共{original_length}字,已截取前{self.MAX_TEXT_LENGTH_FOR_LLM}字用于本体分析)..."
combined_text += f"\n\n[...文档已截断以适应 LLM 上下文窗口。原文共 {original_length} 字,已截取前 {self.MAX_TEXT_LENGTH_FOR_LLM} 字用于本体分析...]"
message = f"""## 模拟需求

View File

@ -5,11 +5,41 @@ LLM客户端封装
import json
import re
import time
import random
import logging
from typing import Optional, Dict, Any, List
from openai import OpenAI
from openai import OpenAI, RateLimitError, APIError, APIConnectionError, APITimeoutError
from ..config import Config
logger = logging.getLogger('foresight.llm_client')
# 重试配置(针对 429 / 5xx / 超时 / 连接错误)
_MAX_RETRIES = 5
_BASE_BACKOFF = 1.0 # 首次重试等 1s
_MAX_BACKOFF = 30.0 # 单次最多等 30s
def _is_rate_limit_error(err: Exception) -> bool:
"""判断是否是速率限制/可重试错误"""
if isinstance(err, (RateLimitError, APIConnectionError, APITimeoutError)):
return True
if isinstance(err, APIError):
# 429 / 500 / 502 / 503 / 504 都可重试
status = getattr(err, "status_code", None) or getattr(err, "code", None)
try:
status = int(status) if status else None
except (ValueError, TypeError):
status = None
if status in (429, 500, 502, 503, 504):
return True
# 智谱 1302 = 速率限制
msg = str(err)
if "1302" in msg or "rate limit" in msg.lower() or "速率限制" in msg or "too many request" in msg.lower():
return True
return False
class LLMClient:
"""LLM客户端"""
@ -61,7 +91,28 @@ class LLMClient:
if response_format:
kwargs["response_format"] = response_format
response = self.client.chat.completions.create(**kwargs)
# 带指数退避的重试:处理 429/5xx/超时等可恢复错误
response = None
last_err = None
for attempt in range(_MAX_RETRIES + 1):
try:
response = self.client.chat.completions.create(**kwargs)
break
except Exception as err:
last_err = err
if attempt >= _MAX_RETRIES or not _is_rate_limit_error(err):
raise
# 指数退避 + 随机抖动
backoff = min(_MAX_BACKOFF, _BASE_BACKOFF * (2 ** attempt))
backoff += random.uniform(0, backoff * 0.3)
logger.warning(
f"LLM 调用遇到可重试错误 (attempt {attempt + 1}/{_MAX_RETRIES}): "
f"{type(err).__name__}: {str(err)[:200]}. 退避 {backoff:.1f}s 后重试..."
)
time.sleep(backoff)
if response is None:
raise last_err or RuntimeError("LLM 调用失败(未知原因)")
# Token 追踪v0.3 新增):记录每次调用的 token 消耗,按当前 stage 归类
try:
from . import token_tracker

View File

@ -1156,7 +1156,7 @@ async def run_twitter_simulation(
agent_graph=result.agent_graph,
platform=oasis.DefaultPlatformType.TWITTER,
database_path=db_path,
semaphore=100, # 限制最大并发 LLM 请求数,防止 API 过载
semaphore=30, # 限制最大并发 LLM 请求数,防止 API 过载GLM RPM 友好值)
)
await result.env.reset()
@ -1347,7 +1347,7 @@ async def run_reddit_simulation(
agent_graph=result.agent_graph,
platform=oasis.DefaultPlatformType.REDDIT,
database_path=db_path,
semaphore=100, # 限制最大并发 LLM 请求数,防止 API 过载
semaphore=30, # 限制最大并发 LLM 请求数,防止 API 过载GLM RPM 友好值)
)
await result.env.reset()

View File

@ -201,8 +201,13 @@ const initProject = async () => {
const handleNewProject = async () => {
const pending = getPendingUpload()
if (!pending.isPending || pending.files.length === 0) {
error.value = 'No pending files found.'
addLog('Error: No pending files found for new project.')
error.value = 'No pending files found. Redirecting to home to upload documents...'
addLog('No pending upload detected for /process/new. Redirecting to Home.')
// Direct-visit / refresh case: pending state is empty because it's in-memory only.
// Send user back to Home so they can upload files and start a real project.
setTimeout(() => {
router.replace({ name: 'Home' })
}, 800)
return
}

File diff suppressed because it is too large Load Diff