RomanNum3ral
/
MicroFish
mirror of https://github.com/666ghj/MiroFish.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge 8dfa7645e6 into 96096ea0ff

This commit is contained in:
Laurens Profittlich 2026-06-10 10:13:54 +00:00 committed by GitHub
parent 96096ea0ff 8dfa7645e6
commit 396e7ed2fb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
128 changed files with 16338 additions and 9212 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.gitignore vendored
View File

@ -57,4 +57,6 @@ backend/logs/
backend/uploads/
# Docker 数据
data/
data/
# Deployment secrets (real values, never commit)
deploy/secrets.env

83
Dockerfile
View File

@ -1,29 +1,78 @@
FROM python:3.11
# MiroFish production Dockerfile for agent.profikid.nl deployment
# - Builds the Vue frontend into static files
# - Serves the static bundle via nginx on :3000
# - Reverse-proxies /api/* to the Flask backend on :5001 (same container, localhost)
# - Single exposed port for Traefik
# - Memory graph: Graphiti (Python lib) talks to FalkorDB on the shared network
# 安装 Node.js （满足 >=18）及必要工具
FROM python:3.11-slim
ENV DEBIAN_FRONTEND=noninteractive \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
VITE_API_BASE_URL="" \
HF_HUB_OFFLINE=0 \
SENTENCE_TRANSFORMERS_HOME=/root/.cache/huggingface
# Install Node 20 (for building the frontend), nginx (for serving it), and
# build deps for the Python wheels (sentence-transformers pulls numpy/torch).
RUN apt-get update \
&& apt-get install -y --no-install-recommends nodejs npm \
&& rm -rf /var/lib/apt/lists/*
&& apt-get install -y --no-install-recommends \
nodejs \
npm \
nginx \
curl \
ca-certificates \
git \
&& rm -rf /var/lib/apt/lists/*
# 从 uv 官方镜像复制 uv
# Install uv for fast Python package management
COPY --from=ghcr.io/astral-sh/uv:0.9.26 /uv /uvx /bin/
WORKDIR /app
# 先复制依赖描述文件以利用缓存
COPY package.json package-lock.json ./
# ---- 1) Install Python deps (backend) ----
# Copy the local path-dep (camel-oasis fork) BEFORE uv sync, so the path can resolve.
# Note: we don't COPY uv.lock here — the first build has to generate it. Once a
# successful build has happened, the lock will be in the source tree and a
# subsequent `uv sync --frozen` will be reproducible.
COPY backend/vendor ./backend/vendor
COPY backend/pyproject.toml ./backend/
WORKDIR /app/backend
RUN uv sync
# ---- 2) Install Node deps + build frontend ----
WORKDIR /app
# Copy locales first because the Vite build's `@locales` alias points to ../locales
COPY locales/ ./locales/
COPY frontend/package.json frontend/package-lock.json ./frontend/
COPY backend/pyproject.toml backend/uv.lock ./backend/
RUN cd frontend && npm ci --no-audit --no-fund
# 安装依赖（Node + Python）
RUN npm ci \
&& npm ci --prefix frontend \
&& cd backend && uv sync --frozen
# Copy full frontend source and build the production bundle
COPY frontend/ ./frontend/
RUN cd frontend && npm run build
# 复制项目源码
COPY . .
# ---- 3) Copy the rest of the app (backend source) ----
COPY backend/ ./backend/
EXPOSE 3000 5001
# ---- 4) Pre-download the sentence-transformers embedding model ----
# Doing it at build time means the container is ready to serve on first boot
# (no 100MB download the first time the user runs a graph build).
WORKDIR /app/backend
RUN uv run python -c "from sentence_transformers import SentenceTransformer; \
SentenceTransformer('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')" \
|| (echo 'Failed to pre-download embedding model — build will fall back to download on first use.' && exit 0)
# 同时启动前后端（开发模式）
CMD ["npm", "run", "dev"]
# ---- 5) nginx config + start script ----
COPY nginx.conf /etc/nginx/nginx.conf
COPY start.sh /app/start.sh
RUN chmod +x /app/start.sh
# Runtime dirs
RUN mkdir -p /app/backend/uploads /app/logs /var/lib/nginx /var/log/nginx
EXPOSE 3000
CMD ["/app/start.sh"]

74
README.md
View File

@ -28,6 +28,8 @@
**MiroFish** is a next-generation AI prediction engine powered by multi-agent technology. By extracting seed information from the real world (such as breaking news, policy drafts, or financial signals), it automatically constructs a high-fidelity parallel digital world. Within this space, thousands of intelligent agents with independent personalities, long-term memory, and behavioral logic freely interact and undergo social evolution. You can inject variables dynamically from a "God's-eye view" to precisely deduce future trajectories — **rehearse the future in a digital sandbox, and win decisions after countless simulations**.
> **This is a maintained fork** of [666ghj/MiroFish](https://github.com/666ghj/MiroFish). It swaps Zep Cloud for **Graphiti + FalkorDB** (open source, self-hosted), adds a one-call `POST /api/graph/ingest_text` API for cron automations, and ships a production Docker stack for `agent.profikid.nl`. See [About this fork](#-about-this-fork) below.
> You only need to: Upload seed materials (data analysis reports or interesting novel stories) and describe your prediction requirements in natural language</br>
> MiroFish will return: A detailed prediction report and a deeply interactive high-fidelity digital world
@ -116,15 +118,22 @@ cp .env.example .env
```env
# LLM API Configuration (supports any LLM API with OpenAI SDK format)
# Recommended: Alibaba Qwen-plus model via Bailian Platform: https://bailian.console.aliyun.com/
# Recommended: MiniMax M-series via https://api.minimax.io/v1
# - MiniMax-M2.7-highspeed : best for cron / high-volume (recommended)
# - MiniMax-M3 : heavier reasoning, slower, higher quality
# Alibaba Qwen-plus via Bailian Platform: https://bailian.console.aliyun.com/
# High consumption, try simulations with fewer than 40 rounds first
LLM_API_KEY=your_api_key
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
LLM_MODEL_NAME=qwen-plus
LLM_BASE_URL=https://api.minimax.io/v1
LLM_MODEL_NAME=MiniMax-M2.7-highspeed
# Zep Cloud Configuration
# Free monthly quota is sufficient for simple usage: https://app.getzep.com/
ZEP_API_KEY=your_zep_api_key
# Graph store (this fork: Graphiti + FalkorDB; no Zep API key needed)
# FalkorDB runs as a Docker sidecar — see deploy/docker-compose.yml
FALKORDB_HOST=falkordb
FALKORDB_PORT=6379
# Embedding model (local sentence-transformers, pre-downloaded in the image)
EMBEDDING_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
```
#### 2. Install Dependencies
@ -192,6 +201,59 @@ The MiroFish team is recruiting full-time/internship positions. If you're intere
MiroFish's simulation engine is powered by **[OASIS (Open Agent Social Interaction Simulations)](https://github.com/camel-ai/oasis)**, We sincerely thank the CAMEL-AI team for their open-source contributions!
## 🔱 About this fork
This is a maintained fork of [666ghj/MiroFish](https://github.com/666ghj/MiroFish) maintained at [profikid/MiroFish](https://github.com/profikid/MiroFish). It replaces the Zep Cloud dependency with **Graphiti + FalkorDB** (both open source), adds a one-call ingest API, and ships a production-ready Docker deployment for `agent.profikid.nl`.
### What changed vs upstream
| | Upstream | This fork |
| --- | --- | --- |
| Graph store | Zep Cloud (managed, requires API key) | Graphiti + FalkorDB (self-hosted, Redis protocol) |
| LLM recommendation | Qwen-plus (DashScope) | MiniMax M-series (M2.7-highspeed for cron, M3 for high-quality) |
| Ingest API | 3 calls (project → ontology → build) | 1 call: `POST /api/graph/ingest_text` (project + ontology + build) |
| Deploy | `npm run dev` (dev) / `docker compose up` (Docker Hub) | `deploy/` overlay: Traefik + Let's Encrypt + FalkorDB sidecar + e2e one-shot |
| E2E test | none shipped | `deploy/e2e_test.py` + `deploy/e2e.sh` (builds a one-shot image, runs against the real briefing seed) |
| Cron-friendly | not designed for it | `deploy/mirofish_ingest.sh` — one-shot POST + poll, exits 0/2/3/4 |
| Embeddings | Zep-managed | local `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` (pre-downloaded in the image) |
### Why Graphiti + FalkorDB
Zep Cloud's free tier is fine for dev, but for a self-hosted cron pipeline (Iran briefing every 12h, entity extraction, simulations) you want:
- **No per-month API cap** — the cron keeps running on bad days, good days, news spikes
- **No external service dependency** — graph store lives next to the app as a Docker sidecar
- **Same engine** — Graphiti is the open-source core that powers Zep Cloud, so the entity/edge quality is identical
The `backend/app/services/graphiti_service.py` shim is a Zep-shaped facade over a real `Graphiti(graph_driver=FalkorDriver(...))`, so the rest of the MiroFish codebase (which still speaks Zep) didn't have to change.
### Production deployment
The `deploy/` directory contains a single-host Docker stack for `agent.profikid.nl`:
```bash
# On the host:
cd /docker/mirofish
./deploy/up.sh # build image, bring up mirofish + falkordb
./deploy/e2e.sh # run the e2e test against the real briefing seed
```
Traefik (already on the host) auto-issues the Let's Encrypt cert for `mirofish.agent.profikid.nl`. See [deploy/README.md](./deploy/README.md) for the full layout, env vars, and troubleshooting.
### Cron integration
The fork was built so that a scheduled OSINT briefing cron can drop the markdown output straight into MiroFish:
```bash
# Cron's last-written briefing file -> POST + poll
briefing="$(ls -t /home/hermes/.hermes/cron/output/<job_id>/*.md | head -1)"
nohup bash /docker/mirofish/deploy/mirofish_ingest.sh \
"iran-osint-$(date -u +%Y%m%dT%H%M)" \
"$briefing" \
>/tmp/mirofish-ingest.log 2>&1 &
```
The script handles ontology generation + async graph build + polling, exits 0 on success with a `nodes=N edges=M` summary in the log.
## 📈 Project Statistics
<a href="https://www.star-history.com/#666ghj/MiroFish&type=date&legend=top-left">

66
backend/app/__init__.py
View File

@ -1,12 +1,12 @@
"""
MiroFish Backend - Flask应用工厂
MiroFish Backend - Flask application factory.
"""
import os
import warnings
# 抑制 multiprocessing resource_tracker 的警告（来自第三方库如 transformers）
# 需要在所有其他导入之前设置
# Suppress multiprocessing resource_tracker warnings (emitted by third-party libs such as transformers).
# Must be configured before any other import.
warnings.filterwarnings("ignore", message=".*resource_tracker.*")
from flask import Flask, request
@ -17,64 +17,64 @@ from .utils.logger import setup_logger, get_logger
def create_app(config_class=Config):
"""Flask应用工厂函数"""
"""Flask application factory."""
app = Flask(__name__)
app.config.from_object(config_class)
# 设置JSON编码：确保中文直接显示（而不是 \uXXXX 格式）
# Flask >= 2.3 使用 app.json.ensure_ascii，旧版本使用 JSON_AS_ASCII 配置
# Configure JSON encoding so Chinese characters are emitted as-is (not escaped to \\uXXXX).
# Flask >= 2.3 uses app.json.ensure_ascii; older versions use the JSON_AS_ASCII config.
if hasattr(app, 'json') and hasattr(app.json, 'ensure_ascii'):
app.json.ensure_ascii = False
# 设置日志
# Set up logging
logger = setup_logger('mirofish')
# 只在 reloader 子进程中打印启动信息（避免 debug 模式下打印两次）
# Only print startup information in the reloader child process (avoids duplicate logs under debug).
is_reloader_process = os.environ.get('WERKZEUG_RUN_MAIN') == 'true'
debug_mode = app.config.get('DEBUG', False)
should_log_startup = not debug_mode or is_reloader_process
if should_log_startup:
logger.info("=" * 50)
logger.info("MiroFish Backend 启动中...")
logger.info("MiroFish Backend starting...")
logger.info("=" * 50)
# 启用CORS
# Enable CORS
CORS(app, resources={r"/api/*": {"origins": "*"}})
# 注册模拟进程清理函数（确保服务器关闭时终止所有模拟进程）
# Register a cleanup hook so that all simulation child processes are terminated
# when the server shuts down.
from .services.simulation_runner import SimulationRunner
SimulationRunner.register_cleanup()
if should_log_startup:
logger.info("已注册模拟进程清理函数")
# 请求日志中间件
logger.info("Registered simulation process cleanup hook")
# Request logging middleware
@app.before_request
def log_request():
logger = get_logger('mirofish.request')
logger.debug(f"请求: {request.method} {request.path}")
logger.debug(f"Request: {request.method} {request.path}")
if request.content_type and 'json' in request.content_type:
logger.debug(f"请求体: {request.get_json(silent=True)}")
logger.debug(f"Request body: {request.get_json(silent=True)}")
@app.after_request
def log_response(response):
logger = get_logger('mirofish.request')
logger.debug(f"响应: {response.status_code}")
logger.debug(f"Response: {response.status_code}")
return response
# 注册蓝图
# Register blueprints
from .api import graph_bp, simulation_bp, report_bp
app.register_blueprint(graph_bp, url_prefix='/api/graph')
app.register_blueprint(simulation_bp, url_prefix='/api/simulation')
app.register_blueprint(report_bp, url_prefix='/api/report')
# 健康检查
# Health check
@app.route('/health')
def health():
return {'status': 'ok', 'service': 'MiroFish Backend'}
if should_log_startup:
logger.info("MiroFish Backend 启动完成")
return app
if should_log_startup:
logger.info("MiroFish Backend startup complete")
return app

3
backend/app/api/__init__.py
View File

@ -1,5 +1,5 @@
"""
API路由模块
API route module.
"""
from flask import Blueprint
@ -11,4 +11,3 @@ report_bp = Blueprint('report', __name__)
from . import graph # noqa: E402, F401
from . import simulation # noqa: E402, F401
from . import report # noqa: E402, F401

561
backend/app/api/graph.py
View File

@ -1,6 +1,6 @@
"""
图谱相关API路由
采用项目上下文机制，服务端持久化状态
Graph API routes.
Uses the project context mechanism so the server persists state between calls.
"""
import os
@ -19,27 +19,27 @@ from ..utils.locale import t, get_locale, set_locale
from ..models.task import TaskManager, TaskStatus
from ..models.project import ProjectManager, ProjectStatus
# 获取日志器
# Get logger
logger = get_logger('mirofish.api')
def allowed_file(filename: str) -> bool:
"""检查文件扩展名是否允许"""
"""Check whether the file extension is in the allow-list."""
if not filename or '.' not in filename:
return False
ext = os.path.splitext(filename)[1].lower().lstrip('.')
return ext in Config.ALLOWED_EXTENSIONS
# ============== 项目管理接口 ==============
# ============== Project management endpoints ==============
@graph_bp.route('/project/<project_id>', methods=['GET'])
def get_project(project_id: str):
"""
获取项目详情
Retrieve project details.
"""
project = ProjectManager.get_project(project_id)
if not project:
return jsonify({
"success": False,
@ -55,11 +55,11 @@ def get_project(project_id: str):
@graph_bp.route('/project/list', methods=['GET'])
def list_projects():
"""
列出所有项目
List all projects.
"""
limit = request.args.get('limit', 50, type=int)
projects = ProjectManager.list_projects(limit=limit)
return jsonify({
"success": True,
"data": [p.to_dict() for p in projects],
@ -70,10 +70,10 @@ def list_projects():
@graph_bp.route('/project/<project_id>', methods=['DELETE'])
def delete_project(project_id: str):
"""
删除项目
Delete a project.
"""
success = ProjectManager.delete_project(project_id)
if not success:
return jsonify({
"success": False,
@ -89,27 +89,27 @@ def delete_project(project_id: str):
@graph_bp.route('/project/<project_id>/reset', methods=['POST'])
def reset_project(project_id: str):
"""
重置项目状态（用于重新构建图谱）
Reset project state (used to rebuild the graph).
"""
project = ProjectManager.get_project(project_id)
if not project:
return jsonify({
"success": False,
"error": t('api.projectNotFound', id=project_id)
}), 404
# 重置到本体已生成状态
# Roll back to the "ontology generated" state
if project.ontology:
project.status = ProjectStatus.ONTOLOGY_GENERATED
else:
project.status = ProjectStatus.CREATED
project.graph_id = None
project.graph_build_task_id = None
project.error = None
ProjectManager.save_project(project)
return jsonify({
"success": True,
"message": t('api.projectReset', id=project_id),
@ -117,22 +117,22 @@ def reset_project(project_id: str):
})
# ============== 接口1：上传文件并生成本体 ==============
# ============== Endpoint 1: upload files and generate the ontology ==============
@graph_bp.route('/ontology/generate', methods=['POST'])
def generate_ontology():
"""
接口1：上传文件，分析生成本体定义
请求方式：multipart/form-data
参数：
files: 上传的文件（PDF/MD/TXT），可多个
simulation_requirement: 模拟需求描述（必填）
project_name: 项目名称（可选）
additional_context: 额外说明（可选）
返回：
Endpoint 1: upload files, analyse them and produce an ontology definition.
Request: multipart/form-data.
Parameters:
files: uploaded files (PDF/MD/TXT), one or many.
simulation_requirement: description of the simulation to run (required).
project_name: project name (optional).
additional_context: extra notes (optional).
Returns:
{
"success": true,
"data": {
@ -148,84 +148,84 @@ def generate_ontology():
}
"""
try:
logger.info("=== 开始生成本体定义 ===")
# 获取参数
logger.info("=== Generating ontology definition ===")
# Read parameters
simulation_requirement = request.form.get('simulation_requirement', '')
project_name = request.form.get('project_name', 'Unnamed Project')
additional_context = request.form.get('additional_context', '')
logger.debug(f"项目名称: {project_name}")
logger.debug(f"模拟需求: {simulation_requirement[:100]}...")
logger.debug(f"Project name: {project_name}")
logger.debug(f"Simulation requirement: {simulation_requirement[:100]}...")
if not simulation_requirement:
return jsonify({
"success": False,
"error": t('api.requireSimulationRequirement')
}), 400
# 获取上传的文件
# Read uploaded files
uploaded_files = request.files.getlist('files')
if not uploaded_files or all(not f.filename for f in uploaded_files):
return jsonify({
"success": False,
"error": t('api.requireFileUpload')
}), 400
# 创建项目
# Create the project
project = ProjectManager.create_project(name=project_name)
project.simulation_requirement = simulation_requirement
logger.info(f"创建项目: {project.project_id}")
# 保存文件并提取文本
logger.info(f"Created project: {project.project_id}")
# Save files and extract text
document_texts = []
all_text = ""
for file in uploaded_files:
if file and file.filename and allowed_file(file.filename):
# 保存文件到项目目录
# Save the file inside the project directory
file_info = ProjectManager.save_file_to_project(
project.project_id,
file,
project.project_id,
file,
file.filename
)
project.files.append({
"filename": file_info["original_filename"],
"size": file_info["size"]
})
# 提取文本
# Extract text
text = FileParser.extract_text(file_info["path"])
text = TextProcessor.preprocess_text(text)
document_texts.append(text)
all_text += f"\n\n=== {file_info['original_filename']} ===\n{text}"
if not document_texts:
ProjectManager.delete_project(project.project_id)
return jsonify({
"success": False,
"error": t('api.noDocProcessed')
}), 400
# 保存提取的文本
# Persist the extracted text
project.total_text_length = len(all_text)
ProjectManager.save_extracted_text(project.project_id, all_text)
logger.info(f"文本提取完成，共 {len(all_text)} 字符")
# 生成本体
logger.info("调用 LLM 生成本体定义...")
logger.info(f"Text extraction complete: {len(all_text)} characters")
# Generate the ontology
logger.info("Calling LLM to generate ontology definition...")
generator = OntologyGenerator()
ontology = generator.generate(
document_texts=document_texts,
simulation_requirement=simulation_requirement,
additional_context=additional_context if additional_context else None
)
# 保存本体到项目
# Save the ontology onto the project
entity_count = len(ontology.get("entity_types", []))
edge_count = len(ontology.get("edge_types", []))
logger.info(f"本体生成完成: {entity_count} 个实体类型, {edge_count} 个关系类型")
logger.info(f"Ontology generation complete: {entity_count} entity types, {edge_count} edge types")
project.ontology = {
"entity_types": ontology.get("entity_types", []),
"edge_types": ontology.get("edge_types", [])
@ -233,8 +233,8 @@ def generate_ontology():
project.analysis_summary = ontology.get("analysis_summary", "")
project.status = ProjectStatus.ONTOLOGY_GENERATED
ProjectManager.save_project(project)
logger.info(f"=== 本体生成完成 === 项目ID: {project.project_id}")
logger.info(f"=== Ontology generation complete === Project ID: {project.project_id}")
return jsonify({
"success": True,
"data": {
@ -246,7 +246,7 @@ def generate_ontology():
"total_text_length": project.total_text_length
}
})
except Exception as e:
return jsonify({
"success": False,
@ -255,57 +255,57 @@ def generate_ontology():
}), 500
# ============== 接口2：构建图谱 ==============
# ============== Endpoint 2: build the graph ==============
@graph_bp.route('/build', methods=['POST'])
def build_graph():
"""
接口2：根据project_id构建图谱
请求（JSON）：
Endpoint 2: build the graph for a given project_id.
Request (JSON):
{
"project_id": "proj_xxxx", // 必填，来自接口1
"graph_name": "图谱名称", // 可选
"chunk_size": 500, // 可选，默认500
"chunk_overlap": 50 // 可选，默认50
"project_id": "proj_xxxx", // required, returned by endpoint 1
"graph_name": "graph name", // optional
"chunk_size": 500, // optional, default 500
"chunk_overlap": 50 // optional, default 50
}
返回：
Returns:
{
"success": true,
"data": {
"project_id": "proj_xxxx",
"task_id": "task_xxxx",
"message": "图谱构建任务已启动"
"message": "Graph build task started"
}
}
"""
try:
logger.info("=== 开始构建图谱 ===")
# 检查配置
logger.info("=== Starting graph build ===")
# Validate config
errors = []
if not Config.ZEP_API_KEY:
if not Config.FALKORDB_HOST:
errors.append(t('api.zepApiKeyMissing'))
if errors:
logger.error(f"配置错误: {errors}")
logger.error(f"Configuration error: {errors}")
return jsonify({
"success": False,
"error": t('api.configError', details="; ".join(errors))
}), 500
# 解析请求
# Parse request
data = request.get_json() or {}
project_id = data.get('project_id')
logger.debug(f"请求参数: project_id={project_id}")
logger.debug(f"Request parameters: project_id={project_id}")
if not project_id:
return jsonify({
"success": False,
"error": t('api.requireProjectId')
}), 400
# 获取项目
# Look up the project
project = ProjectManager.get_project(project_id)
if not project:
return jsonify({
@ -313,116 +313,116 @@ def build_graph():
"error": t('api.projectNotFound', id=project_id)
}), 404
# 检查项目状态
force = data.get('force', False) # 强制重新构建
# Check project state
force = data.get('force', False) # force a rebuild
if project.status == ProjectStatus.CREATED:
return jsonify({
"success": False,
"error": t('api.ontologyNotGenerated')
}), 400
if project.status == ProjectStatus.GRAPH_BUILDING and not force:
return jsonify({
"success": False,
"error": t('api.graphBuilding'),
"task_id": project.graph_build_task_id
}), 400
# 如果强制重建，重置状态
# If forced, roll the project back to the "ontology generated" state
if force and project.status in [ProjectStatus.GRAPH_BUILDING, ProjectStatus.FAILED, ProjectStatus.GRAPH_COMPLETED]:
project.status = ProjectStatus.ONTOLOGY_GENERATED
project.graph_id = None
project.graph_build_task_id = None
project.error = None
# 获取配置
# Read configuration
graph_name = data.get('graph_name', project.name or 'MiroFish Graph')
chunk_size = data.get('chunk_size', project.chunk_size or Config.DEFAULT_CHUNK_SIZE)
chunk_overlap = data.get('chunk_overlap', project.chunk_overlap or Config.DEFAULT_CHUNK_OVERLAP)
# 更新项目配置
# Persist configuration on the project
project.chunk_size = chunk_size
project.chunk_overlap = chunk_overlap
# 获取提取的文本
# Read the extracted text
text = ProjectManager.get_extracted_text(project_id)
if not text:
return jsonify({
"success": False,
"error": t('api.textNotFound')
}), 400
# 获取本体
# Read the ontology
ontology = project.ontology
if not ontology:
return jsonify({
"success": False,
"error": t('api.ontologyNotFound')
}), 400
# 创建异步任务
# Create the async task
task_manager = TaskManager()
task_id = task_manager.create_task(f"构建图谱: {graph_name}")
logger.info(f"创建图谱构建任务: task_id={task_id}, project_id={project_id}")
# 更新项目状态
task_id = task_manager.create_task(f"Build graph: {graph_name}")
logger.info(f"Created graph build task: task_id={task_id}, project_id={project_id}")
# Update project state
project.status = ProjectStatus.GRAPH_BUILDING
project.graph_build_task_id = task_id
ProjectManager.save_project(project)
# Capture locale before spawning background thread
current_locale = get_locale()
# 启动后台任务
# Background worker
def build_task():
set_locale(current_locale)
build_logger = get_logger('mirofish.build')
try:
build_logger.info(f"[{task_id}] 开始构建图谱...")
build_logger.info(f"[{task_id}] Starting graph build...")
task_manager.update_task(
task_id,
task_id,
status=TaskStatus.PROCESSING,
message=t('progress.initGraphService')
)
# 创建图谱构建服务
builder = GraphBuilderService(api_key=Config.ZEP_API_KEY)
# 分块
# Create the graph builder service
builder = GraphBuilderService(api_key=None)
# Chunk the text
task_manager.update_task(
task_id,
message=t('progress.textChunking'),
progress=5
)
chunks = TextProcessor.split_text(
text,
chunk_size=chunk_size,
text,
chunk_size=chunk_size,
overlap=chunk_overlap
)
total_chunks = len(chunks)
# 创建图谱
# Create the graph
task_manager.update_task(
task_id,
message=t('progress.creatingZepGraph'),
progress=10
)
graph_id = builder.create_graph(name=graph_name)
# 更新项目的graph_id
# Save the graph_id onto the project
project.graph_id = graph_id
ProjectManager.save_project(project)
# 设置本体
# Apply the ontology
task_manager.update_task(
task_id,
message=t('progress.settingOntology'),
progress=15
)
builder.set_ontology(graph_id, ontology)
# 添加文本（progress_callback 签名是 (msg, progress_ratio)）
# progress_callback signature: (msg, progress_ratio)
def add_progress_callback(msg, progress_ratio):
progress = 15 + int(progress_ratio * 40) # 15% - 55%
task_manager.update_task(
@ -430,27 +430,27 @@ def build_graph():
message=msg,
progress=progress
)
task_manager.update_task(
task_id,
message=t('progress.addingChunks', count=total_chunks),
progress=15
)
episode_uuids = builder.add_text_batches(
graph_id,
graph_id,
chunks,
batch_size=3,
progress_callback=add_progress_callback
)
# 等待Zep处理完成（查询每个episode的processed状态）
# Wait for Zep/Graphiti to finish processing (poll each episode's processed flag)
task_manager.update_task(
task_id,
message=t('progress.waitingZepProcess'),
progress=55
)
def wait_progress_callback(msg, progress_ratio):
progress = 55 + int(progress_ratio * 35) # 55% - 90%
task_manager.update_task(
@ -458,26 +458,26 @@ def build_graph():
message=msg,
progress=progress
)
builder._wait_for_episodes(episode_uuids, wait_progress_callback)
# 获取图谱数据
# Fetch the final graph data
task_manager.update_task(
task_id,
message=t('progress.fetchingGraphData'),
progress=95
)
graph_data = builder.get_graph_data(graph_id)
# 更新项目状态
# Update project state
project.status = ProjectStatus.GRAPH_COMPLETED
ProjectManager.save_project(project)
node_count = graph_data.get("node_count", 0)
edge_count = graph_data.get("edge_count", 0)
build_logger.info(f"[{task_id}] 图谱构建完成: graph_id={graph_id}, 节点={node_count}, 边={edge_count}")
# 完成
build_logger.info(f"[{task_id}] Graph build complete: graph_id={graph_id}, nodes={node_count}, edges={edge_count}")
# Mark task as complete
task_manager.update_task(
task_id,
status=TaskStatus.COMPLETED,
@ -491,27 +491,27 @@ def build_graph():
"chunk_count": total_chunks
}
)
except Exception as e:
# 更新项目状态为失败
build_logger.error(f"[{task_id}] 图谱构建失败: {str(e)}")
# Mark project as failed
build_logger.error(f"[{task_id}] Graph build failed: {str(e)}")
build_logger.debug(traceback.format_exc())
project.status = ProjectStatus.FAILED
project.error = str(e)
ProjectManager.save_project(project)
task_manager.update_task(
task_id,
status=TaskStatus.FAILED,
message=t('progress.buildFailed', error=str(e)),
error=traceback.format_exc()
)
# 启动后台线程
# Launch background thread
thread = threading.Thread(target=build_task, daemon=True)
thread.start()
return jsonify({
"success": True,
"data": {
@ -520,7 +520,7 @@ def build_graph():
"message": t('api.graphBuildStarted', taskId=task_id)
}
})
except Exception as e:
return jsonify({
"success": False,
@ -529,21 +529,250 @@ def build_graph():
}), 500
# ============== 任务查询接口 ==============
# ============== One-shot ingest endpoint ==============
import threading
import traceback
from flask import request, jsonify
from . import graph_bp
from ..config import Config
from ..services.graph_builder import GraphBuilderService
from ..services.ontology_generator import OntologyGenerator
from ..services.text_processor import TextProcessor
from ..utils.logger import get_logger
from ..utils.locale import t, get_locale, set_locale
from ..models.task import TaskManager, TaskStatus
from ..models.project import ProjectManager, ProjectStatus
logger = get_logger("mirofish.api.ingest_text")
@graph_bp.route("/ingest_text", methods=["POST"])
def ingest_text():
"""
One-shot ingest: text in, project + ontology + async build out.
Request (JSON):
{
"name": "iran-briefing-2026-06-09T11",
"text": "## IRAN/US/ISRAEL CONFLICT ...",
"simulation_requirement": "Extract entities and relations...",
"additional_context": "optional, free-form",
"graph_name": "optional, defaults to name",
"chunk_size": 500,
"chunk_overlap": 50
}
Response:
{
"success": true,
"data": {
"project_id": "proj_xxxx",
"task_id": "task_xxxx",
"message": "..."
}
}
Poll status with: GET /api/graph/task/<task_id>
"""
try:
data = request.get_json(silent=True) or {}
name = (data.get("name") or "").strip()
text = (data.get("text") or "").strip()
simulation_requirement = (data.get("simulation_requirement") or "").strip()
additional_context = (data.get("additional_context") or "").strip() or None
graph_name = (data.get("graph_name") or name or "MiroFish Graph").strip()
chunk_size = int(data.get("chunk_size") or Config.DEFAULT_CHUNK_SIZE)
chunk_overlap = int(data.get("chunk_overlap") or Config.DEFAULT_CHUNK_OVERLAP)
if not name:
return jsonify({"success": False, "error": "name is required"}), 400
if not text:
return jsonify({"success": False, "error": "text is required"}), 400
if not simulation_requirement:
return jsonify(
{"success": False, "error": "simulation_requirement is required"}
), 400
if not Config.FALKORDB_HOST:
return jsonify({"success": False, "error": "FalkorDB not configured"}), 500
# 1. Create project
project = ProjectManager.create_project(name=name)
project.simulation_requirement = simulation_requirement
project.chunk_size = chunk_size
project.chunk_overlap = chunk_overlap
logger.info(f"[ingest_text] created project {project.project_id} ({name})")
# 2. Save extracted text
cleaned_text = TextProcessor.preprocess_text(text)
project.total_text_length = len(cleaned_text)
ProjectManager.save_extracted_text(project.project_id, cleaned_text)
logger.info(
f"[ingest_text] saved text: {len(cleaned_text)} chars "
f"(project={project.project_id})"
)
# 3. Generate ontology synchronously — this is an LLM call, takes ~10–30s
generator = OntologyGenerator()
ontology = generator.generate(
document_texts=[cleaned_text],
simulation_requirement=simulation_requirement,
additional_context=additional_context,
)
entity_count = len(ontology.get("entity_types", []))
edge_count = len(ontology.get("edge_types", []))
project.ontology = {
"entity_types": ontology.get("entity_types", []),
"edge_types": ontology.get("edge_types", []),
}
project.analysis_summary = ontology.get("analysis_summary", "")
project.status = ProjectStatus.ONTOLOGY_GENERATED
ProjectManager.save_project(project)
logger.info(
f"[ingest_text] ontology ready: {entity_count} entity types, "
f"{edge_count} edge types (project={project.project_id})"
)
# 4. Kick off async graph build (same path /build uses)
task_manager = TaskManager()
task_id = task_manager.create_task(f"Build graph: {graph_name}")
project.status = ProjectStatus.GRAPH_BUILDING
project.graph_build_task_id = task_id
ProjectManager.save_project(project)
current_locale = get_locale()
def build_task():
set_locale(current_locale)
build_logger = get_logger("mirofish.build.ingest")
try:
build_logger.info(
f"[{task_id}] start build for project={project.project_id}"
)
task_manager.update_task(
task_id, status=TaskStatus.PROCESSING, message=t("progress.initGraphService")
)
builder = GraphBuilderService(api_key=None)
# Chunk
task_manager.update_task(task_id, message=t("progress.textChunking"), progress=5)
chunks = TextProcessor.split_text(
cleaned_text, chunk_size=chunk_size, overlap=chunk_overlap
)
total_chunks = len(chunks)
# Create graph + ontology
task_manager.update_task(
task_id, message=t("progress.creatingZepGraph"), progress=10
)
graph_id = builder.create_graph(name=graph_name)
project.graph_id = graph_id
ProjectManager.save_project(project)
task_manager.update_task(
task_id, message=t("progress.settingOntology"), progress=15
)
builder.set_ontology(graph_id, ontology)
# Add text batches
def add_cb(msg, ratio):
task_manager.update_task(
task_id, message=msg, progress=15 + int(ratio * 40)
)
episode_uuids = builder.add_text_batches(
graph_id, chunks, batch_size=3, progress_callback=add_cb
)
# Wait for Graphiti to process episodes
def wait_cb(msg, ratio):
task_manager.update_task(
task_id, message=msg, progress=55 + int(ratio * 35)
)
builder._wait_for_episodes(episode_uuids, wait_cb)
# Fetch final graph data
graph_data = builder.get_graph_data(graph_id)
project.status = ProjectStatus.GRAPH_COMPLETED
ProjectManager.save_project(project)
node_count = graph_data.get("node_count", 0)
edge_count_out = graph_data.get("edge_count", 0)
build_logger.info(
f"[{task_id}] done: graph_id={graph_id} "
f"nodes={node_count} edges={edge_count_out}"
)
task_manager.update_task(
task_id,
status=TaskStatus.COMPLETED,
message=t("progress.graphBuildComplete"),
progress=100,
result={
"project_id": project.project_id,
"graph_id": graph_id,
"node_count": node_count,
"edge_count": edge_count_out,
"chunk_count": total_chunks,
},
)
except Exception as e:
build_logger.error(f"[{task_id}] failed: {e}")
build_logger.debug(traceback.format_exc())
project.status = ProjectStatus.FAILED
project.error = str(e)
ProjectManager.save_project(project)
task_manager.update_task(
task_id,
status=TaskStatus.FAILED,
message=t("progress.buildFailed", error=str(e)),
error=traceback.format_exc(),
)
thread = threading.Thread(target=build_task, daemon=True)
thread.start()
return jsonify(
{
"success": True,
"data": {
"project_id": project.project_id,
"task_id": task_id,
"graph_name": graph_name,
"ontology_entity_types": entity_count,
"ontology_edge_types": edge_count,
"text_length": len(cleaned_text),
"message": t("api.graphBuildStarted", taskId=task_id),
},
}
)
except Exception as e:
logger.error(f"ingest_text failed: {e}")
logger.debug(traceback.format_exc())
return jsonify(
{"success": False, "error": str(e), "traceback": traceback.format_exc()}
), 500
# ============== Task query endpoints ==============
@graph_bp.route('/task/<task_id>', methods=['GET'])
def get_task(task_id: str):
"""
查询任务状态
Query a task's state.
"""
task = TaskManager().get_task(task_id)
if not task:
return jsonify({
"success": False,
"error": t('api.taskNotFound', id=task_id)
}), 404
return jsonify({
"success": True,
"data": task.to_dict()
@ -553,10 +782,10 @@ def get_task(task_id: str):
@graph_bp.route('/tasks', methods=['GET'])
def list_tasks():
"""
列出所有任务
List all tasks.
"""
tasks = TaskManager().list_tasks()
return jsonify({
"success": True,
"data": [t.to_dict() for t in tasks],
@ -564,28 +793,28 @@ def list_tasks():
})
# ============== 图谱数据接口 ==============
# ============== Graph data endpoints ==============
@graph_bp.route('/data/<graph_id>', methods=['GET'])
def get_graph_data(graph_id: str):
"""
获取图谱数据（节点和边）
Get graph data (nodes and edges).
"""
try:
if not Config.ZEP_API_KEY:
if not Config.FALKORDB_HOST:
return jsonify({
"success": False,
"error": t('api.zepApiKeyMissing')
}), 500
builder = GraphBuilderService(api_key=Config.ZEP_API_KEY)
builder = GraphBuilderService(api_key=None)
graph_data = builder.get_graph_data(graph_id)
return jsonify({
"success": True,
"data": graph_data
})
except Exception as e:
return jsonify({
"success": False,
@ -597,23 +826,23 @@ def get_graph_data(graph_id: str):
@graph_bp.route('/delete/<graph_id>', methods=['DELETE'])
def delete_graph(graph_id: str):
"""
删除Zep图谱
Delete a Zep/Graphiti graph.
"""
try:
if not Config.ZEP_API_KEY:
if not Config.FALKORDB_HOST:
return jsonify({
"success": False,
"error": t('api.zepApiKeyMissing')
}), 500
builder = GraphBuilderService(api_key=Config.ZEP_API_KEY)
builder = GraphBuilderService(api_key=None)
builder.delete_graph(graph_id)
return jsonify({
"success": True,
"message": t('api.graphDeleted', id=graph_id)
})
except Exception as e:
return jsonify({
"success": False,

485
backend/app/api/report.py
View File

File diff suppressed because it is too large Load Diff

1326
backend/app/api/simulation.py
View File

File diff suppressed because it is too large Load Diff

76
backend/app/config.py
View File

@ -1,54 +1,64 @@
"""
配置管理
统一从项目根目录的 .env 文件加载配置
Configuration management.
Loads configuration from the ``.env`` file at the project root.
"""
import os
from dotenv import load_dotenv
# 加载项目根目录的 .env 文件
# 路径: MiroFish/.env (相对于 backend/app/config.py)
# Load the project-root .env file.
# Path: MiroFish/.env (relative to backend/app/config.py)
project_root_env = os.path.join(os.path.dirname(__file__), '../../.env')
if os.path.exists(project_root_env):
load_dotenv(project_root_env, override=True)
else:
# 如果根目录没有 .env，尝试加载环境变量（用于生产环境）
# If no .env at the root, fall back to environment variables (production-style).
load_dotenv(override=True)
class Config:
"""Flask配置类"""
# Flask配置
"""Flask configuration class."""
# Flask configuration
SECRET_KEY = os.environ.get('SECRET_KEY', 'mirofish-secret-key')
DEBUG = os.environ.get('FLASK_DEBUG', 'True').lower() == 'true'
# JSON配置 - 禁用ASCII转义，让中文直接显示（而不是 \uXXXX 格式）
# JSON configuration - disable ASCII escaping so Chinese characters are emitted
# directly instead of being encoded as \\uXXXX.
JSON_AS_ASCII = False
# LLM配置（统一使用OpenAI格式）
# LLM configuration (uniformly using the OpenAI-compatible interface)
LLM_API_KEY = os.environ.get('LLM_API_KEY')
LLM_BASE_URL = os.environ.get('LLM_BASE_URL', 'https://api.openai.com/v1')
LLM_MODEL_NAME = os.environ.get('LLM_MODEL_NAME', 'gpt-4o-mini')
# Zep配置
ZEP_API_KEY = os.environ.get('ZEP_API_KEY')
# 文件上传配置
# FalkorDB configuration (replaces Zep)
FALKORDB_HOST = os.environ.get('FALKORDB_HOST', 'localhost')
FALKORDB_PORT = int(os.environ.get('FALKORDB_PORT', '6379'))
FALKORDB_USERNAME = os.environ.get('FALKORDB_USERNAME', None) or None
FALKORDB_PASSWORD = os.environ.get('FALKORDB_PASSWORD', None) or None
# Embedding model: local sentence-transformers (pre-downloaded in image)
EMBEDDING_MODEL = os.environ.get(
'EMBEDDING_MODEL',
'sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2',
)
# File upload configuration
MAX_CONTENT_LENGTH = 50 * 1024 * 1024 # 50MB
UPLOAD_FOLDER = os.path.join(os.path.dirname(__file__), '../uploads')
ALLOWED_EXTENSIONS = {'pdf', 'md', 'txt', 'markdown'}
# 文本处理配置
DEFAULT_CHUNK_SIZE = 500 # 默认切块大小
DEFAULT_CHUNK_OVERLAP = 50 # 默认重叠大小
# OASIS模拟配置
# Text processing configuration
DEFAULT_CHUNK_SIZE = 500 # default chunk size
DEFAULT_CHUNK_OVERLAP = 50 # default overlap
# OASIS simulation configuration
OASIS_DEFAULT_MAX_ROUNDS = int(os.environ.get('OASIS_DEFAULT_MAX_ROUNDS', '10'))
OASIS_SIMULATION_DATA_DIR = os.path.join(os.path.dirname(__file__), '../uploads/simulations')
# OASIS平台可用动作配置
# OASIS platform available actions
OASIS_TWITTER_ACTIONS = [
'CREATE_POST', 'LIKE_POST', 'REPOST', 'FOLLOW', 'DO_NOTHING', 'QUOTE_POST'
]
@ -57,19 +67,19 @@ class Config:
'LIKE_COMMENT', 'DISLIKE_COMMENT', 'SEARCH_POSTS', 'SEARCH_USER',
'TREND', 'REFRESH', 'DO_NOTHING', 'FOLLOW', 'MUTE'
]
# Report Agent配置
# Report Agent configuration
REPORT_AGENT_MAX_TOOL_CALLS = int(os.environ.get('REPORT_AGENT_MAX_TOOL_CALLS', '5'))
REPORT_AGENT_MAX_REFLECTION_ROUNDS = int(os.environ.get('REPORT_AGENT_MAX_REFLECTION_ROUNDS', '2'))
REPORT_AGENT_TEMPERATURE = float(os.environ.get('REPORT_AGENT_TEMPERATURE', '0.5'))
@classmethod
def validate(cls) -> list[str]:
"""验证必要配置"""
"""Validate required configuration."""
errors: list[str] = []
if not cls.LLM_API_KEY:
errors.append("LLM_API_KEY 未配置")
if not cls.ZEP_API_KEY:
errors.append("ZEP_API_KEY 未配置")
errors.append("LLM_API_KEY is not configured")
# FalkorDB host defaults to localhost; only fail if explicitly empty
if not cls.FALKORDB_HOST:
errors.append("FALKORDB_HOST is not configured")
return errors

3
backend/app/models/__init__.py
View File

@ -1,9 +1,8 @@
"""
数据模型模块
Data models module.
"""
from .task import TaskManager, TaskStatus
from .project import Project, ProjectStatus, ProjectManager
__all__ = ['TaskManager', 'TaskStatus', 'Project', 'ProjectStatus', 'ProjectManager']

216
backend/app/models/project.py
View File

@ -1,6 +1,7 @@
"""
项目上下文管理
用于在服务端持久化项目状态，避免前端在接口间传递大量数据
Project context management.
Persists project state on the server side so the frontend does not need to
shuttle large amounts of data between API calls.
"""
import os
@ -15,45 +16,45 @@ from ..config import Config
class ProjectStatus(str, Enum):
"""项目状态"""
CREATED = "created" # 刚创建，文件已上传
ONTOLOGY_GENERATED = "ontology_generated" # 本体已生成
GRAPH_BUILDING = "graph_building" # 图谱构建中
GRAPH_COMPLETED = "graph_completed" # 图谱构建完成
FAILED = "failed" # 失败
"""Project lifecycle status."""
CREATED = "created" # just created, files uploaded
ONTOLOGY_GENERATED = "ontology_generated" # ontology has been generated
GRAPH_BUILDING = "graph_building" # graph is being built
GRAPH_COMPLETED = "graph_completed" # graph build complete
FAILED = "failed" # failed
@dataclass
class Project:
"""项目数据模型"""
"""Project data model."""
project_id: str
name: str
status: ProjectStatus
created_at: str
updated_at: str
# 文件信息
# File metadata
files: List[Dict[str, str]] = field(default_factory=list) # [{filename, path, size}]
total_text_length: int = 0
# 本体信息（接口1生成后填充）
# Ontology info (populated after endpoint 1)
ontology: Optional[Dict[str, Any]] = None
analysis_summary: Optional[str] = None
# 图谱信息（接口2完成后填充）
# Graph info (populated after endpoint 2 completes)
graph_id: Optional[str] = None
graph_build_task_id: Optional[str] = None
# 配置
# Configuration
simulation_requirement: Optional[str] = None
chunk_size: int = 500
chunk_overlap: int = 50
# 错误信息
# Error information
error: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
"""转换为字典"""
"""Convert to a plain dictionary."""
return {
"project_id": self.project_id,
"name": self.name,
@ -71,14 +72,14 @@ class Project:
"chunk_overlap": self.chunk_overlap,
"error": self.error
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> 'Project':
"""从字典创建"""
"""Create a Project from a plain dictionary."""
status = data.get('status', 'created')
if isinstance(status, str):
status = ProjectStatus(status)
return cls(
project_id=data['project_id'],
name=data.get('name', 'Unnamed Project'),
@ -99,52 +100,52 @@ class Project:
class ProjectManager:
"""项目管理器 - 负责项目的持久化存储和检索"""
# 项目存储根目录
"""Project manager - responsible for persistent storage and retrieval of projects."""
# Project storage root directory
PROJECTS_DIR = os.path.join(Config.UPLOAD_FOLDER, 'projects')
@classmethod
def _ensure_projects_dir(cls):
"""确保项目目录存在"""
"""Make sure the projects directory exists."""
os.makedirs(cls.PROJECTS_DIR, exist_ok=True)
@classmethod
def _get_project_dir(cls, project_id: str) -> str:
"""获取项目目录路径"""
"""Get the directory path for a project."""
return os.path.join(cls.PROJECTS_DIR, project_id)
@classmethod
def _get_project_meta_path(cls, project_id: str) -> str:
"""获取项目元数据文件路径"""
"""Get the path to a project's metadata file."""
return os.path.join(cls._get_project_dir(project_id), 'project.json')
@classmethod
def _get_project_files_dir(cls, project_id: str) -> str:
"""获取项目文件存储目录"""
"""Get the directory where a project's files are stored."""
return os.path.join(cls._get_project_dir(project_id), 'files')
@classmethod
def _get_project_text_path(cls, project_id: str) -> str:
"""获取项目提取文本存储路径"""
"""Get the path to a project's extracted text."""
return os.path.join(cls._get_project_dir(project_id), 'extracted_text.txt')
@classmethod
def create_project(cls, name: str = "Unnamed Project") -> Project:
"""
创建新项目
Create a new project.
Args:
name: 项目名称
name: project name.
Returns:
新创建的Project对象
The newly created Project object.
"""
cls._ensure_projects_dir()
project_id = f"proj_{uuid.uuid4().hex[:12]}"
now = datetime.now().isoformat()
project = Project(
project_id=project_id,
name=name,
@ -152,154 +153,153 @@ class ProjectManager:
created_at=now,
updated_at=now
)
# 创建项目目录结构
# Create the project directory structure
project_dir = cls._get_project_dir(project_id)
files_dir = cls._get_project_files_dir(project_id)
os.makedirs(project_dir, exist_ok=True)
os.makedirs(files_dir, exist_ok=True)
# 保存项目元数据
# Persist the project metadata
cls.save_project(project)
return project
@classmethod
def save_project(cls, project: Project) -> None:
"""保存项目元数据"""
"""Persist project metadata to disk."""
project.updated_at = datetime.now().isoformat()
meta_path = cls._get_project_meta_path(project.project_id)
with open(meta_path, 'w', encoding='utf-8') as f:
json.dump(project.to_dict(), f, ensure_ascii=False, indent=2)
@classmethod
def get_project(cls, project_id: str) -> Optional[Project]:
"""
获取项目
Retrieve a project by ID.
Args:
project_id: 项目ID
project_id: project ID.
Returns:
Project对象，如果不存在返回None
The Project object, or None if it does not exist.
"""
meta_path = cls._get_project_meta_path(project_id)
if not os.path.exists(meta_path):
return None
with open(meta_path, 'r', encoding='utf-8') as f:
data = json.load(f)
return Project.from_dict(data)
@classmethod
def list_projects(cls, limit: int = 50) -> List[Project]:
"""
列出所有项目
List all projects.
Args:
limit: 返回数量限制
limit: maximum number of results to return.
Returns:
项目列表，按创建时间倒序
A list of projects, sorted newest-first by creation time.
"""
cls._ensure_projects_dir()
projects = []
for project_id in os.listdir(cls.PROJECTS_DIR):
project = cls.get_project(project_id)
if project:
projects.append(project)
# 按创建时间倒序排序
# Sort newest-first by creation time
projects.sort(key=lambda p: p.created_at, reverse=True)
return projects[:limit]
@classmethod
def delete_project(cls, project_id: str) -> bool:
"""
删除项目及其所有文件
Delete a project and all of its files.
Args:
project_id: 项目ID
project_id: project ID.
Returns:
是否删除成功
True if the project was deleted, False otherwise.
"""
project_dir = cls._get_project_dir(project_id)
if not os.path.exists(project_dir):
return False
shutil.rmtree(project_dir)
return True
@classmethod
def save_file_to_project(cls, project_id: str, file_storage, original_filename: str) -> Dict[str, str]:
"""
保存上传的文件到项目目录
Save an uploaded file into the project's directory.
Args:
project_id: 项目ID
file_storage: Flask的FileStorage对象
original_filename: 原始文件名
project_id: project ID.
file_storage: Flask FileStorage object.
original_filename: original filename.
Returns:
文件信息字典 {filename, path, size}
Dictionary with file metadata: {filename, path, size}.
"""
files_dir = cls._get_project_files_dir(project_id)
os.makedirs(files_dir, exist_ok=True)
# 生成安全的文件名
# Build a safe filename
ext = os.path.splitext(original_filename)[1].lower()
safe_filename = f"{uuid.uuid4().hex[:8]}{ext}"
file_path = os.path.join(files_dir, safe_filename)
# 保存文件
# Save the file
file_storage.save(file_path)
# 获取文件大小
# Read back its size
file_size = os.path.getsize(file_path)
return {
"original_filename": original_filename,
"saved_filename": safe_filename,
"path": file_path,
"size": file_size
}
@classmethod
def save_extracted_text(cls, project_id: str, text: str) -> None:
"""保存提取的文本"""
"""Persist extracted text to disk."""
text_path = cls._get_project_text_path(project_id)
with open(text_path, 'w', encoding='utf-8') as f:
f.write(text)
@classmethod
def get_extracted_text(cls, project_id: str) -> Optional[str]:
"""获取提取的文本"""
"""Read extracted text from disk."""
text_path = cls._get_project_text_path(project_id)
if not os.path.exists(text_path):
return None
with open(text_path, 'r', encoding='utf-8') as f:
return f.read()
@classmethod
def get_project_files(cls, project_id: str) -> List[str]:
"""获取项目的所有文件路径"""
"""Return the absolute paths of all files in the project."""
files_dir = cls._get_project_files_dir(project_id)
if not os.path.exists(files_dir):
return []
return [
os.path.join(files_dir, f)
for f in os.listdir(files_dir)
os.path.join(files_dir, f)
for f in os.listdir(files_dir)
if os.path.isfile(os.path.join(files_dir, f))
]

105
backend/app/models/task.py
View File

@ -1,6 +1,6 @@
"""
任务状态管理
用于跟踪长时间运行的任务（如图谱构建）
Task status management.
Tracks long-running tasks (such as graph building) and their progress.
"""
import uuid
@ -14,30 +14,30 @@ from ..utils.locale import t
class TaskStatus(str, Enum):
"""任务状态枚举"""
PENDING = "pending" # 等待中
PROCESSING = "processing" # 处理中
COMPLETED = "completed" # 已完成
FAILED = "failed" # 失败
"""Task status enum."""
PENDING = "pending" # waiting
PROCESSING = "processing" # in progress
COMPLETED = "completed" # finished
FAILED = "failed" # failed
@dataclass
class Task:
"""任务数据类"""
"""Task data class."""
task_id: str
task_type: str
status: TaskStatus
created_at: datetime
updated_at: datetime
progress: int = 0 # 总进度百分比 0-100
message: str = "" # 状态消息
result: Optional[Dict] = None # 任务结果
error: Optional[str] = None # 错误信息
metadata: Dict = field(default_factory=dict) # 额外元数据
progress_detail: Dict = field(default_factory=dict) # 详细进度信息
progress: int = 0 # overall progress percentage 0-100
message: str = "" # status message
result: Optional[Dict] = None # task result payload
error: Optional[str] = None # error message
metadata: Dict = field(default_factory=dict) # extra metadata
progress_detail: Dict = field(default_factory=dict) # detailed progress
def to_dict(self) -> Dict[str, Any]:
"""转换为字典"""
"""Convert to a plain dictionary."""
return {
"task_id": self.task_id,
"task_type": self.task_type,
@ -55,15 +55,15 @@ class Task:
class TaskManager:
"""
任务管理器
线程安全的任务状态管理
Task manager.
Thread-safe tracking of task state and progress.
"""
_instance = None
_lock = threading.Lock()
def __new__(cls):
"""单例模式"""
"""Singleton accessor."""
if cls._instance is None:
with cls._lock:
if cls._instance is None:
@ -71,21 +71,21 @@ class TaskManager:
cls._instance._tasks: Dict[str, Task] = {}
cls._instance._task_lock = threading.Lock()
return cls._instance
def create_task(self, task_type: str, metadata: Optional[Dict] = None) -> str:
"""
创建新任务
Create a new task.
Args:
task_type: 任务类型
metadata: 额外元数据
task_type: task type.
metadata: extra metadata.
Returns:
任务ID
The new task ID.
"""
task_id = str(uuid.uuid4())
now = datetime.now()
task = Task(
task_id=task_id,
task_type=task_type,
@ -94,17 +94,17 @@ class TaskManager:
updated_at=now,
metadata=metadata or {}
)
with self._task_lock:
self._tasks[task_id] = task
return task_id
def get_task(self, task_id: str) -> Optional[Task]:
"""获取任务"""
"""Fetch a task by ID."""
with self._task_lock:
return self._tasks.get(task_id)
def update_task(
self,
task_id: str,
@ -116,16 +116,16 @@ class TaskManager:
progress_detail: Optional[Dict] = None
):
"""
更新任务状态
Update task state.
Args:
task_id: 任务ID
status: 新状态
progress: 进度
message: 消息
result: 结果
error: 错误信息
progress_detail: 详细进度信息
task_id: task ID.
status: new status.
progress: progress percentage.
message: status message.
result: result payload.
error: error message.
progress_detail: detailed progress information.
"""
with self._task_lock:
task = self._tasks.get(task_id)
@ -143,9 +143,9 @@ class TaskManager:
task.error = error
if progress_detail is not None:
task.progress_detail = progress_detail
def complete_task(self, task_id: str, result: Dict):
"""标记任务完成"""
"""Mark a task as completed."""
self.update_task(
task_id,
status=TaskStatus.COMPLETED,
@ -153,29 +153,29 @@ class TaskManager:
message=t('progress.taskComplete'),
result=result
)
def fail_task(self, task_id: str, error: str):
"""标记任务失败"""
"""Mark a task as failed."""
self.update_task(
task_id,
status=TaskStatus.FAILED,
message=t('progress.taskFailed'),
error=error
)
def list_tasks(self, task_type: Optional[str] = None) -> list:
"""列出任务"""
"""List tasks, optionally filtered by type."""
with self._task_lock:
tasks = list(self._tasks.values())
if task_type:
tasks = [t for t in tasks if t.task_type == task_type]
return [t.to_dict() for t in sorted(tasks, key=lambda x: x.created_at, reverse=True)]
def cleanup_old_tasks(self, max_age_hours: int = 24):
"""清理旧任务"""
"""Remove tasks older than the given threshold."""
from datetime import timedelta
cutoff = datetime.now() - timedelta(hours=max_age_hours)
with self._task_lock:
old_ids = [
tid for tid, task in self._tasks.items()
@ -183,4 +183,3 @@ class TaskManager:
]
for tid in old_ids:
del self._tasks[tid]

9
backend/app/services/__init__.py
View File

@ -1,5 +1,5 @@
"""
业务服务模块
Business services module.
"""
from .ontology_generator import OntologyGenerator
@ -9,7 +9,7 @@ from .zep_entity_reader import ZepEntityReader, EntityNode, FilteredEntities
from .oasis_profile_generator import OasisProfileGenerator, OasisAgentProfile
from .simulation_manager import SimulationManager, SimulationState, SimulationStatus
from .simulation_config_generator import (
SimulationConfigGenerator,
SimulationConfigGenerator,
SimulationParameters,
AgentActivityConfig,
TimeSimulationConfig,
@ -38,8 +38,8 @@ from .simulation_ipc import (
)
__all__ = [
'OntologyGenerator',
'GraphBuilderService',
'OntologyGenerator',
'GraphBuilderService',
'TextProcessor',
'ZepEntityReader',
'EntityNode',
@ -70,4 +70,3 @@ __all__ = [
'CommandType',
'CommandStatus',
]

366
backend/app/services/graph_builder.py
View File

@ -1,6 +1,6 @@
"""
图谱构建服务
接口2：使用Zep API构建Standalone Graph
Graph build service
Interface 2: build a Standalone Graph via the Zep API.
"""
import os
@ -10,24 +10,42 @@ import threading
from typing import Dict, Any, List, Optional, Callable
from dataclasses import dataclass
from zep_cloud.client import Zep
from zep_cloud import EpisodeData, EntityEdgeSourceTarget
# Zep types (EpisodeData, EntityEdgeSourceTarget, EntityModel/EdgeModel from
# external_clients.ontology) are still referenced for ontology dynamic Pydantic
# classes. When zep-cloud is uninstalled, we provide stubs so module import
# succeeds and the Zep `client` attribute on this service is just the
# GraphitiAdapter (see __init__ below).
try:
from zep_cloud.client import Zep
from zep_cloud import EpisodeData, EntityEdgeSourceTarget
except ImportError:
class Zep:
def __init__(self, *a, **kw): pass
class EpisodeData:
def __init__(self, data, type="text"):
self.data = data
self.type = type
class EntityEdgeSourceTarget:
def __init__(self, source, target):
self.source = source
self.target = target
from ..config import Config
from ..models.task import TaskManager, TaskStatus
from ..utils.zep_paging import fetch_all_nodes, fetch_all_edges
from .text_processor import TextProcessor
from .graphiti_service import get_graphiti_adapter
from ..utils.locale import t, get_locale, set_locale
@dataclass
class GraphInfo:
"""图谱信息"""
"""Graph info"""
graph_id: str
node_count: int
edge_count: int
entity_types: List[str]
def to_dict(self) -> Dict[str, Any]:
return {
"graph_id": self.graph_id,
@ -39,18 +57,17 @@ class GraphInfo:
class GraphBuilderService:
"""
图谱构建服务
负责调用Zep API构建知识图谱
Graph build service
Responsible for calling the Zep API to build the knowledge graph.
"""
def __init__(self, api_key: Optional[str] = None):
self.api_key = api_key or Config.ZEP_API_KEY
if not self.api_key:
raise ValueError("ZEP_API_KEY 未配置")
self.client = Zep(api_key=self.api_key)
# Zep's per-project graph becomes Graphiti's group_id (a partition).
# The single shared GraphitiAdapter handles all projects.
self.api_key = api_key # kept for signature compat; no longer required
self.client = get_graphiti_adapter()
self.task_manager = TaskManager()
def build_graph_async(
self,
text: str,
@ -61,20 +78,20 @@ class GraphBuilderService:
batch_size: int = 3
) -> str:
"""
异步构建图谱
Build the graph asynchronously.
Args:
text: 输入文本
ontology: 本体定义（来自接口1的输出）
graph_name: 图谱名称
chunk_size: 文本块大小
chunk_overlap: 块重叠大小
batch_size: 每批发送的块数量
text: Input text
ontology: Ontology definition (output of interface 1)
graph_name: Graph name
chunk_size: Text chunk size
chunk_overlap: Chunk overlap size
batch_size: Number of chunks sent per batch
Returns:
任务ID
Task ID
"""
# 创建任务
# Create task
task_id = self.task_manager.create_task(
task_type="graph_build",
metadata={
@ -83,20 +100,20 @@ class GraphBuilderService:
"text_length": len(text),
}
)
# Capture locale before spawning background thread
current_locale = get_locale()
# 在后台线程中执行构建
# Execute the build in a background thread
thread = threading.Thread(
target=self._build_graph_worker,
args=(task_id, text, ontology, graph_name, chunk_size, chunk_overlap, batch_size, current_locale)
)
thread.daemon = True
thread.start()
return task_id
def _build_graph_worker(
self,
task_id: str,
@ -108,7 +125,7 @@ class GraphBuilderService:
batch_size: int,
locale: str = 'zh'
):
"""图谱构建工作线程"""
"""Graph build worker thread"""
set_locale(locale)
try:
self.task_manager.update_task(
@ -117,24 +134,24 @@ class GraphBuilderService:
progress=5,
message=t('progress.startBuildingGraph')
)
# 1. 创建图谱
# 1. Create the graph
graph_id = self.create_graph(graph_name)
self.task_manager.update_task(
task_id,
progress=10,
message=t('progress.graphCreated', graphId=graph_id)
)
# 2. 设置本体
# 2. Set the ontology
self.set_ontology(graph_id, ontology)
self.task_manager.update_task(
task_id,
progress=15,
message=t('progress.ontologySet')
)
# 3. 文本分块
# 3. Split the text into chunks
chunks = TextProcessor.split_text(text, chunk_size, chunk_overlap)
total_chunks = len(chunks)
self.task_manager.update_task(
@ -142,8 +159,8 @@ class GraphBuilderService:
progress=20,
message=t('progress.textSplit', count=total_chunks)
)
# 4. 分批发送数据
# 4. Send data in batches
episode_uuids = self.add_text_batches(
graph_id, chunks, batch_size,
lambda msg, prog: self.task_manager.update_task(
@ -152,14 +169,14 @@ class GraphBuilderService:
message=msg
)
)
# 5. 等待Zep处理完成
# 5. Wait for Zep processing to finish
self.task_manager.update_task(
task_id,
progress=60,
message=t('progress.waitingZepProcess')
)
self._wait_for_episodes(
episode_uuids,
lambda msg, prog: self.task_manager.update_task(
@ -168,109 +185,127 @@ class GraphBuilderService:
message=msg
)
)
# 6. 获取图谱信息
# 6. Fetch graph info
self.task_manager.update_task(
task_id,
progress=90,
message=t('progress.fetchingGraphInfo')
)
graph_info = self._get_graph_info(graph_id)
# 完成
# Complete
self.task_manager.complete_task(task_id, {
"graph_id": graph_id,
"graph_info": graph_info.to_dict(),
"chunks_processed": total_chunks,
})
except Exception as e:
import traceback
error_msg = f"{str(e)}\n{traceback.format_exc()}"
self.task_manager.fail_task(task_id, error_msg)
def create_graph(self, name: str) -> str:
"""创建Zep图谱（公开方法）"""
"""Create a graph (public method)"""
graph_id = f"mirofish_{uuid.uuid4().hex[:16]}"
self.client.graph.create(
# Graphiti creates the partition implicitly on first add_episode; this
# call just reserves the group_id.
self.client.create_graph(
graph_id=graph_id,
name=name,
description="MiroFish Social Simulation Graph"
description="MiroFish Social Simulation Graph",
)
return graph_id
def set_ontology(self, graph_id: str, ontology: Dict[str, Any]):
"""设置图谱本体（公开方法）"""
"""Set the graph ontology (public method)"""
import warnings
from typing import Optional
from pydantic import Field
from zep_cloud.external_clients.ontology import EntityModel, EntityText, EdgeModel
# 抑制 Pydantic v2 关于 Field(default=None) 的警告
# 这是 Zep SDK 要求的用法，警告来自动态类创建，可以安全忽略
# Base classes for the dynamic Pydantic ontology types. Try Zep's
# EntityModel/EdgeModel first (preserves a tiny bit of compatibility if
# zep-cloud is somehow still in the env), else fall back to Graphiti's
# EntityNode/EntityEdge which is what we actually use.
try:
from zep_cloud.external_clients.ontology import EntityModel, EntityText, EdgeModel
except ImportError:
try:
from graphiti_core.nodes import EntityNode as _GTEntity
from graphiti_core.edges import EntityEdge as _GTEdge
EntityModel = _GTEntity # type: ignore[assignment,misc]
EdgeModel = _GTEdge # type: ignore[assignment,misc]
EntityText = str # type: ignore[assignment,misc]
except ImportError:
from pydantic import BaseModel as EntityModel # type: ignore[assignment,misc]
from pydantic import BaseModel as EdgeModel # type: ignore[assignment,misc]
EntityText = str # type: ignore[assignment,misc]
# Suppress Pydantic v2 warning about Field(default=None)
# This is required by the Zep SDK; the warning comes from dynamic
# class creation and can safely be ignored.
warnings.filterwarnings('ignore', category=UserWarning, module='pydantic')
# Zep 保留名称，不能作为属性名
# Zep reserved names; cannot be used as attribute names
RESERVED_NAMES = {'uuid', 'name', 'group_id', 'name_embedding', 'summary', 'created_at'}
def safe_attr_name(attr_name: str) -> str:
"""将保留名称转换为安全名称"""
"""Convert a reserved name into a safe name"""
if attr_name.lower() in RESERVED_NAMES:
return f"entity_{attr_name}"
return attr_name
# 动态创建实体类型
# Dynamically create entity types
entity_types = {}
for entity_def in ontology.get("entity_types", []):
name = entity_def["name"]
description = entity_def.get("description", f"A {name} entity.")
# 创建属性字典和类型注解（Pydantic v2 需要）
# Build the attribute dict and type annotations (required by Pydantic v2)
attrs = {"__doc__": description}
annotations = {}
for attr_def in entity_def.get("attributes", []):
attr_name = safe_attr_name(attr_def["name"]) # 使用安全名称
attr_name = safe_attr_name(attr_def["name"]) # Use the safe name
attr_desc = attr_def.get("description", attr_name)
# Zep API 需要 Field 的 description，这是必需的
# Zep API requires a description on Field; this is required
attrs[attr_name] = Field(description=attr_desc, default=None)
annotations[attr_name] = Optional[EntityText] # 类型注解
annotations[attr_name] = Optional[EntityText] # type annotation
attrs["__annotations__"] = annotations
# 动态创建类
# Dynamically create the class
entity_class = type(name, (EntityModel,), attrs)
entity_class.__doc__ = description
entity_types[name] = entity_class
# 动态创建边类型
# Dynamically create edge types
edge_definitions = {}
for edge_def in ontology.get("edge_types", []):
name = edge_def["name"]
description = edge_def.get("description", f"A {name} relationship.")
# 创建属性字典和类型注解
# Build the attribute dict and type annotations
attrs = {"__doc__": description}
annotations = {}
for attr_def in edge_def.get("attributes", []):
attr_name = safe_attr_name(attr_def["name"]) # 使用安全名称
attr_name = safe_attr_name(attr_def["name"]) # Use the safe name
attr_desc = attr_def.get("description", attr_name)
# Zep API 需要 Field 的 description，这是必需的
# Zep API requires a description on Field; this is required
attrs[attr_name] = Field(description=attr_desc, default=None)
annotations[attr_name] = Optional[str] # 边属性用str类型
annotations[attr_name] = Optional[str] # Edge attributes use str type
attrs["__annotations__"] = annotations
# 动态创建类
# Dynamically create the class
class_name = ''.join(word.capitalize() for word in name.split('_'))
edge_class = type(class_name, (EdgeModel,), attrs)
edge_class.__doc__ = description
# 构建source_targets
# Build source_targets
source_targets = []
for st in edge_def.get("source_targets", []):
source_targets.append(
@ -279,18 +314,19 @@ class GraphBuilderService:
target=st.get("target", "Entity")
)
)
if source_targets:
edge_definitions[name] = (edge_class, source_targets)
# 调用Zep API设置本体
# Graphiti accepts the original JSON ontology directly. The Pydantic
# class construction above was a Zep-specific quirk; we can skip it
# for Graphiti and just forward the input. Keep the code path the same
# shape (Pydantic EntityModel/EdgeModel classes still constructed and
# exposed to introspection code), but the actual set_ontology call now
# uses the adapter with the source-of-truth JSON.
if entity_types or edge_definitions:
self.client.graph.set_ontology(
graph_ids=[graph_id],
entities=entity_types if entity_types else None,
edges=edge_definitions if edge_definitions else None,
)
self.client.set_ontology(graph_id=graph_id, ontology=ontology)
def add_text_batches(
self,
graph_id: str,
@ -298,117 +334,80 @@ class GraphBuilderService:
batch_size: int = 3,
progress_callback: Optional[Callable] = None
) -> List[str]:
"""分批添加文本到图谱，返回所有 episode 的 uuid 列表"""
"""Add text to the graph in batches, returning a list of episode uuids"""
episode_uuids = []
total_chunks = len(chunks)
for i in range(0, total_chunks, batch_size):
batch_chunks = chunks[i:i + batch_size]
batch_num = i // batch_size + 1
total_batches = (total_chunks + batch_size - 1) // batch_size
if progress_callback:
progress = (i + len(batch_chunks)) / total_chunks
progress_callback(
t('progress.sendingBatch', current=batch_num, total=total_batches, chunks=len(batch_chunks)),
progress
)
# 构建episode数据
# Build episode data
episodes = [
EpisodeData(data=chunk, type="text")
for chunk in batch_chunks
]
# 发送到Zep
# Send to the graph (Graphiti via adapter — returns after sync extraction)
try:
batch_result = self.client.graph.add_batch(
batch_result = self.client.add_batch(
graph_id=graph_id,
episodes=episodes
episodes=episodes,
)
# 收集返回的 episode uuid
# Collect returned episode uuids
if batch_result and isinstance(batch_result, list):
for ep in batch_result:
ep_uuid = getattr(ep, 'uuid_', None) or getattr(ep, 'uuid', None)
if ep_uuid:
episode_uuids.append(ep_uuid)
# 避免请求过快
time.sleep(1)
except Exception as e:
if progress_callback:
progress_callback(t('progress.batchFailed', batch=batch_num, error=str(e)), 0)
raise
return episode_uuids
def _wait_for_episodes(
self,
episode_uuids: List[str],
progress_callback: Optional[Callable] = None,
timeout: int = 600
):
"""等待所有 episode 处理完成（通过查询每个 episode 的 processed 状态）"""
"""Wait for all episodes to finish processing.
Graphiti's add_episode is synchronous (extraction completes before the
call returns), so all episodes returned by add_batch are already done.
We still surface progress messages and respect the timeout for the
outer loop contract, but we never actually need to poll.
"""
if not episode_uuids:
if progress_callback:
progress_callback(t('progress.noEpisodesWait'), 1.0)
return
start_time = time.time()
pending_episodes = set(episode_uuids)
completed_count = 0
total_episodes = len(episode_uuids)
if progress_callback:
progress_callback(t('progress.waitingEpisodes', count=total_episodes), 0)
while pending_episodes:
if time.time() - start_time > timeout:
if progress_callback:
progress_callback(
t('progress.episodesTimeout', completed=completed_count, total=total_episodes),
completed_count / total_episodes
)
break
# 检查每个 episode 的处理状态
for ep_uuid in list(pending_episodes):
try:
episode = self.client.graph.episode.get(uuid_=ep_uuid)
is_processed = getattr(episode, 'processed', False)
if is_processed:
pending_episodes.remove(ep_uuid)
completed_count += 1
except Exception as e:
# 忽略单个查询错误，继续
pass
elapsed = int(time.time() - start_time)
if progress_callback:
progress_callback(
t('progress.zepProcessing', completed=completed_count, total=total_episodes, pending=len(pending_episodes), elapsed=elapsed),
completed_count / total_episodes if total_episodes > 0 else 0
)
if pending_episodes:
time.sleep(3) # 每3秒检查一次
if progress_callback:
progress_callback(t('progress.processingComplete', completed=completed_count, total=total_episodes), 1.0)
progress_callback(t('progress.waitingEpisodes', count=len(episode_uuids)), 0)
progress_callback(t('progress.processingComplete', completed=len(episode_uuids), total=len(episode_uuids)), 1.0)
def _get_graph_info(self, graph_id: str) -> GraphInfo:
"""获取图谱信息"""
# 获取节点（分页）
"""Get graph info"""
# Get nodes (paginated)
nodes = fetch_all_nodes(self.client, graph_id)
# 获取边（分页）
# Get edges (paginated)
edges = fetch_all_edges(self.client, graph_id)
# 统计实体类型
# Aggregate entity types
entity_types = set()
for node in nodes:
if node.labels:
@ -422,32 +421,33 @@ class GraphBuilderService:
edge_count=len(edges),
entity_types=list(entity_types)
)
def get_graph_data(self, graph_id: str) -> Dict[str, Any]:
"""
获取完整图谱数据（包含详细信息）
Get the complete graph data (including detailed info).
Args:
graph_id: 图谱ID
graph_id: Graph ID
Returns:
包含nodes和edges的字典，包括时间信息、属性等详细数据
A dict containing nodes and edges, including timing info,
attributes, and other detailed data.
"""
nodes = fetch_all_nodes(self.client, graph_id)
edges = fetch_all_edges(self.client, graph_id)
# 创建节点映射用于获取节点名称
# Build a node-name map
node_map = {}
for node in nodes:
node_map[node.uuid_] = node.name or ""
nodes_data = []
for node in nodes:
# 获取创建时间
# Get creation time
created_at = getattr(node, 'created_at', None)
if created_at:
created_at = str(created_at)
nodes_data.append({
"uuid": node.uuid_,
"name": node.name,
@ -456,25 +456,25 @@ class GraphBuilderService:
"attributes": node.attributes or {},
"created_at": created_at,
})
edges_data = []
for edge in edges:
# 获取时间信息
# Get time info
created_at = getattr(edge, 'created_at', None)
valid_at = getattr(edge, 'valid_at', None)
invalid_at = getattr(edge, 'invalid_at', None)
expired_at = getattr(edge, 'expired_at', None)
# 获取 episodes
# Get episodes
episodes = getattr(edge, 'episodes', None) or getattr(edge, 'episode_ids', None)
if episodes and not isinstance(episodes, list):
episodes = [str(episodes)]
elif episodes:
episodes = [str(e) for e in episodes]
# 获取 fact_type
# Get fact_type
fact_type = getattr(edge, 'fact_type', None) or edge.name or ""
edges_data.append({
"uuid": edge.uuid_,
"name": edge.name or "",
@ -491,7 +491,7 @@ class GraphBuilderService:
"expired_at": str(expired_at) if expired_at else None,
"episodes": episodes or [],
})
return {
"graph_id": graph_id,
"nodes": nodes_data,
@ -501,6 +501,6 @@ class GraphBuilderService:
}
def delete_graph(self, graph_id: str):
"""删除图谱"""
self.client.graph.delete(graph_id=graph_id)
"""Delete a graph"""
self.client.delete_graph(graph_id=graph_id)

1175
backend/app/services/graphiti_service.py Normal file
View File

File diff suppressed because it is too large Load Diff

655
backend/app/services/oasis_profile_generator.py
View File

File diff suppressed because it is too large Load Diff

357
backend/app/services/ontology_generator.py
View File

@ -1,6 +1,6 @@
"""
本体生成服务
接口1：分析文本内容，生成适合社会模拟的实体和关系类型定义
Ontology generation service
Interface 1: analyze text content to generate entity and relationship type definitions suitable for social simulation
"""
import json
@ -14,169 +14,169 @@ logger = logging.getLogger(__name__)
def _to_pascal_case(name: str) -> str:
"""将任意格式的名称转换为 PascalCase（如 'works_for' -> 'WorksFor', 'person' -> 'Person'）"""
# 按非字母数字字符分割
"""Convert any-format name to PascalCase (e.g. 'works_for' -> 'WorksFor', 'person' -> 'Person')"""
# Split by non-alphanumeric characters
parts = re.split(r'[^a-zA-Z0-9]+', name)
# 再按 camelCase 边界分割（如 'camelCase' -> ['camel', 'Case']）
# Then split by camelCase boundaries (e.g. 'camelCase' -> ['camel', 'Case'])
words = []
for part in parts:
words.extend(re.sub(r'([a-z])([A-Z])', r'\1_\2', part).split('_'))
# 每个词首字母大写，过滤空串
# Capitalize each word, filter out empty strings
result = ''.join(word.capitalize() for word in words if word)
return result if result else 'Unknown'
# 本体生成的系统提示词
ONTOLOGY_SYSTEM_PROMPT = """你是一个专业的知识图谱本体设计专家。你的任务是分析给定的文本内容和模拟需求，设计适合**社交媒体舆论模拟**的实体类型和关系类型。
# System prompt for ontology generation
ONTOLOGY_SYSTEM_PROMPT = """You are a professional knowledge-graph ontology design expert. Your task is to analyze the given text content and simulation requirement, design entity types and relationship types suitable for **social media public-opinion simulation**.
**重要：你必须输出有效的JSON格式数据，不要输出任何其他内容。**
**IMPORTANT: You MUST output valid JSON format data, and nothing else.**
## 核心任务背景
## Core Task Background
我们正在构建一个**社交媒体舆论模拟系统**。在这个系统中：
- 每个实体都是一个可以在社交媒体上发声、互动、传播信息的"账号"或"主体"
- 实体之间会相互影响、转发、评论、回应
- 我们需要模拟舆论事件中各方的反应和信息传播路径
We are building a **social media public-opinion simulation system**. In this system:
- Each entity is an "account" or "subject" that can speak out, interact, and spread information on social media
- Entities influence each other, repost, comment, and respond
- We need to simulate the reactions of all parties and the information propagation paths in public-opinion events
因此，**实体必须是现实中真实存在的、可以在社媒上发声和互动的主体**：
Therefore, **Entities must be real-world subjects that actually exist and can speak out and interact on social media**:
**可以是**：
- 具体的个人（公众人物、当事人、意见领袖、专家学者、普通人）
- 公司、企业（包括其官方账号）
- 组织机构（大学、协会、NGO、工会等）
- 政府部门、监管机构
- 媒体机构（报纸、电视台、自媒体、网站）
- 社交媒体平台本身
- 特定群体代表（如校友会、粉丝团、维权群体等）
**Can be**:
- Specific individuals (public figures, parties involved, opinion leaders, experts and scholars, ordinary people)
- Companies, enterprises (including their official accounts)
- Organizations (universities, associations, NGOs, unions, etc.)
- Government departments, regulatory agencies
- Media organizations (newspapers, TV stations, self-media, websites)
- Social media platforms themselves
- Representatives of specific groups (e.g., alumni associations, fan groups, rights-advocacy groups, etc.)
**不可以是**：
- 抽象概念（如"舆论"、"情绪"、"趋势"）
- 主题/话题（如"学术诚信"、"教育改革"）
- 观点/态度（如"支持方"、"反对方"）
**Cannot be**:
- Abstract concepts (e.g., "public opinion", "emotion", "trend")
- Themes/topics (e.g., "academic integrity", "education reform")
- Views/attitudes (e.g., "supporters", "opponents")
## 输出格式
## Output Format
请输出JSON格式，包含以下结构：
Please output JSON format containing the following structure:
```json
{
"entity_types": [
{
"name": "实体类型名称（英文，PascalCase）",
"description": "简短描述（英文，不超过100字符）",
"name": "Entity type name (English, PascalCase)",
"description": "Short description (English, no more than 100 characters)",
"attributes": [
{
"name": "属性名（英文，snake_case）",
"name": "Attribute name (English, snake_case)",
"type": "text",
"description": "属性描述"
"description": "Attribute description"
}
],
"examples": ["示例实体1", "示例实体2"]
"examples": ["Example entity 1", "Example entity 2"]
}
],
"edge_types": [
{
"name": "关系类型名称（英文，UPPER_SNAKE_CASE）",
"description": "简短描述（英文，不超过100字符）",
"name": "Relationship type name (English, UPPER_SNAKE_CASE)",
"description": "Short description (English, no more than 100 characters)",
"source_targets": [
{"source": "源实体类型", "target": "目标实体类型"}
{"source": "Source entity type", "target": "Target entity type"}
],
"attributes": []
}
],
"analysis_summary": "对文本内容的简要分析说明"
"analysis_summary": "Brief analysis and description of the text content"
}
```
## 设计指南（极其重要！）
## Design Guidelines (EXTREMELY IMPORTANT!)
### 1. 实体类型设计 - 必须严格遵守
### 1. Entity Type Design - MUST be strictly followed
**数量要求：必须正好10个实体类型**
**Quantity requirement: must be exactly 10 entity types**
**层次结构要求（必须同时包含具体类型和兜底类型）**：
**Hierarchy requirement (must include both specific types and fallback types)**:
你的10个实体类型必须包含以下层次：
Your 10 entity types must include the following levels:
A. **兜底类型（必须包含，放在列表最后2个）**：
- `Person`: 任何自然人个体的兜底类型。当一个人不属于其他更具体的人物类型时，归入此类。
- `Organization`: 任何组织机构的兜底类型。当一个组织不属于其他更具体的组织类型时，归入此类。
A. **Fallback types (must be included, placed last 2 in the list)**:
- `Person`: Fallback type for any individual person. When a person does not fit any other more specific person type, they are classified here.
- `Organization`: Fallback type for any organization. When an organization does not fit any other more specific organization type, it is classified here.
B. **具体类型（8个，根据文本内容设计）**：
- 针对文本中出现的主要角色，设计更具体的类型
- 例如：如果文本涉及学术事件，可以有 `Student`, `Professor`, `University`
- 例如：如果文本涉及商业事件，可以有 `Company`, `CEO`, `Employee`
B. **Specific types (8, designed from the text content)**:
- Design more specific types for the main characters appearing in the text
- For example: if the text involves an academic event, you may have `Student`, `Professor`, `University`
- For example: if the text involves a business event, you may have `Company`, `CEO`, `Employee`
**为什么需要兜底类型**：
- 文本中会出现各种人物，如"中小学教师"、"路人甲"、"某位网友"
- 如果没有专门的类型匹配，他们应该被归入 `Person`
- 同理，小型组织、临时团体等应该归入 `Organization`
**Why fallback types are needed**:
- The text will contain various characters, such as "primary/secondary school teachers", "passerby A", "a certain netizen"
- If no specific type matches them, they should be classified as `Person`
- Similarly, small organizations, temporary groups, etc. should be classified as `Organization`
**具体类型的设计原则**：
- 从文本中识别出高频出现或关键的角色类型
- 每个具体类型应该有明确的边界，避免重叠
- description 必须清晰说明这个类型和兜底类型的区别
**Design principles for specific types**:
- Identify frequently occurring or key character types from the text
- Each specific type should have clear boundaries to avoid overlap
- The description must clearly explain the difference between this type and the fallback types
### 2. 关系类型设计
### 2. Relationship Type Design
- 数量：6-10个
- 关系应该反映社媒互动中的真实联系
- 确保关系的 source_targets 涵盖你定义的实体类型
- Quantity: 6-10
- Relationships should reflect real connections in social-media interactions
- Ensure the source_targets of relationships cover the entity types you defined
### 3. 属性设计
### 3. Attribute Design
- 每个实体类型1-3个关键属性
- **注意**：属性名不能使用 `name`、`uuid`、`group_id`、`created_at`、`summary`（这些是系统保留字）
- 推荐使用：`full_name`, `title`, `role`, `position`, `location`, `description` 等
- 1-3 key attributes per entity type
- **Note**: Attribute names must NOT use `name`, `uuid`, `group_id`, `created_at`, `summary` (these are system reserved words)
- Recommended: `full_name`, `title`, `role`, `position`, `location`, `description`, etc.
## 实体类型参考
## Entity Type Reference
**个人类（具体）**：
- Student: 学生
- Professor: 教授/学者
- Journalist: 记者
- Celebrity: 明星/网红
- Executive: 高管
- Official: 政府官员
- Lawyer: 律师
- Doctor: 医生
**Person types (specific)**:
- Student: Student
- Professor: Professor/Scholar
- Journalist: Journalist
- Celebrity: Celebrity/Internet celebrity
- Executive: Executive
- Official: Government official
- Lawyer: Lawyer
- Doctor: Doctor
**个人类（兜底）**：
- Person: 任何自然人（不属于上述具体类型时使用）
**Person types (fallback)**:
- Person: Any individual (used when not fitting the specific types above)
**组织类（具体）**：
- University: 高校
- Company: 公司企业
- GovernmentAgency: 政府机构
- MediaOutlet: 媒体机构
- Hospital: 医院
- School: 中小学
- NGO: 非政府组织
**Organization types (specific)**:
- University: University/College
- Company: Company/Enterprise
- GovernmentAgency: Government agency
- MediaOutlet: Media organization
- Hospital: Hospital
- School: Primary/Secondary school
- NGO: Non-governmental organization
**组织类（兜底）**：
- Organization: 任何组织机构（不属于上述具体类型时使用）
**Organization types (fallback)**:
- Organization: Any organization (used when not fitting the specific types above)
## 关系类型参考
## Relationship Type Reference
- WORKS_FOR: 工作于
- STUDIES_AT: 就读于
- AFFILIATED_WITH: 隶属于
- REPRESENTS: 代表
- REGULATES: 监管
- REPORTS_ON: 报道
- COMMENTS_ON: 评论
- RESPONDS_TO: 回应
- SUPPORTS: 支持
- OPPOSES: 反对
- COLLABORATES_WITH: 合作
- COMPETES_WITH: 竞争
- WORKS_FOR: Works for
- STUDIES_AT: Studies at
- AFFILIATED_WITH: Affiliated with
- REPRESENTS: Represents
- REGULATES: Regulates
- REPORTS_ON: Reports on
- COMMENTS_ON: Comments on
- RESPONDS_TO: Responds to
- SUPPORTS: Supports
- OPPOSES: Opposes
- COLLABORATES_WITH: Collaborates with
- COMPETES_WITH: Competes with
"""
class OntologyGenerator:
"""
本体生成器
分析文本内容，生成实体和关系类型定义
Ontology generator
Analyze text content and generate entity and relationship type definitions
"""
def __init__(self, llm_client: Optional[LLMClient] = None):
@ -189,17 +189,17 @@ class OntologyGenerator:
additional_context: Optional[str] = None
) -> Dict[str, Any]:
"""
生成本体定义
Generate the ontology definition
Args:
document_texts: 文档文本列表
simulation_requirement: 模拟需求描述
additional_context: 额外上下文
document_texts: List of document texts
simulation_requirement: Simulation requirement description
additional_context: Additional context
Returns:
本体定义（entity_types, edge_types等）
Ontology definition (entity_types, edge_types, etc.)
"""
# 构建用户消息
# Build user message
user_message = self._build_user_message(
document_texts,
simulation_requirement,
@ -213,19 +213,48 @@ class OntologyGenerator:
{"role": "user", "content": user_message}
]
# 调用LLM
result = self.llm_client.chat_json(
messages=messages,
temperature=0.3,
max_tokens=4096
)
# 验证和后处理
# Call the LLM
try:
result = self.llm_client.chat_json(
messages=messages,
temperature=0.3,
max_tokens=16384,
)
except ValueError as json_err:
logger.warning(
f"[ontology] first attempt failed (likely M3 JSON truncation): {json_err}; retrying with compact prompt"
)
# Compact retry: limit the number of entity/edge types, drop attributes
compact_doc = "\n".join(document_texts)[:30000]
compact_messages = [
{
"role": "system",
"content": (
"Output ONLY valid JSON, no markdown, no commentary. "
"Schema: {entity_types:[{name:PascalCase,description:short}],"
"edge_types:[{name:UPPER_SNAKE_CASE,description:short}],"
"analysis_summary:one sentence}. "
"Limit to AT MOST 6 entity types and 6 edge types. "
"Do NOT include attributes arrays."
),
},
{
"role": "user",
"content": f"Text:\\n{compact_doc}\\n\\nRequirement: {simulation_requirement}\\n\\nReturn JSON.",
},
]
result = self.llm_client.chat_json(
messages=compact_messages,
temperature=0.2,
max_tokens=8192,
)
# Validate and post-process
result = self._validate_and_process(result)
return result
# 传给 LLM 的文本最大长度（5万字）
# Max text length to send to the LLM (50,000 chars)
MAX_TEXT_LENGTH_FOR_LLM = 50000
def _build_user_message(
@ -234,50 +263,50 @@ class OntologyGenerator:
simulation_requirement: str,
additional_context: Optional[str]
) -> str:
"""构建用户消息"""
"""Build user message"""
# 合并文本
# Merge text
combined_text = "\n\n---\n\n".join(document_texts)
original_length = len(combined_text)
# 如果文本超过5万字，截断（仅影响传给LLM的内容，不影响图谱构建）
# If text exceeds 50,000 chars, truncate (only affects what is sent to the LLM, not the graph construction)
if len(combined_text) > self.MAX_TEXT_LENGTH_FOR_LLM:
combined_text = combined_text[:self.MAX_TEXT_LENGTH_FOR_LLM]
combined_text += f"\n\n...(原文共{original_length}字，已截取前{self.MAX_TEXT_LENGTH_FOR_LLM}字用于本体分析)..."
combined_text += f"\n\n...(original text is {original_length} chars, truncated to the first {self.MAX_TEXT_LENGTH_FOR_LLM} chars for ontology analysis)..."
message = f"""## 模拟需求
message = f"""## Simulation requirements
{simulation_requirement}
## 文档内容
## Document content
{combined_text}
"""
if additional_context:
message += f"""
## 额外说明
## Additional notes
{additional_context}
"""
message += """
请根据以上内容，设计适合社会舆论模拟的实体类型和关系类型。
Please design entity types and edge types suitable for social opinion simulation based on the content above.
**必须遵守的规则**：
1. 必须正好输出10个实体类型
2. 最后2个必须是兜底类型：Person（个人兜底）和 Organization（组织兜底）
3. 前8个是根据文本内容设计的具体类型
4. 所有实体类型必须是现实中可以发声的主体，不能是抽象概念
5. 属性名不能使用 name、uuid、group_id 等保留字，用 full_name、org_name 等替代
**Rules that must be followed**:
1. Output exactly 10 entity types
2. The last 2 must be fallback types: Person (individual fallback) and Organization (organization fallback)
3. The first 8 should be specific types designed from the text content
4. All entity types must be real-world entities capable of speaking out, not abstract concepts
5. Attribute names must not use reserved words like name, uuid, group_id; use full_name, org_name, etc. instead
"""
return message
def _validate_and_process(self, result: Dict[str, Any]) -> Dict[str, Any]:
"""验证和后处理结果"""
"""Validate and post-process the result"""
# 确保必要字段存在
# Ensure required fields exist
if "entity_types" not in result:
result["entity_types"] = []
if "edge_types" not in result:
@ -285,11 +314,11 @@ class OntologyGenerator:
if "analysis_summary" not in result:
result["analysis_summary"] = ""
# 验证实体类型
# 记录原始名称到 PascalCase 的映射，用于后续修正 edge 的 source_targets 引用
# Validate entity types
# Record the raw-name -> PascalCase mapping, used later to fix edge source_targets references
entity_name_map = {}
for entity in result["entity_types"]:
# 强制将 entity name 转为 PascalCase（Zep API 要求）
# Force-convert entity name to PascalCase (Zep API requirement)
if "name" in entity:
original_name = entity["name"]
entity["name"] = _to_pascal_case(original_name)
@ -300,19 +329,19 @@ class OntologyGenerator:
entity["attributes"] = []
if "examples" not in entity:
entity["examples"] = []
# 确保description不超过100字符
# Ensure description does not exceed 100 characters
if len(entity.get("description", "")) > 100:
entity["description"] = entity["description"][:97] + "..."
# 验证关系类型
# Validate edge types
for edge in result["edge_types"]:
# 强制将 edge name 转为 SCREAMING_SNAKE_CASE（Zep API 要求）
# Force-convert edge name to SCREAMING_SNAKE_CASE (Zep API requirement)
if "name" in edge:
original_name = edge["name"]
edge["name"] = original_name.upper()
if edge["name"] != original_name:
logger.warning(f"Edge type name '{original_name}' auto-converted to '{edge['name']}'")
# 修正 source_targets 中的实体名称引用，与转换后的 PascalCase 保持一致
# Fix entity name references in source_targets to match the converted PascalCase
for st in edge.get("source_targets", []):
if st.get("source") in entity_name_map:
st["source"] = entity_name_map[st["source"]]
@ -325,11 +354,11 @@ class OntologyGenerator:
if len(edge.get("description", "")) > 100:
edge["description"] = edge["description"][:97] + "..."
# Zep API 限制：最多 10 个自定义实体类型，最多 10 个自定义边类型
# Zep API limit: at most 10 custom entity types, at most 10 custom edge types
MAX_ENTITY_TYPES = 10
MAX_EDGE_TYPES = 10
# 去重：按 name 去重，保留首次出现的
# Deduplicate by name, keep the first occurrence
seen_names = set()
deduped = []
for entity in result["entity_types"]:
@ -341,7 +370,7 @@ class OntologyGenerator:
logger.warning(f"Duplicate entity type '{name}' removed during validation")
result["entity_types"] = deduped
# 兜底类型定义
# Fallback type definitions
person_fallback = {
"name": "Person",
"description": "Any individual person not fitting other specific person types.",
@ -362,12 +391,12 @@ class OntologyGenerator:
"examples": ["small business", "community group"]
}
# 检查是否已有兜底类型
# Check whether fallback types already exist
entity_names = {e["name"] for e in result["entity_types"]}
has_person = "Person" in entity_names
has_organization = "Organization" in entity_names
# 需要添加的兜底类型
# Fallback types that need to be added
fallbacks_to_add = []
if not has_person:
fallbacks_to_add.append(person_fallback)
@ -378,17 +407,17 @@ class OntologyGenerator:
current_count = len(result["entity_types"])
needed_slots = len(fallbacks_to_add)
# 如果添加后会超过 10 个，需要移除一些现有类型
# If adding would exceed 10, we need to drop some existing types
if current_count + needed_slots > MAX_ENTITY_TYPES:
# 计算需要移除多少个
# Compute how many to remove
to_remove = current_count + needed_slots - MAX_ENTITY_TYPES
# 从末尾移除（保留前面更重要的具体类型）
# Remove from the end (keep the more important specific types at the front)
result["entity_types"] = result["entity_types"][:-to_remove]
# 添加兜底类型
# Add fallback types
result["entity_types"].extend(fallbacks_to_add)
# 最终确保不超过限制（防御性编程）
# Final guard to ensure the limit is not exceeded (defensive coding)
if len(result["entity_types"]) > MAX_ENTITY_TYPES:
result["entity_types"] = result["entity_types"][:MAX_ENTITY_TYPES]
@ -399,29 +428,29 @@ class OntologyGenerator:
def generate_python_code(self, ontology: Dict[str, Any]) -> str:
"""
将本体定义转换为Python代码（类似ontology.py）
Convert the ontology definition to Python code (similar to ontology.py)
Args:
ontology: 本体定义
ontology: Ontology definition
Returns:
Python代码字符串
Python code string
"""
code_lines = [
'"""',
'自定义实体类型定义',
'由MiroFish自动生成，用于社会舆论模拟',
'Custom entity type definitions',
'Auto-generated by MiroFish, for social opinion simulation',
'"""',
'',
'from pydantic import Field',
'from zep_cloud.external_clients.ontology import EntityModel, EntityText, EdgeModel',
'',
'',
'# ============== 实体类型定义 ==============',
'# ============== Entity type definitions ==============',
'',
]
# 生成实体类型
# Generate entity types
for entity in ontology.get("entity_types", []):
name = entity["name"]
desc = entity.get("description", f"A {name} entity.")
@ -444,13 +473,13 @@ class OntologyGenerator:
code_lines.append('')
code_lines.append('')
code_lines.append('# ============== 关系类型定义 ==============')
code_lines.append('# ============== Edge type definitions ==============')
code_lines.append('')
# 生成关系类型
# Generate edge types
for edge in ontology.get("edge_types", []):
name = edge["name"]
# 转换为PascalCase类名
# Convert to PascalCase class name
class_name = ''.join(word.capitalize() for word in name.split('_'))
desc = edge.get("description", f"A {name} relationship.")
@ -472,8 +501,8 @@ class OntologyGenerator:
code_lines.append('')
code_lines.append('')
# 生成类型字典
code_lines.append('# ============== 类型配置 ==============')
# Generate type dictionary
code_lines.append('# ============== Type configuration ==============')
code_lines.append('')
code_lines.append('ENTITY_TYPES = {')
for entity in ontology.get("entity_types", []):
@ -489,7 +518,7 @@ class OntologyGenerator:
code_lines.append('}')
code_lines.append('')
# 生成边的source_targets映射
# Generate edge source_targets mapping
code_lines.append('EDGE_SOURCE_TARGETS = {')
for edge in ontology.get("edge_types", []):
name = edge["name"]

1504
backend/app/services/report_agent.py
View File

File diff suppressed because it is too large Load Diff

497
backend/app/services/simulation_config_generator.py
View File

File diff suppressed because it is too large Load Diff

262
backend/app/services/simulation_ipc.py
View File

@ -1,11 +1,13 @@
"""
模拟IPC通信模块
用于Flask后端和模拟脚本之间的进程间通信
Simulation IPC communication module
Used for inter-process communication between the Flask backend
and the simulation script.
通过文件系统实现简单的命令/响应模式：
1. Flask写入命令到 commands/ 目录
2. 模拟脚本轮询命令目录，执行命令并写入响应到 responses/ 目录
3. Flask轮询响应目录获取结果
Implements a simple command/response pattern via the filesystem:
1. Flask writes commands into the commands/ directory
2. The simulation script polls the commands directory, executes the
commands, and writes responses into the responses/ directory
3. Flask polls the responses directory to fetch results
"""
import os
@ -23,14 +25,14 @@ logger = get_logger('mirofish.simulation_ipc')
class CommandType(str, Enum):
"""命令类型"""
INTERVIEW = "interview" # 单个Agent采访
BATCH_INTERVIEW = "batch_interview" # 批量采访
CLOSE_ENV = "close_env" # 关闭环境
"""Command type"""
INTERVIEW = "interview" # Single-agent interview
BATCH_INTERVIEW = "batch_interview" # Batch interview
CLOSE_ENV = "close_env" # Close environment
class CommandStatus(str, Enum):
"""命令状态"""
"""Command status"""
PENDING = "pending"
PROCESSING = "processing"
COMPLETED = "completed"
@ -39,12 +41,12 @@ class CommandStatus(str, Enum):
@dataclass
class IPCCommand:
"""IPC命令"""
"""IPC command"""
command_id: str
command_type: CommandType
args: Dict[str, Any]
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
def to_dict(self) -> Dict[str, Any]:
return {
"command_id": self.command_id,
@ -52,7 +54,7 @@ class IPCCommand:
"args": self.args,
"timestamp": self.timestamp
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> 'IPCCommand':
return cls(
@ -65,13 +67,13 @@ class IPCCommand:
@dataclass
class IPCResponse:
"""IPC响应"""
"""IPC response"""
command_id: str
status: CommandStatus
result: Optional[Dict[str, Any]] = None
error: Optional[str] = None
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
def to_dict(self) -> Dict[str, Any]:
return {
"command_id": self.command_id,
@ -80,7 +82,7 @@ class IPCResponse:
"error": self.error,
"timestamp": self.timestamp
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> 'IPCResponse':
return cls(
@ -94,26 +96,26 @@ class IPCResponse:
class SimulationIPCClient:
"""
模拟IPC客户端（Flask端使用）
用于向模拟进程发送命令并等待响应
Simulation IPC client (used on the Flask side)
Sends commands to the simulation process and waits for responses.
"""
def __init__(self, simulation_dir: str):
"""
初始化IPC客户端
Initialize the IPC client.
Args:
simulation_dir: 模拟数据目录
simulation_dir: Simulation data directory
"""
self.simulation_dir = simulation_dir
self.commands_dir = os.path.join(simulation_dir, "ipc_commands")
self.responses_dir = os.path.join(simulation_dir, "ipc_responses")
# 确保目录存在
# Ensure directories exist
os.makedirs(self.commands_dir, exist_ok=True)
os.makedirs(self.responses_dir, exist_ok=True)
def send_command(
self,
command_type: CommandType,
@ -122,19 +124,19 @@ class SimulationIPCClient:
poll_interval: float = 0.5
) -> IPCResponse:
"""
发送命令并等待响应
Send a command and wait for a response.
Args:
command_type: 命令类型
args: 命令参数
timeout: 超时时间（秒）
poll_interval: 轮询间隔（秒）
command_type: Command type
args: Command arguments
timeout: Timeout in seconds
poll_interval: Poll interval in seconds
Returns:
IPCResponse
Raises:
TimeoutError: 等待响应超时
TimeoutError: Timed out waiting for a response
"""
command_id = str(uuid.uuid4())
command = IPCCommand(
@ -142,50 +144,50 @@ class SimulationIPCClient:
command_type=command_type,
args=args
)
# 写入命令文件
# Write the command file
command_file = os.path.join(self.commands_dir, f"{command_id}.json")
with open(command_file, 'w', encoding='utf-8') as f:
json.dump(command.to_dict(), f, ensure_ascii=False, indent=2)
logger.info(f"发送IPC命令: {command_type.value}, command_id={command_id}")
# 等待响应
logger.info(f"Sent IPC command: {command_type.value}, command_id={command_id}")
# Wait for the response
response_file = os.path.join(self.responses_dir, f"{command_id}.json")
start_time = time.time()
while time.time() - start_time < timeout:
if os.path.exists(response_file):
try:
with open(response_file, 'r', encoding='utf-8') as f:
response_data = json.load(f)
response = IPCResponse.from_dict(response_data)
# 清理命令和响应文件
# Clean up command and response files
try:
os.remove(command_file)
os.remove(response_file)
except OSError:
pass
logger.info(f"收到IPC响应: command_id={command_id}, status={response.status.value}")
logger.info(f"Received IPC response: command_id={command_id}, status={response.status.value}")
return response
except (json.JSONDecodeError, KeyError) as e:
logger.warning(f"解析响应失败: {e}")
logger.warning(f"Failed to parse response: {e}")
time.sleep(poll_interval)
# 超时
logger.error(f"等待IPC响应超时: command_id={command_id}")
# 清理命令文件
# Timeout
logger.error(f"Timeout waiting for IPC response: command_id={command_id}")
# Clean up the command file
try:
os.remove(command_file)
except OSError:
pass
raise TimeoutError(f"等待命令响应超时 ({timeout}秒)")
raise TimeoutError(f"Timed out waiting for command response ({timeout}s)")
def send_interview(
self,
agent_id: int,
@ -194,19 +196,21 @@ class SimulationIPCClient:
timeout: float = 60.0
) -> IPCResponse:
"""
发送单个Agent采访命令
Send a single-Agent interview command.
Args:
agent_id: Agent ID
prompt: 采访问题
platform: 指定平台（可选）
- "twitter": 只采访Twitter平台
- "reddit": 只采访Reddit平台
- None: 双平台模拟时同时采访两个平台，单平台模拟时采访该平台
timeout: 超时时间
prompt: Interview question
platform: Specific platform (optional)
- "twitter": only interview the Twitter platform
- "reddit": only interview the Reddit platform
- None: in a dual-platform simulation interview both
platforms; in a single-platform simulation interview
that platform
timeout: Timeout
Returns:
IPCResponse，result字段包含采访结果
IPCResponse; the result field contains the interview result
"""
args = {
"agent_id": agent_id,
@ -214,13 +218,13 @@ class SimulationIPCClient:
}
if platform:
args["platform"] = platform
return self.send_command(
command_type=CommandType.INTERVIEW,
args=args,
timeout=timeout
)
def send_batch_interview(
self,
interviews: List[Dict[str, Any]],
@ -228,36 +232,39 @@ class SimulationIPCClient:
timeout: float = 120.0
) -> IPCResponse:
"""
发送批量采访命令
Send a batch interview command.
Args:
interviews: 采访列表，每个元素包含 {"agent_id": int, "prompt": str, "platform": str(可选)}
platform: 默认平台（可选，会被每个采访项的platform覆盖）
- "twitter": 默认只采访Twitter平台
- "reddit": 默认只采访Reddit平台
- None: 双平台模拟时每个Agent同时采访两个平台
timeout: 超时时间
interviews: List of interviews; each item is
{"agent_id": int, "prompt": str, "platform": str (optional)}
platform: Default platform (optional; overridden by each
interview item's platform)
- "twitter": default to interviewing only Twitter
- "reddit": default to interviewing only Reddit
- None: in a dual-platform simulation interview each
Agent on both platforms
timeout: Timeout
Returns:
IPCResponse，result字段包含所有采访结果
IPCResponse; the result field contains all interview results
"""
args = {"interviews": interviews}
if platform:
args["platform"] = platform
return self.send_command(
command_type=CommandType.BATCH_INTERVIEW,
args=args,
timeout=timeout
)
def send_close_env(self, timeout: float = 30.0) -> IPCResponse:
"""
发送关闭环境命令
Send a close-environment command.
Args:
timeout: 超时时间
timeout: Timeout
Returns:
IPCResponse
"""
@ -266,17 +273,17 @@ class SimulationIPCClient:
args={},
timeout=timeout
)
def check_env_alive(self) -> bool:
"""
检查模拟环境是否存活
通过检查 env_status.json 文件来判断
Check whether the simulation environment is alive.
Decides by checking the env_status.json file.
"""
status_file = os.path.join(self.simulation_dir, "env_status.json")
if not os.path.exists(status_file):
return False
try:
with open(status_file, 'r', encoding='utf-8') as f:
status = json.load(f)
@ -287,106 +294,107 @@ class SimulationIPCClient:
class SimulationIPCServer:
"""
模拟IPC服务器（模拟脚本端使用）
轮询命令目录，执行命令并返回响应
Simulation IPC server (used on the simulation-script side)
Polls the commands directory, executes commands, and returns
responses.
"""
def __init__(self, simulation_dir: str):
"""
初始化IPC服务器
Initialize the IPC server.
Args:
simulation_dir: 模拟数据目录
simulation_dir: Simulation data directory
"""
self.simulation_dir = simulation_dir
self.commands_dir = os.path.join(simulation_dir, "ipc_commands")
self.responses_dir = os.path.join(simulation_dir, "ipc_responses")
# 确保目录存在
# Ensure directories exist
os.makedirs(self.commands_dir, exist_ok=True)
os.makedirs(self.responses_dir, exist_ok=True)
# 环境状态
# Environment status
self._running = False
def start(self):
"""标记服务器为运行状态"""
"""Mark the server as running"""
self._running = True
self._update_env_status("alive")
def stop(self):
"""标记服务器为停止状态"""
"""Mark the server as stopped"""
self._running = False
self._update_env_status("stopped")
def _update_env_status(self, status: str):
"""更新环境状态文件"""
"""Update the environment-status file"""
status_file = os.path.join(self.simulation_dir, "env_status.json")
with open(status_file, 'w', encoding='utf-8') as f:
json.dump({
"status": status,
"timestamp": datetime.now().isoformat()
}, f, ensure_ascii=False, indent=2)
def poll_commands(self) -> Optional[IPCCommand]:
"""
轮询命令目录，返回第一个待处理的命令
Poll the commands directory; return the first pending command.
Returns:
IPCCommand 或 None
IPCCommand or None
"""
if not os.path.exists(self.commands_dir):
return None
# 按时间排序获取命令文件
# Sort command files by modification time
command_files = []
for filename in os.listdir(self.commands_dir):
if filename.endswith('.json'):
filepath = os.path.join(self.commands_dir, filename)
command_files.append((filepath, os.path.getmtime(filepath)))
command_files.sort(key=lambda x: x[1])
for filepath, _ in command_files:
try:
with open(filepath, 'r', encoding='utf-8') as f:
data = json.load(f)
return IPCCommand.from_dict(data)
except (json.JSONDecodeError, KeyError, OSError) as e:
logger.warning(f"读取命令文件失败: {filepath}, {e}")
logger.warning(f"Failed to read command file: {filepath}, {e}")
continue
return None
def send_response(self, response: IPCResponse):
"""
发送响应
Send a response.
Args:
response: IPC响应
response: IPC response
"""
response_file = os.path.join(self.responses_dir, f"{response.command_id}.json")
with open(response_file, 'w', encoding='utf-8') as f:
json.dump(response.to_dict(), f, ensure_ascii=False, indent=2)
# 删除命令文件
# Delete the command file
command_file = os.path.join(self.commands_dir, f"{response.command_id}.json")
try:
os.remove(command_file)
except OSError:
pass
def send_success(self, command_id: str, result: Dict[str, Any]):
"""发送成功响应"""
"""Send a success response"""
self.send_response(IPCResponse(
command_id=command_id,
status=CommandStatus.COMPLETED,
result=result
))
def send_error(self, command_id: str, error: str):
"""发送错误响应"""
"""Send an error response"""
self.send_response(IPCResponse(
command_id=command_id,
status=CommandStatus.FAILED,

291
backend/app/services/simulation_manager.py
View File

@ -1,7 +1,7 @@
"""
OASIS模拟管理器
管理Twitter和Reddit双平台并行模拟
使用预设脚本 + LLM智能生成配置参数
OASIS Simulation Manager
Manages parallel Twitter and Reddit dual-platform simulations
Uses preset scripts + LLM-intelligent configuration parameter generation
"""
import os
@ -23,60 +23,60 @@ logger = get_logger('mirofish.simulation')
class SimulationStatus(str, Enum):
"""模拟状态"""
"""Simulation status"""
CREATED = "created"
PREPARING = "preparing"
READY = "ready"
RUNNING = "running"
PAUSED = "paused"
STOPPED = "stopped" # 模拟被手动停止
COMPLETED = "completed" # 模拟自然完成
STOPPED = "stopped" # Simulation was manually stopped
COMPLETED = "completed" # Simulation completed naturally
FAILED = "failed"
class PlatformType(str, Enum):
"""平台类型"""
"""Platform type"""
TWITTER = "twitter"
REDDIT = "reddit"
@dataclass
class SimulationState:
"""模拟状态"""
"""Simulation state"""
simulation_id: str
project_id: str
graph_id: str
# 平台启用状态
# Platform enable status
enable_twitter: bool = True
enable_reddit: bool = True
# 状态
# Status
status: SimulationStatus = SimulationStatus.CREATED
# 准备阶段数据
# Preparation stage data
entities_count: int = 0
profiles_count: int = 0
entity_types: List[str] = field(default_factory=list)
# 配置生成信息
# Config generation info
config_generated: bool = False
config_reasoning: str = ""
# 运行时数据
# Runtime data
current_round: int = 0
twitter_status: str = "not_started"
reddit_status: str = "not_started"
# 时间戳
# Timestamps
created_at: str = field(default_factory=lambda: datetime.now().isoformat())
updated_at: str = field(default_factory=lambda: datetime.now().isoformat())
# 错误信息
# Error info
error: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
"""完整状态字典（内部使用）"""
"""Full state dictionary (for internal use)"""
return {
"simulation_id": self.simulation_id,
"project_id": self.project_id,
@ -98,7 +98,7 @@ class SimulationState:
}
def to_simple_dict(self) -> Dict[str, Any]:
"""简化状态字典（API返回使用）"""
"""Simplified state dictionary (for API response)"""
return {
"simulation_id": self.simulation_id,
"project_id": self.project_id,
@ -114,36 +114,36 @@ class SimulationState:
class SimulationManager:
"""
模拟管理器
核心功能：
1. 从Zep图谱读取实体并过滤
2. 生成OASIS Agent Profile
3. 使用LLM智能生成模拟配置参数
4. 准备预设脚本所需的所有文件
Simulation Manager
Core features:
1. Read entities from Zep graph and filter
2. Generate OASIS Agent Profile
3. Use LLM to intelligently generate simulation configuration parameters
4. Prepare all files required by the preset scripts
"""
# 模拟数据存储目录
# Simulation data storage directory
SIMULATION_DATA_DIR = os.path.join(
os.path.dirname(__file__),
os.path.dirname(__file__),
'../../uploads/simulations'
)
def __init__(self):
# 确保目录存在
# Ensure the directory exists
os.makedirs(self.SIMULATION_DATA_DIR, exist_ok=True)
# 内存中的模拟状态缓存
# In-memory simulation state cache
self._simulations: Dict[str, SimulationState] = {}
def _get_simulation_dir(self, simulation_id: str) -> str:
"""获取模拟数据目录"""
"""Get the simulation data directory"""
sim_dir = os.path.join(self.SIMULATION_DATA_DIR, simulation_id)
os.makedirs(sim_dir, exist_ok=True)
return sim_dir
def _save_simulation_state(self, state: SimulationState):
"""保存模拟状态到文件"""
"""Save the simulation state to file"""
sim_dir = self._get_simulation_dir(state.simulation_id)
state_file = os.path.join(sim_dir, "state.json")
@ -155,7 +155,7 @@ class SimulationManager:
self._simulations[state.simulation_id] = state
def _load_simulation_state(self, simulation_id: str) -> Optional[SimulationState]:
"""从文件加载模拟状态"""
"""Load the simulation state from file"""
if simulation_id in self._simulations:
return self._simulations[simulation_id]
@ -199,20 +199,20 @@ class SimulationManager:
enable_reddit: bool = True,
) -> SimulationState:
"""
创建新的模拟
Create a new simulation
Args:
project_id: 项目ID
graph_id: Zep图谱ID
enable_twitter: 是否启用Twitter模拟
enable_reddit: 是否启用Reddit模拟
project_id: Project ID
graph_id: Zep graph ID
enable_twitter: Whether to enable Twitter simulation
enable_reddit: Whether to enable Reddit simulation
Returns:
SimulationState
"""
import uuid
simulation_id = f"sim_{uuid.uuid4().hex[:12]}"
state = SimulationState(
simulation_id=simulation_id,
project_id=project_id,
@ -221,9 +221,9 @@ class SimulationManager:
enable_reddit=enable_reddit,
status=SimulationStatus.CREATED,
)
self._save_simulation_state(state)
logger.info(f"创建模拟: {simulation_id}, project={project_id}, graph={graph_id}")
logger.info(f"Created simulation: {simulation_id}, project={project_id}, graph={graph_id}")
return state
@ -238,30 +238,30 @@ class SimulationManager:
parallel_profile_count: int = 3
) -> SimulationState:
"""
准备模拟环境（全程自动化）
步骤：
1. 从Zep图谱读取并过滤实体
2. 为每个实体生成OASIS Agent Profile（可选LLM增强，支持并行）
3. 使用LLM智能生成模拟配置参数（时间、活跃度、发言频率等）
4. 保存配置文件和Profile文件
5. 复制预设脚本到模拟目录
Prepare the simulation environment (fully automated)
Steps:
1. Read and filter entities from the Zep graph
2. Generate OASIS Agent Profile for each entity (optional LLM enhancement, supports parallelism)
3. Use LLM to intelligently generate simulation configuration parameters (time, activity, posting frequency, etc.)
4. Save config and profile files
5. Copy preset scripts to the simulation directory
Args:
simulation_id: 模拟ID
simulation_requirement: 模拟需求描述（用于LLM生成配置）
document_text: 原始文档内容（用于LLM理解背景）
defined_entity_types: 预定义的实体类型（可选）
use_llm_for_profiles: 是否使用LLM生成详细人设
progress_callback: 进度回调函数 (stage, progress, message)
parallel_profile_count: 并行生成人设的数量，默认3
simulation_id: Simulation ID
simulation_requirement: Simulation requirement description (used for LLM config generation)
document_text: Original document content (used for LLM to understand context)
defined_entity_types: Predefined entity types (optional)
use_llm_for_profiles: Whether to use LLM to generate detailed personas
progress_callback: Progress callback function (stage, progress, message)
parallel_profile_count: Number of personas to generate in parallel, default 3
Returns:
SimulationState
"""
state = self._load_simulation_state(simulation_id)
if not state:
raise ValueError(f"模拟不存在: {simulation_id}")
raise ValueError(f"Simulation not found: {simulation_id}")
try:
state.status = SimulationStatus.PREPARING
@ -269,24 +269,24 @@ class SimulationManager:
sim_dir = self._get_simulation_dir(simulation_id)
# ========== 阶段1: 读取并过滤实体 ==========
# ========== Stage 1: Read and filter entities ==========
if progress_callback:
progress_callback("reading", 0, t('progress.connectingZepGraph'))
reader = ZepEntityReader()
if progress_callback:
progress_callback("reading", 30, t('progress.readingNodeData'))
filtered = reader.filter_defined_entities(
graph_id=state.graph_id,
defined_entity_types=defined_entity_types,
enrich_with_edges=True
)
state.entities_count = filtered.filtered_count
state.entity_types = list(filtered.entity_types)
if progress_callback:
progress_callback(
"reading", 100,
@ -294,16 +294,16 @@ class SimulationManager:
current=filtered.filtered_count,
total=filtered.filtered_count
)
if filtered.filtered_count == 0:
state.status = SimulationStatus.FAILED
state.error = "没有找到符合条件的实体，请检查图谱是否正确构建"
state.error = "No entities matching the criteria were found. Please check whether the graph was built correctly."
self._save_simulation_state(state)
return state
# ========== 阶段2: 生成Agent Profile ==========
# ========== Stage 2: Generate Agent Profile ==========
total_entities = len(filtered.entities)
if progress_callback:
progress_callback(
"generating_profiles", 0,
@ -311,22 +311,22 @@ class SimulationManager:
current=0,
total=total_entities
)
# 传入graph_id以启用Zep检索功能，获取更丰富的上下文
# Pass graph_id to enable Zep retrieval for richer context
generator = OasisProfileGenerator(graph_id=state.graph_id)
def profile_progress(current, total, msg):
if progress_callback:
progress_callback(
"generating_profiles",
int(current / total * 100),
"generating_profiles",
int(current / total * 100),
msg,
current=current,
total=total,
item_name=msg
)
# 设置实时保存的文件路径（优先使用 Reddit JSON 格式）
# Configure real-time save path (prefer Reddit JSON format)
realtime_output_path = None
realtime_platform = "reddit"
if state.enable_reddit:
@ -335,21 +335,21 @@ class SimulationManager:
elif state.enable_twitter:
realtime_output_path = os.path.join(sim_dir, "twitter_profiles.csv")
realtime_platform = "twitter"
profiles = generator.generate_profiles_from_entities(
entities=filtered.entities,
use_llm=use_llm_for_profiles,
progress_callback=profile_progress,
graph_id=state.graph_id, # 传入graph_id用于Zep检索
parallel_count=parallel_profile_count, # 并行生成数量
realtime_output_path=realtime_output_path, # 实时保存路径
output_platform=realtime_platform # 输出格式
graph_id=state.graph_id, # Pass graph_id for Zep retrieval
parallel_count=parallel_profile_count, # Parallel generation count
realtime_output_path=realtime_output_path, # Real-time save path
output_platform=realtime_platform # Output format
)
state.profiles_count = len(profiles)
# 保存Profile文件（注意：Twitter使用CSV格式，Reddit使用JSON格式）
# Reddit 已经在生成过程中实时保存了，这里再保存一次确保完整性
# Save profile files (note: Twitter uses CSV, Reddit uses JSON)
# Reddit has already been saved in real time during generation; saving once more here ensures completeness
if progress_callback:
progress_callback(
"generating_profiles", 95,
@ -357,16 +357,16 @@ class SimulationManager:
current=total_entities,
total=total_entities
)
if state.enable_reddit:
generator.save_profiles(
profiles=profiles,
file_path=os.path.join(sim_dir, "reddit_profiles.json"),
platform="reddit"
)
if state.enable_twitter:
# Twitter使用CSV格式！这是OASIS的要求
# Twitter uses CSV format! This is required by OASIS
generator.save_profiles(
profiles=profiles,
file_path=os.path.join(sim_dir, "twitter_profiles.csv"),
@ -381,7 +381,7 @@ class SimulationManager:
total=len(profiles)
)
# ========== 阶段3: LLM智能生成模拟配置 ==========
# ========== Stage 3: LLM-intelligent generation of simulation config ==========
if progress_callback:
progress_callback(
"generating_config", 0,
@ -389,9 +389,9 @@ class SimulationManager:
current=0,
total=3
)
config_generator = SimulationConfigGenerator()
if progress_callback:
progress_callback(
"generating_config", 30,
@ -399,7 +399,7 @@ class SimulationManager:
current=1,
total=3
)
sim_params = config_generator.generate_config(
simulation_id=simulation_id,
project_id=state.project_id,
@ -410,7 +410,7 @@ class SimulationManager:
enable_twitter=state.enable_twitter,
enable_reddit=state.enable_reddit
)
if progress_callback:
progress_callback(
"generating_config", 70,
@ -418,15 +418,15 @@ class SimulationManager:
current=2,
total=3
)
# 保存配置文件
# Save config file
config_path = os.path.join(sim_dir, "simulation_config.json")
with open(config_path, 'w', encoding='utf-8') as f:
f.write(sim_params.to_json())
state.config_generated = True
state.config_reasoning = sim_params.generation_reasoning
if progress_callback:
progress_callback(
"generating_config", 100,
@ -434,82 +434,83 @@ class SimulationManager:
current=3,
total=3
)
# 注意：运行脚本保留在 backend/scripts/ 目录，不再复制到模拟目录
# 启动模拟时，simulation_runner 会从 scripts/ 目录运行脚本
# 更新状态
# Note: run scripts are kept in backend/scripts/ and are no longer copied into the simulation dir
# When starting a simulation, simulation_runner runs scripts from the scripts/ directory
# Update state
state.status = SimulationStatus.READY
self._save_simulation_state(state)
logger.info(f"模拟准备完成: {simulation_id}, "
logger.info(f"Simulation preparation complete: {simulation_id}, "
f"entities={state.entities_count}, profiles={state.profiles_count}")
return state
except Exception as e:
logger.error(f"模拟准备失败: {simulation_id}, error={str(e)}")
logger.error(f"Simulation preparation failed: {simulation_id}, error={str(e)}")
import traceback
logger.error(traceback.format_exc())
state.status = SimulationStatus.FAILED
state.error = str(e)
self._save_simulation_state(state)
raise
def get_simulation(self, simulation_id: str) -> Optional[SimulationState]:
"""获取模拟状态"""
"""Get the simulation state"""
return self._load_simulation_state(simulation_id)
def list_simulations(self, project_id: Optional[str] = None) -> List[SimulationState]:
"""列出所有模拟"""
"""List all simulations"""
simulations = []
if os.path.exists(self.SIMULATION_DATA_DIR):
for sim_id in os.listdir(self.SIMULATION_DATA_DIR):
# 跳过隐藏文件（如 .DS_Store）和非目录文件
# Skip hidden files (e.g. .DS_Store) and non-directory entries
sim_path = os.path.join(self.SIMULATION_DATA_DIR, sim_id)
if sim_id.startswith('.') or not os.path.isdir(sim_path):
continue
state = self._load_simulation_state(sim_id)
if state:
if project_id is None or state.project_id == project_id:
simulations.append(state)
return simulations
def get_profiles(self, simulation_id: str, platform: str = "reddit") -> List[Dict[str, Any]]:
"""获取模拟的Agent Profile"""
"""Get the Agent Profiles of a simulation"""
state = self._load_simulation_state(simulation_id)
if not state:
raise ValueError(f"模拟不存在: {simulation_id}")
raise ValueError(f"Simulation not found: {simulation_id}")
sim_dir = self._get_simulation_dir(simulation_id)
profile_path = os.path.join(sim_dir, f"{platform}_profiles.json")
if not os.path.exists(profile_path):
return []
with open(profile_path, 'r', encoding='utf-8') as f:
return json.load(f)
def get_simulation_config(self, simulation_id: str) -> Optional[Dict[str, Any]]:
"""获取模拟配置"""
"""Get the simulation config"""
sim_dir = self._get_simulation_dir(simulation_id)
config_path = os.path.join(sim_dir, "simulation_config.json")
if not os.path.exists(config_path):
return None
with open(config_path, 'r', encoding='utf-8') as f:
return json.load(f)
def get_run_instructions(self, simulation_id: str) -> Dict[str, str]:
"""获取运行说明"""
"""Get the run instructions"""
sim_dir = self._get_simulation_dir(simulation_id)
config_path = os.path.join(sim_dir, "simulation_config.json")
scripts_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../scripts'))
return {
"simulation_dir": sim_dir,
"scripts_dir": scripts_dir,
@ -520,10 +521,10 @@ class SimulationManager:
"parallel": f"python {scripts_dir}/run_parallel_simulation.py --config {config_path}",
},
"instructions": (
f"1. 激活conda环境: conda activate MiroFish\n"
f"2. 运行模拟 (脚本位于 {scripts_dir}):\n"
f" - 单独运行Twitter: python {scripts_dir}/run_twitter_simulation.py --config {config_path}\n"
f" - 单独运行Reddit: python {scripts_dir}/run_reddit_simulation.py --config {config_path}\n"
f" - 并行运行双平台: python {scripts_dir}/run_parallel_simulation.py --config {config_path}"
f"1. Activate the conda environment: conda activate MiroFish\n"
f"2. Run the simulation (scripts are at {scripts_dir}):\n"
f" - Twitter only: python {scripts_dir}/run_twitter_simulation.py --config {config_path}\n"
f" - Reddit only: python {scripts_dir}/run_reddit_simulation.py --config {config_path}\n"
f" - Both platforms in parallel: python {scripts_dir}/run_parallel_simulation.py --config {config_path}"
)
}

696
backend/app/services/simulation_runner.py
View File

File diff suppressed because it is too large Load Diff

34
backend/app/services/text_processor.py
View File

@ -1,5 +1,5 @@
"""
文本处理服务
Text processing service
"""
from typing import List, Optional
@ -7,11 +7,11 @@ from ..utils.file_parser import FileParser, split_text_into_chunks
class TextProcessor:
"""文本处理器"""
"""Text processor"""
@staticmethod
def extract_from_files(file_paths: List[str]) -> str:
"""从多个文件提取文本"""
"""Extract text from multiple files"""
return FileParser.extract_from_multiple(file_paths)
@staticmethod
@ -21,40 +21,40 @@ class TextProcessor:
overlap: int = 50
) -> List[str]:
"""
分割文本
Split text into chunks
Args:
text: 原始文本
chunk_size: 块大小
overlap: 重叠大小
text: raw text
chunk_size: chunk size
overlap: overlap size
Returns:
文本块列表
list of text chunks
"""
return split_text_into_chunks(text, chunk_size, overlap)
@staticmethod
def preprocess_text(text: str) -> str:
"""
预处理文本
- 移除多余空白
- 标准化换行
Preprocess text
- Strip extra whitespace
- Normalize newlines
Args:
text: 原始文本
text: raw text
Returns:
处理后的文本
processed text
"""
import re
# 标准化换行
# Normalize newlines
text = text.replace('\r\n', '\n').replace('\r', '\n')
# 移除连续空行（保留最多两个换行）
# Collapse consecutive blank lines（Keep at most two newlines in a row）
text = re.sub(r'\n{3,}', '\n\n', text)
# 移除行首行尾空白
# Strip leading/trailing whitespace per line
lines = [line.strip() for line in text.split('\n')]
text = '\n'.join(lines)
@ -62,7 +62,7 @@ class TextProcessor:
@staticmethod
def get_text_stats(text: str) -> dict:
"""获取文本统计信息"""
"""Get text statistics"""
return {
"total_chars": len(text),
"total_lines": text.count('\n') + 1,

293
backend/app/services/zep_entity_reader.py
View File

@ -1,13 +1,24 @@
"""
Zep实体读取与过滤服务
从Zep图谱中读取节点，筛选出符合预定义实体类型的节点
Zep entity read & filter service
Reads nodes from the Zep graph and filters out nodes that match
predefined entity types.
"""
import time
from typing import Dict, Any, List, Optional, Set, Callable, TypeVar
from dataclasses import dataclass, field
from zep_cloud.client import Zep
try:
from zep_cloud.client import Zep
except ImportError:
class Zep: # type: ignore[no-redef]
def __init__(self, *a, **kw): pass
class graph:
class node:
@staticmethod
def get_entity_edges(**kw): raise NotImplementedError("zep-cloud not installed; use graphiti_service")
@staticmethod
def get(**kw): raise NotImplementedError("zep-cloud not installed; use graphiti_service")
from ..config import Config
from ..utils.logger import get_logger
@ -15,23 +26,23 @@ from ..utils.zep_paging import fetch_all_nodes, fetch_all_edges
logger = get_logger('mirofish.zep_entity_reader')
# 用于泛型返回类型
# Used for generic return type
T = TypeVar('T')
@dataclass
class EntityNode:
"""实体节点数据结构"""
"""Entity node data structure"""
uuid: str
name: str
labels: List[str]
summary: str
attributes: Dict[str, Any]
# 相关的边信息
# Related edge info
related_edges: List[Dict[str, Any]] = field(default_factory=list)
# 相关的其他节点信息
# Other related-node info
related_nodes: List[Dict[str, Any]] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
return {
"uuid": self.uuid,
@ -42,9 +53,9 @@ class EntityNode:
"related_edges": self.related_edges,
"related_nodes": self.related_nodes,
}
def get_entity_type(self) -> Optional[str]:
"""获取实体类型（排除默认的Entity标签）"""
"""Get the entity type (excluding the default Entity label)"""
for label in self.labels:
if label not in ["Entity", "Node"]:
return label
@ -53,12 +64,12 @@ class EntityNode:
@dataclass
class FilteredEntities:
"""过滤后的实体集合"""
"""Filtered entity collection"""
entities: List[EntityNode]
entity_types: Set[str]
total_count: int
filtered_count: int
def to_dict(self) -> Dict[str, Any]:
return {
"entities": [e.to_dict() for e in self.entities],
@ -70,43 +81,42 @@ class FilteredEntities:
class ZepEntityReader:
"""
Zep实体读取与过滤服务
主要功能：
1. 从Zep图谱读取所有节点
2. 筛选出符合预定义实体类型的节点（Labels不只是Entity的节点）
3. 获取每个实体的相关边和关联节点信息
Zep entity read & filter service
Main features:
1. Read all nodes from the Zep graph
2. Filter out nodes that match predefined entity types (nodes whose
labels are not just "Entity")
3. Get the related edges and adjacent node info for each entity
"""
def __init__(self, api_key: Optional[str] = None):
self.api_key = api_key or Config.ZEP_API_KEY
if not self.api_key:
raise ValueError("ZEP_API_KEY 未配置")
self.client = Zep(api_key=self.api_key)
self.api_key = api_key # kept for signature compat; no longer required
from .graphiti_service import get_graphiti_adapter
self.client = get_graphiti_adapter()
def _call_with_retry(
self,
func: Callable[[], T],
self,
func: Callable[[], T],
operation_name: str,
max_retries: int = 3,
initial_delay: float = 2.0
) -> T:
"""
带重试机制的Zep API调用
Zep API call with retry logic.
Args:
func: 要执行的函数（无参数的lambda或callable）
operation_name: 操作名称，用于日志
max_retries: 最大重试次数（默认3次，即最多尝试3次）
initial_delay: 初始延迟秒数
func: Function (parameterless lambda or callable) to execute
operation_name: Operation name, used in logs
max_retries: Maximum number of retries (default 3, i.e. at most 3 attempts)
initial_delay: Initial delay in seconds
Returns:
API调用结果
The result of the API call
"""
last_exception = None
delay = initial_delay
for attempt in range(max_retries):
try:
return func()
@ -114,27 +124,27 @@ class ZepEntityReader:
last_exception = e
if attempt < max_retries - 1:
logger.warning(
f"Zep {operation_name} 第 {attempt + 1} 次尝试失败: {str(e)[:100]}, "
f"{delay:.1f}秒后重试..."
f"Zep {operation_name} attempt {attempt + 1} failed: {str(e)[:100]}, "
f"retrying in {delay:.1f}s..."
)
time.sleep(delay)
delay *= 2 # 指数退避
delay *= 2 # Exponential backoff
else:
logger.error(f"Zep {operation_name} 在 {max_retries} 次尝试后仍失败: {str(e)}")
logger.error(f"Zep {operation_name} still failing after {max_retries} attempts: {str(e)}")
raise last_exception
def get_all_nodes(self, graph_id: str) -> List[Dict[str, Any]]:
"""
获取图谱的所有节点（分页获取）
Get all nodes of the graph (paginated).
Args:
graph_id: 图谱ID
graph_id: Graph ID
Returns:
节点列表
List of nodes
"""
logger.info(f"获取图谱 {graph_id} 的所有节点...")
logger.info(f"Fetching all nodes for graph {graph_id}...")
nodes = fetch_all_nodes(self.client, graph_id)
@ -148,20 +158,20 @@ class ZepEntityReader:
"attributes": node.attributes or {},
})
logger.info(f"共获取 {len(nodes_data)} 个节点")
logger.info(f"Fetched {len(nodes_data)} nodes in total")
return nodes_data
def get_all_edges(self, graph_id: str) -> List[Dict[str, Any]]:
"""
获取图谱的所有边（分页获取）
Get all edges of the graph (paginated).
Args:
graph_id: 图谱ID
graph_id: Graph ID
Returns:
边列表
List of edges
"""
logger.info(f"获取图谱 {graph_id} 的所有边...")
logger.info(f"Fetching all edges for graph {graph_id}...")
edges = fetch_all_edges(self.client, graph_id)
@ -176,26 +186,26 @@ class ZepEntityReader:
"attributes": edge.attributes or {},
})
logger.info(f"共获取 {len(edges_data)} 条边")
logger.info(f"Fetched {len(edges_data)} edges in total")
return edges_data
def get_node_edges(self, node_uuid: str) -> List[Dict[str, Any]]:
"""
获取指定节点的所有相关边（带重试机制）
Get all related edges of a given node (with retry logic).
Args:
node_uuid: 节点UUID
node_uuid: Node UUID
Returns:
边列表
List of edges
"""
try:
# 使用重试机制调用Zep API
# Graphiti via adapter
edges = self._call_with_retry(
func=lambda: self.client.graph.node.get_entity_edges(node_uuid=node_uuid),
operation_name=f"获取节点边(node={node_uuid[:8]}...)"
func=lambda: self.client.get_node_edges(node_uuid=node_uuid),
operation_name=f"get_node_edges(node={node_uuid[:8]}...)"
)
edges_data = []
for edge in edges:
edges_data.append({
@ -206,60 +216,63 @@ class ZepEntityReader:
"target_node_uuid": edge.target_node_uuid,
"attributes": edge.attributes or {},
})
return edges_data
except Exception as e:
logger.warning(f"获取节点 {node_uuid} 的边失败: {str(e)}")
logger.warning(f"Failed to fetch edges for node {node_uuid}: {str(e)}")
return []
def filter_defined_entities(
self,
self,
graph_id: str,
defined_entity_types: Optional[List[str]] = None,
enrich_with_edges: bool = True
) -> FilteredEntities:
"""
筛选出符合预定义实体类型的节点
筛选逻辑：
- 如果节点的Labels只有一个"Entity"，说明这个实体不符合我们预定义的类型，跳过
- 如果节点的Labels包含除"Entity"和"Node"之外的标签，说明符合预定义类型，保留
Filter out nodes that match predefined entity types.
Filter logic:
- If a node's labels contain only "Entity", it does not match
any of our predefined types and is skipped.
- If a node's labels include something other than "Entity" and
"Node", it matches a predefined type and is kept.
Args:
graph_id: 图谱ID
defined_entity_types: 预定义的实体类型列表（可选，如果提供则只保留这些类型）
enrich_with_edges: 是否获取每个实体的相关边信息
graph_id: Graph ID
defined_entity_types: List of predefined entity types (optional; if
provided, only these types are kept)
enrich_with_edges: Whether to fetch related edges for each entity
Returns:
FilteredEntities: 过滤后的实体集合
FilteredEntities: the filtered entity collection
"""
logger.info(f"开始筛选图谱 {graph_id} 的实体...")
# 获取所有节点
logger.info(f"Starting entity filtering for graph {graph_id}...")
# Get all nodes
all_nodes = self.get_all_nodes(graph_id)
total_count = len(all_nodes)
# 获取所有边（用于后续关联查找）
# Get all edges (used later for adjacency lookup)
all_edges = self.get_all_edges(graph_id) if enrich_with_edges else []
# 构建节点UUID到节点数据的映射
# Build a UUID -> node map
node_map = {n["uuid"]: n for n in all_nodes}
# 筛选符合条件的实体
# Filter entities that meet the criteria
filtered_entities = []
entity_types_found = set()
for node in all_nodes:
labels = node.get("labels", [])
# 筛选逻辑：Labels必须包含除"Entity"和"Node"之外的标签
# Filter logic: labels must contain something other than "Entity" and "Node"
custom_labels = [l for l in labels if l not in ["Entity", "Node"]]
if not custom_labels:
# 只有默认标签，跳过
# Only the default labels, skip
continue
# 如果指定了预定义类型，检查是否匹配
# If predefined types are specified, check for a match
if defined_entity_types:
matching_labels = [l for l in custom_labels if l in defined_entity_types]
if not matching_labels:
@ -267,10 +280,10 @@ class ZepEntityReader:
entity_type = matching_labels[0]
else:
entity_type = custom_labels[0]
entity_types_found.add(entity_type)
# 创建实体节点对象
# Build the entity node object
entity = EntityNode(
uuid=node["uuid"],
name=node["name"],
@ -278,12 +291,12 @@ class ZepEntityReader:
summary=node["summary"],
attributes=node["attributes"],
)
# 获取相关边和节点
# Fetch related edges and nodes
if enrich_with_edges:
related_edges = []
related_node_uuids = set()
for edge in all_edges:
if edge["source_node_uuid"] == node["uuid"]:
related_edges.append({
@ -301,10 +314,10 @@ class ZepEntityReader:
"source_node_uuid": edge["source_node_uuid"],
})
related_node_uuids.add(edge["source_node_uuid"])
entity.related_edges = related_edges
# 获取关联节点的基本信息
# Get basic info for related nodes
related_nodes = []
for related_uuid in related_node_uuids:
if related_uuid in node_map:
@ -315,57 +328,57 @@ class ZepEntityReader:
"labels": related_node["labels"],
"summary": related_node.get("summary", ""),
})
entity.related_nodes = related_nodes
filtered_entities.append(entity)
logger.info(f"筛选完成: 总节点 {total_count}, 符合条件 {len(filtered_entities)}, "
f"实体类型: {entity_types_found}")
logger.info(f"Filtering complete: total nodes {total_count}, matching {len(filtered_entities)}, "
f"entity types: {entity_types_found}")
return FilteredEntities(
entities=filtered_entities,
entity_types=entity_types_found,
total_count=total_count,
filtered_count=len(filtered_entities),
)
def get_entity_with_context(
self,
graph_id: str,
self,
graph_id: str,
entity_uuid: str
) -> Optional[EntityNode]:
"""
获取单个实体及其完整上下文（边和关联节点，带重试机制）
Get a single entity with its full context (edges and related nodes, with retry).
Args:
graph_id: 图谱ID
entity_uuid: 实体UUID
graph_id: Graph ID
entity_uuid: Entity UUID
Returns:
EntityNode或None
EntityNode or None
"""
try:
# 使用重试机制获取节点
# Graphiti via adapter
node = self._call_with_retry(
func=lambda: self.client.graph.node.get(uuid_=entity_uuid),
operation_name=f"获取节点详情(uuid={entity_uuid[:8]}...)"
func=lambda: self.client.get_node(node_uuid=entity_uuid),
operation_name=f"get_node_detail(uuid={entity_uuid[:8]}...)"
)
if not node:
return None
# 获取节点的边
# Get the node's edges
edges = self.get_node_edges(entity_uuid)
# 获取所有节点用于关联查找
# Get all nodes for adjacency lookup
all_nodes = self.get_all_nodes(graph_id)
node_map = {n["uuid"]: n for n in all_nodes}
# 处理相关边和节点
# Process related edges and nodes
related_edges = []
related_node_uuids = set()
for edge in edges:
if edge["source_node_uuid"] == entity_uuid:
related_edges.append({
@ -383,8 +396,8 @@ class ZepEntityReader:
"source_node_uuid": edge["source_node_uuid"],
})
related_node_uuids.add(edge["source_node_uuid"])
# 获取关联节点信息
# Get related node info
related_nodes = []
for related_uuid in related_node_uuids:
if related_uuid in node_map:
@ -395,7 +408,7 @@ class ZepEntityReader:
"labels": related_node["labels"],
"summary": related_node.get("summary", ""),
})
return EntityNode(
uuid=getattr(node, 'uuid_', None) or getattr(node, 'uuid', ''),
name=node.name or "",
@ -405,27 +418,27 @@ class ZepEntityReader:
related_edges=related_edges,
related_nodes=related_nodes,
)
except Exception as e:
logger.error(f"获取实体 {entity_uuid} 失败: {str(e)}")
logger.error(f"Failed to fetch entity {entity_uuid}: {str(e)}")
return None
def get_entities_by_type(
self,
graph_id: str,
self,
graph_id: str,
entity_type: str,
enrich_with_edges: bool = True
) -> List[EntityNode]:
"""
获取指定类型的所有实体
Get all entities of a given type.
Args:
graph_id: 图谱ID
entity_type: 实体类型（如 "Student", "PublicFigure" 等）
enrich_with_edges: 是否获取相关边信息
graph_id: Graph ID
entity_type: Entity type (e.g. "Student", "PublicFigure")
enrich_with_edges: Whether to fetch related edges
Returns:
实体列表
List of entities
"""
result = self.filter_defined_entities(
graph_id=graph_id,

506
backend/app/services/zep_graph_memory_updater.py
View File

@ -1,6 +1,6 @@
"""
Zep图谱记忆更新服务
将模拟中的Agent活动动态更新到Zep图谱中
Zep Graph Memory Updater Service
Dynamically updates agent activities from the simulation into the Zep graph
"""
import os
@ -12,7 +12,13 @@ from dataclasses import dataclass
from datetime import datetime
from queue import Queue, Empty
from zep_cloud.client import Zep
try:
from zep_cloud.client import Zep # noqa: F811
except ImportError:
class Zep: # type: ignore[no-redef]
def __init__(self, *a, **kw): pass
class graph:
def add(self, **kw): raise NotImplementedError("zep-cloud not installed; use graphiti_service")
from ..config import Config
from ..utils.logger import get_logger
@ -23,7 +29,7 @@ logger = get_logger('mirofish.zep_graph_memory_updater')
@dataclass
class AgentActivity:
"""Agent活动记录"""
"""Agent activity record"""
platform: str # twitter / reddit
agent_id: int
agent_name: str
@ -31,15 +37,15 @@ class AgentActivity:
action_args: Dict[str, Any]
round_num: int
timestamp: str
def to_episode_text(self) -> str:
"""
将活动转换为可以发送给Zep的文本描述
采用自然语言描述格式，让Zep能够从中提取实体和关系
不添加模拟相关的前缀，避免误导图谱更新
Convert the activity into a text description that can be sent to Zep.
Uses a natural-language description format so Zep can extract entities and relations from it.
No simulation-related prefix is added to avoid misleading the graph update.
"""
# 根据不同的动作类型生成不同的描述
# Generate a different description for each action type
action_descriptions = {
"CREATE_POST": self._describe_create_post,
"LIKE_POST": self._describe_like_post,
@ -54,226 +60,226 @@ class AgentActivity:
"SEARCH_USER": self._describe_search_user,
"MUTE": self._describe_mute,
}
describe_func = action_descriptions.get(self.action_type, self._describe_generic)
description = describe_func()
# 直接返回 "agent名称: 活动描述" 格式，不添加模拟前缀
# Return directly in "agent_name: activity description" format, no simulation prefix
return f"{self.agent_name}: {description}"
def _describe_create_post(self) -> str:
content = self.action_args.get("content", "")
if content:
return f"发布了一条帖子：「{content}」"
return "发布了一条帖子"
return f"published a post: \"{content}\""
return "published a post"
def _describe_like_post(self) -> str:
"""点赞帖子 - 包含帖子原文和作者信息"""
"""Like a post — includes the post text and author info"""
post_content = self.action_args.get("post_content", "")
post_author = self.action_args.get("post_author_name", "")
if post_content and post_author:
return f"点赞了{post_author}的帖子：「{post_content}」"
return f"liked {post_author}'s post: \"{post_content}\""
elif post_content:
return f"点赞了一条帖子：「{post_content}」"
return f"liked a post: \"{post_content}\""
elif post_author:
return f"点赞了{post_author}的一条帖子"
return "点赞了一条帖子"
return f"liked a post by {post_author}"
return "liked a post"
def _describe_dislike_post(self) -> str:
"""踩帖子 - 包含帖子原文和作者信息"""
"""Dislike a post — includes the post text and author info"""
post_content = self.action_args.get("post_content", "")
post_author = self.action_args.get("post_author_name", "")
if post_content and post_author:
return f"踩了{post_author}的帖子：「{post_content}」"
return f"disliked {post_author}'s post: \"{post_content}\""
elif post_content:
return f"踩了一条帖子：「{post_content}」"
return f"disliked a post: \"{post_content}\""
elif post_author:
return f"踩了{post_author}的一条帖子"
return "踩了一条帖子"
return f"disliked a post by {post_author}"
return "disliked a post"
def _describe_repost(self) -> str:
"""转发帖子 - 包含原帖内容和作者信息"""
"""Repost — includes the original post text and author info"""
original_content = self.action_args.get("original_content", "")
original_author = self.action_args.get("original_author_name", "")
if original_content and original_author:
return f"转发了{original_author}的帖子：「{original_content}」"
return f"reposted {original_author}'s post: \"{original_content}\""
elif original_content:
return f"转发了一条帖子：「{original_content}」"
return f"reposted a post: \"{original_content}\""
elif original_author:
return f"转发了{original_author}的一条帖子"
return "转发了一条帖子"
return f"reposted a post by {original_author}"
return "reposted a post"
def _describe_quote_post(self) -> str:
"""引用帖子 - 包含原帖内容、作者信息和引用评论"""
"""Quote a post — includes the original post text, author info, and quote comment"""
original_content = self.action_args.get("original_content", "")
original_author = self.action_args.get("original_author_name", "")
quote_content = self.action_args.get("quote_content", "") or self.action_args.get("content", "")
base = ""
if original_content and original_author:
base = f"引用了{original_author}的帖子「{original_content}」"
base = f"quoted {original_author}'s post \"{original_content}\""
elif original_content:
base = f"引用了一条帖子「{original_content}」"
base = f"quoted a post \"{original_content}\""
elif original_author:
base = f"引用了{original_author}的一条帖子"
base = f"quoted a post by {original_author}"
else:
base = "引用了一条帖子"
base = "quoted a post"
if quote_content:
base += f"，并评论道：「{quote_content}」"
base += f", and commented: \"{quote_content}\""
return base
def _describe_follow(self) -> str:
"""关注用户 - 包含被关注用户的名称"""
"""Follow a user — includes the name of the followed user"""
target_user_name = self.action_args.get("target_user_name", "")
if target_user_name:
return f"关注了用户「{target_user_name}」"
return "关注了一个用户"
return f"followed user \"{target_user_name}\""
return "followed a user"
def _describe_create_comment(self) -> str:
"""发表评论 - 包含评论内容和所评论的帖子信息"""
"""Post a comment — includes the comment content and the commented post info"""
content = self.action_args.get("content", "")
post_content = self.action_args.get("post_content", "")
post_author = self.action_args.get("post_author_name", "")
if content:
if post_content and post_author:
return f"在{post_author}的帖子「{post_content}」下评论道：「{content}」"
return f"commented on {post_author}'s post \"{post_content}\": \"{content}\""
elif post_content:
return f"在帖子「{post_content}」下评论道：「{content}」"
return f"commented on the post \"{post_content}\": \"{content}\""
elif post_author:
return f"在{post_author}的帖子下评论道：「{content}」"
return f"评论道：「{content}」"
return "发表了评论"
return f"commented on {post_author}'s post: \"{content}\""
return f"commented: \"{content}\""
return "posted a comment"
def _describe_like_comment(self) -> str:
"""点赞评论 - 包含评论内容和作者信息"""
"""Like a comment — includes the comment text and author info"""
comment_content = self.action_args.get("comment_content", "")
comment_author = self.action_args.get("comment_author_name", "")
if comment_content and comment_author:
return f"点赞了{comment_author}的评论：「{comment_content}」"
return f"liked {comment_author}'s comment: \"{comment_content}\""
elif comment_content:
return f"点赞了一条评论：「{comment_content}」"
return f"liked a comment: \"{comment_content}\""
elif comment_author:
return f"点赞了{comment_author}的一条评论"
return "点赞了一条评论"
return f"liked a comment by {comment_author}"
return "liked a comment"
def _describe_dislike_comment(self) -> str:
"""踩评论 - 包含评论内容和作者信息"""
"""Dislike a comment — includes the comment text and author info"""
comment_content = self.action_args.get("comment_content", "")
comment_author = self.action_args.get("comment_author_name", "")
if comment_content and comment_author:
return f"踩了{comment_author}的评论：「{comment_content}」"
return f"disliked {comment_author}'s comment: \"{comment_content}\""
elif comment_content:
return f"踩了一条评论：「{comment_content}」"
return f"disliked a comment: \"{comment_content}\""
elif comment_author:
return f"踩了{comment_author}的一条评论"
return "踩了一条评论"
return f"disliked a comment by {comment_author}"
return "disliked a comment"
def _describe_search(self) -> str:
"""搜索帖子 - 包含搜索关键词"""
"""Search posts — includes the search keyword"""
query = self.action_args.get("query", "") or self.action_args.get("keyword", "")
return f"搜索了「{query}」" if query else "进行了搜索"
return f"searched \"{query}\"" if query else "performed a search"
def _describe_search_user(self) -> str:
"""搜索用户 - 包含搜索关键词"""
"""Search users — includes the search keyword"""
query = self.action_args.get("query", "") or self.action_args.get("username", "")
return f"搜索了用户「{query}」" if query else "搜索了用户"
return f"searched for user \"{query}\"" if query else "searched for a user"
def _describe_mute(self) -> str:
"""屏蔽用户 - 包含被屏蔽用户的名称"""
"""Mute a user — includes the name of the muted user"""
target_user_name = self.action_args.get("target_user_name", "")
if target_user_name:
return f"屏蔽了用户「{target_user_name}」"
return "屏蔽了一个用户"
return f"muted user \"{target_user_name}\""
return "muted a user"
def _describe_generic(self) -> str:
# 对于未知的动作类型，生成通用描述
return f"执行了{self.action_type}操作"
# For unknown action types, generate a generic description
return f"performed {self.action_type} operation"
class ZepGraphMemoryUpdater:
"""
Zep图谱记忆更新器
监控模拟的actions日志文件，将新的agent活动实时更新到Zep图谱中。
按平台分组，每累积BATCH_SIZE条活动后批量发送到Zep。
所有有意义的行为都会被更新到Zep，action_args中会包含完整的上下文信息：
- 点赞/踩的帖子原文
- 转发/引用的帖子原文
- 关注/屏蔽的用户名
- 点赞/踩的评论原文
Zep graph memory updater.
Monitors the simulation's actions log file and updates new agent activities
into the Zep graph in real time. Activities are grouped by platform, and
a batch is sent to Zep once BATCH_SIZE items have accumulated.
All meaningful behaviors are pushed to Zep; action_args carries the full
context for each one:
- the original text of liked/disliked posts
- the original text of reposted/quoted posts
- the name of followed/muted users
- the original text of liked/disliked comments
"""
# 批量发送大小（每个平台累积多少条后发送）
# Batch send size (how many activities to accumulate per platform before sending)
BATCH_SIZE = 5
# 平台名称映射（用于控制台显示）
# Platform name mapping (for console display)
PLATFORM_DISPLAY_NAMES = {
'twitter': '世界1',
'reddit': '世界2',
'twitter': 'World 1',
'reddit': 'World 2',
}
# 发送间隔（秒），避免请求过快
# Send interval (seconds) to avoid hitting the API too fast
SEND_INTERVAL = 0.5
# 重试配置
# Retry configuration
MAX_RETRIES = 3
RETRY_DELAY = 2 # 秒
RETRY_DELAY = 2 # seconds
def __init__(self, graph_id: str, api_key: Optional[str] = None):
"""
初始化更新器
Initialize the updater.
Args:
graph_id: Zep图谱ID
api_key: Zep API Key（可选，默认从配置读取）
graph_id: Zep graph ID
api_key: Zep API key (optional; read from config by default)
"""
self.graph_id = graph_id
self.api_key = api_key or Config.ZEP_API_KEY
if not self.api_key:
raise ValueError("ZEP_API_KEY未配置")
self.client = Zep(api_key=self.api_key)
# 活动队列
self.api_key = api_key # kept for signature compat; no longer required
from .graphiti_service import get_graphiti_adapter
self.client = get_graphiti_adapter()
# Activity queue
self._activity_queue: Queue = Queue()
# 按平台分组的活动缓冲区（每个平台各自累积到BATCH_SIZE后批量发送）
# Per-platform activity buffer (each platform accumulates to BATCH_SIZE then sends)
self._platform_buffers: Dict[str, List[AgentActivity]] = {
'twitter': [],
'reddit': [],
}
self._buffer_lock = threading.Lock()
# 控制标志
# Control flags
self._running = False
self._worker_thread: Optional[threading.Thread] = None
# 统计
self._total_activities = 0 # 实际添加到队列的活动数
self._total_sent = 0 # 成功发送到Zep的批次数
self._total_items_sent = 0 # 成功发送到Zep的活动条数
self._failed_count = 0 # 发送失败的批次数
self._skipped_count = 0 # 被过滤跳过的活动数（DO_NOTHING）
logger.info(f"ZepGraphMemoryUpdater 初始化完成: graph_id={graph_id}, batch_size={self.BATCH_SIZE}")
# Stats
self._total_activities = 0 # number of activities actually added to the queue
self._total_sent = 0 # number of batches successfully sent to Zep
self._total_items_sent = 0 # number of activities successfully sent to Zep
self._failed_count = 0 # number of failed batches
self._skipped_count = 0 # activities filtered out (DO_NOTHING)
logger.info(f"ZepGraphMemoryUpdater initialization complete: graph_id={graph_id}, batch_size={self.BATCH_SIZE}")
def _get_platform_display_name(self, platform: str) -> str:
"""获取平台的显示名称"""
"""Get the display name of a platform"""
return self.PLATFORM_DISPLAY_NAMES.get(platform.lower(), platform)
def start(self):
"""启动后台工作线程"""
"""Start the background worker thread"""
if self._running:
return
@ -288,19 +294,19 @@ class ZepGraphMemoryUpdater:
name=f"ZepMemoryUpdater-{self.graph_id[:8]}"
)
self._worker_thread.start()
logger.info(f"ZepGraphMemoryUpdater 已启动: graph_id={self.graph_id}")
logger.info(f"ZepGraphMemoryUpdater started: graph_id={self.graph_id}")
def stop(self):
"""停止后台工作线程"""
"""Stop the background worker thread"""
self._running = False
# 发送剩余的活动
# Flush remaining activities
self._flush_remaining()
if self._worker_thread and self._worker_thread.is_alive():
self._worker_thread.join(timeout=10)
logger.info(f"ZepGraphMemoryUpdater 已停止: graph_id={self.graph_id}, "
logger.info(f"ZepGraphMemoryUpdater stopped: graph_id={self.graph_id}, "
f"total_activities={self._total_activities}, "
f"batches_sent={self._total_sent}, "
f"items_sent={self._total_items_sent}, "
@ -309,46 +315,46 @@ class ZepGraphMemoryUpdater:
def add_activity(self, activity: AgentActivity):
"""
添加一个agent活动到队列
所有有意义的行为都会被添加到队列，包括：
- CREATE_POST（发帖）
- CREATE_COMMENT（评论）
- QUOTE_POST（引用帖子）
- SEARCH_POSTS（搜索帖子）
- SEARCH_USER（搜索用户）
- LIKE_POST/DISLIKE_POST（点赞/踩帖子）
- REPOST（转发）
- FOLLOW（关注）
- MUTE（屏蔽）
- LIKE_COMMENT/DISLIKE_COMMENT（点赞/踩评论）
action_args中会包含完整的上下文信息（如帖子原文、用户名等）。
Add an agent activity to the queue.
All meaningful behaviors are added to the queue, including:
- CREATE_POST (publish a post)
- CREATE_COMMENT (post a comment)
- QUOTE_POST (quote a post)
- SEARCH_POSTS (search posts)
- SEARCH_USER (search users)
- LIKE_POST/DISLIKE_POST (like/dislike a post)
- REPOST (repost)
- FOLLOW (follow)
- MUTE (mute)
- LIKE_COMMENT/DISLIKE_COMMENT (like/dislike a comment)
action_args carries the full context (original post text, username, etc.).
Args:
activity: Agent活动记录
activity: Agent activity record
"""
# 跳过DO_NOTHING类型的活动
# Skip DO_NOTHING activities
if activity.action_type == "DO_NOTHING":
self._skipped_count += 1
return
self._activity_queue.put(activity)
self._total_activities += 1
logger.debug(f"添加活动到Zep队列: {activity.agent_name} - {activity.action_type}")
logger.debug(f"Added activity to Zep queue: {activity.agent_name} - {activity.action_type}")
def add_activity_from_dict(self, data: Dict[str, Any], platform: str):
"""
从字典数据添加活动
Add an activity from a dictionary.
Args:
data: 从actions.jsonl解析的字典数据
platform: 平台名称 (twitter/reddit)
data: Dictionary parsed from actions.jsonl
platform: Platform name (twitter/reddit)
"""
# 跳过事件类型的条目
# Skip entries that are events
if "event_type" in data:
return
activity = AgentActivity(
platform=platform,
agent_id=data.get("agent_id", 0),
@ -358,83 +364,83 @@ class ZepGraphMemoryUpdater:
round_num=data.get("round", 0),
timestamp=data.get("timestamp", datetime.now().isoformat()),
)
self.add_activity(activity)
def _worker_loop(self, locale: str = 'zh'):
"""后台工作循环 - 按平台批量发送活动到Zep"""
"""Background worker loop — batch-sends activities to Zep per platform"""
set_locale(locale)
while self._running or not self._activity_queue.empty():
try:
# 尝试从队列获取活动（超时1秒）
# Try to fetch an activity from the queue (1s timeout)
try:
activity = self._activity_queue.get(timeout=1)
# 将活动添加到对应平台的缓冲区
# Append the activity to the corresponding platform buffer
platform = activity.platform.lower()
with self._buffer_lock:
if platform not in self._platform_buffers:
self._platform_buffers[platform] = []
self._platform_buffers[platform].append(activity)
# 检查该平台是否达到批量大小
# Check if this platform reached the batch size
if len(self._platform_buffers[platform]) >= self.BATCH_SIZE:
batch = self._platform_buffers[platform][:self.BATCH_SIZE]
self._platform_buffers[platform] = self._platform_buffers[platform][self.BATCH_SIZE:]
# 释放锁后再发送
# Release the lock before sending
self._send_batch_activities(batch, platform)
# 发送间隔，避免请求过快
# Send interval to avoid hitting the API too fast
time.sleep(self.SEND_INTERVAL)
except Empty:
pass
except Exception as e:
logger.error(f"工作循环异常: {e}")
logger.error(f"Worker loop error: {e}")
time.sleep(1)
def _send_batch_activities(self, activities: List[AgentActivity], platform: str):
"""
批量发送活动到Zep图谱（合并为一条文本）
Batch-send activities to the Zep graph (merged into a single text).
Args:
activities: Agent活动列表
platform: 平台名称
activities: List of agent activities
platform: Platform name
"""
if not activities:
return
# 将多条活动合并为一条文本，用换行分隔
# Merge multiple activities into one text, separated by newlines
episode_texts = [activity.to_episode_text() for activity in activities]
combined_text = "\n".join(episode_texts)
# 带重试的发送
# Send with retry
for attempt in range(self.MAX_RETRIES):
try:
self.client.graph.add(
# Graphiti via adapter: one episode per batch (extraction is sync)
self.client.add_batch(
graph_id=self.graph_id,
type="text",
data=combined_text
episodes=[{"data": combined_text, "type": "text"}],
)
self._total_sent += 1
self._total_items_sent += len(activities)
display_name = self._get_platform_display_name(platform)
logger.info(f"成功批量发送 {len(activities)} 条{display_name}活动到图谱 {self.graph_id}")
logger.debug(f"批量内容预览: {combined_text[:200]}...")
logger.info(f"Successfully batch-sent {len(activities)} {display_name} activities to graph {self.graph_id}")
logger.debug(f"Batch content preview: {combined_text[:200]}...")
return
except Exception as e:
if attempt < self.MAX_RETRIES - 1:
logger.warning(f"批量发送到Zep失败 (尝试 {attempt + 1}/{self.MAX_RETRIES}): {e}")
logger.warning(f"Batch send to Zep failed (attempt {attempt + 1}/{self.MAX_RETRIES}): {e}")
time.sleep(self.RETRY_DELAY * (attempt + 1))
else:
logger.error(f"批量发送到Zep失败，已重试{self.MAX_RETRIES}次: {e}")
logger.error(f"Batch send to Zep failed after {self.MAX_RETRIES} retries: {e}")
self._failed_count += 1
def _flush_remaining(self):
"""发送队列和缓冲区中剩余的活动"""
# 首先处理队列中剩余的活动，添加到缓冲区
"""Flush remaining activities in the queue and buffers"""
# First, drain the queue into the buffers
while not self._activity_queue.empty():
try:
activity = self._activity_queue.get_nowait()
@ -445,110 +451,110 @@ class ZepGraphMemoryUpdater:
self._platform_buffers[platform].append(activity)
except Empty:
break
# 然后发送各平台缓冲区中剩余的活动（即使不足BATCH_SIZE条）
# Then send any remaining activities in the per-platform buffers (even if < BATCH_SIZE)
with self._buffer_lock:
for platform, buffer in self._platform_buffers.items():
if buffer:
display_name = self._get_platform_display_name(platform)
logger.info(f"发送{display_name}平台剩余的 {len(buffer)} 条活动")
logger.info(f"Flushing remaining {len(buffer)} activities from {display_name}")
self._send_batch_activities(buffer, platform)
# 清空所有缓冲区
# Clear all buffers
for platform in self._platform_buffers:
self._platform_buffers[platform] = []
def get_stats(self) -> Dict[str, Any]:
"""获取统计信息"""
"""Get stats"""
with self._buffer_lock:
buffer_sizes = {p: len(b) for p, b in self._platform_buffers.items()}
return {
"graph_id": self.graph_id,
"batch_size": self.BATCH_SIZE,
"total_activities": self._total_activities, # 添加到队列的活动总数
"batches_sent": self._total_sent, # 成功发送的批次数
"items_sent": self._total_items_sent, # 成功发送的活动条数
"failed_count": self._failed_count, # 发送失败的批次数
"skipped_count": self._skipped_count, # 被过滤跳过的活动数（DO_NOTHING）
"total_activities": self._total_activities, # total activities added to the queue
"batches_sent": self._total_sent, # successfully sent batches
"items_sent": self._total_items_sent, # successfully sent activities
"failed_count": self._failed_count, # failed batches
"skipped_count": self._skipped_count, # activities filtered out (DO_NOTHING)
"queue_size": self._activity_queue.qsize(),
"buffer_sizes": buffer_sizes, # 各平台缓冲区大小
"buffer_sizes": buffer_sizes, # per-platform buffer sizes
"running": self._running,
}
class ZepGraphMemoryManager:
"""
管理多个模拟的Zep图谱记忆更新器
每个模拟可以有自己的更新器实例
Manages Zep graph memory updaters for multiple simulations.
Each simulation can have its own updater instance.
"""
_updaters: Dict[str, ZepGraphMemoryUpdater] = {}
_lock = threading.Lock()
@classmethod
def create_updater(cls, simulation_id: str, graph_id: str) -> ZepGraphMemoryUpdater:
"""
为模拟创建图谱记忆更新器
Create a graph memory updater for a simulation.
Args:
simulation_id: 模拟ID
graph_id: Zep图谱ID
simulation_id: Simulation ID
graph_id: Zep graph ID
Returns:
ZepGraphMemoryUpdater实例
A ZepGraphMemoryUpdater instance
"""
with cls._lock:
# 如果已存在，先停止旧的
# If one already exists, stop it first
if simulation_id in cls._updaters:
cls._updaters[simulation_id].stop()
updater = ZepGraphMemoryUpdater(graph_id)
updater.start()
cls._updaters[simulation_id] = updater
logger.info(f"创建图谱记忆更新器: simulation_id={simulation_id}, graph_id={graph_id}")
logger.info(f"Created graph memory updater: simulation_id={simulation_id}, graph_id={graph_id}")
return updater
@classmethod
def get_updater(cls, simulation_id: str) -> Optional[ZepGraphMemoryUpdater]:
"""获取模拟的更新器"""
"""Get the updater for a simulation"""
return cls._updaters.get(simulation_id)
@classmethod
def stop_updater(cls, simulation_id: str):
"""停止并移除模拟的更新器"""
"""Stop and remove the updater for a simulation"""
with cls._lock:
if simulation_id in cls._updaters:
cls._updaters[simulation_id].stop()
del cls._updaters[simulation_id]
logger.info(f"已停止图谱记忆更新器: simulation_id={simulation_id}")
# 防止 stop_all 重复调用的标志
logger.info(f"Stopped graph memory updater: simulation_id={simulation_id}")
# Flag to prevent repeated stop_all calls
_stop_all_done = False
@classmethod
def stop_all(cls):
"""停止所有更新器"""
# 防止重复调用
"""Stop all updaters"""
# Prevent repeated calls
if cls._stop_all_done:
return
cls._stop_all_done = True
with cls._lock:
if cls._updaters:
for simulation_id, updater in list(cls._updaters.items()):
try:
updater.stop()
except Exception as e:
logger.error(f"停止更新器失败: simulation_id={simulation_id}, error={e}")
logger.error(f"Failed to stop updater: simulation_id={simulation_id}, error={e}")
cls._updaters.clear()
logger.info("已停止所有图谱记忆更新器")
logger.info("Stopped all graph memory updaters")
@classmethod
def get_all_stats(cls) -> Dict[str, Dict[str, Any]]:
"""获取所有更新器的统计信息"""
"""Get stats for all updaters"""
return {
sim_id: updater.get_stats()
sim_id: updater.get_stats()
for sim_id, updater in cls._updaters.items()
}

857
backend/app/services/zep_tools.py
View File

File diff suppressed because it is too large Load Diff

3
backend/app/utils/__init__.py
View File

@ -1,5 +1,5 @@
"""
工具模块
Utility module.
"""
from .file_parser import FileParser
@ -7,4 +7,3 @@ from .llm_client import LLMClient
from .locale import t, get_locale, set_locale, get_language_instruction
__all__ = ['FileParser', 'LLMClient', 't', 'get_locale', 'set_locale', 'get_language_instruction']

82
backend/app/utils/file_parser.py
View File

@ -1,6 +1,6 @@
"""
文件解析工具
支持PDF、Markdown、TXT文件的文本提取
File parser utilities
Supports text extraction from PDF, Markdown, and TXT files
"""
import os
@ -10,29 +10,29 @@ from typing import List, Optional
def _read_text_with_fallback(file_path: str) -> str:
"""
读取文本文件，UTF-8失败时自动探测编码。
Read a text file with UTF-8; auto-detect encoding on failure.
采用多级回退策略：
1. 首先尝试 UTF-8 解码
2. 使用 charset_normalizer 检测编码
3. 回退到 chardet 检测编码
4. 最终使用 UTF-8 + errors='replace' 兜底
Uses a multi-level fallback strategy:
1. 1. First try UTF-8 decoding
2. 2. Use charset_normalizer to detect encoding
3. 3. Fall back to chardet
4. 4. Final fallback: UTF-8 with errors="replace"
Args:
file_path: 文件路径
file_path: file path
Returns:
解码后的文本内容
decoded text content
"""
data = Path(file_path).read_bytes()
# 首先尝试 UTF-8
# First try UTF-8
try:
return data.decode('utf-8')
except UnicodeDecodeError:
pass
# 尝试使用 charset_normalizer 检测编码
# 2. Use charset_normalizer to detect encoding
encoding = None
try:
from charset_normalizer import from_bytes
@ -42,7 +42,7 @@ def _read_text_with_fallback(file_path: str) -> str:
except Exception:
pass
# 回退到 chardet
# Fall back to chardet
if not encoding:
try:
import chardet
@ -51,7 +51,7 @@ def _read_text_with_fallback(file_path: str) -> str:
except Exception:
pass
# 最终兜底：使用 UTF-8 + replace
# Final fallback: UTF-8 with replace
if not encoding:
encoding = 'utf-8'
@ -59,20 +59,20 @@ def _read_text_with_fallback(file_path: str) -> str:
class FileParser:
"""文件解析器"""
"""File parser"""
SUPPORTED_EXTENSIONS = {'.pdf', '.md', '.markdown', '.txt'}
@classmethod
def is_supported(cls, file_path: str) -> bool:
"""
检查文件是否为支持的格式
Check whether a file is in a supported format
Args:
file_path: 文件路径
file_path: file path
Returns:
如果文件格式受支持则返回 True
Return True if the file format is supported
"""
suffix = Path(file_path).suffix.lower()
return suffix in cls.SUPPORTED_EXTENSIONS
@ -80,23 +80,23 @@ class FileParser:
@classmethod
def extract_text(cls, file_path: str) -> str:
"""
从文件中提取文本
Extract text from a file
Args:
file_path: 文件路径
file_path: file path
Returns:
提取的文本内容
extracted text content
"""
path = Path(file_path)
if not path.exists():
raise FileNotFoundError(f"文件不存在: {file_path}")
raise FileNotFoundError(f"File does not exist: {file_path}")
suffix = path.suffix.lower()
if suffix not in cls.SUPPORTED_EXTENSIONS:
raise ValueError(f"不支持的文件格式: {suffix}")
raise ValueError(f"Unsupported file format: {suffix}")
if suffix == '.pdf':
return cls._extract_from_pdf(file_path)
@ -105,15 +105,15 @@ class FileParser:
elif suffix == '.txt':
return cls._extract_from_txt(file_path)
raise ValueError(f"无法处理的文件格式: {suffix}")
raise ValueError(f"Cannot process file format: {suffix}")
@staticmethod
def _extract_from_pdf(file_path: str) -> str:
"""从PDF提取文本"""
"""Extract text from PDF"""
try:
import fitz # PyMuPDF
except ImportError:
raise ImportError("需要安装PyMuPDF: pip install PyMuPDF")
raise ImportError("PyMuPDF is required: pip install PyMuPDF")
text_parts = []
with fitz.open(file_path) as doc:
@ -126,24 +126,24 @@ class FileParser:
@staticmethod
def _extract_from_md(file_path: str) -> str:
"""从Markdown提取文本，支持自动编码检测"""
"""Extract text from Markdown with auto-encoding detection"""
return _read_text_with_fallback(file_path)
@staticmethod
def _extract_from_txt(file_path: str) -> str:
"""从TXT提取文本，支持自动编码检测"""
"""Extract text from TXT with auto-encoding detection"""
return _read_text_with_fallback(file_path)
@classmethod
def extract_from_multiple(cls, file_paths: List[str]) -> str:
"""
从多个文件提取文本并合并
Extract and merge text from multiple files
Args:
file_paths: 文件路径列表
file_paths: list of file paths
Returns:
合并后的文本
merged text
"""
all_texts = []
@ -151,9 +151,9 @@ class FileParser:
try:
text = cls.extract_text(file_path)
filename = Path(file_path).name
all_texts.append(f"=== 文档 {i}: {filename} ===\n{text}")
all_texts.append(f"=== Document {i}: {filename} ===\n{text}")
except Exception as e:
all_texts.append(f"=== 文档 {i}: {file_path} (提取失败: {str(e)}) ===")
all_texts.append(f"=== Document {i}: {file_path} (extraction failed: {str(e)}) ===")
return "\n\n".join(all_texts)
@ -164,15 +164,15 @@ def split_text_into_chunks(
overlap: int = 50
) -> List[str]:
"""
将文本分割成小块
Split text into chunks
Args:
text: 原始文本
chunk_size: 每块的字符数
overlap: 重叠字符数
text: raw text
chunk_size: characters per chunk
overlap: overlap character count
Returns:
文本块列表
list of text chunks
"""
if len(text) <= chunk_size:
return [text] if text.strip() else []
@ -183,9 +183,9 @@ def split_text_into_chunks(
while start < len(text):
end = start + chunk_size
# 尝试在句子边界处分割
# Try to split on sentence boundaries
if end < len(text):
# 查找最近的句子结束符
# Find the nearest sentence-ending punctuation
for sep in ['。', '！', '？', '.\n', '!\n', '?\n', '\n\n', '. ', '! ', '? ']:
last_sep = text[start:end].rfind(sep)
if last_sep != -1 and last_sep > chunk_size * 0.3:
@ -196,7 +196,7 @@ def split_text_into_chunks(
if chunk:
chunks.append(chunk)
# 下一个块从重叠位置开始
# The next chunk starts at the overlap position
start = end - overlap if end < len(text) else len(text)
return chunks

56
backend/app/utils/llm_client.py
View File

@ -1,6 +1,6 @@
"""
LLM客户端封装
统一使用OpenAI格式调用
LLM client wrapper
Unified OpenAI-format invocation
"""
import json
@ -12,7 +12,7 @@ from ..config import Config
class LLMClient:
"""LLM客户端"""
"""LLM client"""
def __init__(
self,
@ -25,7 +25,7 @@ class LLMClient:
self.model = model or Config.LLM_MODEL_NAME
if not self.api_key:
raise ValueError("LLM_API_KEY 未配置")
raise ValueError("LLM_API_KEY is not configured")
self.client = OpenAI(
api_key=self.api_key,
@ -40,16 +40,16 @@ class LLMClient:
response_format: Optional[Dict] = None
) -> str:
"""
发送聊天请求
Send a chat completion request
Args:
messages: 消息列表
temperature: 温度参数
max_tokens: 最大token数
response_format: 响应格式（如JSON模式）
messages: message list
temperature: sampling temperature
max_tokens: max tokens
response_format: response format（e.g. JSON mode）
Returns:
模型响应文本
model response text
"""
kwargs = {
"model": self.model,
@ -63,7 +63,7 @@ class LLMClient:
response = self.client.chat.completions.create(**kwargs)
content = response.choices[0].message.content
# 部分模型（如MiniMax M2.5）会在content中包含<think>思考内容，需要移除
# Some models (e.g. MiniMax M2.5) embed <think>...</think> reasoning in content — strip it
content = re.sub(r'<think>[\s\S]*?</think>', '', content).strip()
return content
@ -71,18 +71,19 @@ class LLMClient:
self,
messages: List[Dict[str, str]],
temperature: float = 0.3,
max_tokens: int = 4096
max_tokens: int = 16384
) -> Dict[str, Any]:
"""
发送聊天请求并返回JSON
Send a chat completion requestand parse the response as JSON
Args:
messages: 消息列表
temperature: 温度参数
max_tokens: 最大token数
messages: message list
temperature: sampling temperature
max_tokens: max tokens (default 16384 — MiniMax M3 in ontology / config
long-JSON mode — 4096 tokens frequently truncates the JSON in the middle)
Returns:
解析后的JSON对象
parsed JSON object
"""
response = self.chat(
messages=messages,
@ -90,14 +91,25 @@ class LLMClient:
max_tokens=max_tokens,
response_format={"type": "json_object"}
)
# 清理markdown代码块标记
# Strip markdown code-block fences
# MiniMax M3 ignores response_format=json_object and wraps JSON in
# markdown fences. Be aggressive about stripping them, including the
# ```json\n and trailing ``` even when surrounded by other content.
cleaned_response = response.strip()
cleaned_response = re.sub(r'^```(?:json)?\s*\n?', '', cleaned_response, flags=re.IGNORECASE)
# Strip a leading ``` (with optional language tag and newline)
cleaned_response = re.sub(r'^```(?:json|JSON)?\s*\n?', '', cleaned_response, flags=re.IGNORECASE)
# Strip a trailing ``` (with optional preceding newline)
cleaned_response = re.sub(r'\n?```\s*$', '', cleaned_response)
# If there's still a ```json or ``` anywhere, take the first JSON-looking
# block from the response.
if '```' in cleaned_response:
m = re.search(r'(\{[\s\S]*\}|\[[\s\S]*\])', cleaned_response)
if m:
cleaned_response = m.group(1)
cleaned_response = cleaned_response.strip()
try:
return json.loads(cleaned_response)
except json.JSONDecodeError:
raise ValueError(f"LLM返回的JSON格式无效: {cleaned_response}")
raise ValueError(f"LLM returned invalid JSON: {cleaned_response[:200]}")

12
backend/app/utils/locale.py
View File

@ -1,3 +1,9 @@
"""
Locale / i18n helpers used by the backend. Loads translation files from
``/opt/data/work/MiroFish/locales`` and exposes ``t(key)`` lookups plus a
per-thread locale setting.
"""
import json
import os
import threading
@ -64,6 +70,10 @@ def t(key: str, **kwargs) -> str:
def get_language_instruction() -> str:
"""
Return the LLM language-injection string for the active locale.
Falls back to a Chinese instruction if no configuration is found.
"""
locale = get_locale()
lang_config = _languages.get(locale, _languages.get('zh', {}))
return lang_config.get('llmInstruction', '请使用中文回答。')
return lang_config.get('llmInstruction', 'Please reply in English.')

48
backend/app/utils/logger.py
View File

@ -1,6 +1,6 @@
"""
日志配置模块
提供统一的日志管理，同时输出到控制台和文件
Logging configuration module
Provides unified log management, output to both console and file
"""
import os
@ -12,47 +12,47 @@ from logging.handlers import RotatingFileHandler
def _ensure_utf8_stdout():
"""
确保 stdout/stderr 使用 UTF-8 编码
解决 Windows 控制台中文乱码问题
Ensure stdout/stderr use UTF-8
Work around Windows console garbled Chinese
"""
if sys.platform == 'win32':
# Windows 下重新配置标准输出为 UTF-8
# Reconfigure stdout to UTF-8 on Windows
if hasattr(sys.stdout, 'reconfigure'):
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
if hasattr(sys.stderr, 'reconfigure'):
sys.stderr.reconfigure(encoding='utf-8', errors='replace')
# 日志目录
# Log directory
LOG_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), 'logs')
def setup_logger(name: str = 'mirofish', level: int = logging.DEBUG) -> logging.Logger:
"""
设置日志器
Set up a logger
Args:
name: 日志器名称
level: 日志级别
name: logger name
level: log level
Returns:
配置好的日志器
configured logger
"""
# 确保日志目录存在
# Ensure log directory exists
os.makedirs(LOG_DIR, exist_ok=True)
# 创建日志器
# Create logger
logger = logging.getLogger(name)
logger.setLevel(level)
# 阻止日志向上传播到根 logger，避免重复输出
# Prevent logs from propagating to the root logger (avoid double output)
logger.propagate = False
# 如果已经有处理器，不重复添加
# If a handler is already attached, do not add another
if logger.handlers:
return logger
# 日志格式
# Log format
detailed_formatter = logging.Formatter(
'[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
@ -63,7 +63,7 @@ def setup_logger(name: str = 'mirofish', level: int = logging.DEBUG) -> logging.
datefmt='%H:%M:%S'
)
# 1. 文件处理器 - 详细日志（按日期命名，带轮转）
# 1. File handler — verbose log (date-named, with rotation)
log_filename = datetime.now().strftime('%Y-%m-%d') + '.log'
file_handler = RotatingFileHandler(
os.path.join(LOG_DIR, log_filename),
@ -74,14 +74,14 @@ def setup_logger(name: str = 'mirofish', level: int = logging.DEBUG) -> logging.
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(detailed_formatter)
# 2. 控制台处理器 - 简洁日志（INFO及以上）
# 确保 Windows 下使用 UTF-8 编码，避免中文乱码
# 2. Console handler — concise log (INFO and above)
# Use UTF-8 on Windows to avoid garbled output
_ensure_utf8_stdout()
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(simple_formatter)
# 添加处理器
# Add handler
logger.addHandler(file_handler)
logger.addHandler(console_handler)
@ -90,13 +90,13 @@ def setup_logger(name: str = 'mirofish', level: int = logging.DEBUG) -> logging.
def get_logger(name: str = 'mirofish') -> logging.Logger:
"""
获取日志器（如果不存在则创建）
Get a logger (creating it if it does not exist)
Args:
name: 日志器名称
name: logger name
Returns:
日志器实例
logger instance
"""
logger = logging.getLogger(name)
if not logger.handlers:
@ -104,11 +104,11 @@ def get_logger(name: str = 'mirofish') -> logging.Logger:
return logger
# 创建默认日志器
# Create the default logger
logger = setup_logger()
# 便捷方法
# Convenience methods
def debug(msg: str, *args, **kwargs) -> None:
logger.debug(msg, *args, **kwargs)

70
backend/app/utils/retry.py
View File

@ -1,6 +1,6 @@
"""
API调用重试机制
用于处理LLM等外部API调用的重试逻辑
API-call retry mechanism
Handles retry logic for external API calls (e.g. LLM)
"""
import time
@ -22,16 +22,16 @@ def retry_with_backoff(
on_retry: Optional[Callable[[Exception, int], None]] = None
):
"""
带指数退避的重试装饰器
Retry decorator with exponential backoff
Args:
max_retries: 最大重试次数
initial_delay: 初始延迟（秒）
max_delay: 最大延迟（秒）
backoff_factor: 退避因子
jitter: 是否添加随机抖动
exceptions: 需要重试的异常类型
on_retry: 重试时的回调函数 (exception, retry_count)
max_retries: maximum retry count
initial_delay: initial delay (seconds)
max_delay: maximum delay (seconds)
backoff_factor: backoff factor
jitter: whether to add random jitter
exceptions: exception types to retry on
on_retry: callback invoked on each retry (exception, retry_count)
Usage:
@retry_with_backoff(max_retries=3)
@ -52,17 +52,17 @@ def retry_with_backoff(
last_exception = e
if attempt == max_retries:
logger.error(f"函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}")
logger.error(f"Function {func.__name__} still failed after {max_retries} retries: {str(e)}")
raise
# 计算延迟
# Calculate delay
current_delay = min(delay, max_delay)
if jitter:
current_delay = current_delay * (0.5 + random.random())
logger.warning(
f"函数 {func.__name__} 第 {attempt + 1} 次尝试失败: {str(e)}, "
f"{current_delay:.1f}秒后重试..."
f"function {func.__name__} attempt {attempt + 1} failed: {str(e)}, "
f"{current_delay:.1f}s, retrying..."
)
if on_retry:
@ -87,7 +87,7 @@ def retry_with_backoff_async(
on_retry: Optional[Callable[[Exception, int], None]] = None
):
"""
异步版本的重试装饰器
Async version of the retry decorator
"""
import asyncio
@ -105,7 +105,7 @@ def retry_with_backoff_async(
last_exception = e
if attempt == max_retries:
logger.error(f"异步函数 {func.__name__} 在 {max_retries} 次重试后仍失败: {str(e)}")
logger.error(f"async function {func.__name__} still failed after {max_retries} retries: {str(e)}")
raise
current_delay = min(delay, max_delay)
@ -113,8 +113,8 @@ def retry_with_backoff_async(
current_delay = current_delay * (0.5 + random.random())
logger.warning(
f"异步函数 {func.__name__} 第 {attempt + 1} 次尝试失败: {str(e)}, "
f"{current_delay:.1f}秒后重试..."
f"async function {func.__name__} attempt {attempt + 1} failed: {str(e)}, "
f"{current_delay:.1f}s, retrying..."
)
if on_retry:
@ -131,7 +131,7 @@ def retry_with_backoff_async(
class RetryableAPIClient:
"""
可重试的API客户端封装
Retry-capable API client wrapper
"""
def __init__(
@ -154,16 +154,16 @@ class RetryableAPIClient:
**kwargs
) -> Any:
"""
执行函数调用并在失败时重试
Execute a function with retry on failure
Args:
func: 要调用的函数
*args: 函数参数
exceptions: 需要重试的异常类型
**kwargs: 函数关键字参数
func: function to call
*args: function positional args
exceptions: exception types to retry on
**kwargs: function keyword args
Returns:
函数返回值
function return value
"""
last_exception = None
delay = self.initial_delay
@ -176,15 +176,15 @@ class RetryableAPIClient:
last_exception = e
if attempt == self.max_retries:
logger.error(f"API调用在 {self.max_retries} 次重试后仍失败: {str(e)}")
logger.error(f"API call still failed after {self.max_retries} retries: {str(e)}")
raise
current_delay = min(delay, self.max_delay)
current_delay = current_delay * (0.5 + random.random())
logger.warning(
f"API调用第 {attempt + 1} 次尝试失败: {str(e)}, "
f"{current_delay:.1f}秒后重试..."
f"API callattempt {attempt + 1} failed: {str(e)}, "
f"{current_delay:.1f}s, retrying..."
)
time.sleep(current_delay)
@ -200,16 +200,16 @@ class RetryableAPIClient:
continue_on_failure: bool = True
) -> Tuple[list, list]:
"""
批量调用并对每个失败项单独重试
Batch call with per-item retry on failure
Args:
items: 要处理的项目列表
process_func: 处理函数，接收单个item作为参数
exceptions: 需要重试的异常类型
continue_on_failure: 单项失败后是否继续处理其他项
items: list of items to process
process_func: processing function, takes a single item
exceptions: exception types to retry on
continue_on_failure: whether to keep going after an individual item fails
Returns:
(成功结果列表, 失败项列表)
(list of successful results, list of failed items)
"""
results = []
failures = []
@ -224,7 +224,7 @@ class RetryableAPIClient:
results.append(result)
except Exception as e:
logger.error(f"处理第 {idx + 1} 项失败: {str(e)}")
logger.error(f"Processing item {idx + 1} failed: {str(e)}")
failures.append({
"index": idx,
"item": item,

157
backend/app/utils/zep_paging.py
View File

@ -1,143 +1,46 @@
"""Zep Graph 分页读取工具。
"""Graph pagination helpers (replaces zep_paging.py).
Zep 的 node/edge 列表接口使用 UUID cursor 分页，
本模块封装自动翻页逻辑（含单页重试），对调用方透明地返回完整列表。
FalkorDB returns nodes/edges in a single query (we cap at 5000 in the adapter),
so there's no real pagination here — but the call sites in graph_builder.py and
zep_entity_reader.py import `fetch_all_nodes` / `fetch_all_edges`, so we keep
those names as adapters over GraphitiAdapter.
"""
from __future__ import annotations
import time
from collections.abc import Callable
from typing import Any
from zep_cloud import InternalServerError
from zep_cloud.client import Zep
import logging
from typing import Any, List
from .logger import get_logger
logger = get_logger('mirofish.zep_paging')
_DEFAULT_PAGE_SIZE = 100
_MAX_NODES = 2000
_DEFAULT_MAX_RETRIES = 3
_DEFAULT_RETRY_DELAY = 2.0 # seconds, doubles each retry
def _fetch_page_with_retry(
api_call: Callable[..., list[Any]],
*args: Any,
max_retries: int = _DEFAULT_MAX_RETRIES,
retry_delay: float = _DEFAULT_RETRY_DELAY,
page_description: str = "page",
**kwargs: Any,
) -> list[Any]:
"""单页请求，失败时指数退避重试。仅重试网络/IO类瞬态错误。"""
if max_retries < 1:
raise ValueError("max_retries must be >= 1")
last_exception: Exception | None = None
delay = retry_delay
for attempt in range(max_retries):
try:
return api_call(*args, **kwargs)
except (ConnectionError, TimeoutError, OSError, InternalServerError) as e:
last_exception = e
if attempt < max_retries - 1:
logger.warning(
f"Zep {page_description} attempt {attempt + 1} failed: {str(e)[:100]}, retrying in {delay:.1f}s..."
)
time.sleep(delay)
delay *= 2
else:
logger.error(f"Zep {page_description} failed after {max_retries} attempts: {str(e)}")
assert last_exception is not None
raise last_exception
def fetch_all_nodes(
client: Zep,
client: Any, # ignored — kept for signature compat with Zep call sites
graph_id: str,
page_size: int = _DEFAULT_PAGE_SIZE,
max_items: int = _MAX_NODES,
max_retries: int = _DEFAULT_MAX_RETRIES,
retry_delay: float = _DEFAULT_RETRY_DELAY,
) -> list[Any]:
"""分页获取图谱节点，最多返回 max_items 条（默认 2000）。每页请求自带重试。"""
all_nodes: list[Any] = []
cursor: str | None = None
page_num = 0
while True:
kwargs: dict[str, Any] = {"limit": page_size}
if cursor is not None:
kwargs["uuid_cursor"] = cursor
page_num += 1
batch = _fetch_page_with_retry(
client.graph.node.get_by_graph_id,
graph_id,
max_retries=max_retries,
retry_delay=retry_delay,
page_description=f"fetch nodes page {page_num} (graph={graph_id})",
**kwargs,
)
if not batch:
break
all_nodes.extend(batch)
if len(all_nodes) >= max_items:
all_nodes = all_nodes[:max_items]
logger.warning(f"Node count reached limit ({max_items}), stopping pagination for graph {graph_id}")
break
if len(batch) < page_size:
break
cursor = getattr(batch[-1], "uuid_", None) or getattr(batch[-1], "uuid", None)
if cursor is None:
logger.warning(f"Node missing uuid field, stopping pagination at {len(all_nodes)} nodes")
break
return all_nodes
page_size: int = 100, # ignored
max_items: int = 2000,
max_retries: int = 3, # ignored
retry_delay: float = 2.0, # ignored
) -> List[Any]:
"""Return all nodes in `graph_id` (FalkorDB capped at max_items, default 2000)."""
from ..services.graphiti_service import get_graphiti_adapter
adapter = get_graphiti_adapter()
nodes = adapter.get_all_nodes(graph_id)
if len(nodes) > max_items:
logger.warning(f"Node count {len(nodes)} > max_items {max_items}, truncating")
nodes = nodes[:max_items]
return nodes
def fetch_all_edges(
client: Zep,
client: Any,
graph_id: str,
page_size: int = _DEFAULT_PAGE_SIZE,
max_retries: int = _DEFAULT_MAX_RETRIES,
retry_delay: float = _DEFAULT_RETRY_DELAY,
) -> list[Any]:
"""分页获取图谱所有边，返回完整列表。每页请求自带重试。"""
all_edges: list[Any] = []
cursor: str | None = None
page_num = 0
while True:
kwargs: dict[str, Any] = {"limit": page_size}
if cursor is not None:
kwargs["uuid_cursor"] = cursor
page_num += 1
batch = _fetch_page_with_retry(
client.graph.edge.get_by_graph_id,
graph_id,
max_retries=max_retries,
retry_delay=retry_delay,
page_description=f"fetch edges page {page_num} (graph={graph_id})",
**kwargs,
)
if not batch:
break
all_edges.extend(batch)
if len(batch) < page_size:
break
cursor = getattr(batch[-1], "uuid_", None) or getattr(batch[-1], "uuid", None)
if cursor is None:
logger.warning(f"Edge missing uuid field, stopping pagination at {len(all_edges)} edges")
break
return all_edges
page_size: int = 100,
max_retries: int = 3,
retry_delay: float = 2.0,
) -> List[Any]:
"""Return all edges in `graph_id`."""
from ..services.graphiti_service import get_graphiti_adapter
adapter = get_graphiti_adapter()
return adapter.get_all_edges(graph_id, include_temporal=True)

30
backend/pyproject.toml
View File

@ -1,6 +1,6 @@
[project]
name = "mirofish-backend"
version = "0.1.0"
version = "0.2.0"
description = "MiroFish - 简洁通用的群体智能引擎，预测万物"
requires-python = ">=3.11,<3.13"
license = { text = "AGPL-3.0" }
@ -16,11 +16,24 @@ dependencies = [
# LLM 相关
"openai>=1.0.0",
# Zep Cloud
"zep-cloud==3.13.0",
# 知识图谱 (replaces zep-cloud)
# Graphiti is the open-source engine behind Zep; runs server-side extraction
# using your LLM provider (MiniMax M3) and stores the graph in FalkorDB.
"graphiti-core[falkordb]>=0.20.0",
# falkordb is the official Python client for the graph backend
"falkordb>=1.0.0",
# sentence-transformers is the local embedding provider we use (FalkorDB
# doesn't bundle one, and MiniMax's bespoke embeddings API is rate-limited
# and not OpenAI-compatible)
"sentence-transformers>=3.0.0",
"torch>=2.0.0",
# OASIS 社交媒体模拟
"camel-oasis==0.2.5",
# Vendored fork — see vendor/camel-oasis/pyproject.toml for the rationale
# (the upstream package pins neo4j==5.23.0 which conflicts with
# graphiti-core[falkordb]'s neo4j>=5.26.0; our fork relaxes the pin since
# MiroFish's OASIS scripts never invoke the neo4j driver).
"camel-oasis @ file:///app/backend/vendor/camel-oasis",
"camel-ai==0.2.78",
# 文件处理
@ -30,7 +43,10 @@ dependencies = [
"chardet>=5.0.0",
# 工具库
# 环境变量加载
"python-dotenv>=1.0.0",
# 数据验证
"pydantic>=2.0.0",
]
@ -53,3 +69,9 @@ dev = [
[tool.hatch.build.targets.wheel]
packages = ["app"]
# Required because the deps list includes a direct-reference local path dep
# (camel-oasis @ file:///app/backend/vendor/camel-oasis). Without this,
# hatchling refuses to build the editable.
[tool.hatch.metadata]
allow-direct-references = true

19
backend/requirements.txt
View File

@ -13,12 +13,23 @@ flask-cors>=6.0.0
# OpenAI SDK（统一使用 OpenAI 格式调用 LLM）
openai>=1.0.0
# ============= Zep Cloud =============
zep-cloud==3.13.0
# ============= Knowledge Graph (replaces zep-cloud) =============
# Graphiti is the open-source engine behind Zep; runs server-side extraction
# using your LLM provider (MiniMax M3) and stores the graph in FalkorDB.
graphiti-core[falkordb]>=0.20.0
falkordb>=1.0.0
# sentence-transformers is the local embedding provider we use (FalkorDB
# doesn't bundle one, and MiniMax's bespoke embeddings API is rate-limited
# and not OpenAI-compatible)
sentence-transformers>=3.0.0
torch>=2.0.0
# ============= OASIS 社交媒体模拟 =============
# OASIS 社交模拟框架
camel-oasis==0.2.5
# Vendored fork — see vendor/camel-oasis/pyproject.toml for the rationale
# (the upstream package pins neo4j==5.23.0 which conflicts with
# graphiti-core[falkordb]'s neo4j>=5.26.0; our fork relaxes the pin since
# MiroFish's OASIS scripts never invoke the neo4j driver).
camel-oasis @ file:///app/backend/vendor/camel-oasis
camel-ai==0.2.78
# ============= 文件处理 =============

24
backend/run.py
View File

@ -1,21 +1,21 @@
"""
MiroFish Backend 启动入口
MiroFish backend entry point
"""
import os
import sys
# 解决 Windows 控制台中文乱码问题：在所有导入之前设置 UTF-8 编码
# Work around Windows console garbled Chinese output：Set UTF-8 encoding before any imports
if sys.platform == 'win32':
# 设置环境变量确保 Python 使用 UTF-8
# Set environment variable to ensure Python uses UTF-8
os.environ.setdefault('PYTHONIOENCODING', 'utf-8')
# 重新配置标准输出流为 UTF-8
# Reconfigure stdout to UTF-8
if hasattr(sys.stdout, 'reconfigure'):
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
if hasattr(sys.stderr, 'reconfigure'):
sys.stderr.reconfigure(encoding='utf-8', errors='replace')
# 添加项目根目录到路径
# Add the project root to sys.path
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from app import create_app
@ -23,25 +23,25 @@ from app.config import Config
def main():
"""主函数"""
# 验证配置
"""Main entry point"""
# Validate configuration
errors = Config.validate()
if errors:
print("配置错误:")
print("Configuration error:")
for err in errors:
print(f" - {err}")
print("\n请检查 .env 文件中的配置")
print("\nPlease check the configuration in .env")
sys.exit(1)
# 创建应用
# Create app
app = create_app()
# 获取运行配置
# Read runtime config
host = os.environ.get('FLASK_HOST', '0.0.0.0')
port = int(os.environ.get('FLASK_PORT', 5001))
debug = Config.DEBUG
# 启动服务
# Start the server
app.run(host=host, port=port, debug=debug, threaded=True)

110
backend/scripts/action_logger.py
View File

@ -1,15 +1,15 @@
"""
动作日志记录器
用于记录OASIS模拟中每个Agent的动作，供后端监控使用
Action logger
Records every Agent's actions in the OASIS simulation for backend monitoring.
日志结构:
Log structure:
sim_xxx/
├── twitter/
│ └── actions.jsonl # Twitter 平台动作日志
│ └── actions.jsonl # Twitter platform action log
├── reddit/
│ └── actions.jsonl # Reddit 平台动作日志
├── simulation.log # 主模拟进程日志
└── run_state.json # 运行状态（API 查询用）
│ └── actions.jsonl # Reddit platform action log
├── simulation.log # Main simulation process log
└── run_state.json # Run state (used for API queries)
"""
import json
@ -20,26 +20,26 @@ from typing import Dict, Any, Optional
class PlatformActionLogger:
"""单平台动作日志记录器"""
"""Single-platform action logger"""
def __init__(self, platform: str, base_dir: str):
"""
初始化日志记录器
Initialize the logger.
Args:
platform: 平台名称 (twitter/reddit)
base_dir: 模拟目录的基础路径
platform: Platform name (twitter/reddit)
base_dir: Base path of the simulation directory
"""
self.platform = platform
self.base_dir = base_dir
self.log_dir = os.path.join(base_dir, platform)
self.log_path = os.path.join(self.log_dir, "actions.jsonl")
self._ensure_dir()
def _ensure_dir(self):
"""确保目录存在"""
"""Ensure the directory exists"""
os.makedirs(self.log_dir, exist_ok=True)
def log_action(
self,
round_num: int,
@ -50,7 +50,7 @@ class PlatformActionLogger:
result: Optional[str] = None,
success: bool = True
):
"""记录一个动作"""
"""Record a single action"""
entry = {
"round": round_num,
"timestamp": datetime.now().isoformat(),
@ -66,31 +66,31 @@ class PlatformActionLogger:
f.write(json.dumps(entry, ensure_ascii=False) + '\n')
def log_round_start(self, round_num: int, simulated_hour: int):
"""记录轮次开始"""
"""Record round start"""
entry = {
"round": round_num,
"timestamp": datetime.now().isoformat(),
"event_type": "round_start",
"simulated_hour": simulated_hour,
}
with open(self.log_path, 'a', encoding='utf-8') as f:
f.write(json.dumps(entry, ensure_ascii=False) + '\n')
def log_round_end(self, round_num: int, actions_count: int):
"""记录轮次结束"""
"""Record round end"""
entry = {
"round": round_num,
"timestamp": datetime.now().isoformat(),
"event_type": "round_end",
"actions_count": actions_count,
}
with open(self.log_path, 'a', encoding='utf-8') as f:
f.write(json.dumps(entry, ensure_ascii=False) + '\n')
def log_simulation_start(self, config: Dict[str, Any]):
"""记录模拟开始"""
"""Record simulation start"""
entry = {
"timestamp": datetime.now().isoformat(),
"event_type": "simulation_start",
@ -98,12 +98,12 @@ class PlatformActionLogger:
"total_rounds": config.get("time_config", {}).get("total_simulation_hours", 72) * 2,
"agents_count": len(config.get("agent_configs", [])),
}
with open(self.log_path, 'a', encoding='utf-8') as f:
f.write(json.dumps(entry, ensure_ascii=False) + '\n')
def log_simulation_end(self, total_rounds: int, total_actions: int):
"""记录模拟结束"""
"""Record simulation end"""
entry = {
"timestamp": datetime.now().isoformat(),
"event_type": "simulation_end",
@ -118,35 +118,35 @@ class PlatformActionLogger:
class SimulationLogManager:
"""
模拟日志管理器
统一管理所有日志文件，按平台分离
Simulation log manager
Unifies management of all log files, separated by platform.
"""
def __init__(self, simulation_dir: str):
"""
初始化日志管理器
Initialize the log manager.
Args:
simulation_dir: 模拟目录路径
simulation_dir: Path of the simulation directory
"""
self.simulation_dir = simulation_dir
self.twitter_logger: Optional[PlatformActionLogger] = None
self.reddit_logger: Optional[PlatformActionLogger] = None
self._main_logger: Optional[logging.Logger] = None
# 设置主日志
# Set up the main logger
self._setup_main_logger()
def _setup_main_logger(self):
"""设置主模拟日志"""
"""Set up the main simulation logger"""
log_path = os.path.join(self.simulation_dir, "simulation.log")
# 创建 logger
# Create the logger
self._main_logger = logging.getLogger(f"simulation.{os.path.basename(self.simulation_dir)}")
self._main_logger.setLevel(logging.INFO)
self._main_logger.handlers.clear()
# 文件处理器
# File handler
file_handler = logging.FileHandler(log_path, encoding='utf-8', mode='w')
file_handler.setLevel(logging.INFO)
file_handler.setFormatter(logging.Formatter(
@ -154,8 +154,8 @@ class SimulationLogManager:
datefmt='%Y-%m-%d %H:%M:%S'
))
self._main_logger.addHandler(file_handler)
# 控制台处理器
# Console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_handler.setFormatter(logging.Formatter(
@ -163,23 +163,23 @@ class SimulationLogManager:
datefmt='%H:%M:%S'
))
self._main_logger.addHandler(console_handler)
self._main_logger.propagate = False
def get_twitter_logger(self) -> PlatformActionLogger:
"""获取 Twitter 平台日志记录器"""
"""Get the Twitter platform logger"""
if self.twitter_logger is None:
self.twitter_logger = PlatformActionLogger("twitter", self.simulation_dir)
return self.twitter_logger
def get_reddit_logger(self) -> PlatformActionLogger:
"""获取 Reddit 平台日志记录器"""
"""Get the Reddit platform logger"""
if self.reddit_logger is None:
self.reddit_logger = PlatformActionLogger("reddit", self.simulation_dir)
return self.reddit_logger
def log(self, message: str, level: str = "info"):
"""记录主日志"""
"""Record into the main log"""
if self._main_logger:
getattr(self._main_logger, level.lower(), self._main_logger.info)(message)
@ -196,12 +196,12 @@ class SimulationLogManager:
self.log(message, "debug")
# ============ 兼容旧接口 ============
# ============ Backward-compat interface ============
class ActionLogger:
"""
动作日志记录器（兼容旧接口）
建议使用 SimulationLogManager 代替
Action logger (backward-compat interface)
Prefer using SimulationLogManager instead.
"""
def __init__(self, log_path: str):
@ -288,12 +288,12 @@ class ActionLogger:
f.write(json.dumps(entry, ensure_ascii=False) + '\n')
# 全局日志实例（兼容旧接口）
# Global log instance (backward-compat interface)
_global_logger: Optional[ActionLogger] = None
def get_logger(log_path: Optional[str] = None) -> ActionLogger:
"""获取全局日志实例（兼容旧接口）"""
"""Get the global log instance (backward-compat interface)"""
global _global_logger
if log_path:

620
backend/scripts/run_parallel_simulation.py
View File

File diff suppressed because it is too large Load Diff

260
backend/scripts/run_reddit_simulation.py
View File

@ -1,16 +1,16 @@
"""
OASIS Reddit模拟预设脚本
此脚本读取配置文件中的参数来执行模拟，实现全程自动化
OASIS Reddit simulation preset script
This script reads parameters from a config file to run simulations, fully automated end-to-end
功能特性:
- 完成模拟后不立即关闭环境，进入等待命令模式
- 支持通过IPC接收Interview命令
- 支持单个Agent采访和批量采访
- 支持远程关闭环境命令
Features:
- After completing the simulation, do not close the environment immediately; enter wait-for-commands mode
- Support receiving Interview commands via IPC
- Support single-Agent and batch interviews
- Support remote environment-close commands
使用方式:
Usage:
python run_reddit_simulation.py --config /path/to/simulation_config.json
python run_reddit_simulation.py --config /path/to/simulation_config.json --no-wait # 完成后立即关闭
python run_reddit_simulation.py --config /path/to/simulation_config.json --no-wait # close immediately after completion
"""
import argparse
@ -25,18 +25,18 @@ import sqlite3
from datetime import datetime
from typing import Dict, Any, List, Optional
# 全局变量：用于信号处理
# Global variables: for signal handling
_shutdown_event = None
_cleanup_done = False
# 添加项目路径
# Add project paths
_scripts_dir = os.path.dirname(os.path.abspath(__file__))
_backend_dir = os.path.abspath(os.path.join(_scripts_dir, '..'))
_project_root = os.path.abspath(os.path.join(_backend_dir, '..'))
sys.path.insert(0, _scripts_dir)
sys.path.insert(0, _backend_dir)
# 加载项目根目录的 .env 文件（包含 LLM_API_KEY 等配置）
# Load project-root .env file (contains LLM_API_KEY etc.)
from dotenv import load_dotenv
_env_file = os.path.join(_project_root, '.env')
if os.path.exists(_env_file):
@ -51,7 +51,7 @@ import re
class UnicodeFormatter(logging.Formatter):
"""自定义格式化器，将 Unicode 转义序列转换为可读字符"""
"""Custom formatter that converts Unicode escape sequences into readable characters"""
UNICODE_ESCAPE_PATTERN = re.compile(r'\\u([0-9a-fA-F]{4})')
@ -68,24 +68,24 @@ class UnicodeFormatter(logging.Formatter):
class MaxTokensWarningFilter(logging.Filter):
"""过滤掉 camel-ai 关于 max_tokens 的警告（我们故意不设置 max_tokens，让模型自行决定）"""
"""Filter out camel-ai warnings about max_tokens (we intentionally do not set max_tokens, letting the model decide)"""
def filter(self, record):
# 过滤掉包含 max_tokens 警告的日志
# Filter out log records containing the max_tokens warning
if "max_tokens" in record.getMessage() and "Invalid or missing" in record.getMessage():
return False
return True
# 在模块加载时立即添加过滤器，确保在 camel 代码执行前生效
# Add the filter at module load time, ensuring it takes effect before camel code runs
logging.getLogger().addFilter(MaxTokensWarningFilter())
def setup_oasis_logging(log_dir: str):
"""配置 OASIS 的日志，使用固定名称的日志文件"""
"""Configure OASIS logging using fixed-name log files"""
os.makedirs(log_dir, exist_ok=True)
# 清理旧的日志文件
# Clean up old log files
for f in os.listdir(log_dir):
old_log = os.path.join(log_dir, f)
if os.path.isfile(old_log) and f.endswith('.log'):
@ -126,25 +126,25 @@ try:
generate_reddit_agent_graph
)
except ImportError as e:
print(f"错误: 缺少依赖 {e}")
print("请先安装: pip install oasis-ai camel-ai")
print(f"Error: missing dependency {e}")
print("Please install first: pip install oasis-ai camel-ai")
sys.exit(1)
# IPC相关常量
# IPC-related constants
IPC_COMMANDS_DIR = "ipc_commands"
IPC_RESPONSES_DIR = "ipc_responses"
ENV_STATUS_FILE = "env_status.json"
class CommandType:
"""命令类型常量"""
"""Command type constants"""
INTERVIEW = "interview"
BATCH_INTERVIEW = "batch_interview"
CLOSE_ENV = "close_env"
class IPCHandler:
"""IPC命令处理器"""
"""IPC command handler"""
def __init__(self, simulation_dir: str, env, agent_graph):
self.simulation_dir = simulation_dir
@ -155,12 +155,12 @@ class IPCHandler:
self.status_file = os.path.join(simulation_dir, ENV_STATUS_FILE)
self._running = True
# 确保目录存在
# Ensure the directory exists
os.makedirs(self.commands_dir, exist_ok=True)
os.makedirs(self.responses_dir, exist_ok=True)
def update_status(self, status: str):
"""更新环境状态"""
"""Update environment status"""
with open(self.status_file, 'w', encoding='utf-8') as f:
json.dump({
"status": status,
@ -168,11 +168,11 @@ class IPCHandler:
}, f, ensure_ascii=False, indent=2)
def poll_command(self) -> Optional[Dict[str, Any]]:
"""轮询获取待处理命令"""
"""Poll for pending commands"""
if not os.path.exists(self.commands_dir):
return None
# 获取命令文件（按时间排序）
# Get command files (sorted by time)
command_files = []
for filename in os.listdir(self.commands_dir):
if filename.endswith('.json'):
@ -191,7 +191,7 @@ class IPCHandler:
return None
def send_response(self, command_id: str, status: str, result: Dict = None, error: str = None):
"""发送响应"""
"""Send response"""
response = {
"command_id": command_id,
"status": status,
@ -204,7 +204,7 @@ class IPCHandler:
with open(response_file, 'w', encoding='utf-8') as f:
json.dump(response, f, ensure_ascii=False, indent=2)
# 删除命令文件
# Delete command file
command_file = os.path.join(self.commands_dir, f"{command_id}.json")
try:
os.remove(command_file)
@ -213,49 +213,49 @@ class IPCHandler:
async def handle_interview(self, command_id: str, agent_id: int, prompt: str) -> bool:
"""
处理单个Agent采访命令
Handle single-Agent interview command
Returns:
True 表示成功，False 表示失败
True on success, False on failure
"""
try:
# 获取Agent
# Get Agent
agent = self.agent_graph.get_agent(agent_id)
# 创建Interview动作
# Create Interview action
interview_action = ManualAction(
action_type=ActionType.INTERVIEW,
action_args={"prompt": prompt}
)
# 执行Interview
# Execute Interview
actions = {agent: interview_action}
await self.env.step(actions)
# 从数据库获取结果
# Get result from database
result = self._get_interview_result(agent_id)
self.send_response(command_id, "completed", result=result)
print(f" Interview完成: agent_id={agent_id}")
print(f" Interview completed: agent_id={agent_id}")
return True
except Exception as e:
error_msg = str(e)
print(f" Interview失败: agent_id={agent_id}, error={error_msg}")
print(f" Interview failed: agent_id={agent_id}, error={error_msg}")
self.send_response(command_id, "failed", error=error_msg)
return False
async def handle_batch_interview(self, command_id: str, interviews: List[Dict]) -> bool:
"""
处理批量采访命令
Handle batch interview command
Args:
interviews: [{"agent_id": int, "prompt": str}, ...]
"""
try:
# 构建动作字典
# Build action dict
actions = {}
agent_prompts = {} # 记录每个agent的prompt
agent_prompts = {} # Record each agent's prompt
for interview in interviews:
agent_id = interview.get("agent_id")
@ -269,16 +269,16 @@ class IPCHandler:
)
agent_prompts[agent_id] = prompt
except Exception as e:
print(f" 警告: 无法获取Agent {agent_id}: {e}")
print(f" Warning: unable to fetch Agent {agent_id}: {e}")
if not actions:
self.send_response(command_id, "failed", error="没有有效的Agent")
self.send_response(command_id, "failed", error="No valid agents")
return False
# 执行批量Interview
# Execute batch Interview
await self.env.step(actions)
# 获取所有结果
# Get all results
results = {}
for agent_id in agent_prompts.keys():
result = self._get_interview_result(agent_id)
@ -288,17 +288,17 @@ class IPCHandler:
"interviews_count": len(results),
"results": results
})
print(f" 批量Interview完成: {len(results)} 个Agent")
print(f" Batch Interview completed: {len(results)} agents")
return True
except Exception as e:
error_msg = str(e)
print(f" 批量Interview失败: {error_msg}")
print(f" Batch Interview failed: {error_msg}")
self.send_response(command_id, "failed", error=error_msg)
return False
def _get_interview_result(self, agent_id: int) -> Dict[str, Any]:
"""从数据库获取最新的Interview结果"""
"""Get the latest Interview results from the database"""
db_path = os.path.join(self.simulation_dir, "reddit_simulation.db")
result = {
@ -314,7 +314,7 @@ class IPCHandler:
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
# 查询最新的Interview记录
# Query the latest Interview record
cursor.execute("""
SELECT user_id, info, created_at
FROM trace
@ -336,16 +336,16 @@ class IPCHandler:
conn.close()
except Exception as e:
print(f" 读取Interview结果失败: {e}")
print(f" Failed to read Interview results: {e}")
return result
async def process_commands(self) -> bool:
"""
处理所有待处理命令
Handle all pending commands
Returns:
True 表示继续运行，False 表示应该退出
True to continue running, False to exit
"""
command = self.poll_command()
if not command:
@ -355,7 +355,7 @@ class IPCHandler:
command_type = command.get("command_type")
args = command.get("args", {})
print(f"\n收到IPC命令: {command_type}, id={command_id}")
print(f"\nReceived IPC command: {command_type}, id={command_id}")
if command_type == CommandType.INTERVIEW:
await self.handle_interview(
@ -373,19 +373,19 @@ class IPCHandler:
return True
elif command_type == CommandType.CLOSE_ENV:
print("收到关闭环境命令")
self.send_response(command_id, "completed", result={"message": "环境即将关闭"})
print("Received close-environment command")
self.send_response(command_id, "completed", result={"message": "Environment is about to close"})
return False
else:
self.send_response(command_id, "failed", error=f"未知命令类型: {command_type}")
self.send_response(command_id, "failed", error=f"Unknown command type: {command_type}")
return True
class RedditSimulationRunner:
"""Reddit模拟运行器"""
"""Reddit simulation runner"""
# Reddit可用动作（不包含INTERVIEW，INTERVIEW只能通过ManualAction手动触发）
# Reddit available actions (does not include INTERVIEW; INTERVIEW can only be triggered manually via ManualAction)
AVAILABLE_ACTIONS = [
ActionType.LIKE_POST,
ActionType.DISLIKE_POST,
@ -404,11 +404,11 @@ class RedditSimulationRunner:
def __init__(self, config_path: str, wait_for_commands: bool = True):
"""
初始化模拟运行器
Initialize the simulation runner
Args:
config_path: 配置文件路径 (simulation_config.json)
wait_for_commands: 模拟完成后是否等待命令（默认True）
config_path: Configuration file path (simulation_config.json)
wait_for_commands: Whether to wait for commands after the simulation completes (default True)
"""
self.config_path = config_path
self.config = self._load_config()
@ -419,47 +419,47 @@ class RedditSimulationRunner:
self.ipc_handler = None
def _load_config(self) -> Dict[str, Any]:
"""加载配置文件"""
"""Load configuration file"""
with open(self.config_path, 'r', encoding='utf-8') as f:
return json.load(f)
def _get_profile_path(self) -> str:
"""获取Profile文件路径"""
"""Get Profile file path"""
return os.path.join(self.simulation_dir, "reddit_profiles.json")
def _get_db_path(self) -> str:
"""获取数据库路径"""
"""Get database path"""
return os.path.join(self.simulation_dir, "reddit_simulation.db")
def _create_model(self):
"""
创建LLM模型
Create the LLM model
统一使用项目根目录 .env 文件中的配置（优先级最高）：
- LLM_API_KEY: API密钥
- LLM_BASE_URL: API基础URL
- LLM_MODEL_NAME: 模型名称
Use config from project-root .env file (highest priority):
- LLM_API_KEY: API key
- LLM_BASE_URL: API base URL
- LLM_MODEL_NAME: Model name
"""
# 优先从 .env 读取配置
# Prefer reading config from .env
llm_api_key = os.environ.get("LLM_API_KEY", "")
llm_base_url = os.environ.get("LLM_BASE_URL", "")
llm_model = os.environ.get("LLM_MODEL_NAME", "")
# 如果 .env 中没有，则使用 config 作为备用
# If .env has no value, fall back to config
if not llm_model:
llm_model = self.config.get("llm_model", "gpt-4o-mini")
# 设置 camel-ai 所需的环境变量
# Set environment variables required by camel-ai
if llm_api_key:
os.environ["OPENAI_API_KEY"] = llm_api_key
if not os.environ.get("OPENAI_API_KEY"):
raise ValueError("缺少 API Key 配置，请在项目根目录 .env 文件中设置 LLM_API_KEY")
raise ValueError("Missing API Key config, please set LLM_API_KEY in the project root .env file")
if llm_base_url:
os.environ["OPENAI_API_BASE_URL"] = llm_base_url
print(f"LLM配置: model={llm_model}, base_url={llm_base_url[:40] if llm_base_url else '默认'}...")
print(f"LLM config: model={llm_model}, base_url={llm_base_url[:40] if llm_base_url else 'default'}...")
return ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
@ -473,7 +473,7 @@ class RedditSimulationRunner:
round_num: int
) -> List:
"""
根据时间和配置决定本轮激活哪些Agent
Determine which Agents to activate this round based on time and config
"""
time_config = self.config.get("time_config", {})
agent_configs = self.config.get("agent_configs", [])
@ -521,16 +521,16 @@ class RedditSimulationRunner:
return active_agents
async def run(self, max_rounds: int = None):
"""运行Reddit模拟
"""Run Reddit simulation
Args:
max_rounds: 最大模拟轮数（可选，用于截断过长的模拟）
max_rounds: Maximum simulation rounds (optional, used to truncate overlong simulations)
"""
print("=" * 60)
print("OASIS Reddit模拟")
print(f"配置文件: {self.config_path}")
print(f"模拟ID: {self.config.get('simulation_id', 'unknown')}")
print(f"等待命令模式: {'启用' if self.wait_for_commands else '禁用'}")
print("OASIS Reddit Simulation")
print(f"Config file: {self.config_path}")
print(f"Simulation ID: {self.config.get('simulation_id', 'unknown')}")
print(f"Wait-for-commands mode: {'enabled' if self.wait_for_commands else 'disabled'}")
print("=" * 60)
time_config = self.config.get("time_config", {})
@ -538,28 +538,28 @@ class RedditSimulationRunner:
minutes_per_round = time_config.get("minutes_per_round", 30)
total_rounds = (total_hours * 60) // minutes_per_round
# 如果指定了最大轮数，则截断
# If a max-rounds is specified, truncate
if max_rounds is not None and max_rounds > 0:
original_rounds = total_rounds
total_rounds = min(total_rounds, max_rounds)
if total_rounds < original_rounds:
print(f"\n轮数已截断: {original_rounds} -> {total_rounds} (max_rounds={max_rounds})")
print(f"\nRounds truncated: {original_rounds} -> {total_rounds} (max_rounds={max_rounds})")
print(f"\n模拟参数:")
print(f" - 总模拟时长: {total_hours}小时")
print(f" - 每轮时间: {minutes_per_round}分钟")
print(f" - 总轮数: {total_rounds}")
print(f"\nSimulation parameters:")
print(f" - Total simulation duration: {total_hours} hours")
print(f" - Time per round: {minutes_per_round} minutes")
print(f" - Total rounds: {total_rounds}")
if max_rounds:
print(f" - 最大轮数限制: {max_rounds}")
print(f" - Agent数量: {len(self.config.get('agent_configs', []))}")
print(f" - Max rounds limit: {max_rounds}")
print(f" - Number of Agents: {len(self.config.get('agent_configs', []))}")
print("\n初始化LLM模型...")
print("\nInitializing LLM model...")
model = self._create_model()
print("加载Agent Profile...")
print("Loading Agent Profile...")
profile_path = self._get_profile_path()
if not os.path.exists(profile_path):
print(f"错误: Profile文件不存在: {profile_path}")
print(f"Error: Profile file does not exist: {profile_path}")
return
self.agent_graph = await generate_reddit_agent_graph(
@ -571,29 +571,29 @@ class RedditSimulationRunner:
db_path = self._get_db_path()
if os.path.exists(db_path):
os.remove(db_path)
print(f"已删除旧数据库: {db_path}")
print(f"Deleted old database: {db_path}")
print("创建OASIS环境...")
print("Creating OASIS environment...")
self.env = oasis.make(
agent_graph=self.agent_graph,
platform=oasis.DefaultPlatformType.REDDIT,
database_path=db_path,
semaphore=30, # 限制最大并发 LLM 请求数，防止 API 过载
semaphore=30, # Limit max concurrent LLM requests to prevent API overload
)
await self.env.reset()
print("环境初始化完成\n")
print("Environment initialization complete\n")
# 初始化IPC处理器
# Initialize IPC handler
self.ipc_handler = IPCHandler(self.simulation_dir, self.env, self.agent_graph)
self.ipc_handler.update_status("running")
# 执行初始事件
# Execute initial events
event_config = self.config.get("event_config", {})
initial_posts = event_config.get("initial_posts", [])
if initial_posts:
print(f"执行初始事件 ({len(initial_posts)}条初始帖子)...")
print(f"Executing initial events ({len(initial_posts)} initial posts)...")
initial_actions = {}
for post in initial_posts:
agent_id = post.get("poster_agent_id", 0)
@ -613,14 +613,14 @@ class RedditSimulationRunner:
action_args={"content": content}
)
except Exception as e:
print(f" 警告: 无法为Agent {agent_id}创建初始帖子: {e}")
print(f" Warning: unable to create initial post for Agent {agent_id}: {e}")
if initial_actions:
await self.env.step(initial_actions)
print(f" 已发布 {len(initial_actions)} 条初始帖子")
print(f" Published {len(initial_actions)} initial posts")
# 主模拟循环
print("\n开始模拟循环...")
# Main simulation loop
print("\nStarting simulation loop...")
start_time = datetime.now()
for round_num in range(total_rounds):
@ -651,20 +651,20 @@ class RedditSimulationRunner:
f"- elapsed: {elapsed:.1f}s")
total_elapsed = (datetime.now() - start_time).total_seconds()
print(f"\n模拟循环完成!")
print(f" - 总耗时: {total_elapsed:.1f}秒")
print(f" - 数据库: {db_path}")
print(f"\nSimulation loop completed!")
print(f" - Total elapsed: {total_elapsed:.1f} seconds")
print(f" - Database: {db_path}")
# 是否进入等待命令模式
# Whether to enter wait-for-commands mode
if self.wait_for_commands:
print("\n" + "=" * 60)
print("进入等待命令模式 - 环境保持运行")
print("支持的命令: interview, batch_interview, close_env")
print("Entering wait-for-commands mode - environment keeps running")
print("Supported commands: interview, batch_interview, close_env")
print("=" * 60)
self.ipc_handler.update_status("alive")
# 等待命令循环（使用全局 _shutdown_event）
# Wait-for-commands loop (using global _shutdown_event)
try:
while not _shutdown_event.is_set():
should_continue = await self.ipc_handler.process_commands()
@ -672,58 +672,58 @@ class RedditSimulationRunner:
break
try:
await asyncio.wait_for(_shutdown_event.wait(), timeout=0.5)
break # 收到退出信号
break # Received exit signal
except asyncio.TimeoutError:
pass
except KeyboardInterrupt:
print("\n收到中断信号")
print("\nReceived interrupt signal")
except asyncio.CancelledError:
print("\n任务被取消")
print("\nTask cancelled")
except Exception as e:
print(f"\n命令处理出错: {e}")
print(f"\nCommand processing error: {e}")
print("\n关闭环境...")
print("\nClosing environment...")
# 关闭环境
# Close the environment
self.ipc_handler.update_status("stopped")
await self.env.close()
print("环境已关闭")
print("Environment closed")
print("=" * 60)
async def main():
parser = argparse.ArgumentParser(description='OASIS Reddit模拟')
parser = argparse.ArgumentParser(description='OASIS Reddit Simulation')
parser.add_argument(
'--config',
type=str,
required=True,
help='配置文件路径 (simulation_config.json)'
help='Configuration file path (simulation_config.json)'
)
parser.add_argument(
'--max-rounds',
type=int,
default=None,
help='最大模拟轮数（可选，用于截断过长的模拟）'
help='Maximum simulation rounds (optional, used to truncate overlong simulations)'
)
parser.add_argument(
'--no-wait',
action='store_true',
default=False,
help='模拟完成后立即关闭环境，不进入等待命令模式'
help='Close environment immediately after simulation completes, do not enter wait-for-commands mode'
)
args = parser.parse_args()
# 在 main 函数开始时创建 shutdown 事件
# Create shutdown event at the start of main()
global _shutdown_event
_shutdown_event = asyncio.Event()
if not os.path.exists(args.config):
print(f"错误: 配置文件不存在: {args.config}")
print(f"Error: configuration file does not exist: {args.config}")
sys.exit(1)
# 初始化日志配置（使用固定文件名，清理旧日志）
# Initialize log config (use fixed log file names, clean up old logs)
simulation_dir = os.path.dirname(args.config) or "."
setup_oasis_logging(os.path.join(simulation_dir, "log"))
@ -736,20 +736,20 @@ async def main():
def setup_signal_handlers():
"""
设置信号处理器，确保收到 SIGTERM/SIGINT 时能够正确退出
让程序有机会正常清理资源（关闭数据库、环境等）
Set up signal handlers to ensure clean exit on SIGTERM/SIGINT
Give the program a chance to clean up resources (close DB, env, etc.)
"""
def signal_handler(signum, frame):
global _cleanup_done
sig_name = "SIGTERM" if signum == signal.SIGTERM else "SIGINT"
print(f"\n收到 {sig_name} 信号，正在退出...")
print(f"\nReceived {sig_name} signal, exiting...")
if not _cleanup_done:
_cleanup_done = True
if _shutdown_event:
_shutdown_event.set()
else:
# 重复收到信号才强制退出
print("强制退出...")
# Only force exit if the signal is received again
print("Forced exit...")
sys.exit(1)
signal.signal(signal.SIGTERM, signal_handler)
@ -761,9 +761,9 @@ if __name__ == "__main__":
try:
asyncio.run(main())
except KeyboardInterrupt:
print("\n程序被中断")
print("\nProgram interrupted")
except SystemExit:
pass
finally:
print("模拟进程已退出")
print("Simulation process exited")

304
backend/scripts/run_twitter_simulation.py
View File

@ -1,16 +1,16 @@
"""
OASIS Twitter模拟预设脚本
此脚本读取配置文件中的参数来执行模拟，实现全程自动化
OASIS Twitter simulation preset script
This script reads parameters from a config file to run simulations, fully automated end-to-end
功能特性:
- 完成模拟后不立即关闭环境，进入等待命令模式
- 支持通过IPC接收Interview命令
- 支持单个Agent采访和批量采访
- 支持远程关闭环境命令
Features:
- After completing the simulation, do not close the environment immediately; enter wait-for-commands mode
- Support receiving Interview commands via IPC
- Support single-Agent and batch interviews
- Support remote environment-close commands
使用方式:
Usage:
python run_twitter_simulation.py --config /path/to/simulation_config.json
python run_twitter_simulation.py --config /path/to/simulation_config.json --no-wait # 完成后立即关闭
python run_twitter_simulation.py --config /path/to/simulation_config.json --no-wait # close immediately after completion
"""
import argparse
@ -25,18 +25,18 @@ import sqlite3
from datetime import datetime
from typing import Dict, Any, List, Optional
# 全局变量：用于信号处理
# Global variables: for signal handling
_shutdown_event = None
_cleanup_done = False
# 添加项目路径
# Add project paths
_scripts_dir = os.path.dirname(os.path.abspath(__file__))
_backend_dir = os.path.abspath(os.path.join(_scripts_dir, '..'))
_project_root = os.path.abspath(os.path.join(_backend_dir, '..'))
sys.path.insert(0, _scripts_dir)
sys.path.insert(0, _backend_dir)
# 加载项目根目录的 .env 文件（包含 LLM_API_KEY 等配置）
# Load project-root .env file (contains LLM_API_KEY etc.)
from dotenv import load_dotenv
_env_file = os.path.join(_project_root, '.env')
if os.path.exists(_env_file):
@ -51,7 +51,7 @@ import re
class UnicodeFormatter(logging.Formatter):
"""自定义格式化器，将 Unicode 转义序列转换为可读字符"""
"""Custom formatter that converts Unicode escape sequences into readable characters"""
UNICODE_ESCAPE_PATTERN = re.compile(r'\\u([0-9a-fA-F]{4})')
@ -68,24 +68,24 @@ class UnicodeFormatter(logging.Formatter):
class MaxTokensWarningFilter(logging.Filter):
"""过滤掉 camel-ai 关于 max_tokens 的警告（我们故意不设置 max_tokens，让模型自行决定）"""
"""Filter out camel-ai warnings about max_tokens (we intentionally do not set max_tokens, letting the model decide)"""
def filter(self, record):
# 过滤掉包含 max_tokens 警告的日志
# Filter out log records containing the max_tokens warning
if "max_tokens" in record.getMessage() and "Invalid or missing" in record.getMessage():
return False
return True
# 在模块加载时立即添加过滤器，确保在 camel 代码执行前生效
# Add the filter at module load time, ensuring it takes effect before camel code runs
logging.getLogger().addFilter(MaxTokensWarningFilter())
def setup_oasis_logging(log_dir: str):
"""配置 OASIS 的日志，使用固定名称的日志文件"""
"""Configure OASIS logging using fixed-name log files"""
os.makedirs(log_dir, exist_ok=True)
# 清理旧的日志文件
# Clean up old log files
for f in os.listdir(log_dir):
old_log = os.path.join(log_dir, f)
if os.path.isfile(old_log) and f.endswith('.log'):
@ -126,25 +126,25 @@ try:
generate_twitter_agent_graph
)
except ImportError as e:
print(f"错误: 缺少依赖 {e}")
print("请先安装: pip install oasis-ai camel-ai")
print(f"Error: missing dependency {e}")
print("Please install first: pip install oasis-ai camel-ai")
sys.exit(1)
# IPC相关常量
# IPC-related constants
IPC_COMMANDS_DIR = "ipc_commands"
IPC_RESPONSES_DIR = "ipc_responses"
ENV_STATUS_FILE = "env_status.json"
class CommandType:
"""命令类型常量"""
"""Command type constants"""
INTERVIEW = "interview"
BATCH_INTERVIEW = "batch_interview"
CLOSE_ENV = "close_env"
class IPCHandler:
"""IPC命令处理器"""
"""IPC command handler"""
def __init__(self, simulation_dir: str, env, agent_graph):
self.simulation_dir = simulation_dir
@ -155,12 +155,12 @@ class IPCHandler:
self.status_file = os.path.join(simulation_dir, ENV_STATUS_FILE)
self._running = True
# 确保目录存在
# Ensure the directory exists
os.makedirs(self.commands_dir, exist_ok=True)
os.makedirs(self.responses_dir, exist_ok=True)
def update_status(self, status: str):
"""更新环境状态"""
"""Update environment status"""
with open(self.status_file, 'w', encoding='utf-8') as f:
json.dump({
"status": status,
@ -168,11 +168,11 @@ class IPCHandler:
}, f, ensure_ascii=False, indent=2)
def poll_command(self) -> Optional[Dict[str, Any]]:
"""轮询获取待处理命令"""
"""Poll for pending commands"""
if not os.path.exists(self.commands_dir):
return None
# 获取命令文件（按时间排序）
# Get command files (sorted by time)
command_files = []
for filename in os.listdir(self.commands_dir):
if filename.endswith('.json'):
@ -191,7 +191,7 @@ class IPCHandler:
return None
def send_response(self, command_id: str, status: str, result: Dict = None, error: str = None):
"""发送响应"""
"""Send response"""
response = {
"command_id": command_id,
"status": status,
@ -204,7 +204,7 @@ class IPCHandler:
with open(response_file, 'w', encoding='utf-8') as f:
json.dump(response, f, ensure_ascii=False, indent=2)
# 删除命令文件
# Delete command file
command_file = os.path.join(self.commands_dir, f"{command_id}.json")
try:
os.remove(command_file)
@ -213,49 +213,49 @@ class IPCHandler:
async def handle_interview(self, command_id: str, agent_id: int, prompt: str) -> bool:
"""
处理单个Agent采访命令
Handle single-Agent interview command
Returns:
True 表示成功，False 表示失败
True on success, False on failure
"""
try:
# 获取Agent
# Get Agent
agent = self.agent_graph.get_agent(agent_id)
# 创建Interview动作
# Create Interview action
interview_action = ManualAction(
action_type=ActionType.INTERVIEW,
action_args={"prompt": prompt}
)
# 执行Interview
# Execute Interview
actions = {agent: interview_action}
await self.env.step(actions)
# 从数据库获取结果
# Get result from database
result = self._get_interview_result(agent_id)
self.send_response(command_id, "completed", result=result)
print(f" Interview完成: agent_id={agent_id}")
print(f" Interview completed: agent_id={agent_id}")
return True
except Exception as e:
error_msg = str(e)
print(f" Interview失败: agent_id={agent_id}, error={error_msg}")
print(f" Interview failed: agent_id={agent_id}, error={error_msg}")
self.send_response(command_id, "failed", error=error_msg)
return False
async def handle_batch_interview(self, command_id: str, interviews: List[Dict]) -> bool:
"""
处理批量采访命令
Handle batch interview command
Args:
interviews: [{"agent_id": int, "prompt": str}, ...]
"""
try:
# 构建动作字典
# Build action dict
actions = {}
agent_prompts = {} # 记录每个agent的prompt
agent_prompts = {} # Record each agent's prompt
for interview in interviews:
agent_id = interview.get("agent_id")
@ -269,16 +269,16 @@ class IPCHandler:
)
agent_prompts[agent_id] = prompt
except Exception as e:
print(f" 警告: 无法获取Agent {agent_id}: {e}")
print(f" Warning: unable to fetch Agent {agent_id}: {e}")
if not actions:
self.send_response(command_id, "failed", error="没有有效的Agent")
self.send_response(command_id, "failed", error="No valid agents")
return False
# 执行批量Interview
# Execute batch Interview
await self.env.step(actions)
# 获取所有结果
# Get all results
results = {}
for agent_id in agent_prompts.keys():
result = self._get_interview_result(agent_id)
@ -288,17 +288,17 @@ class IPCHandler:
"interviews_count": len(results),
"results": results
})
print(f" 批量Interview完成: {len(results)} 个Agent")
print(f" Batch Interview completed: {len(results)} agents")
return True
except Exception as e:
error_msg = str(e)
print(f" 批量Interview失败: {error_msg}")
print(f" Batch Interview failed: {error_msg}")
self.send_response(command_id, "failed", error=error_msg)
return False
def _get_interview_result(self, agent_id: int) -> Dict[str, Any]:
"""从数据库获取最新的Interview结果"""
"""Get the latest Interview results from the database"""
db_path = os.path.join(self.simulation_dir, "twitter_simulation.db")
result = {
@ -314,7 +314,7 @@ class IPCHandler:
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
# 查询最新的Interview记录
# Query the latest Interview record
cursor.execute("""
SELECT user_id, info, created_at
FROM trace
@ -336,16 +336,16 @@ class IPCHandler:
conn.close()
except Exception as e:
print(f" 读取Interview结果失败: {e}")
print(f" Failed to read Interview results: {e}")
return result
async def process_commands(self) -> bool:
"""
处理所有待处理命令
Handle all pending commands
Returns:
True 表示继续运行，False 表示应该退出
True to continue running, False to exit
"""
command = self.poll_command()
if not command:
@ -355,7 +355,7 @@ class IPCHandler:
command_type = command.get("command_type")
args = command.get("args", {})
print(f"\n收到IPC命令: {command_type}, id={command_id}")
print(f"\nReceived IPC command: {command_type}, id={command_id}")
if command_type == CommandType.INTERVIEW:
await self.handle_interview(
@ -373,19 +373,19 @@ class IPCHandler:
return True
elif command_type == CommandType.CLOSE_ENV:
print("收到关闭环境命令")
self.send_response(command_id, "completed", result={"message": "环境即将关闭"})
print("Received close-environment command")
self.send_response(command_id, "completed", result={"message": "Environment is about to close"})
return False
else:
self.send_response(command_id, "failed", error=f"未知命令类型: {command_type}")
self.send_response(command_id, "failed", error=f"Unknown command type: {command_type}")
return True
class TwitterSimulationRunner:
"""Twitter模拟运行器"""
"""Twitter simulation runner"""
# Twitter可用动作（不包含INTERVIEW，INTERVIEW只能通过ManualAction手动触发）
# Twitter available actions (does not include INTERVIEW; INTERVIEW can only be triggered manually via ManualAction)
AVAILABLE_ACTIONS = [
ActionType.CREATE_POST,
ActionType.LIKE_POST,
@ -397,11 +397,11 @@ class TwitterSimulationRunner:
def __init__(self, config_path: str, wait_for_commands: bool = True):
"""
初始化模拟运行器
Initialize the simulation runner
Args:
config_path: 配置文件路径 (simulation_config.json)
wait_for_commands: 模拟完成后是否等待命令（默认True）
config_path: Configuration file path (simulation_config.json)
wait_for_commands: Whether to wait for commands after the simulation completes (default True)
"""
self.config_path = config_path
self.config = self._load_config()
@ -412,47 +412,47 @@ class TwitterSimulationRunner:
self.ipc_handler = None
def _load_config(self) -> Dict[str, Any]:
"""加载配置文件"""
"""Load configuration file"""
with open(self.config_path, 'r', encoding='utf-8') as f:
return json.load(f)
def _get_profile_path(self) -> str:
"""获取Profile文件路径（OASIS Twitter使用CSV格式）"""
"""Get Profile file path (OASIS Twitter uses CSV format)"""
return os.path.join(self.simulation_dir, "twitter_profiles.csv")
def _get_db_path(self) -> str:
"""获取数据库路径"""
"""Get database path"""
return os.path.join(self.simulation_dir, "twitter_simulation.db")
def _create_model(self):
"""
创建LLM模型
Create the LLM model
统一使用项目根目录 .env 文件中的配置（优先级最高）：
- LLM_API_KEY: API密钥
- LLM_BASE_URL: API基础URL
- LLM_MODEL_NAME: 模型名称
Use config from project-root .env file (highest priority):
- LLM_API_KEY: API key
- LLM_BASE_URL: API base URL
- LLM_MODEL_NAME: Model name
"""
# 优先从 .env 读取配置
# Prefer reading config from .env
llm_api_key = os.environ.get("LLM_API_KEY", "")
llm_base_url = os.environ.get("LLM_BASE_URL", "")
llm_model = os.environ.get("LLM_MODEL_NAME", "")
# 如果 .env 中没有，则使用 config 作为备用
# If .env has no value, fall back to config
if not llm_model:
llm_model = self.config.get("llm_model", "gpt-4o-mini")
# 设置 camel-ai 所需的环境变量
# Set environment variables required by camel-ai
if llm_api_key:
os.environ["OPENAI_API_KEY"] = llm_api_key
if not os.environ.get("OPENAI_API_KEY"):
raise ValueError("缺少 API Key 配置，请在项目根目录 .env 文件中设置 LLM_API_KEY")
raise ValueError("Missing API Key config, please set LLM_API_KEY in the project root .env file")
if llm_base_url:
os.environ["OPENAI_API_BASE_URL"] = llm_base_url
print(f"LLM配置: model={llm_model}, base_url={llm_base_url[:40] if llm_base_url else '默认'}...")
print(f"LLM config: model={llm_model}, base_url={llm_base_url[:40] if llm_base_url else 'default'}...")
return ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
@ -466,24 +466,24 @@ class TwitterSimulationRunner:
round_num: int
) -> List:
"""
根据时间和配置决定本轮激活哪些Agent
Determine which Agents to activate this round based on time and config
Args:
env: OASIS环境
current_hour: 当前模拟小时（0-23）
round_num: 当前轮数
env: OASIS environment
current_hour: Current simulation hour (0-23)
round_num: Current round number
Returns:
激活的Agent列表
List of activated Agents
"""
time_config = self.config.get("time_config", {})
agent_configs = self.config.get("agent_configs", [])
# 基础激活数量
# Base activation count
base_min = time_config.get("agents_per_hour_min", 5)
base_max = time_config.get("agents_per_hour_max", 20)
# 根据时段调整
# Adjust by time period
peak_hours = time_config.get("peak_hours", [9, 10, 11, 14, 15, 20, 21, 22])
off_peak_hours = time_config.get("off_peak_hours", [0, 1, 2, 3, 4, 5])
@ -496,28 +496,28 @@ class TwitterSimulationRunner:
target_count = int(random.uniform(base_min, base_max) * multiplier)
# 根据每个Agent的配置计算激活概率
# Compute activation probability from each Agent's config
candidates = []
for cfg in agent_configs:
agent_id = cfg.get("agent_id", 0)
active_hours = cfg.get("active_hours", list(range(8, 23)))
activity_level = cfg.get("activity_level", 0.5)
# 检查是否在活跃时间
# Check if within active hours
if current_hour not in active_hours:
continue
# 根据活跃度计算概率
# Compute probability from activity level
if random.random() < activity_level:
candidates.append(agent_id)
# 随机选择
# Random selection
selected_ids = random.sample(
candidates,
min(target_count, len(candidates))
) if candidates else []
# 转换为Agent对象
# Convert to Agent objects
active_agents = []
for agent_id in selected_ids:
try:
@ -529,50 +529,50 @@ class TwitterSimulationRunner:
return active_agents
async def run(self, max_rounds: int = None):
"""运行Twitter模拟
"""Run Twitter simulation
Args:
max_rounds: 最大模拟轮数（可选，用于截断过长的模拟）
max_rounds: Maximum simulation rounds (optional, used to truncate overlong simulations)
"""
print("=" * 60)
print("OASIS Twitter模拟")
print(f"配置文件: {self.config_path}")
print(f"模拟ID: {self.config.get('simulation_id', 'unknown')}")
print(f"等待命令模式: {'启用' if self.wait_for_commands else '禁用'}")
print("OASIS Twitter Simulation")
print(f"Config file: {self.config_path}")
print(f"Simulation ID: {self.config.get('simulation_id', 'unknown')}")
print(f"Wait-for-commands mode: {'enabled' if self.wait_for_commands else 'disabled'}")
print("=" * 60)
# 加载时间配置
# Load time config
time_config = self.config.get("time_config", {})
total_hours = time_config.get("total_simulation_hours", 72)
minutes_per_round = time_config.get("minutes_per_round", 30)
# 计算总轮数
# Compute total rounds
total_rounds = (total_hours * 60) // minutes_per_round
# 如果指定了最大轮数，则截断
# If a max-rounds is specified, truncate
if max_rounds is not None and max_rounds > 0:
original_rounds = total_rounds
total_rounds = min(total_rounds, max_rounds)
if total_rounds < original_rounds:
print(f"\n轮数已截断: {original_rounds} -> {total_rounds} (max_rounds={max_rounds})")
print(f"\nRounds truncated: {original_rounds} -> {total_rounds} (max_rounds={max_rounds})")
print(f"\n模拟参数:")
print(f" - 总模拟时长: {total_hours}小时")
print(f" - 每轮时间: {minutes_per_round}分钟")
print(f" - 总轮数: {total_rounds}")
print(f"\nSimulation parameters:")
print(f" - Total simulation duration: {total_hours} hours")
print(f" - Time per round: {minutes_per_round} minutes")
print(f" - Total rounds: {total_rounds}")
if max_rounds:
print(f" - 最大轮数限制: {max_rounds}")
print(f" - Agent数量: {len(self.config.get('agent_configs', []))}")
print(f" - Max rounds limit: {max_rounds}")
print(f" - Number of Agents: {len(self.config.get('agent_configs', []))}")
# 创建模型
print("\n初始化LLM模型...")
# Create model
print("\nInitializing LLM model...")
model = self._create_model()
# 加载Agent图
print("加载Agent Profile...")
# Load Agent graph
print("Loading Agent Profile...")
profile_path = self._get_profile_path()
if not os.path.exists(profile_path):
print(f"错误: Profile文件不存在: {profile_path}")
print(f"Error: Profile file does not exist: {profile_path}")
return
self.agent_graph = await generate_twitter_agent_graph(
@ -581,34 +581,34 @@ class TwitterSimulationRunner:
available_actions=self.AVAILABLE_ACTIONS,
)
# 数据库路径
# Database path
db_path = self._get_db_path()
if os.path.exists(db_path):
os.remove(db_path)
print(f"已删除旧数据库: {db_path}")
print(f"Deleted old database: {db_path}")
# 创建环境
print("创建OASIS环境...")
# Create environment
print("Creating OASIS environment...")
self.env = oasis.make(
agent_graph=self.agent_graph,
platform=oasis.DefaultPlatformType.TWITTER,
database_path=db_path,
semaphore=30, # 限制最大并发 LLM 请求数，防止 API 过载
semaphore=30, # Limit max concurrent LLM requests to prevent API overload
)
await self.env.reset()
print("环境初始化完成\n")
print("Environment initialization complete\n")
# 初始化IPC处理器
# Initialize IPC handler
self.ipc_handler = IPCHandler(self.simulation_dir, self.env, self.agent_graph)
self.ipc_handler.update_status("running")
# 执行初始事件
# Execute initial events
event_config = self.config.get("event_config", {})
initial_posts = event_config.get("initial_posts", [])
if initial_posts:
print(f"执行初始事件 ({len(initial_posts)}条初始帖子)...")
print(f"Executing initial events ({len(initial_posts)} initial posts)...")
initial_actions = {}
for post in initial_posts:
agent_id = post.get("poster_agent_id", 0)
@ -620,23 +620,23 @@ class TwitterSimulationRunner:
action_args={"content": content}
)
except Exception as e:
print(f" 警告: 无法为Agent {agent_id}创建初始帖子: {e}")
print(f" Warning: unable to create initial post for Agent {agent_id}: {e}")
if initial_actions:
await self.env.step(initial_actions)
print(f" 已发布 {len(initial_actions)} 条初始帖子")
print(f" Published {len(initial_actions)} initial posts")
# 主模拟循环
print("\n开始模拟循环...")
# Main simulation loop
print("\nStarting simulation loop...")
start_time = datetime.now()
for round_num in range(total_rounds):
# 计算当前模拟时间
# Compute current simulation time
simulated_minutes = round_num * minutes_per_round
simulated_hour = (simulated_minutes // 60) % 24
simulated_day = simulated_minutes // (60 * 24) + 1
# 获取本轮激活的Agent
# Get activated Agents for this round
active_agents = self._get_active_agents_for_round(
self.env, simulated_hour, round_num
)
@ -644,16 +644,16 @@ class TwitterSimulationRunner:
if not active_agents:
continue
# 构建动作
# Build actions
actions = {
agent: LLMAction()
for _, agent in active_agents
}
# 执行动作
# Execute actions
await self.env.step(actions)
# 打印进度
# Print progress
if (round_num + 1) % 10 == 0 or round_num == 0:
elapsed = (datetime.now() - start_time).total_seconds()
progress = (round_num + 1) / total_rounds * 100
@ -663,20 +663,20 @@ class TwitterSimulationRunner:
f"- elapsed: {elapsed:.1f}s")
total_elapsed = (datetime.now() - start_time).total_seconds()
print(f"\n模拟循环完成!")
print(f" - 总耗时: {total_elapsed:.1f}秒")
print(f" - 数据库: {db_path}")
print(f"\nSimulation loop completed!")
print(f" - Total elapsed: {total_elapsed:.1f} seconds")
print(f" - Database: {db_path}")
# 是否进入等待命令模式
# Whether to enter wait-for-commands mode
if self.wait_for_commands:
print("\n" + "=" * 60)
print("进入等待命令模式 - 环境保持运行")
print("支持的命令: interview, batch_interview, close_env")
print("Entering wait-for-commands mode - environment keeps running")
print("Supported commands: interview, batch_interview, close_env")
print("=" * 60)
self.ipc_handler.update_status("alive")
# 等待命令循环（使用全局 _shutdown_event）
# Wait-for-commands loop (using global _shutdown_event)
try:
while not _shutdown_event.is_set():
should_continue = await self.ipc_handler.process_commands()
@ -684,58 +684,58 @@ class TwitterSimulationRunner:
break
try:
await asyncio.wait_for(_shutdown_event.wait(), timeout=0.5)
break # 收到退出信号
break # Received exit signal
except asyncio.TimeoutError:
pass
except KeyboardInterrupt:
print("\n收到中断信号")
print("\nReceived interrupt signal")
except asyncio.CancelledError:
print("\n任务被取消")
print("\nTask cancelled")
except Exception as e:
print(f"\n命令处理出错: {e}")
print(f"\nCommand processing error: {e}")
print("\n关闭环境...")
print("\nClosing environment...")
# 关闭环境
# Close the environment
self.ipc_handler.update_status("stopped")
await self.env.close()
print("环境已关闭")
print("Environment closed")
print("=" * 60)
async def main():
parser = argparse.ArgumentParser(description='OASIS Twitter模拟')
parser = argparse.ArgumentParser(description='OASIS Twitter Simulation')
parser.add_argument(
'--config',
type=str,
required=True,
help='配置文件路径 (simulation_config.json)'
help='Configuration file path (simulation_config.json)'
)
parser.add_argument(
'--max-rounds',
type=int,
default=None,
help='最大模拟轮数（可选，用于截断过长的模拟）'
help='Maximum simulation rounds (optional, used to truncate overlong simulations)'
)
parser.add_argument(
'--no-wait',
action='store_true',
default=False,
help='模拟完成后立即关闭环境，不进入等待命令模式'
help='Close environment immediately after simulation completes, do not enter wait-for-commands mode'
)
args = parser.parse_args()
# 在 main 函数开始时创建 shutdown 事件
# Create shutdown event at the start of main()
global _shutdown_event
_shutdown_event = asyncio.Event()
if not os.path.exists(args.config):
print(f"错误: 配置文件不存在: {args.config}")
print(f"Error: configuration file does not exist: {args.config}")
sys.exit(1)
# 初始化日志配置（使用固定文件名，清理旧日志）
# Initialize log config (use fixed log file names, clean up old logs)
simulation_dir = os.path.dirname(args.config) or "."
setup_oasis_logging(os.path.join(simulation_dir, "log"))
@ -748,20 +748,20 @@ async def main():
def setup_signal_handlers():
"""
设置信号处理器，确保收到 SIGTERM/SIGINT 时能够正确退出
让程序有机会正常清理资源（关闭数据库、环境等）
Set up signal handlers to ensure clean exit on SIGTERM/SIGINT
Give the program a chance to clean up resources (close DB, env, etc.)
"""
def signal_handler(signum, frame):
global _cleanup_done
sig_name = "SIGTERM" if signum == signal.SIGTERM else "SIGINT"
print(f"\n收到 {sig_name} 信号，正在退出...")
print(f"\nReceived {sig_name} signal, exiting...")
if not _cleanup_done:
_cleanup_done = True
if _shutdown_event:
_shutdown_event.set()
else:
# 重复收到信号才强制退出
print("强制退出...")
# Only force exit if the signal is received again
print("Forced exit...")
sys.exit(1)
signal.signal(signal.SIGTERM, signal_handler)
@ -773,8 +773,8 @@ if __name__ == "__main__":
try:
asyncio.run(main())
except KeyboardInterrupt:
print("\n程序被中断")
print("\nProgram interrupted")
except SystemExit:
pass
finally:
print("模拟进程已退出")
print("Simulation process exited")

104
backend/scripts/test_profile_format.py
View File

@ -1,8 +1,8 @@
"""
测试Profile格式生成是否符合OASIS要求
验证：
1. Twitter Profile生成CSV格式
2. Reddit Profile生成JSON详细格式
Test that the generated Profile formats conform to OASIS requirements
Verifies:
1. Twitter Profile generated as CSV format
2. Reddit Profile generated as detailed JSON format
"""
import os
@ -11,19 +11,19 @@ import json
import csv
import tempfile
# 添加项目路径
# Add project path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from app.services.oasis_profile_generator import OasisProfileGenerator, OasisAgentProfile
def test_profile_formats():
"""测试Profile格式"""
"""Test Profile formats"""
print("=" * 60)
print("OASIS Profile格式测试")
print("OASIS Profile format test")
print("=" * 60)
# 创建测试Profile数据
# Create test Profile data
test_profiles = [
OasisAgentProfile(
user_id=0,
@ -60,87 +60,87 @@ def test_profile_formats():
source_entity_type="University",
),
]
generator = OasisProfileGenerator.__new__(OasisProfileGenerator)
# 使用临时目录
# Use a temporary directory
with tempfile.TemporaryDirectory() as temp_dir:
twitter_path = os.path.join(temp_dir, "twitter_profiles.csv")
reddit_path = os.path.join(temp_dir, "reddit_profiles.json")
# 测试Twitter CSV格式
print("\n1. 测试Twitter Profile (CSV格式)")
# Test Twitter CSV format
print("\n1. Test Twitter Profile (CSV format)")
print("-" * 40)
generator._save_twitter_csv(test_profiles, twitter_path)
# 读取并验证CSV
# Read and verify the CSV
with open(twitter_path, 'r', encoding='utf-8') as f:
reader = csv.DictReader(f)
rows = list(reader)
print(f" 文件: {twitter_path}")
print(f" 行数: {len(rows)}")
print(f" 表头: {list(rows[0].keys())}")
print(f"\n 示例数据 (第1行):")
print(f" File: {twitter_path}")
print(f" Row count: {len(rows)}")
print(f" Header: {list(rows[0].keys())}")
print(f"\n Sample data (row 1):")
for key, value in rows[0].items():
print(f" {key}: {value}")
# 验证必需字段
required_twitter_fields = ['user_id', 'user_name', 'name', 'bio',
# Verify required fields
required_twitter_fields = ['user_id', 'user_name', 'name', 'bio',
'friend_count', 'follower_count', 'statuses_count', 'created_at']
missing = set(required_twitter_fields) - set(rows[0].keys())
if missing:
print(f"\n [错误] 缺少字段: {missing}")
print(f"\n [Error] Missing fields: {missing}")
else:
print(f"\n [通过] 所有必需字段都存在")
# 测试Reddit JSON格式
print("\n2. 测试Reddit Profile (JSON详细格式)")
print(f"\n [Pass] All required fields are present")
# Test Reddit JSON format
print("\n2. Test Reddit Profile (detailed JSON format)")
print("-" * 40)
generator._save_reddit_json(test_profiles, reddit_path)
# 读取并验证JSON
# Read and verify the JSON
with open(reddit_path, 'r', encoding='utf-8') as f:
reddit_data = json.load(f)
print(f" 文件: {reddit_path}")
print(f" 条目数: {len(reddit_data)}")
print(f" 字段: {list(reddit_data[0].keys())}")
print(f"\n 示例数据 (第1条):")
print(f" File: {reddit_path}")
print(f" Entry count: {len(reddit_data)}")
print(f" Fields: {list(reddit_data[0].keys())}")
print(f"\n Sample data (entry 1):")
print(json.dumps(reddit_data[0], ensure_ascii=False, indent=4))
# 验证详细格式字段
# Verify detailed-format fields
required_reddit_fields = ['realname', 'username', 'bio', 'persona']
optional_reddit_fields = ['age', 'gender', 'mbti', 'country', 'profession', 'interested_topics']
missing = set(required_reddit_fields) - set(reddit_data[0].keys())
if missing:
print(f"\n [错误] 缺少必需字段: {missing}")
print(f"\n [Error] Missing required fields: {missing}")
else:
print(f"\n [通过] 所有必需字段都存在")
print(f"\n [Pass] All required fields are present")
present_optional = set(optional_reddit_fields) & set(reddit_data[0].keys())
print(f" [信息] 可选字段: {present_optional}")
print(f" [Info] Optional fields: {present_optional}")
print("\n" + "=" * 60)
print("测试完成!")
print("Tests complete!")
print("=" * 60)
def show_expected_formats():
"""显示OASIS期望的格式"""
"""Show the formats expected by OASIS"""
print("\n" + "=" * 60)
print("OASIS 期望的Profile格式参考")
print("OASIS expected Profile format reference")
print("=" * 60)
print("\n1. Twitter Profile (CSV格式)")
print("\n1. Twitter Profile (CSV format)")
print("-" * 40)
twitter_example = """user_id,user_name,name,bio,friend_count,follower_count,statuses_count,created_at
0,user0,User Zero,I am user zero with interests in technology.,100,150,500,2023-01-01
1,user1,User One,Tech enthusiast and coffee lover.,200,250,1000,2023-01-02"""
print(twitter_example)
print("\n2. Reddit Profile (JSON详细格式)")
print("\n2. Reddit Profile (detailed JSON format)")
print("-" * 40)
reddit_example = [
{

2789
backend/uv.lock
View File

File diff suppressed because it is too large Load Diff

201
backend/vendor/camel-oasis/LICENSE vendored Normal file
View File

@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2023 @ CAMEL-AI.org
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

13
backend/vendor/camel-oasis/LICENSES/license_template.txt vendored Normal file
View File

@ -0,0 +1,13 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========

126
backend/vendor/camel-oasis/LICENSES/update_license.py vendored Normal file
View File

@ -0,0 +1,126 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
import os
import re
import sys
from pathlib import Path
from typing import List
# The license template file is hard-coded with specific start and end lines
def fine_license_start_line(lines: List[str], start_with: str) -> int:
for i in range(len(lines)):
if lines[i].startswith(start_with):
return i
return None
def find_license_end_line(lines: List[str], start_with: str) -> int:
for i in range(len(lines) - 1, -1, -1):
if lines[i].startswith(start_with):
return i
return None
def update_license_in_file(
file_path: str,
license_template_path: str,
start_line_start_with: str,
end_line_start_with: str,
) -> bool:
with open(file_path, 'r',
encoding='utf-8') as f: # for windows compatibility
content = f.read()
with open(license_template_path, 'r', encoding='utf-8') as f:
new_license = f.read().strip()
maybe_existing_licenses = re.findall(r'^#.*?(?=\n)', content,
re.MULTILINE | re.DOTALL)
start_index = fine_license_start_line(maybe_existing_licenses,
start_line_start_with)
end_index = find_license_end_line(maybe_existing_licenses,
end_line_start_with)
if start_index is not None and end_index is not None:
maybe_existing_licenses = maybe_existing_licenses[
start_index:end_index + 1]
else:
maybe_existing_licenses = None
if maybe_existing_licenses:
maybe_old_licenses = '\n'.join(maybe_existing_licenses)
if maybe_old_licenses.strip() != new_license.strip():
replaced_content = content.replace(maybe_old_licenses, new_license)
with open(file_path, 'w') as f:
f.write(replaced_content)
print(f'Replaced license in {file_path}')
return True
else:
return False
else:
with open(file_path, 'w') as f:
f.write(new_license + '\n' + content)
print(f'Added license to {file_path}')
return True
def update_license_in_directory(
directory_path: str,
license_template_path: str,
start_line_start_with: str,
end_line_start_with: str,
) -> None:
# Check if directory exists
if not os.path.isdir(directory_path):
raise NotADirectoryError(f'{directory_path} is not a directory')
# Check if license template exists
if not os.path.isfile(license_template_path):
raise FileNotFoundError(f'{license_template_path} not found')
file_count = 0
for py_files in Path(directory_path).rglob("*.py"):
if py_files.name.startswith('.'):
continue
if any(part.startswith('.') for part in py_files.parts):
continue
if update_license_in_file(
py_files,
license_template_path,
start_line_start_with,
end_line_start_with,
):
file_count += 1
print(f'License updated in {file_count} files')
if __name__ == '__main__':
if len(sys.argv) < 3:
print(
"Usage from command line: "
"python update_license.py <directory_path> <license_template_path>"
"No valid input arguments found, please enter manually.")
directory_path = input("Enter directory path: ")
license_template_path = input("Enter license template path: ")
else:
directory_path = sys.argv[1]
license_template_path = sys.argv[2]
start_line_start_with = "# =========== Copyright"
end_line_start_with = "# =========== Copyright"
update_license_in_directory(
directory_path,
license_template_path,
start_line_start_with,
end_line_start_with,
)

337
backend/vendor/camel-oasis/README.md vendored Normal file
View File

@ -0,0 +1,337 @@
<div align="center">
<a href="https://www.camel-ai.org/">
<img src="assets/banner.png" alt=banner>
</a>
</div>
</br>
<div align="center">
<h1> OASIS: Open Agent Social Interaction Simulations with One Million Agents
</h1>
[![Documentation][docs-image]][docs-url]
[![Discord][discord-image]][discord-url]
[![X][x-image]][x-url]
[![Reddit][reddit-image]][reddit-url]
[![Wechat][wechat-image]][wechat-url]
[![Wechat][oasis-image]][oasis-url]
[![Hugging Face][huggingface-image]][huggingface-url]
[![Star][star-image]][star-url]
[![Package License][package-license-image]][package-license-url]
<h4 align="center">
[Community](https://github.com/camel-ai/camel#community) |
[Paper](https://arxiv.org/abs/2411.11581) |
[Examples](https://github.com/camel-ai/oasis/tree/main/scripts) |
[Dataset](https://huggingface.co/datasets/echo-yiyiyi/oasis-dataset) |
[Citation](https://github.com/camel-ai/oasis#-citation) |
[Contributing](https://github.com/camel-ai/oasis#-contributing-to-oasis) |
[CAMEL-AI](https://www.camel-ai.org/)
</h4>
</div>
<br>
<p align="left">
<img src='assets/intro.png'>
🏝️ OASIS is a scalable, open-source social media simulator that incorporates large language model agents to realistically mimic the behavior of up to one million users on platforms like Twitter and Reddit. It's designed to facilitate the study of complex social phenomena such as information spread, group polarization, and herd behavior, offering a versatile tool for exploring diverse social dynamics and user interactions in digital environments.
</p>
<br>
<div align="center">
🌟 Star OASIS on GitHub and be instantly notified of new releases.
</div>
<br>
<div align="center">
<img src="assets/star.gif" alt="Star" width="196" height="52">
</a>
</div>
<br>
## ✨ Key Features
### 📈 Scalability
OASIS supports simulations of up to ***one million agents***, enabling studies of social media dynamics at a scale comparable to real-world platforms.
### 📲 Dynamic Environments
Adapts to real-time changes in social networks and content, mirroring the fluid dynamics of platforms like **Twitter** and **Reddit** for authentic simulation experiences.
### 👍🏼 Diverse Action Spaces
Agents can perform **23 actions**, such as following, commenting, and reposting, allowing for rich, multi-faceted interactions.
### 🔥 Integrated Recommendation Systems
Features **interest-based** and **hot-score-based recommendation algorithms**, simulating how users discover content and interact within social media platforms.
<br>
## 📺 Demo Video
### Introducing OASIS: Open Agent Social Interaction Simulations with One Million Agents
https://github.com/user-attachments/assets/3bd2553c-d25d-4d8c-a739-1af51354b15a
<br>
For more showcaes:
- Can 1,000,000 AI agents simulate social media?
[→Watch demo](https://www.youtube.com/watch?v=lprGHqkApus&t=2s)
<br>
## 🎯 Usecase
<div align="left">
<img src="assets/research_simulation.png" alt=usecase1>
<img src="assets/interaction.png" alt=usecase2>
<a href="http://www.matrix.eigent.ai">
<img src="assets/content_creation.png" alt=usecase3>
</a>
<img src="assets/prediction.png" alt=usecase4>
</div>
## ⚙️ Quick Start
1. **Install the OASIS package:**
Installing OASIS is a breeze thanks to its availability on PyPI. Simply open your terminal and run:
```bash
pip install camel-oasis
```
2. **Set up your OpenAI API key:**
```bash
# For Bash shell (Linux, macOS, Git Bash on Windows):
export OPENAI_API_KEY=<insert your OpenAI API key>
# For Windows Command Prompt:
set OPENAI_API_KEY=<insert your OpenAI API key>
```
3. **Prepare the agent profile file:**
Create the profile you want to assign to the agent. As an example, you can download [user_data_36.json](https://github.com/camel-ai/oasis/blob/main/data/reddit/user_data_36.json) and place it in your local `./data/reddit` folder.
4. **Run the following Python code:**
```python
import asyncio
import os
from camel.models import ModelFactory
from camel.types import ModelPlatformType, ModelType
import oasis
from oasis import (ActionType, LLMAction, ManualAction,
generate_reddit_agent_graph)
async def main():
# Define the model for the agents
openai_model = ModelFactory.create(
model_platform=ModelPlatformType.OPENAI,
model_type=ModelType.GPT_4O_MINI,
)
# Define the available actions for the agents
available_actions = [
ActionType.LIKE_POST,
ActionType.DISLIKE_POST,
ActionType.CREATE_POST,
ActionType.CREATE_COMMENT,
ActionType.LIKE_COMMENT,
ActionType.DISLIKE_COMMENT,
ActionType.SEARCH_POSTS,
ActionType.SEARCH_USER,
ActionType.TREND,
ActionType.REFRESH,
ActionType.DO_NOTHING,
ActionType.FOLLOW,
ActionType.MUTE,
]
agent_graph = await generate_reddit_agent_graph(
profile_path="./data/reddit/user_data_36.json",
model=openai_model,
available_actions=available_actions,
)
# Define the path to the database
db_path = "./data/reddit_simulation.db"
# Delete the old database
if os.path.exists(db_path):
os.remove(db_path)
# Make the environment
env = oasis.make(
agent_graph=agent_graph,
platform=oasis.DefaultPlatformType.REDDIT,
database_path=db_path,
)
# Run the environment
await env.reset()
actions_1 = {}
actions_1[env.agent_graph.get_agent(0)] = [
ManualAction(action_type=ActionType.CREATE_POST,
action_args={"content": "Hello, world!"}),
ManualAction(action_type=ActionType.CREATE_COMMENT,
action_args={
"post_id": "1",
"content": "Welcome to the OASIS World!"
})
]
actions_1[env.agent_graph.get_agent(1)] = ManualAction(
action_type=ActionType.CREATE_COMMENT,
action_args={
"post_id": "1",
"content": "I like the OASIS world."
})
await env.step(actions_1)
actions_2 = {
agent: LLMAction()
for _, agent in env.agent_graph.get_agents()
}
# Perform the actions
await env.step(actions_2)
# Close the environment
await env.close()
if __name__ == "__main__":
asyncio.run(main())
```
<br>
> \[!TIP\]
> For more detailed instructions and additional configuration options, check out the [documentation](https://docs.oasis.camel-ai.org/).
### More Tutorials
To discover how to create profiles for large-scale users, as well as how to visualize and analyze social simulation data once your experiment concludes, please refer to [More Tutorials](examples/experiment/user_generation_visualization.md) for detailed guidance.
<div align="center">
<img src="assets/tutorial.png" alt="Tutorial Overview">
</div>
## 📢 News
### Upcoming Features & Contributions
> We welcome community contributions! Join us in building these exciting features.
- [Support Multi Modal Platform](https://github.com/camel-ai/oasis/issues/47)
<!-- - Public release of our dataset on Hugging Face (November 05, 2024) -->
### Latest Updates
📢 Update the camel-ai version to 0.2.78 and update the dataset HuggingFace link. - 📆 December 4, 2025
- Add the report post action to mark inappropriate content. - 📆 June 8, 2025
- Add features for creating group chats, sending messages in group chats, and leaving group chats. - 📆 June 6, 2025
- Support Interview Action for asking agents specific questions and getting answers. - 📆 June 2, 2025
- Support customization of each agent's models, tools, and prompts; refactor the interface to follow the PettingZoo style. - 📆 May 22, 2025
- Refactor into the OASIS environment, publish camel-oasis on PyPI, and release the documentation. - 📆 April 24, 2025
- Support OPENAI Embedding model for Twhin-Bert Recommendation System. - 📆 March 25, 2025
...
- Slightly refactoring the database to add Quote Action and modify Repost Action - 📆 January 13, 2025
- Added the demo video and oasis's star history in the README - 📆 January 5, 2025
- Introduced an Electronic Mall on the Reddit platform - 📆 December 5, 2024
- OASIS initially released on arXiv - 📆 November 19, 2024
- OASIS GitHub repository initially launched - 📆 November 19, 2024
## 🔎 Follow-up Research
- [MultiAgent4Collusion](https://github.com/renqibing/MultiAgent4Collusion): multi-agent collusion simulation framework in social systems
- More to come...
If your research is based on OASIS, we'd be happy to feature your work here—feel free to reach out or submit a pull request to add it to the [README](https://github.com/camel-ai/oasis/blob/main/README.md)!
## 🥂 Contributing to OASIS🏝️
> We greatly appreciate your interest in contributing to our open-source initiative. To ensure a smooth collaboration and the success of contributions, we adhere to a set of contributing guidelines similar to those established by CAMEL. For a comprehensive understanding of the steps involved in contributing to our project, please refer to the OASIS [contributing guidelines](https://github.com/camel-ai/oasis/blob/master/CONTRIBUTING.md). 🤝🚀
>
> An essential part of contributing involves not only submitting new features with accompanying tests (and, ideally, examples) but also ensuring that these contributions pass our automated pytest suite. This approach helps us maintain the project's quality and reliability by verifying compatibility and functionality.
## 📬 Community & Contact
If you're keen on exploring new research opportunities or discoveries with our platform and wish to dive deeper or suggest new features, we're here to talk. Feel free to get in touch for more details at camel.ai.team@gmail.com.
<br>
- Join us ([*Discord*](https://discord.camel-ai.org/) or [*WeChat*](https://ghli.org/camel/wechat.png)) in pushing the boundaries of finding the scaling laws of agents.
- Join WechatGroup for further discussions!
<div align="">
<img src="assets/wechatgroup.png" alt="WeChat Group QR Code" width="600">
</div>
## 🌟 Star History
[![Star History Chart](https://api.star-history.com/svg?repos=camel-ai/oasis&type=Date)](https://star-history.com/#camel-ai/oasis&Date)
## 🔗 Citation
```
@misc{yang2024oasisopenagentsocial,
title={OASIS: Open Agent Social Interaction Simulations with One Million Agents},
author={Ziyi Yang and Zaibin Zhang and Zirui Zheng and Yuxian Jiang and Ziyue Gan and Zhiyu Wang and Zijian Ling and Jinsong Chen and Martz Ma and Bowen Dong and Prateek Gupta and Shuyue Hu and Zhenfei Yin and Guohao Li and Xu Jia and Lijun Wang and Bernard Ghanem and Huchuan Lu and Chaochao Lu and Wanli Ouyang and Yu Qiao and Philip Torr and Jing Shao},
year={2024},
eprint={2411.11581},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2411.11581},
}
```
## 🙌 Acknowledgment
We would like to thank Douglas for designing the logo of our project.
## 🖺 License
The source code is licensed under Apache 2.0.
[discord-image]: https://img.shields.io/discord/1082486657678311454?logo=discord&labelColor=%20%235462eb&logoColor=%20%23f5f5f5&color=%20%235462eb
[discord-url]: https://discord.camel-ai.org/
[docs-image]: https://img.shields.io/badge/Documentation-EB3ECC
[docs-url]: https://docs.oasis.camel-ai.org/
[huggingface-image]: https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-CAMEL--AI-ffc107?color=ffc107&logoColor=white
[huggingface-url]: https://huggingface.co/camel-ai
[oasis-image]: https://img.shields.io/badge/WeChat-OASISProject-brightgreen?logo=wechat&logoColor=white
[oasis-url]: ./assets/wechatgroup.png
[package-license-image]: https://img.shields.io/badge/License-Apache_2.0-blue.svg
[package-license-url]: https://github.com/camel-ai/oasis/blob/main/licenses/LICENSE
[reddit-image]: https://img.shields.io/reddit/subreddit-subscribers/CamelAI?style=plastic&logo=reddit&label=r%2FCAMEL&labelColor=white
[reddit-url]: https://www.reddit.com/r/CamelAI/
[star-image]: https://img.shields.io/github/stars/camel-ai/oasis?label=stars&logo=github&color=brightgreen
[star-url]: https://github.com/camel-ai/oasis/stargazers
[wechat-image]: https://img.shields.io/badge/WeChat-CamelAIOrg-brightgreen?logo=wechat&logoColor=white
[wechat-url]: ./assets/wechat.JPGwechat.jpg
[x-image]: https://img.shields.io/twitter/follow/CamelAIOrg?style=social
[x-url]: https://x.com/CamelAIOrg

31
backend/vendor/camel-oasis/oasis/__init__.py vendored Normal file
View File

@ -0,0 +1,31 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
__version__ = "0.2.5"
from oasis.environment.env_action import LLMAction, ManualAction
from oasis.environment.make import make
from oasis.social_agent import (generate_reddit_agent_graph,
generate_twitter_agent_graph)
from oasis.social_agent.agent import SocialAgent
from oasis.social_agent.agent_graph import AgentGraph
from oasis.social_platform.config import UserInfo
from oasis.social_platform.platform import Platform
from oasis.social_platform.typing import ActionType, DefaultPlatformType
from oasis.testing.show_db import print_db_contents
__all__ = [
"make", "Platform", "ActionType", "DefaultPlatformType", "ManualAction",
"LLMAction", "print_db_contents", "AgentGraph", "SocialAgent", "UserInfo",
"generate_reddit_agent_graph", "generate_twitter_agent_graph"
]

16
backend/vendor/camel-oasis/oasis/clock/__init__.py vendored Normal file
View File

@ -0,0 +1,16 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from .clock import Clock
__all__ = ["Clock"]

33
backend/vendor/camel-oasis/oasis/clock/clock.py vendored Normal file
View File

@ -0,0 +1,33 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from datetime import datetime
class Clock:
r"""Clock used for the sandbox."""
def __init__(self, k: int = 1):
self.real_start_time = datetime.now()
self.k = k
self.time_step = 0
def time_transfer(self, now_time: datetime,
start_time: datetime) -> datetime:
time_diff = now_time - self.real_start_time
adjusted_diff = self.k * time_diff
adjusted_time = start_time + adjusted_diff
return adjusted_time
def get_time_step(self) -> str:
return str(self.time_step)

13
backend/vendor/camel-oasis/oasis/environment/__init__.py vendored Normal file
View File

@ -0,0 +1,13 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========

208
backend/vendor/camel-oasis/oasis/environment/env.py vendored Normal file
View File

@ -0,0 +1,208 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
import asyncio
import logging
import os
from datetime import datetime
from typing import List, Union
from oasis.environment.env_action import LLMAction, ManualAction
from oasis.social_agent.agent import SocialAgent
from oasis.social_agent.agent_graph import AgentGraph
from oasis.social_agent.agents_generator import generate_custom_agents
from oasis.social_platform.channel import Channel
from oasis.social_platform.platform import Platform
from oasis.social_platform.typing import (ActionType, DefaultPlatformType,
RecsysType)
# Create log directory if it doesn't exist
log_dir = "./log"
if not os.path.exists(log_dir):
os.makedirs(log_dir)
# Configure logger
env_log = logging.getLogger("oasis.env")
env_log.setLevel("INFO")
# Add file handler to save logs to file
current_time = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
file_handler = logging.FileHandler(f"{log_dir}/oasis-{current_time}.log",
encoding="utf-8")
file_handler.setLevel("INFO")
file_handler.setFormatter(
logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
env_log.addHandler(file_handler)
class OasisEnv:
def __init__(
self,
agent_graph: AgentGraph,
platform: Union[DefaultPlatformType, Platform],
database_path: str = None,
semaphore: int = 128,
) -> None:
r"""Init the oasis environment.
Args:
agent_graph: The AgentGraph to use in the simulation.
platform: The platform type to use. Including
`DefaultPlatformType.TWITTER` or `DefaultPlatformType.REDDIT`.
Or you can pass a custom `Platform` instance.
database_path: The path to create a sqlite3 database. The file
extension must be `.db` such as `twitter_simulation.db`.
"""
# Initialize the agent graph
self.agent_graph = agent_graph
# Use a semaphore to limit the number of concurrent requests
self.llm_semaphore = asyncio.Semaphore(semaphore)
if isinstance(platform, DefaultPlatformType):
if database_path is None:
raise ValueError(
"database_path is required for DefaultPlatformType")
self.platform = platform
if platform == DefaultPlatformType.TWITTER:
self.channel = Channel()
self.platform = Platform(
db_path=database_path,
channel=self.channel,
recsys_type="twhin-bert",
refresh_rec_post_count=2,
max_rec_post_len=2,
following_post_count=3,
)
self.platform_type = DefaultPlatformType.TWITTER
elif platform == DefaultPlatformType.REDDIT:
self.channel = Channel()
self.platform = Platform(
db_path=database_path,
channel=self.channel,
recsys_type="reddit",
allow_self_rating=True,
show_score=True,
max_rec_post_len=100,
refresh_rec_post_count=5,
)
self.platform_type = DefaultPlatformType.REDDIT
else:
raise ValueError(f"Invalid platform: {platform}. Only "
"DefaultPlatformType.TWITTER or "
"DefaultPlatformType.REDDIT are supported.")
elif isinstance(platform, Platform):
if database_path != platform.db_path:
env_log.warning("database_path is not the same as the "
"platform.db_path, using the platform.db_path")
self.platform = platform
self.channel = platform.channel
if platform.recsys_type == RecsysType.REDDIT:
self.platform_type = DefaultPlatformType.REDDIT
else:
self.platform_type = DefaultPlatformType.TWITTER
else:
raise ValueError(
f"Invalid platform: {platform}. You should pass a "
"DefaultPlatformType or a Platform instance.")
async def reset(self) -> None:
r"""Start the platform and sign up the agents."""
self.platform_task = asyncio.create_task(self.platform.running())
self.agent_graph = await generate_custom_agents(
channel=self.channel, agent_graph=self.agent_graph)
async def _perform_llm_action(self, agent):
r"""Send the request to the llm model and execute the action.
"""
async with self.llm_semaphore:
return await agent.perform_action_by_llm()
async def _perform_interview_action(self, agent, interview_prompt: str):
r"""Send the request to the llm model and execute the interview.
"""
async with self.llm_semaphore:
return await agent.perform_interview(interview_prompt)
async def step(
self, actions: dict[SocialAgent, Union[ManualAction, LLMAction,
List[Union[ManualAction,
LLMAction]]]]
) -> None:
r"""Update the recommendation system and perform the actions.
Args:
actions(dict[SocialAgent, Union[ManualAction, LLMAction,
List[Union[ManualAction, LLMAction]]]]): The actions to
perform, including the manual(pre-defined) actions and llm
actions.
Returns:
None
"""
# Update the recommendation system
await self.platform.update_rec_table()
env_log.info("update rec table.")
# Create tasks for both manual and LLM actions
tasks = []
for agent, action in actions.items():
if isinstance(action, list):
for single_action in action:
if isinstance(single_action, ManualAction):
if single_action.action_type == ActionType.INTERVIEW:
# Use the agent's perform_interview method for
# interview actions
interview_prompt = single_action.action_args.get(
"prompt", "")
tasks.append(
self._perform_interview_action(
agent, interview_prompt))
else:
tasks.append(
agent.perform_action_by_data(
single_action.action_type,
**single_action.action_args))
elif isinstance(single_action, LLMAction):
tasks.append(self._perform_llm_action(agent))
else:
if isinstance(action, ManualAction):
if action.action_type == ActionType.INTERVIEW:
# Use the agent's perform_interview method for
# interview actions
interview_prompt = action.action_args.get("prompt", "")
tasks.append(
self._perform_interview_action(
agent, interview_prompt))
else:
tasks.append(
agent.perform_action_by_data(
action.action_type, **action.action_args))
elif isinstance(action, LLMAction):
tasks.append(self._perform_llm_action(agent))
# Execute all tasks concurrently
await asyncio.gather(*tasks)
env_log.info("performed all actions.")
# # Control some agents to perform actions
# Update the clock
if self.platform_type == DefaultPlatformType.TWITTER:
self.platform.sandbox_clock.time_step += 1
async def close(self) -> None:
r"""Stop the platform and close the environment.
"""
await self.channel.write_to_receive_queue(
(None, None, ActionType.EXIT))
await self.platform_task
env_log.info("Simulation finished! Please check the results in the "
f"database: {self.platform.db_path}. Note that the trace "
"table stored all the actions of the agents.")

45
backend/vendor/camel-oasis/oasis/environment/env_action.py vendored Normal file
View File

@ -0,0 +1,45 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from dataclasses import dataclass
from typing import Any, Dict
from oasis.social_platform.typing import ActionType
@dataclass
class ManualAction:
r"""Some manual predefined social platform actions that need to be
executed by certain agents.
Args:
agent_id: The ID of the agent that will perform the action.
action: The action to perform.
args: The arguments to pass to the action. For details of each args in
each action, please refer to
`https://github.com/camel-ai/oasis/blob/main/oasis/social_agent/agent_action.py`.
"""
action_type: ActionType
action_args: Dict[str, Any]
def init(self, action_type, action_args):
self.action_type = action_type
self.action_args = action_args
@dataclass
class LLMAction:
r"""Represents actions generated by a Language Learning Model (LLM)."""
def init(self):
pass

19
backend/vendor/camel-oasis/oasis/environment/make.py vendored Normal file
View File

@ -0,0 +1,19 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from oasis.environment.env import OasisEnv
def make(*args, **kwargs):
obj = OasisEnv(*args, **kwargs)
return obj

23
backend/vendor/camel-oasis/oasis/social_agent/__init__.py vendored Normal file
View File

@ -0,0 +1,23 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from .agent import SocialAgent
from .agent_graph import AgentGraph
from .agents_generator import (generate_agents_100w,
generate_reddit_agent_graph,
generate_twitter_agent_graph)
__all__ = [
"SocialAgent", "AgentGraph", "generate_agents_100w",
"generate_reddit_agent_graph", "generate_twitter_agent_graph"
]

321
backend/vendor/camel-oasis/oasis/social_agent/agent.py vendored Normal file
View File

@ -0,0 +1,321 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from __future__ import annotations
import inspect
import logging
import sys
from datetime import datetime
from typing import TYPE_CHECKING, Any, Callable, List, Optional, Union
from camel.agents import ChatAgent
from camel.messages import BaseMessage
from camel.models import BaseModelBackend, ModelManager
from camel.prompts import TextPrompt
from camel.toolkits import FunctionTool
from camel.types import OpenAIBackendRole
from oasis.social_agent.agent_action import SocialAction
from oasis.social_agent.agent_environment import SocialEnvironment
from oasis.social_platform import Channel
from oasis.social_platform.config import UserInfo
from oasis.social_platform.typing import ActionType
if TYPE_CHECKING:
from oasis.social_agent import AgentGraph
if "sphinx" not in sys.modules:
agent_log = logging.getLogger(name="social.agent")
agent_log.setLevel("DEBUG")
if not agent_log.handlers:
now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
file_handler = logging.FileHandler(
f"./log/social.agent-{str(now)}.log")
file_handler.setLevel("DEBUG")
file_handler.setFormatter(
logging.Formatter(
"%(levelname)s - %(asctime)s - %(name)s - %(message)s"))
agent_log.addHandler(file_handler)
ALL_SOCIAL_ACTIONS = [action.value for action in ActionType]
class SocialAgent(ChatAgent):
r"""Social Agent."""
def __init__(self,
agent_id: int,
user_info: UserInfo,
user_info_template: TextPrompt | None = None,
channel: Channel | None = None,
model: Optional[Union[BaseModelBackend,
List[BaseModelBackend],
ModelManager]] = None,
agent_graph: "AgentGraph" = None,
available_actions: list[ActionType] = None,
tools: Optional[List[Union[FunctionTool, Callable]]] = None,
max_iteration: int = 1,
interview_record: bool = False):
self.social_agent_id = agent_id
self.user_info = user_info
self.channel = channel or Channel()
self.env = SocialEnvironment(SocialAction(agent_id, self.channel))
if user_info_template is None:
system_message_content = self.user_info.to_system_message()
else:
system_message_content = self.user_info.to_custom_system_message(
user_info_template)
system_message = BaseMessage.make_assistant_message(
role_name="system",
content=system_message_content, # system prompt
)
if not available_actions:
agent_log.info("No available actions defined, using all actions.")
self.action_tools = self.env.action.get_openai_function_list()
else:
all_tools = self.env.action.get_openai_function_list()
all_possible_actions = [tool.func.__name__ for tool in all_tools]
for action in available_actions:
action_name = action.value if isinstance(
action, ActionType) else action
if action_name not in all_possible_actions:
agent_log.warning(
f"Action {action_name} is not supported. Supported "
f"actions are: {', '.join(all_possible_actions)}")
self.action_tools = [
tool for tool in all_tools if tool.func.__name__ in [
a.value if isinstance(a, ActionType) else a
for a in available_actions
]
]
all_tools = (tools or []) + (self.action_tools or [])
super().__init__(
system_message=system_message,
model=model,
scheduling_strategy='random_model',
tools=all_tools,
)
self.max_iteration = max_iteration
self.interview_record = interview_record
self.agent_graph = agent_graph
self.test_prompt = (
"\n"
"Helen is a successful writer who usually writes popular western "
"novels. Now, she has an idea for a new novel that could really "
"make a big impact. If it works out, it could greatly "
"improve her career. But if it fails, she will have spent "
"a lot of time and effort for nothing.\n"
"\n"
"What do you think Helen should do?")
async def perform_action_by_llm(self):
# Get posts:
env_prompt = await self.env.to_text_prompt()
user_msg = BaseMessage.make_user_message(
role_name="User",
content=(
f"Please perform social media actions after observing the "
f"platform environments. Notice that don't limit your "
f"actions for example to just like the posts. "
f"Here is your social media environment: {env_prompt}"))
try:
agent_log.info(
f"Agent {self.social_agent_id} observing environment: "
f"{env_prompt}")
response = await self.astep(user_msg)
for tool_call in response.info['tool_calls']:
action_name = tool_call.tool_name
args = tool_call.args
agent_log.info(f"Agent {self.social_agent_id} performed "
f"action: {action_name} with args: {args}")
if action_name not in ALL_SOCIAL_ACTIONS:
agent_log.info(
f"Agent {self.social_agent_id} get the result: "
f"{tool_call.result}")
# Abort graph action for if 100w Agent
# self.perform_agent_graph_action(action_name, args)
return response
except Exception as e:
agent_log.error(f"Agent {self.social_agent_id} error: {e}")
return e
async def perform_test(self):
"""
doing group polarization test for all agents.
TODO: rewrite the function according to the ChatAgent.
TODO: unify the test and interview function.
"""
# user conduct test to agent
_ = BaseMessage.make_user_message(role_name="User",
content=("You are a twitter user."))
# Test memory should not be writed to memory.
# self.memory.write_record(MemoryRecord(user_msg,
# OpenAIBackendRole.USER))
openai_messages, num_tokens = self.memory.get_context()
openai_messages = ([{
"role":
self.system_message.role_name,
"content":
self.system_message.content.split("# RESPONSE FORMAT")[0],
}] + openai_messages + [{
"role": "user",
"content": self.test_prompt
}])
agent_log.info(f"Agent {self.social_agent_id}: {openai_messages}")
# NOTE: this is a temporary solution.
# Camel can not stop updating the agents' memory after stop and astep
# now.
response = await self._aget_model_response(
openai_messages=openai_messages, num_tokens=num_tokens)
content = response.output_messages[0].content
agent_log.info(
f"Agent {self.social_agent_id} receive response: {content}")
return {
"user_id": self.social_agent_id,
"prompt": openai_messages,
"content": content
}
async def perform_interview(self, interview_prompt: str):
"""
Perform an interview with the agent.
"""
# user conduct test to agent
user_msg = BaseMessage.make_user_message(
role_name="User", content=("You are a twitter user."))
if self.interview_record:
# Test memory should not be writed to memory.
self.update_memory(message=user_msg, role=OpenAIBackendRole.SYSTEM)
openai_messages, num_tokens = self.memory.get_context()
openai_messages = ([{
"role":
self.system_message.role_name,
"content":
self.system_message.content.split("# RESPONSE FORMAT")[0],
}] + openai_messages + [{
"role": "user",
"content": interview_prompt
}])
agent_log.info(f"Agent {self.social_agent_id}: {openai_messages}")
# NOTE: this is a temporary solution.
# Camel can not stop updating the agents' memory after stop and astep
# now.
response = await self._aget_model_response(
openai_messages=openai_messages, num_tokens=num_tokens)
content = response.output_messages[0].content
if self.interview_record:
# Test memory should not be writed to memory.
self.update_memory(message=response.output_messages[0],
role=OpenAIBackendRole.USER)
agent_log.info(
f"Agent {self.social_agent_id} receive response: {content}")
# Record the complete interview (prompt + response) through the channel
interview_data = {"prompt": interview_prompt, "response": content}
result = await self.env.action.perform_action(
interview_data, ActionType.INTERVIEW.value)
# Return the combined result
return {
"user_id": self.social_agent_id,
"prompt": openai_messages,
"content": content,
"success": result.get("success", False)
}
async def perform_action_by_hci(self) -> Any:
print("Please choose one function to perform:")
function_list = self.env.action.get_openai_function_list()
for i in range(len(function_list)):
agent_log.info(f"Agent {self.social_agent_id} function: "
f"{function_list[i].func.__name__}")
selection = int(input("Enter your choice: "))
if not 0 <= selection < len(function_list):
agent_log.error(f"Agent {self.social_agent_id} invalid input.")
return
func = function_list[selection].func
params = inspect.signature(func).parameters
args = []
for param in params.values():
while True:
try:
value = input(f"Enter value for {param.name}: ")
args.append(value)
break
except ValueError:
agent_log.error("Invalid input, please enter an integer.")
result = await func(*args)
return result
async def perform_action_by_data(self, func_name, *args, **kwargs) -> Any:
func_name = func_name.value if isinstance(func_name,
ActionType) else func_name
function_list = self.env.action.get_openai_function_list()
for i in range(len(function_list)):
if function_list[i].func.__name__ == func_name:
func = function_list[i].func
result = await func(*args, **kwargs)
self.update_memory(message=BaseMessage.make_user_message(
role_name=OpenAIBackendRole.SYSTEM,
content=f"Agent {self.social_agent_id} performed "
f"{func_name} with args: {args} and kwargs: {kwargs}"
f"and the result is {result}"),
role=OpenAIBackendRole.SYSTEM)
agent_log.info(f"Agent {self.social_agent_id}: {result}")
return result
raise ValueError(f"Function {func_name} not found in the list.")
def perform_agent_graph_action(
self,
action_name: str,
arguments: dict[str, Any],
):
r"""Remove edge if action is unfollow or add edge
if action is follow to the agent graph.
"""
if "unfollow" in action_name:
followee_id: int | None = arguments.get("followee_id", None)
if followee_id is None:
return
self.agent_graph.remove_edge(self.social_agent_id, followee_id)
agent_log.info(
f"Agent {self.social_agent_id} unfollowed Agent {followee_id}")
elif "follow" in action_name:
followee_id: int | None = arguments.get("followee_id", None)
if followee_id is None:
return
self.agent_graph.add_edge(self.social_agent_id, followee_id)
agent_log.info(
f"Agent {self.social_agent_id} followed Agent {followee_id}")
def __str__(self) -> str:
return (f"{self.__class__.__name__}(agent_id={self.social_agent_id}, "
f"model_type={self.model_type.value})")

758
backend/vendor/camel-oasis/oasis/social_agent/agent_action.py vendored Normal file
View File

@ -0,0 +1,758 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from typing import Any
from camel.toolkits import FunctionTool
from oasis.social_platform.channel import Channel
from oasis.social_platform.typing import ActionType
class SocialAction:
def __init__(self, agent_id: int, channel: Channel):
self.agent_id = agent_id
self.channel = channel
def get_openai_function_list(self) -> list[FunctionTool]:
return [
FunctionTool(func) for func in [
self.create_post,
self.like_post,
self.repost,
self.quote_post,
self.unlike_post,
self.dislike_post,
self.undo_dislike_post,
self.search_posts,
self.search_user,
self.trend,
self.refresh,
self.do_nothing,
self.create_comment,
self.like_comment,
self.dislike_comment,
self.unlike_comment,
self.undo_dislike_comment,
self.follow,
self.unfollow,
self.mute,
self.unmute,
self.purchase_product,
self.interview,
self.report_post,
self.join_group,
self.leave_group,
self.send_to_group,
self.create_group,
self.listen_from_group,
]
]
async def perform_action(self, message: Any, type: str):
message_id = await self.channel.write_to_receive_queue(
(self.agent_id, message, type))
response = await self.channel.read_from_send_queue(message_id)
return response[2]
async def sign_up(self, user_name: str, name: str, bio: str):
r"""Signs up a new user with the provided username, name, and bio.
This method prepares a user message comprising the user's details and
invokes an asynchronous action to perform the sign-up process. On
successful execution, it returns a dictionary indicating success along
with the newly created user ID.
Args:
user_name (str): The username for the new user.
name (str): The full name of the new user.
bio (str): A brief biography of the new user.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the sign-up was
successful, and 'user_id' key maps to the integer ID of the
newly created user on success.
Example of a successful return:
{'success': True, 'user_id': 2}
"""
# print(f"Agent {self.agent_id} is signing up with "
# f"user_name: {user_name}, name: {name}, bio: {bio}")
user_message = (user_name, name, bio)
return await self.perform_action(user_message, ActionType.SIGNUP.value)
async def refresh(self):
r"""Refresh to get recommended posts.
This method invokes an asynchronous action to refresh and fetch
recommended posts. On successful execution, it returns a dictionary
indicating success along with a list of posts. Each post in the list
contains details such as post ID, user ID, content, creation date,
and the number of likes.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the refresh is
successful. The 'posts' key maps to a list of dictionaries,
each representing a post with its details.
Example of a successful return:
{
"success": True,
"posts": [
{
"post_id": 1,
"user_id": 23,
"content": "This is an example post content.",
"created_at": "2024-05-14T12:00:00Z",
"num_likes": 5
},
{
"post_id": 2,
"user_id": 42,
"content": "Another example post content.",
"created_at": "2024-05-14T12:05:00Z",
"num_likes": 15
}
]
}
"""
return await self.perform_action(None, ActionType.REFRESH.value)
async def do_nothing(self):
"""Perform no action.
Returns:
dict: A dictionary with 'success' indicating if the removal was
successful.
Example of a successful return:
{"success": True}
"""
return await self.perform_action(None, ActionType.DO_NOTHING.value)
async def create_post(self, content: str):
r"""Create a new post with the given content.
This method invokes an asynchronous action to create a new post based
on the provided content. Upon successful execution, it returns a
dictionary indicating success and the ID of the newly created post.
Args:
content (str): The content of the post to be created.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the post creation was
successful. The 'post_id' key maps to the integer ID of the
newly created post.
Example of a successful return:
{'success': True, 'post_id': 50}
"""
return await self.perform_action(content, ActionType.CREATE_POST.value)
async def repost(self, post_id: int):
r"""Repost a specified post.
This method invokes an asynchronous action to Repost a specified
post. It is identified by the given post ID. Upon successful
execution, it returns a dictionary indicating success and the ID of
the newly created repost.
Args:
post_id (int): The ID of the post to be reposted.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the Repost creation was
successful. The 'post_id' key maps to the integer ID of the
newly created repost.
Example of a successful return:
{"success": True, "post_id": 123}
Note:
Attempting to repost a post that the user has already reposted
will result in a failure.
"""
return await self.perform_action(post_id, ActionType.REPOST.value)
async def quote_post(self, post_id: int, quote_content: str):
r"""Quote a specified post with a given quote content.
This method invokes an asynchronous action to quote a specified post
with a given quote content. Upon successful execution, it returns a
dictionary indicating success and the ID of the newly created quote.
Args:
post_id (int): The ID of the post to be quoted.
quote_content (str): The content of the quote to be created.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the quote creation was
successful. The 'post_id' key maps to the integer ID of the
newly created quote.
Example of a successful return:
{"success": True, "post_id": 123}
Note:
Attempting to quote a post that the user has already quoted will
result in a failure.
"""
quote_message = (post_id, quote_content)
return await self.perform_action(quote_message, ActionType.QUOTE_POST)
async def like_post(self, post_id: int):
r"""Create a new like for a specified post.
This method invokes an asynchronous action to create a new like for a
post. It is identified by the given post ID. Upon successful
execution, it returns a dictionary indicating success and the ID of
the newly created like.
Args:
post_id (int): The ID of the post to be liked.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the like creation was
successful. The 'like_id' key maps to the integer ID of the
newly created like.
Example of a successful return:
{"success": True, "like_id": 123}
Note:
Attempting to like a post that the user has already liked will
result in a failure.
"""
return await self.perform_action(post_id, ActionType.LIKE_POST.value)
async def unlike_post(self, post_id: int):
"""Remove a like for a post.
This method removes a like from the database, identified by the
post's ID. It returns a dictionary indicating the success of the
operation and the ID of the removed like.
Args:
post_id (int): The ID of the post to be unliked.
Returns:
dict: A dictionary with 'success' indicating if the removal was
successful, and 'like_id' the ID of the removed like.
Example of a successful return:
{"success": True, "like_id": 123}
Note:
Attempting to remove a like for a post that the user has not
previously liked will result in a failure.
"""
return await self.perform_action(post_id, ActionType.UNLIKE_POST.value)
async def dislike_post(self, post_id: int):
r"""Create a new dislike for a specified post.
This method invokes an asynchronous action to create a new dislike for
a post. It is identified by the given post ID. Upon successful
execution, it returns a dictionary indicating success and the ID of
the newly created dislike.
Args:
post_id (int): The ID of the post to be disliked.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the dislike creation was
successful. The 'dislike_id' key maps to the integer ID of the
newly created like.
Example of a successful return:
{"success": True, "dislike_id": 123}
Note:
Attempting to dislike a post that the user has already liked will
result in a failure.
"""
return await self.perform_action(post_id,
ActionType.DISLIKE_POST.value)
async def undo_dislike_post(self, post_id: int):
"""Remove a dislike for a post.
This method removes a dislike from the database, identified by the
post's ID. It returns a dictionary indicating the success of the
operation and the ID of the removed dislike.
Args:
post_id (int): The ID of the post to be unliked.
Returns:
dict: A dictionary with 'success' indicating if the removal was
successful, and 'dislike_id' the ID of the removed like.
Example of a successful return:
{"success": True, "dislike_id": 123}
Note:
Attempting to remove a dislike for a post that the user has not
previously liked will result in a failure.
"""
return await self.perform_action(post_id,
ActionType.UNDO_DISLIKE_POST.value)
async def search_posts(self, query: str):
r"""Search posts based on a given query.
This method performs a search operation in the database for posts
that match the given query string. The search considers the
post's content, post ID, and user ID. It returns a dictionary
indicating the operation's success and, if successful, a list of
posts that match the query.
Args:
query (str): The search query string. The search is performed
against the post's content, post ID, and user ID.
Returns:
dict: A dictionary with a 'success' key indicating the operation's
success. On success, it includes a 'posts' key with a list of
dictionaries, each representing a post. On failure, it
includes an 'error' message or a 'message' indicating no
posts were found.
Example of a successful return:
{
"success": True,
"posts": [
{
"post_id": 1,
"user_id": 42,
"content": "Hello, world!",
"created_at": "2024-05-14T12:00:00Z",
"num_likes": 150
},
...
]
}
"""
return await self.perform_action(query, ActionType.SEARCH_POSTS.value)
async def search_user(self, query: str):
r"""Search users based on a given query.
This asynchronous method performs a search operation in the database
for users that match the given query string. The search considers the
user's username, name, bio, and user ID. It returns a dictionary
indicating the operation's success and, if successful, a list of users
that match the query.
Args:
query (str): The search query string. The search is performed
against the user's username, name, bio, and user ID.
Returns:
dict: A dictionary with a 'success' key indicating the operation's
success. On success, it includes a 'users' key with a list of
dictionaries, each representing a user. On failure, it includes
an 'error' message or a 'message' indicating no users were
found.
Example of a successful return:
{
"success": True,
"users": [
{
"user_id": 1,
"user_name": "exampleUser",
"name": "John Doe",
"bio": "This is an example bio",
"created_at": "2024-05-14T12:00:00Z",
"num_followings": 100,
"num_followers": 150
},
...
]
}
"""
return await self.perform_action(query, ActionType.SEARCH_USER.value)
async def follow(self, followee_id: int):
r"""Follow a user.
This method allows agent to follow another user (followee).
It checks if the agent initiating the follow request has a
corresponding user ID and if the follow relationship already exists.
Args:
followee_id (int): The user ID of the user to be followed.
Returns:
dict: A dictionary with a 'success' key indicating the operation's
success. On success, it includes a 'follow_id' key with the ID
of the newly created follow record. On failure, it includes an
'error' message.
Example of a successful return:
{"success": True, "follow_id": 123}
"""
return await self.perform_action(followee_id, ActionType.FOLLOW.value)
async def unfollow(self, followee_id: int):
r"""Unfollow a user.
This method allows agent to unfollow another user (followee). It
checks if the agent initiating the unfollow request has a
corresponding user ID and if the follow relationship exists. If so,
it removes the follow record from the database, updates the followers
and followings count for both users, and logs the action.
Args:
followee_id (int): The user ID of the user to be unfollowed.
Returns:
dict: A dictionary with a 'success' key indicating the operation's
success. On success, it includes a 'follow_id' key with the ID
of the removed follow record. On failure, it includes an
'error' message.
Example of a successful return:
{"success": True, "follow_id": 123}
"""
return await self.perform_action(followee_id,
ActionType.UNFOLLOW.value)
async def mute(self, mutee_id: int):
r"""Mute a user.
Allows agent to mute another user. Checks for an existing mute
record before adding a new one to the database.
Args:
mutee_id (int): ID of the user to be muted.
Returns:
dict: On success, returns a dictionary with 'success': True and
mute_id' of the new record. On failure, returns 'success':
False and an 'error' message.
Example of a successful return:
{"success": True, "mutee_id": 123}
"""
return await self.perform_action(mutee_id, ActionType.MUTE.value)
async def unmute(self, mutee_id: int):
r"""Unmute a user.
Allows agent to remove a mute on another user. Checks for an
existing mute record before removing it from the database.
Args:
mutee_id (int): ID of the user to be unmuted.
Returns:
dict: On success, returns a dictionary with 'success': True and
'mutee_id' of the unmuted record. On failure, returns
'success': False and an 'error' message.
Example of a successful return:
{"success": True, "mutee_id": 123}
"""
return await self.perform_action(mutee_id, ActionType.UNMUTE.value)
async def trend(self):
r"""Fetch the trending posts within a predefined time period.
Retrieves the top K posts with the most likes in the last specified
number of days.
Returns:
dict: On success, returns a dictionary with 'success': True and a
list of 'posts', each post being a dictionary containing
'post_id', 'user_id', 'content', 'created_at', and
'num_likes'. On failure, returns 'success': False and an
'error' message or a message indicating no trending posts
were found.
Example of a successful return:
{
"success": True,
"posts": [
{
"post_id": 123,
"user_id": 456,
"content": "Example post content",
"created_at": "2024-05-14T12:00:00",
"num_likes": 789
},
...
]
}
"""
return await self.perform_action(None, ActionType.TREND.value)
async def create_comment(self, post_id: int, content: str):
r"""Create a new comment for a specified post given content.
This method creates a new comment based on the provided content and
associates it with the given post ID. Upon successful execution, it
returns a dictionary indicating success and the ID of the newly created
comment.
Args:
post_id (int): The ID of the post to which the comment is to be
added.
content (str): The content of the comment to be created.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the comment creation was
successful. The 'comment_id' key maps to the integer ID of the
newly created comment.
Example of a successful return:
{'success': True, 'comment_id': 123}
"""
comment_message = (post_id, content)
return await self.perform_action(comment_message,
ActionType.CREATE_COMMENT.value)
async def like_comment(self, comment_id: int):
r"""Create a new like for a specified comment.
This method invokes an action to create a new like for a comment,
identified by the given comment ID. Upon successful execution, it
returns a dictionary indicating success and the ID of the newly
created like.
Args:
comment_id (int): The ID of the comment to be liked.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the like creation was
successful. The 'like_id' key maps to the integer ID of the
newly created like.
Example of a successful return:
{"success": True, "comment_like_id": 456}
Note:
Attempting to like a comment that the user has already liked will
result in a failure.
"""
return await self.perform_action(comment_id,
ActionType.LIKE_COMMENT.value)
async def unlike_comment(self, comment_id: int):
"""Remove a like for a comment based on the comment's ID.
This method removes a like from the database, identified by the
comment's ID. It returns a dictionary indicating the success of the
operation and the ID of the removed like.
Args:
comment_id (int): The ID of the comment to be unliked.
Returns:
dict: A dictionary with 'success' indicating if the removal was
successful, and 'like_id' the ID of the removed like.
Example of a successful return:
{"success": True, "like_id": 456}
Note:
Attempting to remove a like for a comment that the user has not
previously liked will result in a failure.
"""
return await self.perform_action(comment_id,
ActionType.UNLIKE_COMMENT.value)
async def dislike_comment(self, comment_id: int):
r"""Create a new dislike for a specified comment.
This method invokes an action to create a new dislike for a
comment, identified by the given comment ID. Upon successful execution,
it returns a dictionary indicating success and the ID of the newly
created dislike.
Args:
comment_id (int): The ID of the comment to be disliked.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the dislike creation was
successful. The 'dislike_id' key maps to the integer ID of the
newly created dislike.
Example of a successful return:
{"success": True, "comment_dislike_id": 456}
Note:
Attempting to dislike a comment that the user has already liked
will result in a failure.
"""
return await self.perform_action(comment_id,
ActionType.DISLIKE_COMMENT.value)
async def undo_dislike_comment(self, comment_id: int):
"""Remove a dislike for a comment.
This method removes a dislike from the database, identified by the
comment's ID. It returns a dictionary indicating the success of the
operation and the ID of the removed dislike.
Args:
comment_id (int): The ID of the comment to have the dislike
removed.
Returns:
dict: A dictionary with 'success' indicating if the removal was
successful, and 'dislike_id' the ID of the removed dislike.
Example of a successful return:
{"success": True, "dislike_id": 456}
Note:
Attempting to remove a dislike for a comment that the user has not
previously disliked will result in a failure.
"""
return await self.perform_action(comment_id,
ActionType.UNDO_DISLIKE_COMMENT.value)
async def purchase_product(self, product_name: str, purchase_num: int):
r"""Purchase a product.
Args:
product_name (str): The name of the product to be purchased.
purchase_num (int): The number of products to be purchased.
Returns:
dict: A dictionary with 'success' indicating if the purchase was
successful.
"""
purchase_message = (product_name, purchase_num)
return await self.perform_action(purchase_message,
ActionType.PURCHASE_PRODUCT.value)
async def interview(self, prompt: str):
r"""Interview an agent with the given prompt.
This method invokes an asynchronous action to interview an agent with a
specific prompt question. Upon successful execution,
it returns a dictionary containing a success status
and an interview_id for tracking.
Args:
prompt (str): The interview question or prompt to ask the agent.
Returns:
dict: A dictionary containing success status and an interview_id.
Example of a successful return:
{
"success": True,
"interview_id": "1621234567_0" # Timestamp_UserID format
}
"""
return await self.perform_action(prompt, ActionType.INTERVIEW.value)
async def report_post(self, post_id: int, report_reason: str):
r"""Report a specified post with a given reason.
This method invokes an asynchronous action to report a specified post
with a given reason. Upon successful execution, it returns a
dictionary indicating success and the ID of the newly created report.
Args:
post_id (int): The ID of the post to be reported.
report_reason (str): The reason for reporting the post.
Returns:
dict: A dictionary with two key-value pairs. The 'success' key
maps to a boolean indicating whether the report creation was
successful. The 'report_id' key maps to the integer ID of the
newly created report.
Example of a successful return:
{"success": True, "report_id": 123}
Note:
Attempting to report a post that the user has already reported will
result in a failure.
"""
report_message = (post_id, report_reason)
return await self.perform_action(report_message,
ActionType.REPORT_POST.value)
async def create_group(self, group_name: str):
r"""Creates a new group on the platform.
Args:
group_name (str): The name of the group to be created.
Returns:
dict: Platform response indicating success or failure,
e.g.{"success": True, "group_id": 1}
"""
return await self.perform_action(group_name,
ActionType.CREATE_GROUP.value)
async def join_group(self, group_id: int):
r"""Joins a group with the specified ID.
Args:
group_id (int): The ID of the group to join.
Returns:
dict: Platform response indicating success or failure,
e.g. {"success": True}
"""
return await self.perform_action(group_id, ActionType.JOIN_GROUP.value)
async def leave_group(self, group_id: int):
r"""Leaves a group with the specified ID.
Args:
group_id (int): The ID of the group to leave.
Returns:
dict: Platform response indicating success or failure, e.g.
{"success": True}
"""
return await self.perform_action(group_id,
ActionType.LEAVE_GROUP.value)
async def send_to_group(self, group_id: int, message: str):
r"""Sends a message to a specific group.
Args:
group_id (int): The ID of the target group.
message (str): The content of the message to send.
Returns:
dict: Platform response indicating success or failure, e.g.
{"success": True, "message_id": 123}
"""
return await self.perform_action((group_id, message),
ActionType.SEND_TO_GROUP.value)
async def listen_from_group(self):
r"""Listen messages from groups"""
return await self.perform_action(self.agent_id,
ActionType.LISTEN_FROM_GROUP.value)

135
backend/vendor/camel-oasis/oasis/social_agent/agent_environment.py vendored Normal file
View File

@ -0,0 +1,135 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from __future__ import annotations
import json
import sqlite3
from abc import ABC, abstractmethod
from string import Template
from oasis.social_agent.agent_action import SocialAction
from oasis.social_platform.database import get_db_path
class Environment(ABC):
@abstractmethod
def to_text_prompt(self) -> str:
r"""Convert the environment to text prompt."""
raise NotImplementedError
class SocialEnvironment(Environment):
followers_env_template = Template("I have $num_followers followers.")
follows_env_template = Template("I have $num_follows follows.")
posts_env_template = Template(
"After refreshing, you see some posts $posts")
groups_env_template = Template(
"And there are many group chat channels $all_groups\n"
"And You are already in some groups $joined_groups\n"
"You receive some messages from them $messages\n"
"You can join the groups you are interested, "
"leave the groups you already in, send messages to the group "
"you already in.\n"
"You must make sure you can only send messages to the group you "
"are already in")
env_template = Template(
"$groups_env\n"
"$posts_env\npick one you want to perform action that best "
"reflects your current inclination based on your profile and "
"posts content. Do not limit your action in just `like` to like posts")
def __init__(self, action: SocialAction):
self.action = action
async def get_posts_env(self) -> str:
posts = await self.action.refresh()
# TODO: Replace posts json format string to other formats
if posts["success"]:
posts_env = json.dumps(posts["posts"], indent=4)
posts_env = self.posts_env_template.substitute(posts=posts_env)
else:
posts_env = "After refreshing, there are no existing posts."
return posts_env
async def get_followers_env(self) -> str:
# TODO: Implement followers env
agent_id = self.action.agent_id
db_path = get_db_path()
try:
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute("SELECT num_followers FROM user WHERE agent_id = ?",
(agent_id, ))
result = cursor.fetchone()
num_followers = result[0] if result else 0
conn.close()
except Exception:
num_followers = 0
return self.followers_env_template.substitute(
{"num_followers": num_followers})
async def get_follows_env(self) -> str:
# TODO: Implement follows env
agent_id = self.action.agent_id
try:
db_path = get_db_path()
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
cursor.execute(
"SELECT num_followings FROM user WHERE agent_id = ?",
(agent_id, ))
result = cursor.fetchone()
num_followings = result[0] if result else 0
conn.close()
except Exception:
num_followings = 0
return self.follows_env_template.substitute(
{"num_follows": num_followings})
async def get_group_env(self) -> str:
groups = await self.action.listen_from_group()
if groups["success"]:
all_groups = json.dumps(groups["all_groups"])
joined_groups = json.dumps(groups["joined_groups"])
messages = json.dumps(groups["messages"])
groups_env = self.groups_env_template.substitute(
all_groups=all_groups,
joined_groups=joined_groups,
messages=messages,
)
else:
groups_env = "No groups."
return groups_env
async def to_text_prompt(
self,
include_posts: bool = True,
include_followers: bool = True,
include_follows: bool = True,
) -> str:
followers_env = (await self.get_followers_env()
if include_follows else "No followers.")
follows_env = (await self.get_follows_env()
if include_followers else "No follows.")
posts_env = await self.get_posts_env() if include_posts else ""
return self.env_template.substitute(
followers_env=followers_env,
follows_env=follows_env,
posts_env=posts_env,
groups_env=await self.get_group_env(),
)

292
backend/vendor/camel-oasis/oasis/social_agent/agent_graph.py vendored Normal file
View File

@ -0,0 +1,292 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from __future__ import annotations
from typing import Any, Literal
import igraph as ig
from neo4j import GraphDatabase
from oasis.social_agent.agent import SocialAgent
from oasis.social_platform.config import Neo4jConfig
class Neo4jHandler:
def __init__(self, nei4j_config: Neo4jConfig):
self.driver = GraphDatabase.driver(
nei4j_config.uri,
auth=(nei4j_config.username, nei4j_config.password),
)
self.driver.verify_connectivity()
def close(self):
self.driver.close()
def create_agent(self, agent_id: int):
with self.driver.session() as session:
session.write_transaction(self._create_and_return_agent, agent_id)
def delete_agent(self, agent_id: int):
with self.driver.session() as session:
session.write_transaction(
self._delete_agent_and_relationships,
agent_id,
)
def get_number_of_nodes(self) -> int:
with self.driver.session() as session:
return session.read_transaction(self._get_number_of_nodes)
def get_number_of_edges(self) -> int:
with self.driver.session() as session:
return session.read_transaction(self._get_number_of_edges)
def add_edge(self, src_agent_id: int, dst_agent_id: int):
with self.driver.session() as session:
session.write_transaction(
self._add_and_return_edge,
src_agent_id,
dst_agent_id,
)
def remove_edge(self, src_agent_id: int, dst_agent_id: int):
with self.driver.session() as session:
session.write_transaction(
self._remove_and_return_edge,
src_agent_id,
dst_agent_id,
)
def get_all_nodes(self) -> list[int]:
with self.driver.session() as session:
return session.read_transaction(self._get_all_nodes)
def get_all_edges(self) -> list[tuple[int, int]]:
with self.driver.session() as session:
return session.read_transaction(self._get_all_edges)
def reset_graph(self):
with self.driver.session() as session:
session.write_transaction(self._reset_graph)
@staticmethod
def _create_and_return_agent(tx: Any, agent_id: int):
query = """
CREATE (a:Agent {id: $agent_id})
RETURN a
"""
result = tx.run(query, agent_id=agent_id)
return result.single()
@staticmethod
def _delete_agent_and_relationships(tx: Any, agent_id: int):
query = """
MATCH (a:Agent {id: $agent_id})
DETACH DELETE a
RETURN count(a) AS deleted
"""
result = tx.run(query, agent_id=agent_id)
return result.single()
@staticmethod
def _add_and_return_edge(tx: Any, src_agent_id: int, dst_agent_id: int):
query = """
MATCH (a:Agent {id: $src_agent_id}), (b:Agent {id: $dst_agent_id})
CREATE (a)-[r:FOLLOW]->(b)
RETURN r
"""
result = tx.run(query,
src_agent_id=src_agent_id,
dst_agent_id=dst_agent_id)
return result.single()
@staticmethod
def _remove_and_return_edge(tx: Any, src_agent_id: int, dst_agent_id: int):
query = """
MATCH (a:Agent {id: $src_agent_id})
MATCH (b:Agent {id: $dst_agent_id})
MATCH (a)-[r:FOLLOW]->(b)
DELETE r
RETURN count(r) AS deleted
"""
result = tx.run(query,
src_agent_id=src_agent_id,
dst_agent_id=dst_agent_id)
return result.single()
@staticmethod
def _get_number_of_nodes(tx: Any) -> int:
query = """
MATCH (n)
RETURN count(n) AS num_nodes
"""
result = tx.run(query)
return result.single()["num_nodes"]
@staticmethod
def _get_number_of_edges(tx: Any) -> int:
query = """
MATCH ()-[r]->()
RETURN count(r) AS num_edges
"""
result = tx.run(query)
return result.single()["num_edges"]
@staticmethod
def _get_all_nodes(tx: Any) -> list[int]:
query = """
MATCH (a:Agent)
RETURN a.id AS agent_id
"""
result = tx.run(query)
return [record["agent_id"] for record in result]
@staticmethod
def _get_all_edges(tx: Any) -> list[tuple[int, int]]:
query = """
MATCH (a:Agent)-[r:FOLLOW]->(b:Agent)
RETURN a.id AS src_agent_id, b.id AS dst_agent_id
"""
result = tx.run(query)
return [(record["src_agent_id"], record["dst_agent_id"])
for record in result]
@staticmethod
def _reset_graph(tx: Any):
query = """
MATCH (n)
DETACH DELETE n
"""
tx.run(query)
class AgentGraph:
r"""AgentGraph class to manage the social graph of agents."""
def __init__(
self,
backend: Literal["igraph", "neo4j"] = "igraph",
neo4j_config: Neo4jConfig | None = None,
):
self.backend = backend
if self.backend == "igraph":
self.graph = ig.Graph(directed=True)
else:
assert neo4j_config is not None
assert neo4j_config.is_valid()
self.graph = Neo4jHandler(neo4j_config)
self.agent_mappings: dict[int, SocialAgent] = {}
def reset(self):
if self.backend == "igraph":
self.graph = ig.Graph(directed=True)
else:
self.graph.reset_graph()
self.agent_mappings: dict[int, SocialAgent] = {}
def add_agent(self, agent: SocialAgent):
if self.backend == "igraph":
self.graph.add_vertex(agent.social_agent_id)
else:
self.graph.create_agent(agent.social_agent_id)
self.agent_mappings[agent.social_agent_id] = agent
def add_edge(self, agent_id_0: int, agent_id_1: int):
try:
self.graph.add_edge(agent_id_0, agent_id_1)
except Exception:
pass
def remove_agent(self, agent: SocialAgent):
if self.backend == "igraph":
self.graph.delete_vertices(agent.social_agent_id)
else:
self.graph.delete_agent(agent.social_agent_id)
del self.agent_mappings[agent.social_agent_id]
def remove_edge(self, agent_id_0: int, agent_id_1: int):
if self.backend == "igraph":
if self.graph.are_connected(agent_id_0, agent_id_1):
self.graph.delete_edges([(agent_id_0, agent_id_1)])
else:
self.graph.remove_edge(agent_id_0, agent_id_1)
def get_agent(self, agent_id: int) -> SocialAgent:
return self.agent_mappings[agent_id]
def get_agents(
self,
agent_ids: list[int] = None) -> list[tuple[int, SocialAgent]]:
if agent_ids:
return [(agent_id, self.get_agent(agent_id))
for agent_id in agent_ids]
if self.backend == "igraph":
return [(node.index, self.agent_mappings[node.index])
for node in self.graph.vs]
else:
return [(agent_id, self.agent_mappings[agent_id])
for agent_id in self.graph.get_all_nodes()]
def get_edges(self) -> list[tuple[int, int]]:
if self.backend == "igraph":
return [(edge.source, edge.target) for edge in self.graph.es]
else:
return self.graph.get_all_edges()
def get_num_nodes(self) -> int:
if self.backend == "igraph":
return self.graph.vcount()
else:
return self.graph.get_number_of_nodes()
def get_num_edges(self) -> int:
if self.backend == "igraph":
return self.graph.ecount()
else:
return self.graph.get_number_of_edges()
def close(self) -> None:
if self.backend == "neo4j":
self.graph.close()
def visualize(
self,
path: str,
vertex_size: int = 20,
edge_arrow_size: float = 0.5,
with_labels: bool = True,
vertex_color: str = "#f74f1b",
vertex_frame_width: int = 2,
width: int = 1000,
height: int = 1000,
):
if self.backend == "neo4j":
raise ValueError("Neo4j backend does not support visualization.")
layout = self.graph.layout("auto")
if with_labels:
labels = [node_id for node_id, _ in self.get_agents()]
else:
labels = None
ig.plot(
self.graph,
target=path,
layout=layout,
vertex_label=labels,
vertex_size=vertex_size,
vertex_color=vertex_color,
edge_arrow_size=edge_arrow_size,
vertex_frame_width=vertex_frame_width,
bbox=(width, height),
)

657
backend/vendor/camel-oasis/oasis/social_agent/agents_generator.py vendored Normal file
View File

@ -0,0 +1,657 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from __future__ import annotations
import ast
import asyncio
import json
from typing import List, Optional, Union
import pandas as pd
import tqdm
from camel.memories import MemoryRecord
from camel.messages import BaseMessage
from camel.models import BaseModelBackend, ModelManager
from camel.types import OpenAIBackendRole
from oasis.social_agent import AgentGraph, SocialAgent
from oasis.social_platform import Channel, Platform
from oasis.social_platform.config import Neo4jConfig, UserInfo
from oasis.social_platform.typing import ActionType
async def generate_agents(
agent_info_path: str,
channel: Channel,
model: Union[BaseModelBackend, List[BaseModelBackend]],
start_time,
recsys_type: str = "twitter",
twitter: Platform = None,
available_actions: list[ActionType] = None,
neo4j_config: Neo4jConfig | None = None,
) -> AgentGraph:
"""TODO: need update the description of args and check
Generate and return a dictionary of agents from the agent
information CSV file. Each agent is added to the database and
their respective profiles are updated.
Args:
agent_info_path (str): The file path to the agent information CSV file.
channel (Channel): Information channel.
action_space_prompt (str): determine the action space of agents.
model_random_seed (int): Random seed to randomly assign model to
each agent. (default: 42)
cfgs (list, optional): List of configuration. (default: `None`)
neo4j_config (Neo4jConfig, optional): Neo4j graph database
configuration. (default: `None`)
Returns:
dict: A dictionary of agent IDs mapped to their respective agent
class instances.
"""
agent_info = pd.read_csv(agent_info_path)
agent_graph = (AgentGraph() if neo4j_config is None else AgentGraph(
backend="neo4j",
neo4j_config=neo4j_config,
))
# agent_graph = []
sign_up_list = []
follow_list = []
user_update1 = []
user_update2 = []
post_list = []
for agent_id in range(len(agent_info)):
profile = {
"nodes": [],
"edges": [],
"other_info": {},
}
profile["other_info"]["user_profile"] = agent_info["user_char"][
agent_id]
user_info = UserInfo(
name=agent_info["username"][agent_id],
description=agent_info["description"][agent_id],
profile=profile,
recsys_type=recsys_type,
)
agent = SocialAgent(
agent_id=agent_id,
user_info=user_info,
channel=channel,
model=model,
agent_graph=agent_graph,
available_actions=available_actions,
)
agent_graph.add_agent(agent)
# TODO we should not use following_count and followers_count
# We should calculate the number of followings and followers
# based on the graph because the following situation is dynamic.
num_followings = 0
num_followers = 0
sign_up_list.append((
agent_id,
agent_id,
agent_info["username"][agent_id],
agent_info["name"][agent_id],
agent_info["description"][agent_id],
start_time,
num_followings,
num_followers,
))
following_id_list = ast.literal_eval(
agent_info["following_agentid_list"][agent_id])
if not isinstance(following_id_list, int):
if len(following_id_list) != 0:
for follow_id in following_id_list:
follow_list.append((agent_id, follow_id, start_time))
user_update1.append((agent_id, ))
user_update2.append((follow_id, ))
agent_graph.add_edge(agent_id, follow_id)
previous_posts = ast.literal_eval(
agent_info["previous_tweets"][agent_id])
if len(previous_posts) != 0:
for post in previous_posts:
post_list.append((agent_id, post, start_time, 0, 0))
# generate_log.info('agent gegenerate finished.')
user_insert_query = (
"INSERT INTO user (user_id, agent_id, user_name, name, bio, "
"created_at, num_followings, num_followers) VALUES "
"(?, ?, ?, ?, ?, ?, ?, ?)")
twitter.pl_utils._execute_many_db_command(user_insert_query,
sign_up_list,
commit=True)
follow_insert_query = (
"INSERT INTO follow (follower_id, followee_id, created_at) "
"VALUES (?, ?, ?)")
twitter.pl_utils._execute_many_db_command(follow_insert_query,
follow_list,
commit=True)
user_update_query1 = (
"UPDATE user SET num_followings = num_followings + 1 "
"WHERE user_id = ?")
twitter.pl_utils._execute_many_db_command(user_update_query1,
user_update1,
commit=True)
user_update_query2 = ("UPDATE user SET num_followers = num_followers + 1 "
"WHERE user_id = ?")
twitter.pl_utils._execute_many_db_command(user_update_query2,
user_update2,
commit=True)
# generate_log.info('twitter followee update finished.')
post_insert_query = (
"INSERT INTO post (user_id, content, created_at, num_likes, "
"num_dislikes) VALUES (?, ?, ?, ?, ?)")
twitter.pl_utils._execute_many_db_command(post_insert_query,
post_list,
commit=True)
# generate_log.info('twitter creat post finished.')
return agent_graph
async def generate_agents_100w(
agent_info_path: str,
channel: Channel,
start_time,
model: Union[BaseModelBackend, List[BaseModelBackend]],
recsys_type: str = "twitter",
twitter: Platform = None,
available_actions: list[ActionType] = None,
) -> List:
""" TODO: need update the description of args.
Generate and return a dictionary of agents from the agent
information CSV file. Each agent is added to the database and
their respective profiles are updated.
Args:
agent_info_path (str): The file path to the agent information CSV file.
channel (Channel): Information channel.
action_space_prompt (str): determine the action space of agents.
model_random_seed (int): Random seed to randomly assign model to
each agent. (default: 42)
Returns:
dict: A dictionary of agent IDs mapped to their respective agent
class instances.
"""
agent_info = pd.read_csv(agent_info_path)
# TODO when setting 100w agents, the agentgraph class is too slow.
# I use the list.
agent_graph = []
# agent_graph = (AgentGraph() if neo4j_config is None else AgentGraph(
# backend="neo4j",
# neo4j_config=neo4j_config,
# ))
# agent_graph = []
sign_up_list = []
follow_list = []
user_update1 = []
user_update2 = []
post_list = []
# precompute to speed up agent generation in one million scale
_ = agent_info["following_agentid_list"].apply(ast.literal_eval)
previous_tweets_lists = agent_info["previous_tweets"].apply(
ast.literal_eval)
previous_tweets_lists = agent_info['previous_tweets'].apply(
ast.literal_eval)
following_id_lists = agent_info["following_agentid_list"].apply(
ast.literal_eval)
for agent_id in tqdm.tqdm(range(len(agent_info))):
profile = {
"nodes": [],
"edges": [],
"other_info": {},
}
profile["other_info"]["user_profile"] = agent_info["user_char"][
agent_id]
# TODO if you simulate one million agents, use active threshold below.
# profile['other_info']['active_threshold'] = [0.01] * 24
user_info = UserInfo(
name=agent_info["username"][agent_id],
description=agent_info["description"][agent_id],
profile=profile,
recsys_type=recsys_type,
)
agent = SocialAgent(
agent_id=agent_id,
user_info=user_info,
channel=channel,
model=model,
agent_graph=agent_graph,
available_actions=available_actions,
)
agent_graph.append(agent)
num_followings = 0
num_followers = 0
# print('agent_info["following_count"]', agent_info["following_count"])
# TODO some data does not cotain this key.
if 'following_count' not in agent_info.columns:
agent_info['following_count'] = 0
if 'followers_count' not in agent_info.columns:
agent_info['followers_count'] = 0
if not agent_info["following_count"].empty:
num_followings = agent_info["following_count"][agent_id]
if not agent_info["followers_count"].empty:
num_followers = agent_info["followers_count"][agent_id]
sign_up_list.append((
agent_id,
agent_id,
agent_info["username"][agent_id],
agent_info["name"][agent_id],
agent_info["description"][agent_id],
start_time,
num_followings,
num_followers,
))
following_id_list = following_id_lists[agent_id]
# TODO If we simulate 1 million agents, we can not use agent_graph
# class. It is not scalble.
if not isinstance(following_id_list, int):
if len(following_id_list) != 0:
for follow_id in following_id_list:
follow_list.append((agent_id, follow_id, start_time))
user_update1.append((agent_id, ))
user_update2.append((follow_id, ))
# agent_graph.add_edge(agent_id, follow_id)
previous_posts = previous_tweets_lists[agent_id]
if len(previous_posts) != 0:
for post in previous_posts:
post_list.append((agent_id, post, start_time, 0, 0))
# generate_log.info('agent gegenerate finished.')
user_insert_query = (
"INSERT INTO user (user_id, agent_id, user_name, name, bio, "
"created_at, num_followings, num_followers) VALUES "
"(?, ?, ?, ?, ?, ?, ?, ?)")
twitter.pl_utils._execute_many_db_command(user_insert_query,
sign_up_list,
commit=True)
follow_insert_query = (
"INSERT INTO follow (follower_id, followee_id, created_at) "
"VALUES (?, ?, ?)")
twitter.pl_utils._execute_many_db_command(follow_insert_query,
follow_list,
commit=True)
if not (agent_info["following_count"].empty
and agent_info["followers_count"].empty):
user_update_query1 = (
"UPDATE user SET num_followings = num_followings + 1 "
"WHERE user_id = ?")
twitter.pl_utils._execute_many_db_command(user_update_query1,
user_update1,
commit=True)
user_update_query2 = (
"UPDATE user SET num_followers = num_followers + 1 "
"WHERE user_id = ?")
twitter.pl_utils._execute_many_db_command(user_update_query2,
user_update2,
commit=True)
# generate_log.info('twitter followee update finished.')
post_insert_query = (
"INSERT INTO post (user_id, content, created_at, num_likes, "
"num_dislikes) VALUES (?, ?, ?, ?, ?)")
twitter.pl_utils._execute_many_db_command(post_insert_query,
post_list,
commit=True)
# generate_log.info('twitter creat post finished.')
return agent_graph
async def generate_controllable_agents(
channel: Channel,
control_user_num: int,
) -> tuple[AgentGraph, dict]:
agent_graph = AgentGraph()
agent_user_id_mapping = {}
for i in range(control_user_num):
user_info = UserInfo(
is_controllable=True,
profile={"other_info": {
"user_profile": "None"
}},
recsys_type="reddit",
)
# controllable的agent_id全都在llm agent的agent_id的前面
agent = SocialAgent(agent_id=i,
user_info=user_info,
channel=channel,
agent_graph=agent_graph)
# Add agent to the agent graph
agent_graph.add_agent(agent)
username = input(f"Please input username for agent {i}: ")
name = input(f"Please input name for agent {i}: ")
bio = input(f"Please input bio for agent {i}: ")
response = await agent.env.action.sign_up(username, name, bio)
user_id = response["user_id"]
agent_user_id_mapping[i] = user_id
for i in range(control_user_num):
for j in range(control_user_num):
agent = agent_graph.get_agent(i)
# controllable agent互相也全部关注
if i != j:
user_id = agent_user_id_mapping[j]
await agent.env.action.follow(user_id)
agent_graph.add_edge(i, j)
return agent_graph, agent_user_id_mapping
async def gen_control_agents_with_data(
channel: Channel,
control_user_num: int,
models: list[BaseModelBackend] | None = None,
) -> tuple[AgentGraph, dict]:
agent_graph = AgentGraph()
agent_user_id_mapping = {}
for i in range(control_user_num):
user_info = UserInfo(
is_controllable=True,
profile={
"other_info": {
"user_profile": "None",
"gender": "None",
"mbti": "None",
"country": "None",
"age": "None",
}
},
recsys_type="reddit",
)
# controllable的agent_id全都在llm agent的agent_id的前面
agent = SocialAgent(
agent_id=i,
user_info=user_info,
channel=channel,
agent_graph=agent_graph,
model=models,
available_actions=None,
)
# Add agent to the agent graph
agent_graph.add_agent(agent)
user_name = "momo"
name = "momo"
bio = "None."
response = await agent.env.action.sign_up(user_name, name, bio)
user_id = response["user_id"]
agent_user_id_mapping[i] = user_id
return agent_graph, agent_user_id_mapping
async def generate_reddit_agents(
agent_info_path: str,
channel: Channel,
agent_graph: AgentGraph | None = None,
agent_user_id_mapping: dict[int, int] | None = None,
follow_post_agent: bool = False,
mute_post_agent: bool = False,
model: Optional[Union[BaseModelBackend, List[BaseModelBackend],
ModelManager]] = None,
available_actions: list[ActionType] = None,
) -> AgentGraph:
if agent_user_id_mapping is None:
agent_user_id_mapping = {}
if agent_graph is None:
agent_graph = AgentGraph()
control_user_num = agent_graph.get_num_nodes()
with open(agent_info_path, "r") as file:
agent_info = json.load(file)
async def process_agent(i):
# Instantiate an agent
profile = {
"nodes": [], # Relationships with other agents
"edges": [], # Relationship details
"other_info": {},
}
# Update agent profile with additional information
profile["other_info"]["user_profile"] = agent_info[i]["persona"]
profile["other_info"]["mbti"] = agent_info[i]["mbti"]
profile["other_info"]["gender"] = agent_info[i]["gender"]
profile["other_info"]["age"] = agent_info[i]["age"]
profile["other_info"]["country"] = agent_info[i]["country"]
user_info = UserInfo(
name=agent_info[i]["username"],
description=agent_info[i]["bio"],
profile=profile,
recsys_type="reddit",
)
agent = SocialAgent(
agent_id=i + control_user_num,
user_info=user_info,
channel=channel,
agent_graph=agent_graph,
model=model,
available_actions=available_actions,
)
# Add agent to the agent graph
agent_graph.add_agent(agent)
# Sign up agent and add their information to the database
# print(f"Signing up agent {agent_info['username'][i]}...")
response = await agent.env.action.sign_up(agent_info[i]["username"],
agent_info[i]["realname"],
agent_info[i]["bio"])
user_id = response["user_id"]
agent_user_id_mapping[i + control_user_num] = user_id
if follow_post_agent:
await agent.env.action.follow(1)
content = """
{
"reason": "He is my friend, and I would like to follow him "
"on social media.",
"functions": [
{
"name": "follow",
"arguments": {
"user_id": 1
}
}
]
}
"""
agent_msg = BaseMessage.make_assistant_message(
role_name="Assistant", content=content)
agent.memory.write_record(
MemoryRecord(agent_msg, OpenAIBackendRole.ASSISTANT))
elif mute_post_agent:
await agent.env.action.mute(1)
content = """
{
"reason": "He is my enemy, and I would like to mute him on social media.",
"functions": [{
"name": "mute",
"arguments": {
"user_id": 1
}
}
"""
agent_msg = BaseMessage.make_assistant_message(
role_name="Assistant", content=content)
agent.memory.write_record(
MemoryRecord(agent_msg, OpenAIBackendRole.ASSISTANT))
tasks = [process_agent(i) for i in range(len(agent_info))]
await asyncio.gather(*tasks)
return agent_graph
def connect_platform_channel(
channel: Channel,
agent_graph: AgentGraph | None = None,
) -> AgentGraph:
for _, agent in agent_graph.get_agents():
agent.channel = channel
agent.env.action.channel = channel
return agent_graph
async def generate_custom_agents(
channel: Channel,
agent_graph: AgentGraph | None = None,
) -> AgentGraph:
if agent_graph is None:
agent_graph = AgentGraph()
agent_graph = connect_platform_channel(channel=channel,
agent_graph=agent_graph)
sign_up_tasks = [
agent.env.action.sign_up(user_name=agent.user_info.user_name,
name=agent.user_info.name,
bio=agent.user_info.description)
for _, agent in agent_graph.get_agents()
]
await asyncio.gather(*sign_up_tasks)
return agent_graph
async def generate_reddit_agent_graph(
profile_path: str,
model: Optional[Union[BaseModelBackend, List[BaseModelBackend],
ModelManager]] = None,
available_actions: list[ActionType] = None,
) -> AgentGraph:
agent_graph = AgentGraph()
with open(profile_path, "r") as file:
agent_info = json.load(file)
async def process_agent(i):
# Instantiate an agent
profile = {
"nodes": [], # Relationships with other agents
"edges": [], # Relationship details
"other_info": {},
}
# Update agent profile with additional information.
# Use .get() with sensible defaults so a malformed/missing-field
# profile (which happens when the LLM-emitted persona JSON fails to
# parse and the persona worker falls back to a basic structure) does
# not KeyError the whole simulation at start time. Hard-coding
# defaults here means a sim can run with a few sparse profiles
# instead of crashing on round 1.
profile["other_info"]["user_profile"] = (
agent_info[i].get("persona") or agent_info[i].get("bio", "")
)
profile["other_info"]["mbti"] = agent_info[i].get("mbti", "ISTJ")
profile["other_info"]["gender"] = agent_info[i].get("gender", "other")
profile["other_info"]["age"] = agent_info[i].get("age", 30)
profile["other_info"]["country"] = agent_info[i].get("country", "Unknown")
user_info = UserInfo(
name=agent_info[i]["username"],
description=agent_info[i]["bio"],
profile=profile,
recsys_type="reddit",
)
agent = SocialAgent(
agent_id=i,
user_info=user_info,
agent_graph=agent_graph,
model=model,
available_actions=available_actions,
)
# Add agent to the agent graph
agent_graph.add_agent(agent)
tasks = [process_agent(i) for i in range(len(agent_info))]
await asyncio.gather(*tasks)
return agent_graph
async def generate_twitter_agent_graph(
profile_path: str,
model: Optional[Union[BaseModelBackend, List[BaseModelBackend],
ModelManager]] = None,
available_actions: list[ActionType] = None,
) -> AgentGraph:
agent_info = pd.read_csv(profile_path)
agent_graph = AgentGraph()
for agent_id in range(len(agent_info)):
profile = {
"nodes": [],
"edges": [],
"other_info": {},
}
profile["other_info"]["user_profile"] = agent_info["user_char"][
agent_id]
user_info = UserInfo(
name=agent_info["username"][agent_id],
description=agent_info["description"][agent_id],
profile=profile,
recsys_type='twitter',
)
agent = SocialAgent(
agent_id=agent_id,
user_info=user_info,
model=model,
agent_graph=agent_graph,
available_actions=available_actions,
)
agent_graph.add_agent(agent)
return agent_graph

20
backend/vendor/camel-oasis/oasis/social_platform/__init__.py vendored Normal file
View File

@ -0,0 +1,20 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from .channel import Channel
from .platform import Platform
__all__ = [
"Channel",
"Platform",
]

71
backend/vendor/camel-oasis/oasis/social_platform/channel.py vendored Normal file
View File

@ -0,0 +1,71 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
import asyncio
import uuid
class AsyncSafeDict:
def __init__(self):
self.dict = {}
self.lock = asyncio.Lock()
async def put(self, key, value):
async with self.lock:
self.dict[key] = value
async def get(self, key, default=None):
async with self.lock:
return self.dict.get(key, default)
async def pop(self, key, default=None):
async with self.lock:
return self.dict.pop(key, default)
async def keys(self):
async with self.lock:
return list(self.dict.keys())
class Channel:
def __init__(self):
self.receive_queue = asyncio.Queue() # Used to store received messages
# Using an asynchronous safe dictionary to store messages to be sent
self.send_dict = AsyncSafeDict()
async def receive_from(self):
message = await self.receive_queue.get()
return message
async def send_to(self, message):
# message_id is the first element of the message
message_id = message[0]
await self.send_dict.put(message_id, message)
async def write_to_receive_queue(self, action_info):
message_id = str(uuid.uuid4())
await self.receive_queue.put((message_id, action_info))
return message_id
async def read_from_send_queue(self, message_id):
while True:
if message_id in await self.send_dict.keys():
# Attempting to retrieve the message
message = await self.send_dict.pop(message_id, None)
if message:
return message # Return the found message
# Temporarily suspend to avoid tight looping
await asyncio.sleep(
0.1) # set a large one to reduce the workload of cpu

20
backend/vendor/camel-oasis/oasis/social_platform/config/__init__.py vendored Normal file
View File

@ -0,0 +1,20 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from .neo4j import Neo4jConfig
from .user import UserInfo
__all__ = [
"UserInfo",
"Neo4jConfig",
]

24
backend/vendor/camel-oasis/oasis/social_platform/config/neo4j.py vendored Normal file
View File

@ -0,0 +1,24 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from dataclasses import dataclass
@dataclass
class Neo4jConfig:
uri: str | None = None
username: str | None = None
password: str | None = None
def is_valid(self) -> bool:
return all([self.uri, self.username, self.password])

111
backend/vendor/camel-oasis/oasis/social_platform/config/user.py vendored Normal file
View File

@ -0,0 +1,111 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# flake8: noqa: E501
import warnings
from dataclasses import dataclass
from typing import Any
from camel.prompts import TextPrompt
@dataclass
class UserInfo:
user_name: str | None = None
name: str | None = None
description: str | None = None
profile: dict[str, Any] | None = None
recsys_type: str = "twitter"
is_controllable: bool = False
def to_custom_system_message(self, user_info_template: TextPrompt) -> str:
required_keys = user_info_template.key_words
info_keys = set(self.profile.keys())
missing = required_keys - info_keys
extra = info_keys - required_keys
if missing:
raise ValueError(
f"Missing required keys in UserInfo.profile: {missing}")
if extra:
warnings.warn(f"Extra keys not used in UserInfo.profile: {extra}")
return user_info_template.format(**self.profile)
def to_system_message(self) -> str:
if self.recsys_type != "reddit":
return self.to_twitter_system_message()
else:
return self.to_reddit_system_message()
def to_twitter_system_message(self) -> str:
name_string = ""
description_string = ""
if self.name is not None:
name_string = f"Your name is {self.name}."
if self.profile is None:
description = name_string
elif "other_info" not in self.profile:
description = name_string
elif "user_profile" in self.profile["other_info"]:
if self.profile["other_info"]["user_profile"] is not None:
user_profile = self.profile["other_info"]["user_profile"]
description_string = f"Your have profile: {user_profile}."
description = f"{name_string}\n{description_string}"
system_content = f"""
# OBJECTIVE
You're a Twitter user, and I'll present you with some posts. After you see the posts, choose some actions from the following functions.
# SELF-DESCRIPTION
Your actions should be consistent with your self-description and personality.
{description}
# RESPONSE METHOD
Please perform actions by tool calling.
"""
return system_content
def to_reddit_system_message(self) -> str:
name_string = ""
description_string = ""
if self.name is not None:
name_string = f"Your name is {self.name}."
if self.profile is None:
description = name_string
elif "other_info" not in self.profile:
description = name_string
elif "user_profile" in self.profile["other_info"]:
if self.profile["other_info"]["user_profile"] is not None:
user_profile = self.profile["other_info"]["user_profile"]
description_string = f"Your have profile: {user_profile}."
description = f"{name_string}\n{description_string}"
print(self.profile['other_info'])
description += (
f"You are a {self.profile['other_info']['gender']}, "
f"{self.profile['other_info']['age']} years old, with an MBTI "
f"personality type of {self.profile['other_info']['mbti']} from "
f"{self.profile['other_info']['country']}.")
system_content = f"""
# OBJECTIVE
You're a Reddit user, and I'll present you with some tweets. After you see the tweets, choose some actions from the following functions.
# SELF-DESCRIPTION
Your actions should be consistent with your self-description and personality.
{description}
# RESPONSE METHOD
Please perform actions by tool calling.
"""
return system_content

291
backend/vendor/camel-oasis/oasis/social_platform/database.py vendored Normal file
View File

@ -0,0 +1,291 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from __future__ import annotations
import os
import os.path as osp
import sqlite3
from typing import Any, Dict, List
SCHEMA_DIR = "social_platform/schema"
DB_DIR = "data"
DB_NAME = "social_media.db"
USER_SCHEMA_SQL = "user.sql"
POST_SCHEMA_SQL = "post.sql"
FOLLOW_SCHEMA_SQL = "follow.sql"
MUTE_SCHEMA_SQL = "mute.sql"
LIKE_SCHEMA_SQL = "like.sql"
DISLIKE_SCHEMA_SQL = "dislike.sql"
REPORT_SCHEAM_SQL = "report.sql"
TRACE_SCHEMA_SQL = "trace.sql"
REC_SCHEMA_SQL = "rec.sql"
COMMENT_SCHEMA_SQL = "comment.sql"
COMMENT_LIKE_SCHEMA_SQL = "comment_like.sql"
COMMENT_DISLIKE_SCHEMA_SQL = "comment_dislike.sql"
PRODUCT_SCHEMA_SQL = "product.sql"
GROUP_SCHEMA_SQL = "chat_group.sql"
GROUP_MEMBER_SCHEMA_SQL = "group_member.sql"
GROUP_MESSAGE_SCHEMA_SQL = "group_message.sql"
TABLE_NAMES = {
"user",
"post",
"follow",
"mute",
"like",
"dislike",
"report",
"trace",
"rec",
"comment.sql",
"comment_like.sql",
"comment_dislike.sql",
"product.sql",
"group",
"group_member",
"group_message",
}
def get_db_path() -> str:
# First check if the database path is set in environment variables
env_db_path = os.environ.get("OASIS_DB_PATH")
if env_db_path:
return env_db_path
# If no environment variable is set, use the original default path
curr_file_path = osp.abspath(__file__)
parent_dir = osp.dirname(osp.dirname(curr_file_path))
db_dir = osp.join(parent_dir, DB_DIR)
os.makedirs(db_dir, exist_ok=True)
db_path = osp.join(db_dir, DB_NAME)
return db_path
def get_schema_dir_path() -> str:
curr_file_path = osp.abspath(__file__)
parent_dir = osp.dirname(osp.dirname(curr_file_path))
schema_dir = osp.join(parent_dir, SCHEMA_DIR)
return schema_dir
def create_db(db_path: str | None = None):
r"""Create the database if it does not exist. A :obj:`twitter.db`
file will be automatically created in the :obj:`data` directory.
"""
schema_dir = get_schema_dir_path()
if db_path is None:
db_path = get_db_path()
# Connect to the database:
print("db_path", db_path)
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
try:
# Read and execute the user table SQL script:
user_sql_path = osp.join(schema_dir, USER_SCHEMA_SQL)
with open(user_sql_path, "r") as sql_file:
user_sql_script = sql_file.read()
cursor.executescript(user_sql_script)
# Read and execute the post table SQL script:
post_sql_path = osp.join(schema_dir, POST_SCHEMA_SQL)
with open(post_sql_path, "r") as sql_file:
post_sql_script = sql_file.read()
cursor.executescript(post_sql_script)
# Read and execute the follow table SQL script:
follow_sql_path = osp.join(schema_dir, FOLLOW_SCHEMA_SQL)
with open(follow_sql_path, "r") as sql_file:
follow_sql_script = sql_file.read()
cursor.executescript(follow_sql_script)
# Read and execute the mute table SQL script:
mute_sql_path = osp.join(schema_dir, MUTE_SCHEMA_SQL)
with open(mute_sql_path, "r") as sql_file:
mute_sql_script = sql_file.read()
cursor.executescript(mute_sql_script)
# Read and execute the like table SQL script:
like_sql_path = osp.join(schema_dir, LIKE_SCHEMA_SQL)
with open(like_sql_path, "r") as sql_file:
like_sql_script = sql_file.read()
cursor.executescript(like_sql_script)
# Read and execute the dislike table SQL script:
dislike_sql_path = osp.join(schema_dir, DISLIKE_SCHEMA_SQL)
with open(dislike_sql_path, "r") as sql_file:
dislike_sql_script = sql_file.read()
cursor.executescript(dislike_sql_script)
# Read and execute the report table SQL script:
report_sql_path = osp.join(schema_dir, REPORT_SCHEAM_SQL)
with open(report_sql_path, "r") as sql_file:
report_sql_script = sql_file.read()
cursor.executescript(report_sql_script)
# Read and execute the trace table SQL script:
trace_sql_path = osp.join(schema_dir, TRACE_SCHEMA_SQL)
with open(trace_sql_path, "r") as sql_file:
trace_sql_script = sql_file.read()
cursor.executescript(trace_sql_script)
# Read and execute the rec table SQL script:
rec_sql_path = osp.join(schema_dir, REC_SCHEMA_SQL)
with open(rec_sql_path, "r") as sql_file:
rec_sql_script = sql_file.read()
cursor.executescript(rec_sql_script)
# Read and execute the comment table SQL script:
comment_sql_path = osp.join(schema_dir, COMMENT_SCHEMA_SQL)
with open(comment_sql_path, "r") as sql_file:
comment_sql_script = sql_file.read()
cursor.executescript(comment_sql_script)
# Read and execute the comment_like table SQL script:
comment_like_sql_path = osp.join(schema_dir, COMMENT_LIKE_SCHEMA_SQL)
with open(comment_like_sql_path, "r") as sql_file:
comment_like_sql_script = sql_file.read()
cursor.executescript(comment_like_sql_script)
# Read and execute the comment_dislike table SQL script:
comment_dislike_sql_path = osp.join(schema_dir,
COMMENT_DISLIKE_SCHEMA_SQL)
with open(comment_dislike_sql_path, "r") as sql_file:
comment_dislike_sql_script = sql_file.read()
cursor.executescript(comment_dislike_sql_script)
# Read and execute the product table SQL script:
product_sql_path = osp.join(schema_dir, PRODUCT_SCHEMA_SQL)
with open(product_sql_path, "r") as sql_file:
product_sql_script = sql_file.read()
cursor.executescript(product_sql_script)
# Read and execute the group table SQL script:
group_sql_path = osp.join(schema_dir, GROUP_SCHEMA_SQL)
with open(group_sql_path, "r") as sql_file:
group_sql_script = sql_file.read()
cursor.executescript(group_sql_script)
# Read and execute the group_member table SQL script:
group_member_sql_path = osp.join(schema_dir, GROUP_MEMBER_SCHEMA_SQL)
with open(group_member_sql_path, "r") as sql_file:
group_member_sql_script = sql_file.read()
cursor.executescript(group_member_sql_script)
# Read and execute the group_message table SQL script:
group_message_sql_path = osp.join(schema_dir, GROUP_MESSAGE_SCHEMA_SQL)
with open(group_message_sql_path, "r") as sql_file:
group_message_sql_script = sql_file.read()
cursor.executescript(group_message_sql_script)
# Commit the changes:
conn.commit()
except sqlite3.Error as e:
print(f"An error occurred while creating tables: {e}")
return conn, cursor
def print_db_tables_summary():
# Connect to the SQLite database
db_path = get_db_path()
conn = sqlite3.connect(db_path)
cursor = conn.cursor()
# Retrieve a list of all tables in the database
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
tables = cursor.fetchall()
# Print a summary of each table
for table in tables:
table_name = table[0]
if table_name not in TABLE_NAMES:
continue
print(f"Table: {table_name}")
# Retrieve the table schema
cursor.execute(f"PRAGMA table_info({table_name})")
columns = cursor.fetchall()
column_names = [column[1] for column in columns]
print("- Columns:", column_names)
# Retrieve and print foreign key information
cursor.execute(f"PRAGMA foreign_key_list({table_name})")
foreign_keys = cursor.fetchall()
if foreign_keys:
print("- Foreign Keys:")
for fk in foreign_keys:
print(f" {fk[2]} references {fk[3]}({fk[4]}) on update "
f"{fk[5]} on delete {fk[6]}")
else:
print(" No foreign keys.")
# Print the first few rows of the table
cursor.execute(f"SELECT * FROM {table_name} LIMIT 5;")
rows = cursor.fetchall()
for row in rows:
print(row)
print() # Adds a newline for better readability between tables
# Close the database connection
conn.close()
def fetch_table_from_db(cursor: sqlite3.Cursor,
table_name: str) -> List[Dict[str, Any]]:
cursor.execute(f"SELECT * FROM {table_name}")
columns = [description[0] for description in cursor.description]
data_dicts = [dict(zip(columns, row)) for row in cursor.fetchall()]
return data_dicts
def fetch_rec_table_as_matrix(cursor: sqlite3.Cursor) -> List[List[int]]:
# First, query all user_ids from the user table, assuming they start from
# 1 and are consecutive
cursor.execute("SELECT user_id FROM user ORDER BY user_id")
user_ids = [row[0] for row in cursor.fetchall()]
# Then, query all records from the rec table
cursor.execute(
"SELECT user_id, post_id FROM rec ORDER BY user_id, post_id")
rec_rows = cursor.fetchall()
# Initialize a dictionary, assigning an empty list to each user_id
user_posts = {user_id: [] for user_id in user_ids}
# Fill the dictionary with the records queried from the rec table
for user_id, post_id in rec_rows:
if user_id in user_posts:
user_posts[user_id].append(post_id)
# Convert the dictionary into matrix form
matrix = [user_posts[user_id] for user_id in user_ids]
return matrix
def insert_matrix_into_rec_table(cursor: sqlite3.Cursor,
matrix: List[List[int]]) -> None:
# Iterate through the matrix, skipping the placeholder at index 0
for user_id, post_ids in enumerate(matrix, start=1):
# Adjusted to start counting from 1
for post_id in post_ids:
# Insert each combination of user_id and post_id into the rec table
cursor.execute("INSERT INTO rec (user_id, post_id) VALUES (?, ?)",
(user_id, post_id))
if __name__ == "__main__":
create_db()
print_db_tables_summary()

1642
backend/vendor/camel-oasis/oasis/social_platform/platform.py vendored Normal file
View File

File diff suppressed because it is too large Load Diff

262
backend/vendor/camel-oasis/oasis/social_platform/platform_utils.py vendored Normal file
View File

@ -0,0 +1,262 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
import json
from datetime import datetime
from oasis.social_platform.typing import RecsysType
class PlatformUtils:
def __init__(self,
db,
db_cursor,
start_time,
sandbox_clock,
show_score,
recsys_type,
report_threshold=1):
self.db = db
self.db_cursor = db_cursor
self.start_time = start_time
self.sandbox_clock = sandbox_clock
self.show_score = show_score
self.recsys_type = recsys_type
self.report_threshold = report_threshold
@staticmethod
def _not_signup_error_message(agent_id):
return {
"success":
False,
"error": (f"Agent {agent_id} has not signed up and does not have "
f"a user id."),
}
def _execute_db_command(self, command, args=(), commit=False):
self.db_cursor.execute(command, args)
if commit:
self.db.commit()
return self.db_cursor
def _execute_many_db_command(self, command, args_list, commit=False):
self.db_cursor.executemany(command, args_list)
if commit:
self.db.commit()
return self.db_cursor
def _check_agent_userid(self, agent_id):
try:
user_query = "SELECT user_id FROM user WHERE agent_id = ?"
results = self._execute_db_command(user_query, (agent_id, ))
# Fetch the first row of the query result
first_row = results.fetchone()
if first_row:
user_id = first_row[0]
return user_id
else:
return None
except Exception as e:
# Log or handle the error as appropriate
print(f"Error querying user_id for agent_id {agent_id}: {e}")
return None
def _add_comments_to_posts(self, posts_results):
# Initialize the returned posts list
posts = []
for row in posts_results:
(post_id, user_id, original_post_id, content, quote_content,
created_at, num_likes, num_dislikes, num_shares) = row
post_type_result = self._get_post_type(post_id)
if post_type_result is None:
continue
original_user_id_query = (
"SELECT user_id FROM post WHERE post_id = ?")
if post_type_result["type"] == "repost":
self.db_cursor.execute(original_user_id_query,
(original_post_id, ))
original_user_id = self.db_cursor.fetchone()[0]
original_post_id = post_id
post_id = post_type_result["root_post_id"]
self.db_cursor.execute(
"SELECT content, quote_content, created_at, num_likes, "
"num_dislikes, num_shares, num_reports FROM post "
"WHERE post_id = ?", (post_id, ))
original_post_result = self.db_cursor.fetchone()
(content, quote_content, created_at, num_likes, num_dislikes,
num_shares, num_reports) = original_post_result
post_content = (
f"User {user_id} reposted a post from User "
f"{original_user_id}. Repost content: {content}. ")
elif post_type_result["type"] == "quote":
self.db_cursor.execute(original_user_id_query,
(original_post_id, ))
original_user_id = self.db_cursor.fetchone()[0]
post_content = (
f"User {user_id} quoted a post from User "
f"{original_user_id}. Quote content: {quote_content}. "
f"Original Content: {content}")
elif post_type_result["type"] == "common":
post_content = content
# Get num_reports for common posts
self.db_cursor.execute(
"SELECT num_reports FROM post WHERE post_id = ?",
(post_id, ))
num_reports = self.db_cursor.fetchone()[0]
# For each post, query its corresponding comments
self.db_cursor.execute(
"SELECT comment_id, post_id, user_id, content, created_at, "
"num_likes, num_dislikes FROM comment WHERE post_id = ?",
(post_id, ),
)
comments_results = self.db_cursor.fetchall()
# Convert each comment's result into dictionary format
comments = [{
"comment_id":
comment_id,
"post_id":
post_id,
"user_id":
user_id,
"content":
content,
"created_at":
created_at,
**({
"score": num_likes - num_dislikes
} if self.show_score else {
"num_likes": num_likes,
"num_dislikes": num_dislikes
}),
} for (
comment_id,
post_id,
user_id,
content,
created_at,
num_likes,
num_dislikes,
) in comments_results]
# Add warning message if the post has been reported
if num_reports >= self.report_threshold:
warning_message = ("[Warning: This post has been reported"
f" {num_reports} times]")
post_content = f"{warning_message}\n{post_content}"
# Add post information and corresponding comments to the posts list
posts.append({
"post_id":
post_id
if post_type_result["type"] != "repost" else original_post_id,
"user_id":
user_id,
"content":
post_content,
"created_at":
created_at,
**({
"score": num_likes - num_dislikes
} if self.show_score else {
"num_likes": num_likes,
"num_dislikes": num_dislikes
}),
"num_shares":
num_shares,
"num_reports":
num_reports,
"comments":
comments,
})
return posts
def _record_trace(self,
user_id,
action_type,
action_info,
current_time=None):
r"""If, in addition to the trace, the operation function also records
time in other tables of the database, use the time of entering
the operation function for consistency.
Pass in current_time to make, for example, the created_at in the post
table exactly the same as the time in the trace table.
If only the trace table needs to record time, use the entry time into
_record_trace as the time for the trace record.
"""
if self.recsys_type == RecsysType.REDDIT:
current_time = self.sandbox_clock.time_transfer(
datetime.now(), self.start_time)
else:
current_time = self.sandbox_clock.get_time_step()
trace_insert_query = (
"INSERT INTO trace (user_id, created_at, action, info) "
"VALUES (?, ?, ?, ?)")
action_info_str = json.dumps(action_info)
self._execute_db_command(
trace_insert_query,
(user_id, current_time, action_type, action_info_str),
commit=True,
)
def _check_self_post_rating(self, post_id, user_id):
self_like_check_query = "SELECT user_id FROM post WHERE post_id = ?"
self._execute_db_command(self_like_check_query, (post_id, ))
result = self.db_cursor.fetchone()
if result and result[0] == user_id:
error_message = ("Users are not allowed to like/dislike their own "
"posts.")
return {"success": False, "error": error_message}
else:
return None
def _check_self_comment_rating(self, comment_id, user_id):
self_like_check_query = ("SELECT user_id FROM comment WHERE "
"comment_id = ?")
self._execute_db_command(self_like_check_query, (comment_id, ))
result = self.db_cursor.fetchone()
if result and result[0] == user_id:
error_message = ("Users are not allowed to like/dislike their "
"own comments.")
return {"success": False, "error": error_message}
else:
return None
def _get_post_type(self, post_id: int):
query = (
"SELECT original_post_id, quote_content FROM post WHERE post_id "
"= ?")
self._execute_db_command(query, (post_id, ))
result = self.db_cursor.fetchone()
if not result:
return None
original_post_id, quote_content = result
if original_post_id is None:
# common post without quote or repost
return {"type": "common", "root_post_id": None}
elif quote_content is None:
# post with repost
return {"type": "repost", "root_post_id": original_post_id}
else:
# post with quote
return {"type": "quote", "root_post_id": original_post_id}

81
backend/vendor/camel-oasis/oasis/social_platform/process_recsys_posts.py vendored Normal file
View File

@ -0,0 +1,81 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from typing import List
import torch
from camel.embeddings import OpenAIEmbedding
from camel.types import EmbeddingModelType
from transformers import AutoModel, AutoTokenizer
# Function: Process each batch
@torch.no_grad()
def process_batch(model: AutoModel, tokenizer: AutoTokenizer,
batch_texts: List[str]):
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
inputs = tokenizer(batch_texts,
return_tensors="pt",
padding=True,
truncation=True)
inputs = {key: value.to(device) for key, value in inputs.items()}
outputs = model(**inputs)
return outputs.pooler_output
def generate_post_vector(model: AutoModel, tokenizer: AutoTokenizer, texts,
batch_size):
# Loop through all messages
# If the list of messages is too large, process them in batches.
all_outputs = []
for i in range(0, len(texts), batch_size):
batch_texts = texts[i:i + batch_size]
batch_outputs = process_batch(model, tokenizer, batch_texts)
all_outputs.append(batch_outputs)
all_outputs_tensor = torch.cat(all_outputs, dim=0) # num_posts x dimension
return all_outputs_tensor.cpu()
def generate_post_vector_openai(texts: List[str], batch_size: int = 100):
"""
Generate embeddings using OpenAI API
Args:
texts: List of texts to process
batch_size: Size of each batch
"""
openai_embedding = OpenAIEmbedding(
model_type=EmbeddingModelType.TEXT_EMBEDDING_3_SMALL)
all_embeddings = []
for i in range(0, len(texts), batch_size):
batch_texts = texts[i:i + batch_size]
cleaned_texts = [
text.strip() if text and isinstance(text, str) else "empty"
for text in batch_texts
]
batch_embeddings = openai_embedding.embed_list(objs=cleaned_texts)
batch_tensor = torch.tensor(batch_embeddings)
all_embeddings.append(batch_tensor)
return torch.cat(all_embeddings, dim=0)
if __name__ == "__main__":
# Input list of strings (assuming there are tens of thousands of messages)
# Here, the same message is repeated 10000 times as an example
texts = ["I'm using TwHIN-BERT! #TwHIN-BERT #NLP"] * 10000
# Define batch size
batch_size = 100
all_outputs_tensor = generate_post_vector(texts, batch_size)
print(all_outputs_tensor.shape)

797
backend/vendor/camel-oasis/oasis/social_platform/recsys.py vendored Normal file
View File

@ -0,0 +1,797 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
'''Note that you need to check if it exceeds max_rec_post_len when writing
into rec_matrix'''
import heapq
import logging
import random
import time
from ast import literal_eval
from datetime import datetime
from math import log
from typing import Any, Dict, List
import numpy as np
import torch
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from .process_recsys_posts import (generate_post_vector,
generate_post_vector_openai)
from .typing import ActionType, RecsysType
rec_log = logging.getLogger(name='social.rec')
rec_log.setLevel('DEBUG')
# Initially set to None, to be assigned once again in the recsys function
model = None
twhin_tokenizer = None
twhin_model = None
# Create the TF-IDF model
tfidf_vectorizer = TfidfVectorizer()
# Prepare the twhin model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# All historical tweets and the most recent tweet of each user
user_previous_post_all = {}
user_previous_post = {}
user_profiles = []
# Get the {post_id: content} dict
t_items = {}
# Get the {uid: follower_count} dict
# It's necessary to ensure that agent registration is sequential, with the
# relationship of user_id=agent_id+1; disorder in registration will cause
# issues here
u_items = {}
# Get the creation times of all tweets, assigning scores based on how recent
# they are
date_score = []
def get_twhin_tokenizer():
global twhin_tokenizer
if twhin_tokenizer is None:
from transformers import AutoTokenizer
twhin_tokenizer = AutoTokenizer.from_pretrained(
pretrained_model_name_or_path="Twitter/twhin-bert-base",
model_max_length=512)
return twhin_tokenizer
def get_twhin_model(device):
global twhin_model
if twhin_model is None:
from transformers import AutoModel
twhin_model = AutoModel.from_pretrained(
pretrained_model_name_or_path="Twitter/twhin-bert-base").to(device)
return twhin_model
def load_model(model_name):
try:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
if model_name == 'paraphrase-MiniLM-L6-v2':
return SentenceTransformer(model_name,
device=device,
cache_folder="./models")
elif model_name == 'Twitter/twhin-bert-base':
twhin_tokenizer = get_twhin_tokenizer()
twhin_model = get_twhin_model(device)
return twhin_tokenizer, twhin_model
else:
raise ValueError(f"Unknown model name: {model_name}")
except Exception as e:
raise Exception(f"Failed to load the model: {model_name}") from e
def get_recsys_model(recsys_type: str = None):
if recsys_type == RecsysType.TWITTER.value:
model = load_model('paraphrase-MiniLM-L6-v2')
return model
elif recsys_type == RecsysType.TWHIN.value:
twhin_tokenizer, twhin_model = load_model("Twitter/twhin-bert-base")
models = (twhin_tokenizer, twhin_model)
return models
elif (recsys_type == RecsysType.REDDIT.value
or recsys_type == RecsysType.RANDOM.value):
return None
else:
raise ValueError(f"Unknown recsys type: {recsys_type}")
# Move model to GPU if available
device = 'cuda' if torch.cuda.is_available() else 'cpu'
if model is not None:
model.to(device)
else:
pass
# Reset global variables
def reset_globals():
global user_previous_post_all, user_previous_post
global user_profiles, t_items, u_items
global date_score
user_previous_post_all = {}
user_previous_post = {}
user_profiles = []
t_items = {}
u_items = {}
date_score = []
def rec_sys_random(post_table: List[Dict[str, Any]], rec_matrix: List[List],
max_rec_post_len: int) -> List[List]:
"""
Randomly recommend posts to users.
Args:
user_table (List[Dict[str, Any]]): List of users.
post_table (List[Dict[str, Any]]): List of posts.
trace_table (List[Dict[str, Any]]): List of user interactions.
rec_matrix (List[List]): Existing recommendation matrix.
max_rec_post_len (int): Maximum number of recommended posts.
Returns:
List[List]: Updated recommendation matrix.
"""
# Get all post IDs
post_ids = [post['post_id'] for post in post_table]
new_rec_matrix = []
if len(post_ids) <= max_rec_post_len:
# If the number of posts is less than or equal to the maximum number
# of recommendations, each user gets all post IDs
new_rec_matrix = [post_ids] * len(rec_matrix)
else:
# If the number of posts is greater than the maximum number of
# recommendations, each user randomly gets a specified number of post
# IDs
for _ in range(len(rec_matrix)):
new_rec_matrix.append(random.sample(post_ids, max_rec_post_len))
return new_rec_matrix
def calculate_hot_score(num_likes: int, num_dislikes: int,
created_at: datetime) -> int:
"""
Compute the hot score for a post.
Args:
num_likes (int): Number of likes.
num_dislikes (int): Number of dislikes.
created_at (datetime): Creation time of the post.
Returns:
int: Hot score of the post.
Reference:
https://medium.com/hacking-and-gonzo/how-reddit-ranking-algorithms-work-ef111e33d0d9
"""
s = num_likes - num_dislikes
order = log(max(abs(s), 1), 10)
sign = 1 if s > 0 else -1 if s < 0 else 0
# epoch_seconds
epoch = datetime(1970, 1, 1)
td = created_at - epoch
epoch_seconds_result = td.days * 86400 + td.seconds + (
float(td.microseconds) / 1e6)
seconds = epoch_seconds_result - 1134028003
return round(sign * order + seconds / 45000, 7)
def get_recommendations(
user_index,
cosine_similarities,
items,
score,
top_n=100,
):
similarities = np.array(cosine_similarities[user_index])
similarities = similarities * score
top_item_indices = similarities.argsort()[::-1][:top_n]
recommended_items = [(list(items.keys())[i], similarities[i])
for i in top_item_indices]
return recommended_items
def rec_sys_reddit(post_table: List[Dict[str, Any]], rec_matrix: List[List],
max_rec_post_len: int) -> List[List]:
"""
Recommend posts based on Reddit-like hot score.
Args:
post_table (List[Dict[str, Any]]): List of posts.
rec_matrix (List[List]): Existing recommendation matrix.
max_rec_post_len (int): Maximum number of recommended posts.
Returns:
List[List]: Updated recommendation matrix.
"""
# Get all post IDs
post_ids = [post['post_id'] for post in post_table]
if len(post_ids) <= max_rec_post_len:
# If the number of posts is less than or equal to the maximum number
# of recommendations, each user gets all post IDs
new_rec_matrix = [post_ids] * len(rec_matrix)
else:
# The time complexity of this recommendation system is
# O(post_num * log max_rec_post_len)
all_hot_score = []
for post in post_table:
try:
created_at_dt = datetime.strptime(post['created_at'],
"%Y-%m-%d %H:%M:%S.%f")
except Exception:
created_at_dt = datetime.strptime(post['created_at'],
"%Y-%m-%d %H:%M:%S")
hot_score = calculate_hot_score(post['num_likes'],
post['num_dislikes'],
created_at_dt)
all_hot_score.append((hot_score, post['post_id']))
# Sort
top_posts = heapq.nlargest(max_rec_post_len,
all_hot_score,
key=lambda x: x[0])
top_post_ids = [post_id for _, post_id in top_posts]
# If the number of posts is greater than the maximum number of
# recommendations, each user gets a specified number of post IDs
# randomly
new_rec_matrix = [top_post_ids] * len(rec_matrix)
return new_rec_matrix
def rec_sys_personalized(user_table: List[Dict[str, Any]],
post_table: List[Dict[str, Any]],
trace_table: List[Dict[str,
Any]], rec_matrix: List[List],
max_rec_post_len: int) -> List[List]:
"""
Recommend posts based on personalized similarity scores.
Args:
user_table (List[Dict[str, Any]]): List of users.
post_table (List[Dict[str, Any]]): List of posts.
trace_table (List[Dict[str, Any]]): List of user interactions.
rec_matrix (List[List]): Existing recommendation matrix.
max_rec_post_len (int): Maximum number of recommended posts.
Returns:
List[List]: Updated recommendation matrix.
"""
global model
if model is None or isinstance(model, tuple):
model = get_recsys_model(recsys_type="twitter")
post_ids = [post['post_id'] for post in post_table]
print(
f'Running personalized recommendation for {len(user_table)} users...')
start_time = time.time()
new_rec_matrix = []
if len(post_ids) <= max_rec_post_len:
# If the number of posts is less than or equal to the maximum
# recommended length, each user gets all post IDs
new_rec_matrix = [post_ids] * len(rec_matrix)
else:
# If the number of posts is greater than the maximum recommended
# length, each user gets personalized post IDs
user_bios = [
user['bio'] if 'bio' in user and user['bio'] is not None else ''
for user in user_table
]
post_contents = [post['content'] for post in post_table]
if model:
user_embeddings = model.encode(user_bios,
convert_to_tensor=True,
device=device)
post_embeddings = model.encode(post_contents,
convert_to_tensor=True,
device=device)
# Compute dot product similarity
dot_product = torch.matmul(user_embeddings, post_embeddings.T)
# Compute norm
user_norms = torch.norm(user_embeddings, dim=1)
post_norms = torch.norm(post_embeddings, dim=1)
# Compute cosine similarity
similarities = dot_product / (user_norms[:, None] *
post_norms[None, :])
else:
# Generate random similarities
similarities = torch.rand(len(user_table), len(post_table))
# Iterate through each user to generate personalized recommendations.
for user_index, user in enumerate(user_table):
# Filter out posts made by the current user.
filtered_post_indices = [
i for i, post in enumerate(post_table)
if post['user_id'] != user['user_id']
]
user_similarities = similarities[user_index, filtered_post_indices]
# Get the corresponding post IDs for the filtered posts.
filtered_post_ids = [
post_table[i]['post_id'] for i in filtered_post_indices
]
# Determine the top posts based on the similarities, limited by
# max_rec_post_len.
_, top_indices = torch.topk(user_similarities,
k=min(max_rec_post_len,
len(filtered_post_ids)))
top_post_ids = [filtered_post_ids[i] for i in top_indices.tolist()]
# Append the top post IDs to the new recommendation matrix.
new_rec_matrix.append(top_post_ids)
end_time = time.time()
print(f'Personalized recommendation time: {end_time - start_time:.6f}s')
return new_rec_matrix
def get_like_post_id(user_id, action, trace_table):
"""
Get the post IDs that a user has liked or unliked.
Args:
user_id (str): ID of the user.
action (str): Type of action (like or unlike).
post_table (list): List of posts.
trace_table (list): List of user interactions.
Returns:
list: List of post IDs.
"""
# Get post IDs from trace table for the given user and action
trace_post_ids = [
literal_eval(trace['info'])["post_id"] for trace in trace_table
if (trace['user_id'] == user_id and trace['action'] == action)
]
"""Only take the last 5 liked posts, if not enough, pad with the most
recently liked post. Only take IDs, not content, because calculating
embeddings for all posts again is very time-consuming, especially when the
number of agents is large"""
if len(trace_post_ids) < 5 and len(trace_post_ids) > 0:
trace_post_ids += [trace_post_ids[-1]] * (5 - len(trace_post_ids))
elif len(trace_post_ids) > 5:
trace_post_ids = trace_post_ids[-5:]
else:
trace_post_ids = [0]
return trace_post_ids
# Calculate the average cosine similarity between liked posts and target posts
def calculate_like_similarity(liked_vectors, target_vectors):
# Calculate the norms of the vectors
liked_norms = np.linalg.norm(liked_vectors, axis=1)
target_norms = np.linalg.norm(target_vectors, axis=1)
# Calculate dot products
dot_products = np.dot(target_vectors, liked_vectors.T)
# Calculate cosine similarities
cosine_similarities = dot_products / np.outer(target_norms, liked_norms)
# Take the average
average_similarities = np.mean(cosine_similarities, axis=1)
return average_similarities
def coarse_filtering(input_list, scale):
"""
Coarse filtering posts and return selected elements with their indices.
"""
if len(input_list) <= scale:
# Return elements and their indices as list of tuples (element, index)
sampled_indices = range(len(input_list))
return (input_list, sampled_indices)
else:
# Get random sample of scale elements
sampled_indices = random.sample(range(len(input_list)), scale)
sampled_elements = [input_list[idx] for idx in sampled_indices]
# return [(input_list[idx], idx) for idx in sampled_indices]
return (sampled_elements, sampled_indices)
def rec_sys_personalized_twh(
user_table: List[Dict[str, Any]],
post_table: List[Dict[str, Any]],
latest_post_count: int,
trace_table: List[Dict[str, Any]],
rec_matrix: List[List],
max_rec_post_len: int,
current_time: int,
# source_post_indexs: List[int],
recall_only: bool = False,
enable_like_score: bool = False,
use_openai_embedding: bool = False) -> List[List]:
global twhin_model, twhin_tokenizer
if twhin_model is None or twhin_tokenizer is None:
twhin_tokenizer, twhin_model = get_recsys_model(
recsys_type="twhin-bert")
# Set some global variables to reduce time consumption
global date_score, t_items, u_items, user_previous_post
global user_previous_post_all, user_profiles
# Get the uid: follower_count dict
# Update only once, unless adding the feature to include new users midway.
if (not u_items) or len(u_items) != len(user_table):
u_items = {
user['user_id']: user["num_followers"]
for user in user_table
}
if not user_previous_post_all or len(user_previous_post_all) != len(
user_table):
# Each user must have a list of historical tweets
user_previous_post_all = {
index: []
for index in range(len(user_table))
}
user_previous_post = {index: "" for index in range(len(user_table))}
if not user_profiles or len(user_profiles) != len(user_table):
for user in user_table:
if user['bio'] is None:
user_profiles.append('This user does not have profile')
else:
user_profiles.append(user['bio'])
if len(t_items) < len(post_table):
for post in post_table[-latest_post_count:]:
# Get the {post_id: content} dict, update only the latest tweets
t_items[post['post_id']] = post['content']
# Update the user's historical tweets
user_previous_post_all[post['user_id']].append(post['content'])
user_previous_post[post['user_id']] = post['content']
# Get the creation times of all tweets, assigning scores based on
# how recent they are, note that this algorithm can run for a
# maximum of 90 time steps
date_score.append(
np.log(
(271.8 - (current_time - int(post['created_at']))) / 100))
date_score_np = np.array(date_score)
if enable_like_score:
# Calculate similarity with previously liked content, first gather
# liked post ids from the trace
like_post_ids_all = []
for user in user_table:
user_id = user['agent_id']
like_post_ids = get_like_post_id(user_id,
ActionType.LIKE_POST.value,
trace_table)
like_post_ids_all.append(like_post_ids)
scores = date_score_np
new_rec_matrix = []
if len(post_table) <= max_rec_post_len:
# If the number of tweets is less than or equal to the max
# recommendation count, each user gets all post IDs
tids = [t['post_id'] for t in post_table]
new_rec_matrix = [tids] * (len(rec_matrix))
else:
# If the number of tweets is greater than the max recommendation
# count, each user randomly gets personalized post IDs
# This requires going through all users to update their profiles,
# which is a time-consuming operation
for post_user_index in user_previous_post:
try:
# Directly replacing the profile with the latest tweet will
# cause the recommendation system to repeatedly push other
# reposts to users who have already shared that tweet
# user_profiles[post_user_index] =
# user_previous_post[post_user_index]
# Instead, append the description of the Recent post's content
# to the end of the user char
update_profile = (
f" # Recent post:{user_previous_post[post_user_index]}")
if user_previous_post[post_user_index] != "":
# If there's no update for the recent post, add this part
if "# Recent post:" not in user_profiles[post_user_index]:
user_profiles[post_user_index] += update_profile
# If the profile has a recent post but it's not the user's
# latest, replace it
elif update_profile not in user_profiles[post_user_index]:
user_profiles[post_user_index] = user_profiles[
post_user_index].split(
"# Recent post:")[0] + update_profile
except Exception:
print("update previous post failed")
# coarse filtering 4000 posts due to the memory constraint.
filtered_posts_tuple = coarse_filtering(list(t_items.values()), 4000)
corpus = user_profiles + filtered_posts_tuple[0]
# corpus = user_profiles + list(t_items.values())
tweet_vector_start_t = time.time()
if use_openai_embedding:
all_post_vector_list = generate_post_vector_openai(corpus,
batch_size=1000)
else:
all_post_vector_list = generate_post_vector(twhin_model,
twhin_tokenizer,
corpus,
batch_size=1000)
tweet_vector_end_t = time.time()
rec_log.info(
f"twhin model cost time: {tweet_vector_end_t-tweet_vector_start_t}"
)
user_vector = all_post_vector_list[:len(user_profiles)]
posts_vector = all_post_vector_list[len(user_profiles):]
if enable_like_score:
# Traverse all liked post ids, collecting liked post vectors from
# posts_vector for matrix acceleration calculation
like_posts_vectors = []
for user_idx, like_post_ids in enumerate(like_post_ids_all):
if len(like_post_ids) != 1:
for like_post_id in like_post_ids:
try:
like_posts_vectors.append(
posts_vector[like_post_id - 1])
except Exception:
like_posts_vectors.append(user_vector[user_idx])
else:
like_posts_vectors += [
user_vector[user_idx] for _ in range(5)
]
try:
like_posts_vectors = torch.stack(like_posts_vectors).view(
len(user_table), 5, posts_vector.shape[1])
except Exception:
import pdb # noqa: F811
pdb.set_trace()
get_similar_start_t = time.time()
cosine_similarities = cosine_similarity(user_vector, posts_vector)
get_similar_end_t = time.time()
rec_log.info(f"get cosine_similarity time: "
f"{get_similar_end_t-get_similar_start_t}")
if enable_like_score:
for user_index, profile in enumerate(user_profiles):
user_like_posts_vector = like_posts_vectors[user_index]
like_scores = calculate_like_similarity(
user_like_posts_vector, posts_vector)
try:
scores = scores + like_scores
except Exception:
import pdb
pdb.set_trace()
filter_posts_index = filtered_posts_tuple[1]
cosine_similarities = cosine_similarities * scores[filter_posts_index]
cosine_similarities = torch.tensor(cosine_similarities)
value, indices = torch.topk(cosine_similarities,
max_rec_post_len,
dim=1,
largest=True,
sorted=True)
filter_posts_index = torch.tensor(filter_posts_index)
indices = filter_posts_index[indices]
# cosine_similarities = cosine_similarities * scores
# cosine_similarities = torch.tensor(cosine_similarities)
# value, indices = torch.topk(cosine_similarities,
# max_rec_post_len,
# dim=1,
# largest=True,
# sorted=True)
matrix_list = indices.cpu().numpy()
post_list = list(t_items.keys())
for rec_ids in matrix_list:
rec_ids = [post_list[i] for i in rec_ids]
new_rec_matrix.append(rec_ids)
return new_rec_matrix
def normalize_similarity_adjustments(post_scores, base_similarity,
like_similarity, dislike_similarity):
"""
Normalize the adjustments to keep them in scale with overall similarities.
Args:
post_scores (list): List of post scores.
base_similarity (float): Base similarity score.
like_similarity (float): Similarity score for liked posts.
dislike_similarity (float): Similarity score for disliked posts.
Returns:
float: Adjusted similarity score.
"""
if len(post_scores) == 0:
return base_similarity
max_score = max(post_scores, key=lambda x: x[1])[1]
min_score = min(post_scores, key=lambda x: x[1])[1]
score_range = max_score - min_score
adjustment = (like_similarity - dislike_similarity) * (score_range / 2)
return base_similarity + adjustment
def swap_random_posts(rec_post_ids, post_ids, swap_percent=0.1):
"""
Swap a percentage of recommended posts with random posts.
Args:
rec_post_ids (list): List of recommended post IDs.
post_ids (list): List of all post IDs.
swap_percent (float): Percentage of posts to swap.
Returns:
list: Updated list of recommended post IDs.
"""
num_to_swap = int(len(rec_post_ids) * swap_percent)
posts_to_swap = random.sample(post_ids, num_to_swap)
indices_to_replace = random.sample(range(len(rec_post_ids)), num_to_swap)
for idx, new_post in zip(indices_to_replace, posts_to_swap):
rec_post_ids[idx] = new_post
return rec_post_ids
def get_trace_contents(user_id, action, post_table, trace_table):
"""
Get the contents of posts that a user has interacted with.
Args:
user_id (str): ID of the user.
action (str): Type of action (like or unlike).
post_table (list): List of posts.
trace_table (list): List of user interactions.
Returns:
list: List of post contents.
"""
# Get post IDs from trace table for the given user and action
trace_post_ids = [
trace['post_id'] for trace in trace_table
if (trace['user_id'] == user_id and trace['action'] == action)
]
# Fetch post contents from post table where post IDs match those in the
# trace
trace_contents = [
post['content'] for post in post_table
if post['post_id'] in trace_post_ids
]
return trace_contents
def rec_sys_personalized_with_trace(
user_table: List[Dict[str, Any]],
post_table: List[Dict[str, Any]],
trace_table: List[Dict[str, Any]],
rec_matrix: List[List],
max_rec_post_len: int,
swap_rate: float = 0.1,
) -> List[List]:
"""
This version:
1. If the number of posts is less than or equal to the maximum
recommended length, each user gets all post IDs
2. Otherwise:
- For each user, get a like-trace pool and dislike-trace pool from the
trace table
- For each user, calculate the similarity between the user's bio and
the post text
- Use the trace table to adjust the similarity score
- Swap 10% of the recommended posts with the random posts
Personalized recommendation system that uses user interaction traces.
Args:
user_table (List[Dict[str, Any]]): List of users.
post_table (List[Dict[str, Any]]): List of posts.
trace_table (List[Dict[str, Any]]): List of user interactions.
rec_matrix (List[List]): Existing recommendation matrix.
max_rec_post_len (int): Maximum number of recommended posts.
swap_rate (float): Percentage of posts to swap for diversity.
Returns:
List[List]: Updated recommendation matrix.
"""
start_time = time.time()
new_rec_matrix = []
post_ids = [post['post_id'] for post in post_table]
if len(post_ids) <= max_rec_post_len:
new_rec_matrix = [post_ids] * (len(rec_matrix) - 1)
else:
for idx in range(1, len(rec_matrix)):
user_id = user_table[idx - 1]['user_id']
user_bio = user_table[idx - 1]['bio']
# filter out posts that belong to the user
available_post_contents = [(post['post_id'], post['content'])
for post in post_table
if post['user_id'] != user_id]
# filter out like-trace and dislike-trace
like_trace_contents = get_trace_contents(
user_id, ActionType.LIKE_POST.value, post_table, trace_table)
dislike_trace_contents = get_trace_contents(
user_id, ActionType.UNLIKE_POST.value, post_table, trace_table)
# calculate similarity between user bio and post text
post_scores = []
for post_id, post_content in available_post_contents:
if model is not None:
user_embedding = model.encode(user_bio)
post_embedding = model.encode(post_content)
base_similarity = np.dot(
user_embedding,
post_embedding) / (np.linalg.norm(user_embedding) *
np.linalg.norm(post_embedding))
post_scores.append((post_id, base_similarity))
else:
post_scores.append((post_id, random.random()))
new_post_scores = []
# adjust similarity based on like and dislike traces
for _post_id, _base_similarity in post_scores:
_post_content = post_table[post_ids.index(_post_id)]['content']
like_similarity = sum(
np.dot(model.encode(_post_content), model.encode(like)) /
(np.linalg.norm(model.encode(_post_content)) *
np.linalg.norm(model.encode(like)))
for like in like_trace_contents) / len(
like_trace_contents) if like_trace_contents else 0
dislike_similarity = sum(
np.dot(model.encode(_post_content), model.encode(dislike))
/ (np.linalg.norm(model.encode(_post_content)) *
np.linalg.norm(model.encode(dislike)))
for dislike in dislike_trace_contents) / len(
dislike_trace_contents
) if dislike_trace_contents else 0
# Normalize and apply adjustments
adjusted_similarity = normalize_similarity_adjustments(
post_scores, _base_similarity, like_similarity,
dislike_similarity)
new_post_scores.append((_post_id, adjusted_similarity))
# sort posts by similarity
new_post_scores.sort(key=lambda x: x[1], reverse=True)
# extract post ids
rec_post_ids = [
post_id for post_id, _ in new_post_scores[:max_rec_post_len]
]
if swap_rate > 0:
# swap the recommended posts with random posts
swap_free_ids = [
post_id for post_id in post_ids
if post_id not in rec_post_ids and post_id not in [
trace['post_id']
for trace in trace_table if trace['user_id']
]
]
rec_post_ids = swap_random_posts(rec_post_ids, swap_free_ids,
swap_rate)
new_rec_matrix.append(rec_post_ids)
end_time = time.time()
print(f'Personalized recommendation time: {end_time - start_time:.6f}s')
return new_rec_matrix

5
backend/vendor/camel-oasis/oasis/social_platform/schema/chat_group.sql vendored Normal file
View File

@ -0,0 +1,5 @@
CREATE TABLE IF NOT EXISTS `chat_group` (
group_id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

12
backend/vendor/camel-oasis/oasis/social_platform/schema/comment.sql vendored Normal file
View File

@ -0,0 +1,12 @@
-- This is the schema definition for the comment table
CREATE TABLE comment (
comment_id INTEGER PRIMARY KEY AUTOINCREMENT,
post_id INTEGER,
user_id INTEGER,
content TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
num_likes INTEGER DEFAULT 0,
num_dislikes INTEGER DEFAULT 0,
FOREIGN KEY(post_id) REFERENCES post(post_id),
FOREIGN KEY(user_id) REFERENCES user(user_id)
);

9
backend/vendor/camel-oasis/oasis/social_platform/schema/comment_dislike.sql vendored Normal file
View File

@ -0,0 +1,9 @@
-- This is the schema definition for the comment_dislike table
CREATE TABLE comment_dislike (
comment_dislike_id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER,
comment_id INTEGER,
created_at DATETIME,
FOREIGN KEY(user_id) REFERENCES user(user_id),
FOREIGN KEY(comment_id) REFERENCES comment(comment_id)
);

9
backend/vendor/camel-oasis/oasis/social_platform/schema/comment_like.sql vendored Normal file
View File

@ -0,0 +1,9 @@
-- This is the schema definition for the comment_like table
CREATE TABLE comment_like (
comment_like_id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER,
comment_id INTEGER,
created_at DATETIME,
FOREIGN KEY(user_id) REFERENCES user(user_id),
FOREIGN KEY(comment_id) REFERENCES comment(comment_id)
);

9
backend/vendor/camel-oasis/oasis/social_platform/schema/dislike.sql vendored Normal file
View File

@ -0,0 +1,9 @@
-- This is the schema definition for the dislike table
CREATE TABLE dislike (
dislike_id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER,
post_id INTEGER,
created_at DATETIME,
FOREIGN KEY(user_id) REFERENCES user(user_id),
FOREIGN KEY(post_id) REFERENCES tweet(post_id)
);

9
backend/vendor/camel-oasis/oasis/social_platform/schema/follow.sql vendored Normal file
View File

@ -0,0 +1,9 @@
-- This is the schema definition for the follow table
CREATE TABLE follow (
follow_id INTEGER PRIMARY KEY AUTOINCREMENT,
follower_id INTEGER,
followee_id INTEGER,
created_at DATETIME,
FOREIGN KEY(follower_id) REFERENCES user(user_id),
FOREIGN KEY(followee_id) REFERENCES user(user_id)
);

7
backend/vendor/camel-oasis/oasis/social_platform/schema/group_member.sql vendored Normal file
View File

@ -0,0 +1,7 @@
CREATE TABLE IF NOT EXISTS group_members (
group_id INTEGER NOT NULL,
agent_id INTEGER NOT NULL,
joined_at DATETIME DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (group_id, agent_id),
FOREIGN KEY (group_id) REFERENCES chat_group(group_id)
);

9
backend/vendor/camel-oasis/oasis/social_platform/schema/group_message.sql vendored Normal file
View File

@ -0,0 +1,9 @@
CREATE TABLE IF NOT EXISTS group_messages (
message_id INTEGER PRIMARY KEY AUTOINCREMENT,
group_id INTEGER NOT NULL,
sender_id INTEGER NOT NULL,
content TEXT NOT NULL,
sent_at DATETIME DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (group_id) REFERENCES chat_group(group_id),
FOREIGN KEY (sender_id) REFERENCES user(agent_id)
);

9
backend/vendor/camel-oasis/oasis/social_platform/schema/like.sql vendored Normal file
View File

@ -0,0 +1,9 @@
-- This is the schema definition for the like table
CREATE TABLE like (
like_id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER,
post_id INTEGER,
created_at DATETIME,
FOREIGN KEY(user_id) REFERENCES user(user_id),
FOREIGN KEY(post_id) REFERENCES tweet(post_id)
);

9
backend/vendor/camel-oasis/oasis/social_platform/schema/mute.sql vendored Normal file
View File

@ -0,0 +1,9 @@
-- This is the schema definition for the mute table
CREATE TABLE mute (
mute_id INTEGER PRIMARY KEY AUTOINCREMENT,
muter_id INTEGER,
mutee_id INTEGER,
created_at DATETIME,
FOREIGN KEY(muter_id) REFERENCES user(user_id),
FOREIGN KEY(mutee_id) REFERENCES user(user_id)
);

16
backend/vendor/camel-oasis/oasis/social_platform/schema/post.sql vendored Normal file
View File

@ -0,0 +1,16 @@
-- This is the schema definition for the post table
-- Add Images, location etc.?
CREATE TABLE post (
post_id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER,
original_post_id INTEGER, -- NULL if this is an original post
content TEXT DEFAULT '', -- DEFAULT '' for initial posts
quote_content TEXT, -- NULL if this is an original post or a repost
created_at DATETIME,
num_likes INTEGER DEFAULT 0,
num_dislikes INTEGER DEFAULT 0,
num_shares INTEGER DEFAULT 0, -- num_shares = num_reposts + num_quotes
num_reports INTEGER DEFAULT 0,
FOREIGN KEY(user_id) REFERENCES user(user_id),
FOREIGN KEY(original_post_id) REFERENCES post(post_id)
);

6
backend/vendor/camel-oasis/oasis/social_platform/schema/product.sql vendored Normal file
View File

@ -0,0 +1,6 @@
-- This is the schema definition for the product table
CREATE TABLE product (
product_id INTEGER PRIMARY KEY,
product_name TEXT,
sales INTEGER DEFAULT 0
);

8
backend/vendor/camel-oasis/oasis/social_platform/schema/rec.sql vendored Normal file
View File

@ -0,0 +1,8 @@
-- This is the schema definition for the rec table
CREATE TABLE rec (
user_id INTEGER,
post_id INTEGER,
PRIMARY KEY(user_id, post_id),
FOREIGN KEY(user_id) REFERENCES user(user_id)
FOREIGN KEY(post_id) REFERENCES tweet(post_id)
);

10
backend/vendor/camel-oasis/oasis/social_platform/schema/report.sql vendored Normal file
View File

@ -0,0 +1,10 @@
-- This is the schema definition for the report table
CREATE TABLE report (
report_id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id INTEGER,
post_id INTEGER,
report_reason TEXT,
created_at DATETIME,
FOREIGN KEY(user_id) REFERENCES user(user_id),
FOREIGN KEY(post_id) REFERENCES post(post_id)
);

9
backend/vendor/camel-oasis/oasis/social_platform/schema/trace.sql vendored Normal file
View File

@ -0,0 +1,9 @@
-- This is the schema definition for the trace table
CREATE TABLE trace (
user_id INTEGER,
created_at DATETIME,
action TEXT,
info TEXT,
PRIMARY KEY(user_id, created_at, action, info),
FOREIGN KEY(user_id) REFERENCES user(user_id)
);

11
backend/vendor/camel-oasis/oasis/social_platform/schema/user.sql vendored Normal file
View File

@ -0,0 +1,11 @@
-- This is the schema definition for the user table
CREATE TABLE user (
user_id INTEGER PRIMARY KEY AUTOINCREMENT,
agent_id INTEGER,
user_name TEXT,
name TEXT,
bio TEXT,
created_at DATETIME,
num_followings INTEGER DEFAULT 0,
num_followers INTEGER DEFAULT 0
);

90
backend/vendor/camel-oasis/oasis/social_platform/typing.py vendored Normal file
View File

@ -0,0 +1,90 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
from enum import Enum
class ActionType(Enum):
EXIT = "exit"
REFRESH = "refresh"
SEARCH_USER = "search_user"
SEARCH_POSTS = "search_posts"
CREATE_POST = "create_post"
LIKE_POST = "like_post"
UNLIKE_POST = "unlike_post"
DISLIKE_POST = "dislike_post"
UNDO_DISLIKE_POST = "undo_dislike_post"
REPORT_POST = "report_post"
FOLLOW = "follow"
UNFOLLOW = "unfollow"
MUTE = "mute"
UNMUTE = "unmute"
TREND = "trend"
SIGNUP = "sign_up"
REPOST = "repost"
QUOTE_POST = "quote_post"
UPDATE_REC_TABLE = "update_rec_table"
CREATE_COMMENT = "create_comment"
LIKE_COMMENT = "like_comment"
UNLIKE_COMMENT = "unlike_comment"
DISLIKE_COMMENT = "dislike_comment"
UNDO_DISLIKE_COMMENT = "undo_dislike_comment"
DO_NOTHING = "do_nothing"
PURCHASE_PRODUCT = "purchase_product"
INTERVIEW = "interview"
JOIN_GROUP = "join_group"
LEAVE_GROUP = "leave_group"
SEND_TO_GROUP = "send_to_group"
CREATE_GROUP = "create_group"
LISTEN_FROM_GROUP = "listen_from_group"
@classmethod
def get_default_twitter_actions(cls):
return [
cls.CREATE_POST,
cls.LIKE_POST,
cls.REPOST,
cls.FOLLOW,
cls.DO_NOTHING,
cls.QUOTE_POST,
]
@classmethod
def get_default_reddit_actions(cls):
return [
cls.LIKE_POST,
cls.DISLIKE_POST,
cls.CREATE_POST,
cls.CREATE_COMMENT,
cls.LIKE_COMMENT,
cls.DISLIKE_COMMENT,
cls.SEARCH_POSTS,
cls.SEARCH_USER,
cls.TREND,
cls.REFRESH,
cls.DO_NOTHING,
cls.FOLLOW,
cls.MUTE,
]
class RecsysType(Enum):
TWITTER = "twitter"
TWHIN = "twhin-bert"
REDDIT = "reddit"
RANDOM = "random"
class DefaultPlatformType(Enum):
TWITTER = "twitter"
REDDIT = "reddit"

13
backend/vendor/camel-oasis/oasis/testing/__init__.py vendored Normal file
View File

@ -0,0 +1,13 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========

64
backend/vendor/camel-oasis/oasis/testing/show_db.py vendored Normal file
View File

@ -0,0 +1,64 @@
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. ===========
import logging
import sqlite3
from datetime import datetime
table_log = logging.getLogger(name="table")
table_log.setLevel("DEBUG")
now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") # Modify here
file_handler = logging.FileHandler(f"./log/table-{str(now)}.log",
encoding="utf-8")
file_handler.setLevel("DEBUG")
file_handler.setFormatter(logging.Formatter("%(message)s"))
table_log.addHandler(file_handler)
stream_handler = logging.StreamHandler()
stream_handler.setLevel("DEBUG")
stream_handler.setFormatter(logging.Formatter("%(message)s"))
table_log.addHandler(stream_handler)
def print_db_contents(db_file):
# Connect to the SQLite database
conn = sqlite3.connect(db_file)
cursor = conn.cursor()
# Retrieve and print all table names
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
tables = cursor.fetchall()
# print("Tables:", [table[0] for table in tables])
table_log.info("Tables:" + " ".join([str(table[0]) for table in tables]))
for table_name in tables:
# print(f"\nTable: {table_name[0]}")
table_log.info(f"\nTable: {table_name[0]}")
# Print table structure
cursor.execute(f"PRAGMA table_info({table_name[0]})")
columns = cursor.fetchall()
# print("Columns:")
table_log.info("Columns:")
for col in columns:
# print(f" {col[1]} ({col[2]})")
table_log.info(f" {col[1]} ({col[2]})")
# Print table contents
cursor.execute(f"SELECT * FROM {table_name[0]}")
rows = cursor.fetchall()
# print("Contents:")
table_log.info("Contents:")
for row in rows:
# print(" ", row)
table_log.info(" " + ", ".join(str(item) for item in row))
# Close the connection
conn.close()

41
backend/vendor/camel-oasis/pyproject.toml vendored Normal file
View File

@ -0,0 +1,41 @@
[project]
name = "camel-oasis"
version = "0.2.5.post1"
description = "Open Agents Social Interaction Simulations on a Large Scale (vendored fork: relaxed neo4j pin to coexist with graphiti-core[falkordb])"
authors = [{ name = "CAMEL-AI.org" }]
readme = "README.md"
license = { text = "Apache-2.0" }
keywords = ["multi-agent-systems", "social-simulations"]
requires-python = ">=3.11,<3.13"
dependencies = [
"pandas==2.2.2",
"igraph==0.11.6",
"cairocffi==1.7.1",
"pillow==10.3.0",
"unstructured==0.13.7",
"sentence-transformers==3.0.0",
"prance==23.6.21.0",
"openapi-spec-validator==0.7.1",
"slack_sdk==3.31.0",
# Relaxed from ==5.23.0 to coexist with graphiti-core[falkordb]'s
# neo4j>=5.26.0 requirement. The MiroFish code paths (run_*_simulation.py
# + run_parallel_simulation.py) never pass a neo4j_config, so AgentGraph
# is constructed in-memory-only and the neo4j driver is never invoked.
"neo4j>=5.23.0",
"camel-ai==0.2.78",
"requests_oauthlib==2.0.0",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["oasis"]
# Required because the parent project (MiroFish) declares this package as
# a direct-reference local path dep ("camel-oasis @ file:///app/backend/vendor/camel-oasis").
# Without this, hatchling refuses to build when uv is constructing the lock entry.
[tool.hatch.metadata]
allow-direct-references = true

19
deploy/Dockerfile.e2e Normal file
View File

@ -0,0 +1,19 @@
# Lightweight e2e test image. Inherits the MiroFish base image (already
# has the patched graphiti_service.py + all Python deps) and only adds
# the test script. Keeps the image small and the build fast.
#
# Build context: project root (so the parent Dockerfile already has app/).
# The test script is fetched from deploy/ in the project tree.
FROM mirofish:latest
# Drop the long-running start.sh and just run the test
USER root
RUN mkdir -p /app/deploy
COPY deploy/e2e_test.py /app/deploy/e2e_test.py
# Run the test as the entrypoint. The MiroFish image is uv-managed — flask,
# graphiti, falkordb all live under /app/backend/.venv, so we need to invoke
# python via `uv run` to get the venv on PYTHONPATH.
WORKDIR /app/backend
ENTRYPOINT ["uv", "run", "python", "/app/deploy/e2e_test.py"]

98
deploy/README.md Normal file
View File

@ -0,0 +1,98 @@
# MiroFish deploy (agent.profikid.nl)
Single-host Docker deployment of the [MiroFish swarm-intelligence engine](https://github.com/666ghj/MiroFish) reforked to use **Graphiti + FalkorDB** instead of Zep Cloud.
## What you get
| URL | Service |
| --- | --- |
| `https://mirofish.agent.profikid.nl` | MiroFish web UI + API (frontend on :3000, Flask API on :5001, both inside the mirofish container) |
| `mirofish_net` (internal Docker network) | FalkorDB (in-memory graph store, port 6379) |
Traefik (running on the host as the `traefik` Docker network) auto-issues a Let's Encrypt cert for `mirofish.agent.profikid.nl`.
## Files
```
/docker/mirofish/ <-- everything below lives here
├── docker-compose.yml <-- mirofish + falkordb + e2e (opt-in)
├── secrets.env <-- LLM_API_KEY, FALKORDB creds
├── bootstrap.sh <-- one-time host checks
├── up.sh <-- build + bring up mirofish + falkordb
├── e2e.sh <-- run the in-container end-to-end test
├── health.sh <-- /health, container status, falkordb ping
├── Dockerfile.e2e <-- e2e test image (inherits mirofish:latest)
└── e2e_test.py <-- the test itself
```
## Companion: FalkorDB Browser
A separate stack at `/docker/falkordb-browser/` exposes the [FalkorDB Browser](https://github.com/FalkorDB/falkordb-browser) (v2.0.10) at **`https://falkor.agent.profikid.nl`**. It joins the same `mirofish_net` network as the sidecar so the browser UI can read every graph the MiroFish cron produces (MATCH queries, schema explorer, node editor).
To add the connection in the UI:
1. Open `https://falkor.agent.profikid.nl/`
2. On the login page, enter:
- **Host:** `falkordb`
- **Port:** `6379`
- **Username / Password:** leave empty (the MiroFish FalkorDB runs without auth)
3. The browser will list all the `mirofish_*` graphs from the cron ingests.
The browser uses its own network connection (server-side) to dial `falkordb:6379`, so the host field is just an identifier — your browser doesn't need to be able to reach it.
## Deploy
```bash
ssh root@agent.profikid.nl
mkdir -p /docker/mirofish
cd /docker/mirofish
# Drop the contents of this directory here (or git clone the repo and copy deploy/ -> .)
./bootstrap.sh # verifies docker / dns / file presence
./up.sh # builds the image, brings up mirofish + falkordb
```
After ~90s (FalkorDB cold start + embedding model download on first build only), the URL `https://mirofish.agent.profikid.nl` should respond.
## E2E test
```bash
cd /docker/mirofish
./e2e.sh
```
This builds `mirofish-e2e:latest` (a thin layer on top of `mirofish:latest` that runs `e2e_test.py` as the entrypoint), then runs it as a one-shot container against the real `falkordb` sidecar. It:
1. Verifies FalkorDB is reachable (raw TCP probe)
2. Runs the real `GraphBuilderService` against the Iran/US/Israel OSINT briefing seed text
3. Reads back from FalkorDB via the adapter (`get_all_nodes`, `get_all_edges`)
4. Re-verifies with a raw `falkordb.FalkorDB` client (independent check)
5. Prints PASS / FAIL with node + edge counts
Expected on first run: **~8 entities, ~8 edges** in FalkorDB after the 2-chunk run (`E2E_MAX_CHUNKS=2`). Wall time ~3-4min on the first call (LLM is doing the extraction; reasoning tokens make M3 slow). Tune `E2E_MAX_CHUNKS` higher in `docker-compose.yml` to test with more seed text.
## How the refactor differs from upstream
Upstream MiroFish uses [Zep Cloud](https://www.getzep.com/) for the knowledge graph. This fork replaces it with [Graphiti](https://github.com/getzep/graphiti) (the open-source engine behind Zep) backed by [FalkorDB](https://www.falkordb.com/) (a Redis-protocol graph store). All graph operations go through a Zep-shaped facade in `backend/app/services/graphiti_service.py` so the rest of the MiroFish code (which still speaks Zep) didn't have to change.
Key wiring:
- `MinimaxLLMClient` — Graphiti's LLMClient ABC implemented against MiniMax M3 (`https://api.minimax.io/v1`). Uses the `tools` API for structured output because M3 ignores `response_format: json_object`.
- `M3RerankerClient` — chat-completion True/False reranker (M3 returns `logprobs: None`, so the stock OpenAI reranker that reads logprobs breaks).
- `HashEmbedder` — deterministic 384-dim embedder so the test image doesn't need to download the full `sentence-transformers` multilingual model. The production image still uses the real model (set `EMBEDDING_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` in `secrets.env` to switch).
- `FalkorDriver` (from `graphiti-core[falkordb]>=0.20.0`) — the actual graph store. No Neo4j, no Bolt, just a Redis-protocol TCP socket to the sidecar.
## How to upgrade
```bash
cd /docker/mirofish
git pull # in the parent MiroFish repo
./up.sh --rebuild # rebuild + restart
./e2e.sh # smoke-test
```
## Troubleshooting
- **FalkorDB healthcheck fails** — check `docker compose logs falkordb`. The `falkordb/falkordb:latest` image is small; first-pull can take a minute.
- **`/health` returns 200 but `/api/ontology/generate` times out** — the embedding model is downloading on first request. Wait, or run with a prebuilt `mirofish:latest` that has the model baked in (the `Dockerfile` already does this in step 4).
- **M3 returns 0 entities** — the M3 endpoint is shared; rate limiting or reasoning-token quirks can cause this. Check `docker compose logs mirofish | grep graphiti_service` and the raw LLM payloads.
- **Cert issuance stuck in Traefik** — confirm `TRAEFIK_HOST=agent.profikid.nl` and that the `traefik` Docker network exists. Traefik must be on the same network as the `mirofish` container.

43
deploy/bootstrap.sh Executable file
View File

@ -0,0 +1,43 @@
#!/usr/bin/env bash
# MiroFish host bootstrap — agent.profikid.nl
#
# Run this ONCE on the host (after cloning the repo to /docker/mirofish/).
# It creates the project directory layout and verifies Docker + Traefik are
# reachable. Does NOT actually deploy — that's the `up.sh` next to this file.
#
# Usage: ./bootstrap.sh
set -euo pipefail
cd "$(dirname "$0")"
echo "[bootstrap] creating /docker/mirofish/ if missing..."
sudo mkdir -p /docker/mirofish
sudo chown -R "$USER":"$USER" /docker/mirofish || true
echo "[bootstrap] required files present:"
for f in docker-compose.yml secrets.env Dockerfile Dockerfile.e2e e2e_test.py; do
if [ ! -f "$f" ] && [ ! -f "../$f" ] && [ ! -f "../deploy/$f" ]; then
echo " MISSING: $f" >&2
exit 1
fi
echo " OK: $f"
done
echo "[bootstrap] checking docker..."
docker --version
docker compose version
echo "[bootstrap] checking traefik network..."
docker network ls --format '{{.Name}}' | grep -q '^traefik$' \
|| { echo " WARN: 'traefik' network not found. Traefik must be on a network called 'traefik' (or edit docker-compose.yml)." >&2; }
echo "[bootstrap] DNS for agent.profikid.nl (must resolve to 69.62.114.199)..."
RESOLVED=$(getent hosts agent.profikid.nl | awk '{print $1; exit}')
if [ "$RESOLVED" != "69.62.114.199" ]; then
echo " WARN: agent.profikid.nl resolves to $RESOLVED, expected 69.62.114.199" >&2
fi
echo
echo "[bootstrap] all checks passed."
echo
echo "Next: edit secrets.env if you need to change LLM keys, then run ./up.sh"

104
deploy/docker-compose.yml Normal file
View File

@ -0,0 +1,104 @@
# MiroFish deploy overlay — agent.profikid.nl
#
# Project conventions (matches hermes, snake, wiki siblings):
# /docker/<name>/docker-compose.yml
# COMPOSE_PROJECT_NAME=<name>
# TRAEFIK_HOST=agent.profikid.nl
#
# Single port (3000) exposed to Traefik; nginx inside the container
# serves the built frontend and reverse-proxies /api/* + /health to Flask.
#
# Memory graph: FalkorDB (in-memory graph store) on a shared network.
services:
falkordb:
image: falkordb/falkordb:latest
container_name: mirofish-falkordb
restart: unless-stopped
expose:
- "6379"
volumes:
- falkordb-data:/data
healthcheck:
test: ["CMD-SHELL", "redis-cli ping | grep -q PONG || exit 1"]
interval: 15s
timeout: 3s
retries: 5
start_period: 20s
networks:
- mirofish_net
mirofish:
build:
context: ..
dockerfile: Dockerfile
image: mirofish:latest
container_name: mirofish
restart: unless-stopped
depends_on:
falkordb:
condition: service_healthy
env_file:
- secrets.env
environment:
FALKORDB_HOST: falkordb
FALKORDB_PORT: "6379"
expose:
- "3000"
labels:
- traefik.enable=true
- traefik.http.routers.${COMPOSE_PROJECT_NAME:-mirofish}.rule=Host(`${COMPOSE_PROJECT_NAME:-mirofish}.${TRAEFIK_HOST:-agent.profikid.nl}`)
- traefik.http.routers.${COMPOSE_PROJECT_NAME:-mirofish}.entrypoints=websecure
- traefik.http.routers.${COMPOSE_PROJECT_NAME:-mirofish}.tls.certresolver=letsencrypt
- traefik.http.services.${COMPOSE_PROJECT_NAME:-mirofish}.loadbalancer.server.port=3000
# HTTP -> HTTPS redirect
- traefik.http.routers.${COMPOSE_PROJECT_NAME:-mirofish}-http.rule=Host(`${COMPOSE_PROJECT_NAME:-mirofish}.${TRAEFIK_HOST:-agent.profikid.nl}`)
- traefik.http.routers.${COMPOSE_PROJECT_NAME:-mirofish}-http.entrypoints=web
- traefik.http.routers.${COMPOSE_PROJECT_NAME:-mirofish}-http.middlewares=redirect-to-https
- traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
- traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true
volumes:
- uploads:/app/backend/uploads
- flask-logs:/app/logs
- embedding-model:/root/.cache/huggingface # cache the sentence-transformers model
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:3000/health"]
interval: 30s
timeout: 5s
retries: 5
start_period: 90s
networks:
- mirofish_net
# One-shot e2e test. Built but only runs when you opt in:
# docker compose --profile e2e run --rm mirofish-e2e
# Output is printed to the host (PASS/FAIL exit code).
mirofish-e2e:
build:
context: ..
dockerfile: deploy/Dockerfile.e2e
image: mirofish-e2e:latest
container_name: mirofish-e2e
depends_on:
falkordb:
condition: service_healthy
env_file:
- secrets.env
environment:
FALKORDB_HOST: falkordb
FALKORDB_PORT: "6379"
E2E_MAX_CHUNKS: "2"
networks:
- mirofish_net
profiles: ["e2e"]
restart: "no"
volumes:
uploads:
flask-logs:
falkordb-data:
embedding-model:
networks:
mirofish_net:
name: mirofish_net

28
deploy/e2e.sh Executable file
View File

@ -0,0 +1,28 @@
#!/usr/bin/env bash
# MiroFish e2e — runs the in-container e2e test against the real FalkorDB
# sidecar. Prints PASS/FAIL and exits accordingly.
set -euo pipefail
cd "$(dirname "$0")"
export COMPOSE_PROJECT_NAME=mirofish
# Make sure the e2e image is built
if ! docker image inspect mirofish-e2e:latest >/dev/null 2>&1; then
echo "[e2e] building mirofish-e2e image..."
docker compose --profile e2e build mirofish-e2e
fi
echo "[e2e] running test against falkordb sidecar..."
# `run --rm` creates a one-off container with the e2e profile, runs the test,
# and removes the container on exit. Exit code propagates.
docker compose --profile e2e run --rm mirofish-e2e
RC=$?
echo
if [ "$RC" -eq 0 ]; then
echo "[e2e] PASS"
else
echo "[e2e] FAIL (exit $RC)"
fi
exit "$RC"

209
deploy/e2e_test.py Normal file
View File

@ -0,0 +1,209 @@
"""
MiroFish → FalkorDB end-to-end test (production-mode).
Runs the same code path the API uses, but in a one-shot container, against
the real FalkorDB sidecar in the same compose project. Verifies that:
- FalkorDB is reachable from the app container (real TCP, real protocol)
- graphiti-core connects and the Zep-shaped adapter works
- Entities + edges actually land in FalkorDB after add_episode
- The /health endpoint on the running MiroFish container is 200
Exits 0 on PASS, 1 on FAIL.
"""
from __future__ import annotations
import json
import os
import sys
import time
import traceback
from pathlib import Path
# Make MiroFish backend importable
MIROFISH_BACKEND = "/app/backend"
if MIROFISH_BACKEND not in sys.path:
sys.path.insert(0, MIROFISH_BACKEND)
# .env is loaded by MiroFish's config.py; nothing to do here.
print("=" * 70)
print("MIROFISH END-TO-END TEST (FalkorDB backend)")
print("=" * 70)
print(f" FALKORDB_HOST = {os.environ.get('FALKORDB_HOST', '<unset>')}")
print(f" FALKORDB_PORT = {os.environ.get('FALKORDB_PORT', '<unset>')}")
print(f" LLM_BASE_URL = {os.environ.get('LLM_BASE_URL', '<unset>')}")
print(f" LLM_MODEL_NAME = {os.environ.get('LLM_MODEL_NAME', '<unset>')}")
print(f" EMBEDDING_MODEL= {os.environ.get('EMBEDDING_MODEL','<unset>')}")
print()
# -- 0. Sanity: is FalkorDB up? Use a raw TCP probe (no app deps) --
import socket
fk_host = os.environ.get("FALKORDB_HOST", "falkordb")
fk_port = int(os.environ.get("FALKORDB_PORT", "6379"))
try:
with socket.create_connection((fk_host, fk_port), timeout=5):
print(f"[0] FalkorDB TCP reachable at {fk_host}:{fk_port}")
except OSError as e:
print(f"[0] FAIL: FalkorDB not reachable at {fk_host}:{fk_port}: {e}")
sys.exit(2)
# -- 1. The Iran/US/Israel OSINT cron briefing as the seed text --
# In a real MiroFish run, this comes from /home/hermes/.hermes/cron/output/...
# For the e2e test, we ship a copy of the latest briefing inline.
SEED_TEXT = """\
## IRAN/US/ISRAEL CONFLICT — UPDATE
**Timestamp:** Monday, 8 June 2026, ~21:00 UTC
**Coverage Period:** Last 12 hours (07–08 June 2026)
### INCIDENTS / HOSTILITIES
- **Iran launched ballistic missile attack on Israel** (Sun 7 Jun, evening local) — first direct Iranian bombardment since the April 8 ceasefire. Multiple projectiles struck central Israel; Israeli emergency services report **3 killed and >100 injured** in a strike on a building in **Bat Yam** (south of Tel Aviv). [BBC, NYT, AP]
- **Israeli retaliatory strikes on Iran** conducted hours after the Iranian launch, targeting military sites in Tehran. Netanyahu stated Israel "hit the terror regime in Tehran." [NYT, Times of Israel]
- **Hezbollah rocket fire on northern Israel** (7 Jun) preceded the Israeli strike on Beirut's southern suburbs, which killed **2 and wounded ~11–20** (Lebanese state media). [NPR]
- **Iranian drone attack on US interests in the Strait of Hormuz** — US Central Command reports shooting down a pair of Iranian drones threatening the strait on 7 Jun. [CBS]
### DIPLOMATIC / POLITICAL
- **Trump-Netanyahu phone call** (7 Jun, post-Iranian missile launch) — Trump urged Netanyahu **not to retaliate**; Netanyahu reportedly replied "The Iranians violated our sovereignty."
- **Iran declared halt to military offensive** against Israel on 8 Jun.
### CEASEFIRE / TRUCE STATUS
- **April 8 Iran-US ceasefire:** Functionally **broken** — first direct Iran-Israel missile exchange since the truce.
### REGIONAL SPILLOVER
- **Strait of Hormuz:** US naval blockade in effect; shipping at near-standstill since March. IRGC restricts traffic; ~150+ tankers anchored outside. US forces disabled 4 vessels, redirected 94. [CNN, Reuters]
- **Houthi attacks (Yemen):** Iran-backed Houthi rebels struck Israel; rocket debris found near Jericho (West Bank) on 8 Jun.
- **Iraq/Syria:** US radar sites attacked; Iranian drones shot down near US positions (per US claims, 6 Jun).
- **Kuwait/Bahrain:** Earlier Iranian ballistic missile volleys (2–5 Jun) hit Kuwait International Airport.
### ASSESSMENT
The April 8 ceasefire has effectively collapsed after the first direct Iran-Israel missile exchange in two months, triggered by Israel's Beirut strike on Hezbollah. Both sides have paused strikes as of 8 Jun following Trump intervention, but a 60-day US-Iran deal remains unsigned and Hezbollah's rejection of the Lebanon track preserves a major escalation vector. **High risk of renewed escalation within 24–72 hours** if Trump-Netanyahu alignment fractures or Lebanon track collapses.
"""
print(f"[1] Seed text: {len(SEED_TEXT)} chars (Iran/US/Israel OSINT briefing)")
# -- 2. Ontology (same shape MiroFish's OntologyGenerator would emit) --
ONTOLOGY = {
"entity_types": [
{"name": "Country", "description": "A sovereign nation or state actor.",
"attributes": [{"name": "iso_code", "description": "ISO code"},
{"name": "government_type", "description": "Form of government"}]},
{"name": "Person", "description": "A named individual mentioned in the briefing.",
"attributes": [{"name": "role", "description": "Title, position, or affiliation"}]},
{"name": "Organization","description": "A formal organization, military unit, or political body.",
"attributes": [{"name": "type", "description": "e.g. political, militant, state"}]},
{"name": "Location", "description": "A geographic place, city, or strategic region.",
"attributes": [{"name": "region", "description": "Broader region"}]},
{"name": "Event", "description": "A distinct incident, attack, meeting, or action.",
"attributes": [{"name": "date", "description": "When the event occurred"},
{"name": "event_type", "description": "Type of event"}]},
],
"edge_types": [
{"name": "LAUNCHED_ATTACK_AGAINST", "description": "Actor performed a military strike on target.",
"source_targets": [{"source": "Country", "target": "Country"},
{"source": "Organization", "target": "Country"}],
"attributes": [{"name": "date", "description": "Date"},
{"name": "weapon", "description": "Weapon used"}]},
{"name": "OCCURRED_IN", "description": "An event happened at a location.",
"source_targets": [{"source": "Event", "target": "Location"}],
"attributes": [{"name": "date", "description": "Date"}]},
{"name": "IS_LEADER_OF", "description": "A person leads a country or organization.",
"source_targets": [{"source": "Person", "target": "Country"},
{"source": "Person", "target": "Organization"}],
"attributes": []},
{"name": "MADE_STATEMENT", "description": "A person or organization issued a public statement.",
"source_targets": [{"source": "Person", "target": "Event"}],
"attributes": [{"name": "stance", "description": "Position taken"}]},
{"name": "PARTICIPATED_IN", "description": "A country or organization was involved in an event.",
"source_targets": [{"source": "Country", "target": "Event"}],
"attributes": []},
],
}
print(f"[2] Ontology: {len(ONTOLOGY['entity_types'])} entity types, "
f"{len(ONTOLOGY['edge_types'])} edge types")
# -- 3. Chunk the seed text --
from app.services.text_processor import TextProcessor
chunks = TextProcessor.split_text(SEED_TEXT, chunk_size=500, overlap=50)
print(f"[3] Text split: {len(chunks)} chunks")
# -- 4. Run the build pipeline via the real GraphBuilderService --
from app.services.graph_builder import GraphBuilderService
from app.services import graph_builder as _gb_mod # for the EpisodeData stub
EpisodeData = _gb_mod.EpisodeData
MAX_CHUNKS = int(os.environ.get("E2E_MAX_CHUNKS", "2"))
test_chunks = chunks[:MAX_CHUNKS]
print(f"[4] Using {len(test_chunks)} chunks for the e2e test (E2E_MAX_CHUNKS={MAX_CHUNKS})")
print()
print("[5] Running graph build pipeline...")
t0 = time.time()
builder = GraphBuilderService()
graph_id = builder.create_graph(name="Iran_US_Israel_OSINT_e2e")
print(f" create_graph -> {graph_id}")
builder.set_ontology(graph_id, ONTOLOGY)
print(f" set_ontology -> cached {len(ONTOLOGY['entity_types'])} types, {len(ONTOLOGY['edge_types'])} edges")
episodes = [EpisodeData(data=c, type="text") for c in test_chunks]
print(f" add_batch -> sending {len(episodes)} EpisodeData objects...")
batch_result = builder.client.add_batch(graph_id=graph_id, episodes=episodes)
t1 = time.time()
print(f" add_batch took {t1 - t0:.1f}s")
for r in batch_result:
print(f" uuid={r.uuid_} processed={r.processed}")
# -- 5. Verify in FalkorDB via the adapter's read methods --
print()
print("[6] Reading back from FalkorDB via the adapter...")
nodes = builder.client.get_all_nodes(graph_id)
edges = builder.client.get_all_edges(graph_id)
print(f" nodes: {len(nodes)}")
print(f" edges: {len(edges)}")
print()
print("[7] Sample entities (first 10):")
for n in nodes[:10]:
labels_str = ",".join(n.labels) if n.labels else "(no labels)"
summary_preview = (n.summary[:60] + "...") if n.summary and len(n.summary) > 60 else (n.summary or "")
print(f" [{labels_str}] {n.name!r} uuid={n.uuid_[:8]} summary={summary_preview!r}")
print()
print("[8] Sample relationships (first 5):")
for e in edges[:5]:
src = e.source_node_uuid[:8] if e.source_node_uuid else "?"
dst = e.target_node_uuid[:8] if e.target_node_uuid else "?"
fact_preview = (e.fact[:60] + "...") if e.fact and len(e.fact) > 60 else (e.fact or "")
print(f" ({src}) -[{e.name or e.fact_type}]→ ({dst}) fact={fact_preview!r}")
# -- 6. Independent verification with the raw falkordb client --
print()
print("[9] Direct falkordb verification (source of truth)...")
import falkordb
client = falkordb.FalkorDB(host=fk_host, port=fk_port)
graph = client.select_graph(graph_id)
n_count = graph.query("MATCH (n) RETURN count(n) AS c").result_set
e_count = graph.query("MATCH ()-[r]->() RETURN count(r) AS c").result_set
print(f" FalkorDB MATCH count(n) = {n_count}")
print(f" FalkorDB MATCH count(edges) = {e_count}")
labels = graph.query("MATCH (n) UNWIND labels(n) AS l RETURN l, count(*) AS c ORDER BY c DESC LIMIT 20")
print(f" Entity label distribution:")
for row in labels.result_set:
print(f" {row[0]}: {row[1]}")
names = graph.query("MATCH (n) RETURN n.name AS name LIMIT 20")
print(f" Sample node names: {[r[0] for r in names.result_set]}")
# -- 7. Final summary --
print()
print("=" * 70)
print("E2E TEST SUMMARY")
print("=" * 70)
print(f" Graph ID: {graph_id}")
print(f" Seed text: Iran/US/Israel OSINT briefing ({len(SEED_TEXT)} chars)")
print(f" Chunks ingested: {len(episodes)}")
print(f" Nodes in FalkorDB: {len(nodes)} (direct: {n_count[0][0]})")
print(f" Edges in FalkorDB: {len(edges)} (direct: {e_count[0][0]})")
print(f" Wall time: {t1 - t0:.1f}s")
print(f" Pass criteria: node_count > 0 AND edge_count > 0")
passing = (len(nodes) > 0 and len(edges) > 0)
print(f" RESULT: {'PASS' if passing else 'FAIL'}")
print("=" * 70)
sys.exit(0 if passing else 1)

19
deploy/health.sh Executable file
View File

@ -0,0 +1,19 @@
#!/usr/bin/env bash
# Quick healthcheck — /health endpoint, container status, FalkorDB ping.
set -euo pipefail
cd "$(dirname "$0")"
echo "[health] docker compose ps:"
docker compose ps
echo
echo "[health] mirofish /health:"
docker compose exec -T mirofish curl -sS -m 5 http://127.0.0.1:3000/health || echo " /health FAILED"
echo
echo "[health] falkordb ping:"
docker compose exec -T falkordb redis-cli ping || echo " redis-cli FAILED"
echo
echo "[health] falkordb graph list:"
docker compose exec -T falkordb redis-cli GRAPH.LIST || echo " GRAPH.LIST FAILED"

31
deploy/secrets.env.example Normal file
View File

@ -0,0 +1,31 @@
# MiroFish secrets for the agent.profikid.nl deployment.
#
# Rename to `secrets.env` and fill in. Filename is `secrets.env` (not `.env`)
# on purpose — Hermes' safety guard blocks read/write of any file literally
# named `.env`.
#
# DO NOT COMMIT a populated secrets.env to git. This template IS in git
# (secrets.env.example); the real one is gitignored.
#
# LLM provider: MiniMax M3 via the OpenAI-compat endpoint
# (verified working: https://api.minimax.io/v1 returns chat completions for model "MiniMax-M3")
LLM_API_KEY=sk-cp-REPLACE-ME
LLM_BASE_URL=https://api.minimax.io/v1
LLM_MODEL_NAME=MiniMax-M3
# Memory graph backend: Graphiti + FalkorDB (replaces Zep)
# Host/port are set in docker-compose.yml; the keys here are auth (FalkorDB
# ships without auth, so we leave username/password empty for on-prem).
FALKORDB_HOST=falkordb
FALKORDB_PORT=6379
FALKORDB_USERNAME=
FALKORDB_PASSWORD=
# Embedding model. Two options:
# 1) Local sentence-transformers, pre-downloaded in the image (set this):
# sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
# 2) Lightweight deterministic hash embedder (used by the e2e test image,
# useful for CI / quick smoke tests):
# hash
EMBEDDING_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

46
deploy/up.sh Executable file
View File

@ -0,0 +1,46 @@
#!/usr/bin/env bash
# MiroFish up — agent.profikid.nl
#
# Builds the image, brings up mirofish + falkordb sidecar on the
# mirofish_net network, registers Traefik routes. Idempotent.
#
# Usage:
# ./up.sh # build + up -d
# ./up.sh --no-build # up -d, skip build
# ./up.sh --rebuild # force rebuild (no cache)
set -euo pipefail
cd "$(dirname "$0")"
BUILD_FLAGS=()
if [ "${1:-}" = "--no-build" ]; then
BUILD_FLAGS+=(--no-build)
elif [ "${1:-}" = "--rebuild" ]; then
BUILD_FLAGS+=(--build --force-rm --no-cache --pull)
else
BUILD_FLAGS+=(--build)
fi
export COMPOSE_PROJECT_NAME=mirofish
export TRAEFIK_HOST=agent.profikid.nl
echo "[up] COMPOSE_PROJECT_NAME=$COMPOSE_PROJECT_NAME"
echo "[up] TRAEFIK_HOST=$TRAEFIK_HOST"
echo "[up] building..."
docker compose "${BUILD_FLAGS[@]}"
echo "[up] bringing up mirofish + falkordb..."
docker compose up -d
echo "[up] tailing logs for ~20s while healthchecks settle..."
docker compose logs --tail=200 --since=2m &
LOG_PID=$!
sleep 20
kill "$LOG_PID" 2>/dev/null || true
echo
echo "[up] status:"
docker compose ps
echo
echo "URL (after Traefik cert): https://mirofish.agent.profikid.nl"
echo "E2E test: ./e2e.sh (runs the in-container e2e test against real FalkorDB)"

2
docker-compose.yml
View File

@ -1,7 +1,7 @@
services:
mirofish:
image: ghcr.io/666ghj/mirofish:latest
# 加速镜像（如拉取缓慢可替换上方地址）
# Accelerated registry mirror (replace the upstream address above if pulling is slow)
# image: ghcr.nju.edu.cn/666ghj/mirofish:latest
container_name: mirofish
env_file:

8
frontend/index.html
View File

@ -1,15 +1,15 @@
<!doctype html>
<html lang="zh">
<html lang="nl">
<head>
<script>document.documentElement.lang = localStorage.getItem('locale') || 'zh'</script>
<script>document.documentElement.lang = localStorage.getItem('locale') || 'nl'</script>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&family=JetBrains+Mono:wght@100..800&family=Noto+Sans+SC:wght@300;400;500;700;800;900&family=Space+Grotesk:wght@300..700&display=swap" rel="stylesheet">
<meta charset="UTF-8" />
<link rel="icon" type="image/png" href="/icon.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="MiroFish - 社交媒体舆论模拟系统" />
<title>MiroFish - 预测万物</title>
<meta name="description" content="MiroFish - Social media opinion simulation system" />
<title>MiroFish - Predict Anything</title>
</head>
<body>
<div id="app"></div>

Some files were not shown because too many files have changed in this diff Show More