Productionize: add ingest_text one-call API, ontology JSON retry, README fork notice, e2e Dockerfile path fix
- backend/app/api/graph.py: add POST /api/graph/ingest_text (one-call: project + ontology + async build + returns project_id/task_id) Designed for cron automations that need to ingest a markdown briefing without the 3-step project -> ontology -> build dance. - backend/app/services/ontology_generator.py: bump max_tokens 4096->16384 and add a retry-with-compact-prompt path on JSON parse failure. M3 truncates long entity-attribute arrays; the retry asks for a stripped schema (no attributes, 6 entity types max, 6 edge types max). - deploy/Dockerfile.e2e: build context is the project root, so COPY needs the deploy/ prefix; entrypoint uses 'uv run python' because the MiroFish image is uv-managed (flask/graphiti/falkordb all live under /app/backend/.venv). - README.md: replace Zep + Qwen references with the Graphiti+FalkorDB stack and MiniMax M-series LLMs; add 'About this fork' section documenting the deploy overlay, ingest_text API, and cron integration.
This commit is contained in:
ENI
parent
9decc6c6f4
commit
ee3609d439
74
README.md
74
README.md
|
|
@ -28,6 +28,8 @@
|
|||
|
||||
**MiroFish** is a next-generation AI prediction engine powered by multi-agent technology. By extracting seed information from the real world (such as breaking news, policy drafts, or financial signals), it automatically constructs a high-fidelity parallel digital world. Within this space, thousands of intelligent agents with independent personalities, long-term memory, and behavioral logic freely interact and undergo social evolution. You can inject variables dynamically from a "God's-eye view" to precisely deduce future trajectories — **rehearse the future in a digital sandbox, and win decisions after countless simulations**.
|
||||
|
||||
> **This is a maintained fork** of [666ghj/MiroFish](https://github.com/666ghj/MiroFish). It swaps Zep Cloud for **Graphiti + FalkorDB** (open source, self-hosted), adds a one-call `POST /api/graph/ingest_text` API for cron automations, and ships a production Docker stack for `agent.profikid.nl`. See [About this fork](#-about-this-fork) below.
|
||||
|
||||
> You only need to: Upload seed materials (data analysis reports or interesting novel stories) and describe your prediction requirements in natural language</br>
|
||||
> MiroFish will return: A detailed prediction report and a deeply interactive high-fidelity digital world
|
||||
|
||||
|
|
@ -116,15 +118,22 @@ cp .env.example .env
|
|||
|
||||
```env
|
||||
# LLM API Configuration (supports any LLM API with OpenAI SDK format)
|
||||
# Recommended: Alibaba Qwen-plus model via Bailian Platform: https://bailian.console.aliyun.com/
|
||||
# Recommended: MiniMax M-series via https://api.minimax.io/v1
|
||||
# - MiniMax-M2.7-highspeed : best for cron / high-volume (recommended)
|
||||
# - MiniMax-M3 : heavier reasoning, slower, higher quality
|
||||
# Alibaba Qwen-plus via Bailian Platform: https://bailian.console.aliyun.com/
|
||||
# High consumption, try simulations with fewer than 40 rounds first
|
||||
LLM_API_KEY=your_api_key
|
||||
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
|
||||
LLM_MODEL_NAME=qwen-plus
|
||||
LLM_BASE_URL=https://api.minimax.io/v1
|
||||
LLM_MODEL_NAME=MiniMax-M2.7-highspeed
|
||||
|
||||
# Zep Cloud Configuration
|
||||
# Free monthly quota is sufficient for simple usage: https://app.getzep.com/
|
||||
ZEP_API_KEY=your_zep_api_key
|
||||
# Graph store (this fork: Graphiti + FalkorDB; no Zep API key needed)
|
||||
# FalkorDB runs as a Docker sidecar — see deploy/docker-compose.yml
|
||||
FALKORDB_HOST=falkordb
|
||||
FALKORDB_PORT=6379
|
||||
|
||||
# Embedding model (local sentence-transformers, pre-downloaded in the image)
|
||||
EMBEDDING_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
|
||||
```
|
||||
|
||||
#### 2. Install Dependencies
|
||||
|
|
@ -192,6 +201,59 @@ The MiroFish team is recruiting full-time/internship positions. If you're intere
|
|||
|
||||
MiroFish's simulation engine is powered by **[OASIS (Open Agent Social Interaction Simulations)](https://github.com/camel-ai/oasis)**, We sincerely thank the CAMEL-AI team for their open-source contributions!
|
||||
|
||||
## 🔱 About this fork
|
||||
|
||||
This is a maintained fork of [666ghj/MiroFish](https://github.com/666ghj/MiroFish) maintained at [profikid/MiroFish](https://github.com/profikid/MiroFish). It replaces the Zep Cloud dependency with **Graphiti + FalkorDB** (both open source), adds a one-call ingest API, and ships a production-ready Docker deployment for `agent.profikid.nl`.
|
||||
|
||||
### What changed vs upstream
|
||||
|
||||
| | Upstream | This fork |
|
||||
| --- | --- | --- |
|
||||
| Graph store | Zep Cloud (managed, requires API key) | Graphiti + FalkorDB (self-hosted, Redis protocol) |
|
||||
| LLM recommendation | Qwen-plus (DashScope) | MiniMax M-series (M2.7-highspeed for cron, M3 for high-quality) |
|
||||
| Ingest API | 3 calls (project → ontology → build) | 1 call: `POST /api/graph/ingest_text` (project + ontology + build) |
|
||||
| Deploy | `npm run dev` (dev) / `docker compose up` (Docker Hub) | `deploy/` overlay: Traefik + Let's Encrypt + FalkorDB sidecar + e2e one-shot |
|
||||
| E2E test | none shipped | `deploy/e2e_test.py` + `deploy/e2e.sh` (builds a one-shot image, runs against the real briefing seed) |
|
||||
| Cron-friendly | not designed for it | `deploy/mirofish_ingest.sh` — one-shot POST + poll, exits 0/2/3/4 |
|
||||
| Embeddings | Zep-managed | local `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` (pre-downloaded in the image) |
|
||||
|
||||
### Why Graphiti + FalkorDB
|
||||
|
||||
Zep Cloud's free tier is fine for dev, but for a self-hosted cron pipeline (Iran briefing every 12h, entity extraction, simulations) you want:
|
||||
- **No per-month API cap** — the cron keeps running on bad days, good days, news spikes
|
||||
- **No external service dependency** — graph store lives next to the app as a Docker sidecar
|
||||
- **Same engine** — Graphiti is the open-source core that powers Zep Cloud, so the entity/edge quality is identical
|
||||
|
||||
The `backend/app/services/graphiti_service.py` shim is a Zep-shaped facade over a real `Graphiti(graph_driver=FalkorDriver(...))`, so the rest of the MiroFish codebase (which still speaks Zep) didn't have to change.
|
||||
|
||||
### Production deployment
|
||||
|
||||
The `deploy/` directory contains a single-host Docker stack for `agent.profikid.nl`:
|
||||
|
||||
```bash
|
||||
# On the host:
|
||||
cd /docker/mirofish
|
||||
./deploy/up.sh # build image, bring up mirofish + falkordb
|
||||
./deploy/e2e.sh # run the e2e test against the real briefing seed
|
||||
```
|
||||
|
||||
Traefik (already on the host) auto-issues the Let's Encrypt cert for `mirofish.agent.profikid.nl`. See [deploy/README.md](./deploy/README.md) for the full layout, env vars, and troubleshooting.
|
||||
|
||||
### Cron integration
|
||||
|
||||
The fork was built so that a scheduled OSINT briefing cron can drop the markdown output straight into MiroFish:
|
||||
|
||||
```bash
|
||||
# Cron's last-written briefing file -> POST + poll
|
||||
briefing="$(ls -t /home/hermes/.hermes/cron/output/<job_id>/*.md | head -1)"
|
||||
nohup bash /docker/mirofish/deploy/mirofish_ingest.sh \
|
||||
"iran-osint-$(date -u +%Y%m%dT%H%M)" \
|
||||
"$briefing" \
|
||||
>/tmp/mirofish-ingest.log 2>&1 &
|
||||
```
|
||||
|
||||
The script handles ontology generation + async graph build + polling, exits 0 on success with a `nodes=N edges=M` summary in the log.
|
||||
|
||||
## 📈 Project Statistics
|
||||
|
||||
<a href="https://www.star-history.com/#666ghj/MiroFish&type=date&legend=top-left">
|
||||
|
|
|
|||
|
|
@ -529,6 +529,236 @@ def build_graph():
|
|||
}), 500
|
||||
|
||||
|
||||
|
||||
|
||||
# ============== 一键摄入接口 ==============
|
||||
|
||||
import threading
|
||||
import traceback
|
||||
|
||||
from flask import request, jsonify
|
||||
|
||||
from . import graph_bp
|
||||
from ..config import Config
|
||||
from ..services.graph_builder import GraphBuilderService
|
||||
from ..services.ontology_generator import OntologyGenerator
|
||||
from ..services.text_processor import TextProcessor
|
||||
from ..utils.logger import get_logger
|
||||
from ..utils.locale import t, get_locale, set_locale
|
||||
from ..models.task import TaskManager, TaskStatus
|
||||
from ..models.project import ProjectManager, ProjectStatus
|
||||
|
||||
logger = get_logger("mirofish.api.ingest_text")
|
||||
|
||||
|
||||
@graph_bp.route("/ingest_text", methods=["POST"])
|
||||
def ingest_text():
|
||||
"""
|
||||
One-shot ingest: text in, project + ontology + async build out.
|
||||
|
||||
Request (JSON):
|
||||
{
|
||||
"name": "iran-briefing-2026-06-09T11",
|
||||
"text": "## IRAN/US/ISRAEL CONFLICT ...",
|
||||
"simulation_requirement": "Extract entities and relations...",
|
||||
"additional_context": "optional, free-form",
|
||||
"graph_name": "optional, defaults to name",
|
||||
"chunk_size": 500,
|
||||
"chunk_overlap": 50
|
||||
}
|
||||
|
||||
Response:
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"project_id": "proj_xxxx",
|
||||
"task_id": "task_xxxx",
|
||||
"message": "..."
|
||||
}
|
||||
}
|
||||
|
||||
Poll status with: GET /api/graph/task/<task_id>
|
||||
"""
|
||||
try:
|
||||
data = request.get_json(silent=True) or {}
|
||||
|
||||
name = (data.get("name") or "").strip()
|
||||
text = (data.get("text") or "").strip()
|
||||
simulation_requirement = (data.get("simulation_requirement") or "").strip()
|
||||
additional_context = (data.get("additional_context") or "").strip() or None
|
||||
graph_name = (data.get("graph_name") or name or "MiroFish Graph").strip()
|
||||
chunk_size = int(data.get("chunk_size") or Config.DEFAULT_CHUNK_SIZE)
|
||||
chunk_overlap = int(data.get("chunk_overlap") or Config.DEFAULT_CHUNK_OVERLAP)
|
||||
|
||||
if not name:
|
||||
return jsonify({"success": False, "error": "name is required"}), 400
|
||||
if not text:
|
||||
return jsonify({"success": False, "error": "text is required"}), 400
|
||||
if not simulation_requirement:
|
||||
return jsonify(
|
||||
{"success": False, "error": "simulation_requirement is required"}
|
||||
), 400
|
||||
if not Config.FALKORDB_HOST:
|
||||
return jsonify({"success": False, "error": "FalkorDB not configured"}), 500
|
||||
|
||||
# 1. Create project
|
||||
project = ProjectManager.create_project(name=name)
|
||||
project.simulation_requirement = simulation_requirement
|
||||
project.chunk_size = chunk_size
|
||||
project.chunk_overlap = chunk_overlap
|
||||
logger.info(f"[ingest_text] created project {project.project_id} ({name})")
|
||||
|
||||
# 2. Save extracted text
|
||||
cleaned_text = TextProcessor.preprocess_text(text)
|
||||
project.total_text_length = len(cleaned_text)
|
||||
ProjectManager.save_extracted_text(project.project_id, cleaned_text)
|
||||
logger.info(
|
||||
f"[ingest_text] saved text: {len(cleaned_text)} chars "
|
||||
f"(project={project.project_id})"
|
||||
)
|
||||
|
||||
# 3. Generate ontology synchronously — this is an LLM call, takes ~10–30s
|
||||
generator = OntologyGenerator()
|
||||
ontology = generator.generate(
|
||||
document_texts=[cleaned_text],
|
||||
simulation_requirement=simulation_requirement,
|
||||
additional_context=additional_context,
|
||||
)
|
||||
entity_count = len(ontology.get("entity_types", []))
|
||||
edge_count = len(ontology.get("edge_types", []))
|
||||
project.ontology = {
|
||||
"entity_types": ontology.get("entity_types", []),
|
||||
"edge_types": ontology.get("edge_types", []),
|
||||
}
|
||||
project.analysis_summary = ontology.get("analysis_summary", "")
|
||||
project.status = ProjectStatus.ONTOLOGY_GENERATED
|
||||
ProjectManager.save_project(project)
|
||||
logger.info(
|
||||
f"[ingest_text] ontology ready: {entity_count} entity types, "
|
||||
f"{edge_count} edge types (project={project.project_id})"
|
||||
)
|
||||
|
||||
# 4. Kick off async graph build (same path /build uses)
|
||||
task_manager = TaskManager()
|
||||
task_id = task_manager.create_task(f"构建图谱: {graph_name}")
|
||||
project.status = ProjectStatus.GRAPH_BUILDING
|
||||
project.graph_build_task_id = task_id
|
||||
ProjectManager.save_project(project)
|
||||
current_locale = get_locale()
|
||||
|
||||
def build_task():
|
||||
set_locale(current_locale)
|
||||
build_logger = get_logger("mirofish.build.ingest")
|
||||
try:
|
||||
build_logger.info(
|
||||
f"[{task_id}] start build for project={project.project_id}"
|
||||
)
|
||||
task_manager.update_task(
|
||||
task_id, status=TaskStatus.PROCESSING, message=t("progress.initGraphService")
|
||||
)
|
||||
builder = GraphBuilderService(api_key=None)
|
||||
|
||||
# Chunk
|
||||
task_manager.update_task(task_id, message=t("progress.textChunking"), progress=5)
|
||||
chunks = TextProcessor.split_text(
|
||||
cleaned_text, chunk_size=chunk_size, overlap=chunk_overlap
|
||||
)
|
||||
total_chunks = len(chunks)
|
||||
|
||||
# Create graph + ontology
|
||||
task_manager.update_task(
|
||||
task_id, message=t("progress.creatingZepGraph"), progress=10
|
||||
)
|
||||
graph_id = builder.create_graph(name=graph_name)
|
||||
project.graph_id = graph_id
|
||||
ProjectManager.save_project(project)
|
||||
|
||||
task_manager.update_task(
|
||||
task_id, message=t("progress.settingOntology"), progress=15
|
||||
)
|
||||
builder.set_ontology(graph_id, ontology)
|
||||
|
||||
# Add text batches
|
||||
def add_cb(msg, ratio):
|
||||
task_manager.update_task(
|
||||
task_id, message=msg, progress=15 + int(ratio * 40)
|
||||
)
|
||||
|
||||
episode_uuids = builder.add_text_batches(
|
||||
graph_id, chunks, batch_size=3, progress_callback=add_cb
|
||||
)
|
||||
|
||||
# Wait for Graphiti to process episodes
|
||||
def wait_cb(msg, ratio):
|
||||
task_manager.update_task(
|
||||
task_id, message=msg, progress=55 + int(ratio * 35)
|
||||
)
|
||||
|
||||
builder._wait_for_episodes(episode_uuids, wait_cb)
|
||||
|
||||
# Fetch final graph data
|
||||
graph_data = builder.get_graph_data(graph_id)
|
||||
project.status = ProjectStatus.GRAPH_COMPLETED
|
||||
ProjectManager.save_project(project)
|
||||
|
||||
node_count = graph_data.get("node_count", 0)
|
||||
edge_count_out = graph_data.get("edge_count", 0)
|
||||
build_logger.info(
|
||||
f"[{task_id}] done: graph_id={graph_id} "
|
||||
f"nodes={node_count} edges={edge_count_out}"
|
||||
)
|
||||
|
||||
task_manager.update_task(
|
||||
task_id,
|
||||
status=TaskStatus.COMPLETED,
|
||||
message=t("progress.graphBuildComplete"),
|
||||
progress=100,
|
||||
result={
|
||||
"project_id": project.project_id,
|
||||
"graph_id": graph_id,
|
||||
"node_count": node_count,
|
||||
"edge_count": edge_count_out,
|
||||
"chunk_count": total_chunks,
|
||||
},
|
||||
)
|
||||
except Exception as e:
|
||||
build_logger.error(f"[{task_id}] failed: {e}")
|
||||
build_logger.debug(traceback.format_exc())
|
||||
project.status = ProjectStatus.FAILED
|
||||
project.error = str(e)
|
||||
ProjectManager.save_project(project)
|
||||
task_manager.update_task(
|
||||
task_id,
|
||||
status=TaskStatus.FAILED,
|
||||
message=t("progress.buildFailed", error=str(e)),
|
||||
error=traceback.format_exc(),
|
||||
)
|
||||
|
||||
thread = threading.Thread(target=build_task, daemon=True)
|
||||
thread.start()
|
||||
|
||||
return jsonify(
|
||||
{
|
||||
"success": True,
|
||||
"data": {
|
||||
"project_id": project.project_id,
|
||||
"task_id": task_id,
|
||||
"graph_name": graph_name,
|
||||
"ontology_entity_types": entity_count,
|
||||
"ontology_edge_types": edge_count,
|
||||
"text_length": len(cleaned_text),
|
||||
"message": t("api.graphBuildStarted", taskId=task_id),
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"ingest_text failed: {e}")
|
||||
logger.debug(traceback.format_exc())
|
||||
return jsonify(
|
||||
{"success": False, "error": str(e), "traceback": traceback.format_exc()}
|
||||
), 500
|
||||
|
||||
# ============== 任务查询接口 ==============
|
||||
|
||||
@graph_bp.route('/task/<task_id>', methods=['GET'])
|
||||
|
|
|
|||
|
|
@ -214,12 +214,41 @@ class OntologyGenerator:
|
|||
]
|
||||
|
||||
# 调用LLM
|
||||
result = self.llm_client.chat_json(
|
||||
messages=messages,
|
||||
temperature=0.3,
|
||||
max_tokens=4096
|
||||
)
|
||||
|
||||
try:
|
||||
result = self.llm_client.chat_json(
|
||||
messages=messages,
|
||||
temperature=0.3,
|
||||
max_tokens=16384,
|
||||
)
|
||||
except ValueError as json_err:
|
||||
logger.warning(
|
||||
f"[ontology] first attempt failed (likely M3 JSON truncation): {json_err}; retrying with compact prompt"
|
||||
)
|
||||
# 紧凑重试:限制 entity/edge types 数量,砍掉 attributes
|
||||
compact_doc = "\n".join(document_texts)[:30000]
|
||||
compact_messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"Output ONLY valid JSON, no markdown, no commentary. "
|
||||
"Schema: {entity_types:[{name:PascalCase,description:short}],"
|
||||
"edge_types:[{name:UPPER_SNAKE_CASE,description:short}],"
|
||||
"analysis_summary:one sentence}. "
|
||||
"Limit to AT MOST 6 entity types and 6 edge types. "
|
||||
"Do NOT include attributes arrays."
|
||||
),
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": f"Text:\\n{compact_doc}\\n\\nRequirement: {simulation_requirement}\\n\\nReturn JSON.",
|
||||
},
|
||||
]
|
||||
result = self.llm_client.chat_json(
|
||||
messages=compact_messages,
|
||||
temperature=0.2,
|
||||
max_tokens=8192,
|
||||
)
|
||||
|
||||
# 验证和后处理
|
||||
result = self._validate_and_process(result)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,13 +1,19 @@
|
|||
# Lightweight e2e test image. Inherits the MiroFish base image (already
|
||||
# has the patched graphiti_service.py + all Python deps) and only adds
|
||||
# the test script. Keeps the image small and the build fast.
|
||||
#
|
||||
# Build context: project root (so the parent Dockerfile already has app/).
|
||||
# The test script is fetched from deploy/ in the project tree.
|
||||
|
||||
FROM mirofish:latest
|
||||
|
||||
# Drop the long-running start.sh and just run the test
|
||||
USER root
|
||||
RUN mkdir -p /app/deploy
|
||||
COPY e2e_test.py /app/deploy/e2e_test.py
|
||||
COPY deploy/e2e_test.py /app/deploy/e2e_test.py
|
||||
|
||||
# Run the test as the entrypoint
|
||||
ENTRYPOINT ["python", "/app/deploy/e2e_test.py"]
|
||||
# Run the test as the entrypoint. The MiroFish image is uv-managed — flask,
|
||||
# graphiti, falkordb all live under /app/backend/.venv, so we need to invoke
|
||||
# python via `uv run` to get the venv on PYTHONPATH.
|
||||
WORKDIR /app/backend
|
||||
ENTRYPOINT ["uv", "run", "python", "/app/deploy/e2e_test.py"]
|
||||
|
|
|
|||
Loading…
Reference in New Issue