diff --git a/README.md b/README.md index de082935..318d4e24 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,8 @@ **MiroFish** is a next-generation AI prediction engine powered by multi-agent technology. By extracting seed information from the real world (such as breaking news, policy drafts, or financial signals), it automatically constructs a high-fidelity parallel digital world. Within this space, thousands of intelligent agents with independent personalities, long-term memory, and behavioral logic freely interact and undergo social evolution. You can inject variables dynamically from a "God's-eye view" to precisely deduce future trajectories โ€” **rehearse the future in a digital sandbox, and win decisions after countless simulations**. +> **This is a maintained fork** of [666ghj/MiroFish](https://github.com/666ghj/MiroFish). It swaps Zep Cloud for **Graphiti + FalkorDB** (open source, self-hosted), adds a one-call `POST /api/graph/ingest_text` API for cron automations, and ships a production Docker stack for `agent.profikid.nl`. See [About this fork](#-about-this-fork) below. + > You only need to: Upload seed materials (data analysis reports or interesting novel stories) and describe your prediction requirements in natural language
> MiroFish will return: A detailed prediction report and a deeply interactive high-fidelity digital world @@ -116,15 +118,22 @@ cp .env.example .env ```env # LLM API Configuration (supports any LLM API with OpenAI SDK format) -# Recommended: Alibaba Qwen-plus model via Bailian Platform: https://bailian.console.aliyun.com/ +# Recommended: MiniMax M-series via https://api.minimax.io/v1 +# - MiniMax-M2.7-highspeed : best for cron / high-volume (recommended) +# - MiniMax-M3 : heavier reasoning, slower, higher quality +# Alibaba Qwen-plus via Bailian Platform: https://bailian.console.aliyun.com/ # High consumption, try simulations with fewer than 40 rounds first LLM_API_KEY=your_api_key -LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 -LLM_MODEL_NAME=qwen-plus +LLM_BASE_URL=https://api.minimax.io/v1 +LLM_MODEL_NAME=MiniMax-M2.7-highspeed -# Zep Cloud Configuration -# Free monthly quota is sufficient for simple usage: https://app.getzep.com/ -ZEP_API_KEY=your_zep_api_key +# Graph store (this fork: Graphiti + FalkorDB; no Zep API key needed) +# FalkorDB runs as a Docker sidecar โ€” see deploy/docker-compose.yml +FALKORDB_HOST=falkordb +FALKORDB_PORT=6379 + +# Embedding model (local sentence-transformers, pre-downloaded in the image) +EMBEDDING_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 ``` #### 2. Install Dependencies @@ -192,6 +201,59 @@ The MiroFish team is recruiting full-time/internship positions. If you're intere MiroFish's simulation engine is powered by **[OASIS (Open Agent Social Interaction Simulations)](https://github.com/camel-ai/oasis)**, We sincerely thank the CAMEL-AI team for their open-source contributions! +## ๐Ÿ”ฑ About this fork + +This is a maintained fork of [666ghj/MiroFish](https://github.com/666ghj/MiroFish) maintained at [profikid/MiroFish](https://github.com/profikid/MiroFish). It replaces the Zep Cloud dependency with **Graphiti + FalkorDB** (both open source), adds a one-call ingest API, and ships a production-ready Docker deployment for `agent.profikid.nl`. + +### What changed vs upstream + +| | Upstream | This fork | +| --- | --- | --- | +| Graph store | Zep Cloud (managed, requires API key) | Graphiti + FalkorDB (self-hosted, Redis protocol) | +| LLM recommendation | Qwen-plus (DashScope) | MiniMax M-series (M2.7-highspeed for cron, M3 for high-quality) | +| Ingest API | 3 calls (project โ†’ ontology โ†’ build) | 1 call: `POST /api/graph/ingest_text` (project + ontology + build) | +| Deploy | `npm run dev` (dev) / `docker compose up` (Docker Hub) | `deploy/` overlay: Traefik + Let's Encrypt + FalkorDB sidecar + e2e one-shot | +| E2E test | none shipped | `deploy/e2e_test.py` + `deploy/e2e.sh` (builds a one-shot image, runs against the real briefing seed) | +| Cron-friendly | not designed for it | `deploy/mirofish_ingest.sh` โ€” one-shot POST + poll, exits 0/2/3/4 | +| Embeddings | Zep-managed | local `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` (pre-downloaded in the image) | + +### Why Graphiti + FalkorDB + +Zep Cloud's free tier is fine for dev, but for a self-hosted cron pipeline (Iran briefing every 12h, entity extraction, simulations) you want: +- **No per-month API cap** โ€” the cron keeps running on bad days, good days, news spikes +- **No external service dependency** โ€” graph store lives next to the app as a Docker sidecar +- **Same engine** โ€” Graphiti is the open-source core that powers Zep Cloud, so the entity/edge quality is identical + +The `backend/app/services/graphiti_service.py` shim is a Zep-shaped facade over a real `Graphiti(graph_driver=FalkorDriver(...))`, so the rest of the MiroFish codebase (which still speaks Zep) didn't have to change. + +### Production deployment + +The `deploy/` directory contains a single-host Docker stack for `agent.profikid.nl`: + +```bash +# On the host: +cd /docker/mirofish +./deploy/up.sh # build image, bring up mirofish + falkordb +./deploy/e2e.sh # run the e2e test against the real briefing seed +``` + +Traefik (already on the host) auto-issues the Let's Encrypt cert for `mirofish.agent.profikid.nl`. See [deploy/README.md](./deploy/README.md) for the full layout, env vars, and troubleshooting. + +### Cron integration + +The fork was built so that a scheduled OSINT briefing cron can drop the markdown output straight into MiroFish: + +```bash +# Cron's last-written briefing file -> POST + poll +briefing="$(ls -t /home/hermes/.hermes/cron/output//*.md | head -1)" +nohup bash /docker/mirofish/deploy/mirofish_ingest.sh \ + "iran-osint-$(date -u +%Y%m%dT%H%M)" \ + "$briefing" \ + >/tmp/mirofish-ingest.log 2>&1 & +``` + +The script handles ontology generation + async graph build + polling, exits 0 on success with a `nodes=N edges=M` summary in the log. + ## ๐Ÿ“ˆ Project Statistics diff --git a/backend/app/api/graph.py b/backend/app/api/graph.py index f1153f18..151d5041 100644 --- a/backend/app/api/graph.py +++ b/backend/app/api/graph.py @@ -529,6 +529,236 @@ def build_graph(): }), 500 + + +# ============== ไธ€้”ฎๆ‘„ๅ…ฅๆŽฅๅฃ ============== + +import threading +import traceback + +from flask import request, jsonify + +from . import graph_bp +from ..config import Config +from ..services.graph_builder import GraphBuilderService +from ..services.ontology_generator import OntologyGenerator +from ..services.text_processor import TextProcessor +from ..utils.logger import get_logger +from ..utils.locale import t, get_locale, set_locale +from ..models.task import TaskManager, TaskStatus +from ..models.project import ProjectManager, ProjectStatus + +logger = get_logger("mirofish.api.ingest_text") + + +@graph_bp.route("/ingest_text", methods=["POST"]) +def ingest_text(): + """ + One-shot ingest: text in, project + ontology + async build out. + + Request (JSON): + { + "name": "iran-briefing-2026-06-09T11", + "text": "## IRAN/US/ISRAEL CONFLICT ...", + "simulation_requirement": "Extract entities and relations...", + "additional_context": "optional, free-form", + "graph_name": "optional, defaults to name", + "chunk_size": 500, + "chunk_overlap": 50 + } + + Response: + { + "success": true, + "data": { + "project_id": "proj_xxxx", + "task_id": "task_xxxx", + "message": "..." + } + } + + Poll status with: GET /api/graph/task/ + """ + try: + data = request.get_json(silent=True) or {} + + name = (data.get("name") or "").strip() + text = (data.get("text") or "").strip() + simulation_requirement = (data.get("simulation_requirement") or "").strip() + additional_context = (data.get("additional_context") or "").strip() or None + graph_name = (data.get("graph_name") or name or "MiroFish Graph").strip() + chunk_size = int(data.get("chunk_size") or Config.DEFAULT_CHUNK_SIZE) + chunk_overlap = int(data.get("chunk_overlap") or Config.DEFAULT_CHUNK_OVERLAP) + + if not name: + return jsonify({"success": False, "error": "name is required"}), 400 + if not text: + return jsonify({"success": False, "error": "text is required"}), 400 + if not simulation_requirement: + return jsonify( + {"success": False, "error": "simulation_requirement is required"} + ), 400 + if not Config.FALKORDB_HOST: + return jsonify({"success": False, "error": "FalkorDB not configured"}), 500 + + # 1. Create project + project = ProjectManager.create_project(name=name) + project.simulation_requirement = simulation_requirement + project.chunk_size = chunk_size + project.chunk_overlap = chunk_overlap + logger.info(f"[ingest_text] created project {project.project_id} ({name})") + + # 2. Save extracted text + cleaned_text = TextProcessor.preprocess_text(text) + project.total_text_length = len(cleaned_text) + ProjectManager.save_extracted_text(project.project_id, cleaned_text) + logger.info( + f"[ingest_text] saved text: {len(cleaned_text)} chars " + f"(project={project.project_id})" + ) + + # 3. Generate ontology synchronously โ€” this is an LLM call, takes ~10โ€“30s + generator = OntologyGenerator() + ontology = generator.generate( + document_texts=[cleaned_text], + simulation_requirement=simulation_requirement, + additional_context=additional_context, + ) + entity_count = len(ontology.get("entity_types", [])) + edge_count = len(ontology.get("edge_types", [])) + project.ontology = { + "entity_types": ontology.get("entity_types", []), + "edge_types": ontology.get("edge_types", []), + } + project.analysis_summary = ontology.get("analysis_summary", "") + project.status = ProjectStatus.ONTOLOGY_GENERATED + ProjectManager.save_project(project) + logger.info( + f"[ingest_text] ontology ready: {entity_count} entity types, " + f"{edge_count} edge types (project={project.project_id})" + ) + + # 4. Kick off async graph build (same path /build uses) + task_manager = TaskManager() + task_id = task_manager.create_task(f"ๆž„ๅปบๅ›พ่ฐฑ: {graph_name}") + project.status = ProjectStatus.GRAPH_BUILDING + project.graph_build_task_id = task_id + ProjectManager.save_project(project) + current_locale = get_locale() + + def build_task(): + set_locale(current_locale) + build_logger = get_logger("mirofish.build.ingest") + try: + build_logger.info( + f"[{task_id}] start build for project={project.project_id}" + ) + task_manager.update_task( + task_id, status=TaskStatus.PROCESSING, message=t("progress.initGraphService") + ) + builder = GraphBuilderService(api_key=None) + + # Chunk + task_manager.update_task(task_id, message=t("progress.textChunking"), progress=5) + chunks = TextProcessor.split_text( + cleaned_text, chunk_size=chunk_size, overlap=chunk_overlap + ) + total_chunks = len(chunks) + + # Create graph + ontology + task_manager.update_task( + task_id, message=t("progress.creatingZepGraph"), progress=10 + ) + graph_id = builder.create_graph(name=graph_name) + project.graph_id = graph_id + ProjectManager.save_project(project) + + task_manager.update_task( + task_id, message=t("progress.settingOntology"), progress=15 + ) + builder.set_ontology(graph_id, ontology) + + # Add text batches + def add_cb(msg, ratio): + task_manager.update_task( + task_id, message=msg, progress=15 + int(ratio * 40) + ) + + episode_uuids = builder.add_text_batches( + graph_id, chunks, batch_size=3, progress_callback=add_cb + ) + + # Wait for Graphiti to process episodes + def wait_cb(msg, ratio): + task_manager.update_task( + task_id, message=msg, progress=55 + int(ratio * 35) + ) + + builder._wait_for_episodes(episode_uuids, wait_cb) + + # Fetch final graph data + graph_data = builder.get_graph_data(graph_id) + project.status = ProjectStatus.GRAPH_COMPLETED + ProjectManager.save_project(project) + + node_count = graph_data.get("node_count", 0) + edge_count_out = graph_data.get("edge_count", 0) + build_logger.info( + f"[{task_id}] done: graph_id={graph_id} " + f"nodes={node_count} edges={edge_count_out}" + ) + + task_manager.update_task( + task_id, + status=TaskStatus.COMPLETED, + message=t("progress.graphBuildComplete"), + progress=100, + result={ + "project_id": project.project_id, + "graph_id": graph_id, + "node_count": node_count, + "edge_count": edge_count_out, + "chunk_count": total_chunks, + }, + ) + except Exception as e: + build_logger.error(f"[{task_id}] failed: {e}") + build_logger.debug(traceback.format_exc()) + project.status = ProjectStatus.FAILED + project.error = str(e) + ProjectManager.save_project(project) + task_manager.update_task( + task_id, + status=TaskStatus.FAILED, + message=t("progress.buildFailed", error=str(e)), + error=traceback.format_exc(), + ) + + thread = threading.Thread(target=build_task, daemon=True) + thread.start() + + return jsonify( + { + "success": True, + "data": { + "project_id": project.project_id, + "task_id": task_id, + "graph_name": graph_name, + "ontology_entity_types": entity_count, + "ontology_edge_types": edge_count, + "text_length": len(cleaned_text), + "message": t("api.graphBuildStarted", taskId=task_id), + }, + } + ) + + except Exception as e: + logger.error(f"ingest_text failed: {e}") + logger.debug(traceback.format_exc()) + return jsonify( + {"success": False, "error": str(e), "traceback": traceback.format_exc()} + ), 500 + # ============== ไปปๅŠกๆŸฅ่ฏขๆŽฅๅฃ ============== @graph_bp.route('/task/', methods=['GET']) diff --git a/backend/app/services/ontology_generator.py b/backend/app/services/ontology_generator.py index 01a3d799..aedf403f 100644 --- a/backend/app/services/ontology_generator.py +++ b/backend/app/services/ontology_generator.py @@ -214,12 +214,41 @@ class OntologyGenerator: ] # ่ฐƒ็”จLLM - result = self.llm_client.chat_json( - messages=messages, - temperature=0.3, - max_tokens=4096 - ) - + try: + result = self.llm_client.chat_json( + messages=messages, + temperature=0.3, + max_tokens=16384, + ) + except ValueError as json_err: + logger.warning( + f"[ontology] first attempt failed (likely M3 JSON truncation): {json_err}; retrying with compact prompt" + ) + # ็ดงๅ‡‘้‡่ฏ•:้™ๅˆถ entity/edge types ๆ•ฐ้‡,็ ๆŽ‰ attributes + compact_doc = "\n".join(document_texts)[:30000] + compact_messages = [ + { + "role": "system", + "content": ( + "Output ONLY valid JSON, no markdown, no commentary. " + "Schema: {entity_types:[{name:PascalCase,description:short}]," + "edge_types:[{name:UPPER_SNAKE_CASE,description:short}]," + "analysis_summary:one sentence}. " + "Limit to AT MOST 6 entity types and 6 edge types. " + "Do NOT include attributes arrays." + ), + }, + { + "role": "user", + "content": f"Text:\\n{compact_doc}\\n\\nRequirement: {simulation_requirement}\\n\\nReturn JSON.", + }, + ] + result = self.llm_client.chat_json( + messages=compact_messages, + temperature=0.2, + max_tokens=8192, + ) + # ้ชŒ่ฏๅ’ŒๅŽๅค„็† result = self._validate_and_process(result) diff --git a/deploy/Dockerfile.e2e b/deploy/Dockerfile.e2e index f0d660a0..08f2a446 100644 --- a/deploy/Dockerfile.e2e +++ b/deploy/Dockerfile.e2e @@ -1,13 +1,19 @@ # Lightweight e2e test image. Inherits the MiroFish base image (already # has the patched graphiti_service.py + all Python deps) and only adds # the test script. Keeps the image small and the build fast. +# +# Build context: project root (so the parent Dockerfile already has app/). +# The test script is fetched from deploy/ in the project tree. FROM mirofish:latest # Drop the long-running start.sh and just run the test USER root RUN mkdir -p /app/deploy -COPY e2e_test.py /app/deploy/e2e_test.py +COPY deploy/e2e_test.py /app/deploy/e2e_test.py -# Run the test as the entrypoint -ENTRYPOINT ["python", "/app/deploy/e2e_test.py"] +# Run the test as the entrypoint. The MiroFish image is uv-managed โ€” flask, +# graphiti, falkordb all live under /app/backend/.venv, so we need to invoke +# python via `uv run` to get the venv on PYTHONPATH. +WORKDIR /app/backend +ENTRYPOINT ["uv", "run", "python", "/app/deploy/e2e_test.py"]