MicroFish/.kiro/specs/graphiti-ollama-reranker/research.md

Research & Design Decisions — graphiti-ollama-reranker

Summary

• Feature: graphiti-ollama-reranker
• Discovery Scope: Extension (one new service module + factory branch + config + docs).
• Key Findings:
  • CrossEncoderClient.rank(query, passages) -> list[tuple[str, float]] is the only abstract contract Graphiti requires of the reranker. The existing _PassthroughReranker already exercises this contract correctly.
  • Ollama's OpenAI-compatible /v1/chat/completions endpoint does not reliably expose logprobs / logit_bias, so Graphiti's default OpenAI scoring approach (binary YES/NO over token logits) cannot be ported. The reranker must use prompted numeric scoring with text-output parsing.
  • The openai SDK already shipped in backend/.venv (v2.35.1) exposes AsyncOpenAI, which is the right client for the async rank() method without introducing any new dependency.

Research Log

Graphiti's CrossEncoderClient contract

• Context: Need to confirm the precise shape of the rank interface and any other abstract members.
• Sources Consulted: backend/app/services/graphiti_adapter.py:38-51 (_PassthroughReranker); .kiro/specs/graphiti-neo4j-finalize/research.md and gap-analysis.md (which captured the upstream contract on first integration); ticket #39 narrative.
• Findings:
  • _PassthroughReranker subclasses CrossEncoderClient and only overrides async def rank(query: str, passages: list[str]) -> list[tuple[str, float]].
  • Graphiti's internal call site (graphiti_core/graphiti.py:154) constructs the reranker once and calls rank per search. There is no separate batch interface to satisfy.
  • Passages are short text snippets (entity-edge facts / node summaries). Typical N per search ≤ 10 (limit defaulted in _GraphNamespace.search).
• Implications: A drop-in subclass that implements rank is sufficient. No additional abstract methods to wire.

Ollama OpenAI-compatible scoring surface

• Context: Decide how to obtain a relevance score per passage from a small Ollama-served chat model.
• Sources Consulted: Project-internal backend/app/utils/llm_client.py (uses openai.OpenAI + chat.completions.create against Dashscope / OpenAI / Ollama uniformly); ticket #39 "Proposed approach" section enumerating Ollama chat-model scoring vs. embedding cosine.
• Findings:
  • Ollama supports /v1/chat/completions for chat models like qwen2.5:3b, llama3.2:3b, phi3:3.8b. Pulling a model is required (ollama pull <model>).
  • JSON-mode (response_format={"type": "json_object"}) is honored by recent Ollama versions but not universally; project convention is to fall back gracefully (cf. LLMClient.chat_json).
  • Embedding-cosine reranker is feasible (re-embed query and passages with mxbai-embed-large) but produces a weaker ordering signal than an LLM that can reason about the question. Picking LLM scoring matches the ticket's preferred path.
• Implications:
  • Use a chat-completion call per passage with a deterministic temperature (0.0) and a tight system prompt asking for a JSON score in [0.0, 1.0].
  • Parse with the same defensive strategy used elsewhere: strip <think> blocks, strip markdown fences, attempt json.loads, regex-fallback to first float, deterministic low score on hard failure.

Concurrency strategy

• Context: Decide between per-passage parallel calls vs. one batched call.
• Findings:
  • Per-passage with asyncio.gather is simpler to align outputs and resilient — a single bad output only loses one passage's score.
  • Single batched prompt requires the model to emit aligned scores (often by index); LLMs occasionally drop entries or misorder them, demanding additional validation.
  • With typical limit ≤ 10, parallel per-passage calls hit Ollama briefly; on a 3B model this is < 5s for 10 passages.
• Implications: Default to per-passage asyncio.gather. Expose no extra concurrency knob initially (avoid premature configuration surface; YAGNI per project guidelines).

Failure semantics

• Context: Required by R5 — Flask must keep serving on Ollama outage, and graph search should remain functional.
• Sources Consulted: backend/app/services/graphiti_adapter.py:515-517 (_GraphNamespace.search swallows all exceptions and logs a warning); _get_graphiti() runs once at first call.
• Findings:
  • Construction of an openai.AsyncOpenAI client does not perform any network I/O. Therefore OllamaReranker.__init__ can be safe at startup even when Ollama is down.
  • If rank() itself raises, the upstream Graphiti.search may surface the exception. The new reranker should therefore catch its own errors and degrade to passthrough behavior in-method rather than relying on the outer try/except in _GraphNamespace.search.
• Implications: OllamaReranker.rank should never raise. On exception or unparseable output it returns the input passages in the original order with passthrough-style synthetic scores and emits a single WARNING log per failure (rate-limited by intent: one log per rank() call).

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
A: Add class to graphiti_adapter.py	Define OllamaReranker next to _PassthroughReranker in the same file.	Minimal diff; single file to read.	Bloats an already-long adapter; mixes wiring with provider-specific logic.	—
B: New services/ollama_reranker.py module	Dedicated module owns prompt + parse + async client; adapter only selects it.	Single-responsibility module; matches ticket suggestion; reusable in isolation.	One extra import in adapter.	Selected. Aligns with project pattern of one concern per services/* file.
C: Hybrid provider registry	Map RERANKER_PROVIDER → builder in adapter; class still in B's module.	Future providers are a one-line registry change.	Over-engineering for two providers (ollama + none).	Deferred until a third provider is needed.

Design Decisions

Decision: Provider selected via env var, branch lives in _get_graphiti()

• Context: R3 requires env-driven provider selection; only two values supported by this spec (ollama and none).
• Alternatives Considered:
  1. Function-pointer registry (Option C).
  2. Inline if/else in the factory selecting one of two classes.
• Selected Approach: Inline branch in _get_graphiti() reads Config.RERANKER_PROVIDER, picks _build_ollama_reranker() or _PassthroughReranker(), validates unknown values with a ValueError matching the existing _ALLOWED_GRAPHITI_PROVIDERS convention.
• Rationale: Mirrors the established GRAPHITI_LLM_PROVIDER validation pattern (_ALLOWED_GRAPHITI_PROVIDERS) without adding speculative abstraction. Two values, two branches.
• Trade-offs: Adding a third provider later costs one more elif; acceptable.
• Follow-up: Surface the selected provider in the INFO startup log so operators can confirm.

Decision: Per-passage scoring with asyncio.gather, no concurrency knob

• Context: R2.3 requires one score per passage in descending order; R5 requires graceful per-call failure.
• Alternatives Considered:
  1. Single batched prompt with index-aligned output.
  2. Per-passage call with bounded Semaphore.
• Selected Approach: Per-passage asyncio.gather with no explicit limit; rely on default limit ≤ 10 in _GraphNamespace.search.
• Rationale: Simple, deterministic, isolates per-passage failures. Avoids premature configuration knob.
• Trade-offs: If a future caller asks for limit=100, Ollama may queue 100 requests; acceptable for now because no caller does this.
• Follow-up: If real-world rerank latency becomes a concern, add RERANKER_MAX_PARALLEL then.

Decision: Default model = qwen2.5:3b

• Context: Need a small, broadly-available Ollama chat model that reliably emits a numeric score in 1–2 tokens.
• Alternatives Considered:
  1. qwen2.5:3b (Apache-2.0, 3B params, strong instruction following).
  2. llama3.2:3b (Llama community license, 3B).
  3. phi3:3.8b (MIT, 3.8B).
• Selected Approach: qwen2.5:3b.
• Rationale: Matches the Qwen-family alignment of the rest of the project (qwen-plus is the documented LLM default). Apache-2.0 license is permissive. Small enough for typical dev machines.
• Trade-offs: Operators on systems without qwen2.5:3b must ollama pull qwen2.5:3b or override RERANKER_MODEL.
• Follow-up: README will document ollama pull qwen2.5:3b alongside the existing ollama pull mxbai-embed-large step.

Decision: Defensive output parsing (json.loads → regex float → deterministic low score)

• Context: R2.6 requires deterministic handling of unparseable model responses.
• Selected Approach:
  1. Strip <think>...</think> blocks (project convention from llm_client.py:64).
  2. Strip markdown fences (project convention from llm_client.chat_json).
  3. json.loads and read score (float in [0, 1], clipped on out-of-range).
  4. On JSON failure, regex-extract the first float token; clip to [0, 1].
  5. On total failure, assign 0.0 - 0.001 * passage_index (deterministic and below any successfully-parsed score).
• Rationale: Reuses patterns already in the codebase. Keeps every passage in the output (R2.6).
• Trade-offs: One failed parse silently downranks a passage; logged at DEBUG (not WARNING) to avoid log spam.

Risks & Mitigations

• Risk: Ollama service is not running on startup → boot must not fail. Mitigation: Construct only AsyncOpenAI (no network call) during __init__. Defer connectivity to first rank(). R5.4.
• Risk: Model is not pulled → rank() raises 404 from Ollama. Mitigation: Catch within rank(), log WARNING naming model + error class, return passthrough-ordered tuples so search still works. R5.1, R5.3.
• Risk: Operator misconfigures RERANKER_PROVIDER to an unknown value → silent fallthrough to wrong reranker. Mitigation: _get_graphiti() raises ValueError listing allowed values, mirroring _ALLOWED_GRAPHITI_PROVIDERS. R3.5.
• Risk: Multiple concurrent rank() calls overwhelm a small local Ollama daemon. Mitigation: Accept default Graphiti limit ≤ 10; document RERANKER_MAX_PARALLEL as a future follow-up if needed.

References

• backend/app/services/graphiti_adapter.py:38-51 — current passthrough reranker contract.
• backend/app/services/graphiti_adapter.py:142-162 — current _get_graphiti() wiring point.
• backend/app/utils/llm_client.py — project pattern for OpenAI-SDK chat + JSON parsing + reasoning-block stripping.
• .kiro/specs/graphiti-neo4j-finalize/research.md — historical context for why the passthrough was introduced.
• Ticket #39 in .ticket/39.md — feature brief and acceptance criteria.