diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 3983005523..a918c61cdf 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -352,6 +352,19 @@ These browser suites are intended for targeted local verification and CI, not th For normal issue work, start with the smallest targeted check that proves the change. Reserve repo-wide typecheck/build/test runs for PR-ready handoff or changes broad enough that narrow checks do not cover the risk. +### Task search evaluation + +The task search relevance rubric and regression corpus are documented in +[SEARCH.md](SEARCH.md). Run the real PostgreSQL relevance suite with: + +```sh +pnpm exec vitest run server/src/__tests__/task-search-quality.test.ts +``` + +Set `SEARCH_EVAL_SCALE=1` to additionally measure a disposable 10,000-task, +30,000-comment dataset. `SEARCH_EVAL_REPORT=/tmp/search-quality.json` saves +per-query results and latency measurements; scale measurements are opt-in. + ### Recent task ordering The streamlined sidebar keeps five recent tasks per company and account in browser diff --git a/doc/SEARCH.md b/doc/SEARCH.md new file mode 100644 index 0000000000..bdda5ce317 --- /dev/null +++ b/doc/SEARCH.md @@ -0,0 +1,200 @@ +# Task search relevance + +## Product rubric + +Search should help someone reopen work they remember, using whatever fragment +stuck in memory: an ID, a few title words, a technical name, or something in the +conversation. The first screen should contain plausible answers, with enough +context to explain each match. + +| Intent | Good result | Failure | +|---|---|---| +| Known task ID | Exact ID first, case-insensitive; accept `PAP-42`, `pap42`, `PAP 42` | A mention or neighboring ID beats the task | +| Remembered title | Exact title, phrase, then all title words in any order | A recent comment mentioning those words beats the title | +| Several concepts | Every meaningful query term contributes, including short terms such as API/UI | A task matches only one common word | +| Exact phrase | Quoted text stays together and literal | Quotes silently behave like OR or fuzzy search | +| Thread memory | Find words across task text, comments and current documents | Relevant content exists but the task cannot be found | +| Technical text | Preserve underscores, percent signs, paths and numbers | SQL wildcard expansion or fuzzy IDs return unrelated work | +| Typo | Conservative title-word correction; all other terms still required | Ignoring a short term changes the query's meaning | +| Result explanation | Show the best evidence and link to its source | A title hit jumps into an unrelated comment | +| Old work | Strong completed-task matches remain ahead of weak recent hits | Recency/activity replaces relevance | +| Boundaries | Company, visibility, deletion and explicit filters always apply | Content leaks through counts, snippets or typo matches | +| Operations | PostgreSQL only, synchronous current-row reads, bounded query/page sizes | A worker, remote index or eventual-consistency repair is required | + +Judge results on a 0–3 scale: **3** directly answers the remembered task intent, +**2** is useful related work, **1** is only an incidental mention, **0** is +irrelevant. Ambiguous short queries may have several grade-3 answers; do not +invent a unique intended task for them. + +Acceptance gates: + +- Every unambiguous known-task case returns its intended task first. +- Every grade-3 result in the small judged corpus appears in the first five. +- All explicit negative, visibility, filter, freshness and literal-query cases pass. +- Report mean reciprocal rank (first grade-3 result) and nDCG@5 (graded ordering + and recall). Target MRR ≥ 0.95 and nDCG@5 ≥ 0.90 on the authored corpus. +- Measure both the full search page and the command-palette/task-list API. +- Measure database-backed latency separately from relevance. Report dataset + size, warm/cold assumptions and hardware; a small fixture is not scale proof. + Initial target: warm p95 ≤ 250 ms at 10,000 tasks and 30,000 short comments. + A regression greater than 20% from baseline requires investigation and an + explicit explanation of the cost; do not describe a quality improvement as + latency-neutral when it is not. + +The initial corpus is synthetic and deliberately adversarial. It includes +plausible distractors and gives older completed tasks strong relevance labels. +It is not evidence that every real user's search is solved. Add real failed +queries and human judgments as they become available. Do not adjust judgments +just to improve a score. + +## Previous behavior + +The command palette calls the issue-list endpoint. It searched one literal +substring across title, identifier, description and comments, prioritized titles +before identifiers, and did not search documents or recover typos. Reordered +words commonly returned no result. + +Company search used a different algorithm: any token admitted a result, bonuses +from titles, comments and documents accumulated, and title-only token coverage +was indistinguishable from words scattered across a long thread. It ran edit +distance for title words, discarded short terms from fuzzy matching, and also +fuzzed identifiers. Quotes were tokenized but did not constrain other matches. + +## Matching contract + +Both task search paths use `server/src/services/task-search.ts`. Search is lexical: +trim/collapse whitespace, normalize case, keep quoted phrases, remove a small +set of unquoted grammatical filler words, deduplicate terms, and retain up to +8 terms within the existing 200-character query bound. All-filler queries keep +their terms. No synonym service, embedding model or language-specific stemming +is involved. + +All retained terms must match. Full search and task lists allow terms to occur +across task text and current, undeleted conversation/document content. The +Tasks scope requires coverage in task text. Comments and Documents require a +participating match in that source, while retaining the task context. Exact/prefix +identifiers and conservative title-word typo matches are additional task matches. +Typo matching runs only when no literal match satisfies the requested filters. +It never guesses task numbers, loosens a quoted phrase or drops a short query +term. Alphabetic terms of one to three characters must begin a word, so `UI` +does not match `build`, while incomplete longer words still support typeahead. + +Ranking uses disjoint bands: exact ID, ID prefix, exact title, title phrase, +all title terms, all task-text terms, all thread terms, then title typo recovery. +Whole-word title matches and title prefixes break close ties; status has only a +small effect within a band. Full search uses recency and stable IDs for remaining +ties; task lists retain their existing priority/activity tie-breaking. Explicit +created/updated/priority sort modes retain their documented behavior. Other +entity types retain their existing scoring rules, rescaled to keep exact names +ahead of speculative task typo matches. The UI displays the server's order +without regrouping results by source. + +The existing `pg_trgm` indexes support literal substring retrieval. Tagged +comment/document match sets are computed once per search with separate indexed +patterns. Ranking stages carry compact flags; descriptions and matching snippets +are fetched for the result window. The database reads current rows, so creates, edits, deletions and +hidden-task changes take effect without indexing jobs. Bounded edit-distance +checks operate on titles only, run only as a zero-result fallback, and guard +fuzzystrmatch's 255-character argument limit. There is no schema migration or +new extension in this change. + +PostgreSQL documents the existing index support in +[pg_trgm](https://www.postgresql.org/docs/17/pgtrgm.html). + +## Reproduce the evaluation + +```sh +pnpm exec vitest run server/src/__tests__/task-search-quality.test.ts +# Also write per-query rankings and metrics for inspection: +SEARCH_EVAL_REPORT=/tmp/search-quality.json pnpm exec vitest run server/src/__tests__/task-search-quality.test.ts +# Include the larger latency dataset and query plans: +SEARCH_EVAL_SCALE=1 SEARCH_EVAL_REPORT=/tmp/search-scale.json pnpm exec vitest run server/src/__tests__/task-search-quality.test.ts +``` + +The fixture is `server/src/__tests__/fixtures/task-search-corpus.ts`. Tests run +the real services against a temporary embedded PostgreSQL database with the +normal migrations. `SEARCH_EVAL_BASELINE=1` records judgments without asserting +improved behavior. To compare another revision, copy this test, its fixture and +`task-search.ts` into a separate worktree for that revision, leave its actual +`company-search.ts` and `issues.ts` services unchanged, and run with +`SEARCH_EVAL_BASELINE=1`. The copied helper is not used by the baseline services; +its query-plan branch is disabled in baseline mode. + +## Initial evaluation — 2026-09-12 + +Compared against `2083bf6f9` using the same 31-task corpus and 24 queries (23 +queries with intended answers, plus one no-result query). + +| Surface | Intended answer first, before → after | MRR, before → after | nDCG@5, before → after | +|---|---|---|---| +| Full search | 17/23 → 23/23 | 0.828 → 1.000 | 0.904 → 0.999 | +| Quick search / task list | 5/23 → 23/23 | 0.268 → 1.000 | 0.339 → 0.999 | + +The relevance gates pass. These results measure the authored corpus, not general +search accuracy. The no-result query also returns no tasks in both surfaces. + +The scale run adds 10,000 tasks with ~300-character descriptions and 30,000 +~345-character comments. Measurements call the real service methods (including +facets/snippets or task-list hydration), excluding HTTP and UI debounce. Each +query has one separately recorded first request and 20 warm repetitions; p95 +is the 19th sorted warm sample. This is not a cold-disk test. Both revisions used +PostgreSQL 18.1, default planner/memory settings, and `ANALYZE` after seeding. +The host was an Apple M5 Max with 128 GiB RAM, running an x86_64 PostgreSQL +binary and other development tests concurrently. Treat timing deltas as local +measurements, not production capacity or a controlled concurrency benchmark. + +| Query | Full p95 before → after (ms) | Quick p95 before → after (ms) | +|---|---|---| +| `GitHub OAuth` | 139 → 95 | 39 → 128 | +| `OAuth callback GitHub` | 209 → 81 | 26 → 134 | +| `mibile api` | 153 → 131 | 19 → 111 | +| `search` | 172 → 41 | 22 → 67 | +| `quasarxylophone` | 154 → 105 | 18 → 218 | +| `routine` (matches all 10,000 added tasks) | 239 → 253 | 152 → 371 | + +Selective full searches improved. Quick search is more expensive: it now +evaluates term coverage, searches documents, and can scan company titles for +typo recovery. The old quick search returned no answers for the reordered and +typo queries, so its lower cost did not deliver equivalent results. The relative +regression threshold is triggered, and broad-query p95 does **not** meet the +initial 250 ms target. This is an explicit performance limitation of this pass. +The implementation adds no operational service, but it is not latency-neutral. + +`EXPLAIN (ANALYZE, BUFFERS)` confirmed existing trigram indexes on selective +comment/document retrieval and zero fuzzy-branch executions for successful +literal searches. Removing descriptions from intermediate materialized rows +eliminated 1,699 temporary blocks (~13 MiB) of writes in the broad-query core +plan; its final measured execution was 139 ms with no temporary writes. The +quick-search endpoint also performs its existing activity sorting and task +hydration. Larger companies, long threads and sustained concurrent searches +still need production-shaped measurement before a stronger latency claim. + +Browser acceptance used the real built app against an isolated PostgreSQL +database containing this corpus. Starting from the dashboard, the Search link +found an older completed task from reordered title words and opened that task. +Command-K ranked `PAP-42` first for `pap42`; `mibile api` recovered only the +intended mobile API task and carried the query into full search. Quoted +`"connection timeout"` excluded scattered words. `Hermes parser` showed the +document title as evidence and opened the correct plan in the task's side panel. +The desktop result layout was visually inspected. Mobile layout, continuous +transition timing and production data were not part of this walkthrough. + +For a future failed search, record the query, what the person remembered, and +the intended task IDs. Grade the old top five and any missed intended tasks +before changing the ranker, add realistic distractors, then run both entry +points. Keep these judgments independent of the ranking constants. + +Verification: 78 search/parser tests (including the real PostgreSQL scale run), +14 existing task-list search/filter tests, and 29 Search/CommandPalette UI tests +passed. Workspace typecheck, the final server typecheck, production build, +Storybook build and token gates passed. The full repository test run was stopped +after `chat-channels.integration.test.ts` reported one failure in “publishes a +closed-choice question, settles its Slack card, and delivers its exact +continuation response”; that test passed when rerun alone. The remaining broad +suite was not completed, so this is not a claim of a green repository-wide run. + +The API response contracts and company authorization stay unchanged. Artifact, +agent and project ranking are separate from the task relevance rubric. Extraction +search and the specialized blocked-attention queue retain their existing +literal matching. Pure semantic paraphrases and +language-specific word inflections are outside this first lexical rubric. diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index 7484cb9041..49281e7f3b 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -1142,6 +1142,9 @@ The current app also exposes V1-supporting surfaces for: - company-scoped summary slots for projects, the workspaces overview, project workspaces, and individual execution workspaces; execution-workspace slots are keyed by execution workspace id so a new workspace never inherits another workspace's summary - issue thread interactions (`suggest_tasks`, `ask_user_questions`, `request_confirmation`, `request_checkbox_confirmation`, `request_item_verdicts`) with the open-default resolver contract in §9.8.1 - issue approvals, issue references/search, labels, read state, inbox/archive state, and work products +- task search uses shared PostgreSQL matching/ranking for company search and task-list quick search; + all query terms contribute, quoted phrases stay literal, exact identifiers and direct title matches + lead relevance ordering, and the UI preserves server result order (see `doc/SEARCH.md`) - company search through `GET /companies/:companyId/search` plus agent-oriented bulk extraction through `GET /companies/:companyId/search/extract`; extraction accepts a server-escaped literal `contains`, optional server-owned URL expansion, issue/comment/document scopes, status/date filters, issue-level pagination, a diff --git a/doc/SPEC.md b/doc/SPEC.md index 23111a078f..967cd50145 100644 --- a/doc/SPEC.md +++ b/doc/SPEC.md @@ -586,3 +586,12 @@ allocation retains one selected number and individually enabled groups. See [iMessage Photon](connections/IMESSAGE-PHOTON.md) for the implementation contract, setup, recovery, boundaries, and qualification status. + +## Task search relevance + +Task discovery uses PostgreSQL and the existing search indexes, with no external +search service or background indexing job. The task-list quick search and full +company search share lexical matching and ranking. Known identifiers and direct +title matches lead; current conversation and document content supplies supporting +evidence. See [Task search relevance](SEARCH.md) for the evaluation rubric, +matching contract and reproducible quality tests. diff --git a/server/src/__tests__/company-search-service.test.ts b/server/src/__tests__/company-search-service.test.ts index 0db84ccced..a8665665ab 100644 --- a/server/src/__tests__/company-search-service.test.ts +++ b/server/src/__tests__/company-search-service.test.ts @@ -163,6 +163,19 @@ describeEmbeddedPostgres("companySearchService", () => { return id; } + it("keeps exact entity names ahead of speculative task typos and rejects empty quotes", async () => { + const companyId = await createCompany(); + const agentId = await createAgent(companyId, { name: "Mibile" }); + const projectId = await createProject(companyId, { name: "Mibile" }); + const taskId = await createIssue(companyId, { title: "Mobile navigation" }); + const result = await svc.search(companyId, companySearchQuerySchema.parse({ q: "mibile" })); + const ids = result.results.map((row) => row.id); + expect(ids).toContain(taskId); + expect(ids.indexOf(agentId)).toBeLessThan(ids.indexOf(taskId)); + expect(ids.indexOf(projectId)).toBeLessThan(ids.indexOf(taskId)); + expect((await svc.search(companyId, companySearchQuerySchema.parse({ q: '""' }))).results).toEqual([]); + }); + it("ranks exact issue identifiers before weaker title matches", async () => { const companyId = await createCompany(); const exactId = await createIssue(companyId, { @@ -180,7 +193,7 @@ describeEmbeddedPostgres("companySearchService", () => { expect(result.results[0]?.matchedFields).toContain("identifier"); }); - it("ranks phrase and all-token issue matches before partial scattered-token matches", async () => { + it("ranks phrase before reordered title words and rejects partial matches", async () => { const companyId = await createCompany(); const base = new Date("2026-01-01T00:00:00.000Z").getTime(); const partialTokenId = await createIssue(companyId, { @@ -201,7 +214,8 @@ describeEmbeddedPostgres("companySearchService", () => { const result = await svc.search(companyId, companySearchQuerySchema.parse({ q: "alpha beta", scope: "issues" })); - expect(result.results.map((row) => row.id)).toEqual([phraseId, allTokenId, partialTokenId]); + expect(result.results.map((row) => row.id)).toEqual([phraseId, allTokenId]); + expect(result.results.map((row) => row.id)).not.toContain(partialTokenId); }); it("matches multiple tokens across the same issue thread and returns comment snippets", async () => { @@ -682,6 +696,83 @@ describeEmbeddedPostgres("companySearchService", () => { } }); + it("does not interpret short UI terms as the middle of unrelated words", async () => { + const companyId = await createCompany(); + const target = await createIssue(companyId, { title: "Improve mobile UI" }); + await createIssue(companyId, { title: "Build billing reports" }); + const result = await svc.search(companyId, companySearchQuerySchema.parse({ q: "UI" })); + expect(result.results.map((row) => row.id)).toEqual([target]); + }); + + it("keeps typo fallback inside the requested filters", async () => { + const companyId = await createCompany(); + await createIssue(companyId, { title: "Mibile API", status: "done" }); + const target = await createIssue(companyId, { title: "Mobile API", status: "todo" }); + const result = await svc.search(companyId, companySearchQuerySchema.parse({ q: "mibile api", status: "todo" })); + expect(result.results.map((row) => row.id)).toEqual([target]); + }); + + it("keeps exact title hits on the task and chooses the most complete context evidence", async () => { + const companyId = await createCompany(); + const task = await createIssue(companyId, { title: "Aurora callback" }); + await db.insert(issueComments).values({ companyId, issueId: task, body: "Aurora callback is mentioned here too." }); + const exact = await svc.search(companyId, companySearchQuerySchema.parse({ q: "Aurora callback" })); + expect(exact.results[0]?.href).not.toContain("#comment-"); + + const holder = await createIssue(companyId, { title: "Connection investigation" }); + await db.insert(issueComments).values({ companyId, issueId: holder, body: "Aurora was discussed.", updatedAt: new Date("2026-06-01") }); + const strongest = randomUUID(); + await db.insert(issueComments).values({ id: strongest, companyId, issueId: holder, + body: "Aurora loses the callback state.", updatedAt: new Date("2026-01-01") }); + const result = await svc.search(companyId, companySearchQuerySchema.parse({ q: "aurora state callback" })); + expect(result.results[0]?.href).toContain(`#comment-${strongest}`); + expect(result.results[0]?.snippet).toContain("state"); + }); + + it.each(["comment", "document"] as const)("preserves %s evidence when title and identifier both match", async (source) => { + const companyId = await createCompany(); + const task = await createIssue(companyId, { + identifier: "CTX-123", title: "CTX callback investigation", description: "CTX callback details", + }); + const sourceId = randomUUID(); + if (source === "comment") { + await db.insert(issueComments).values({ id: sourceId, companyId, issueId: task, body: "The missing signal is quasar." }); + } else { + await db.insert(documents).values({ id: sourceId, companyId, title: "Investigation plan", latestBody: "The missing signal is quasar.", format: "markdown" }); + await db.insert(issueDocuments).values({ companyId, issueId: task, documentId: sourceId, key: "plan" }); + } + + const result = await svc.search(companyId, companySearchQuerySchema.parse({ q: "CTX quasar" })); + const match = result.results.find((row) => row.id === task)!; + expect(match.matchedFields).toEqual(expect.arrayContaining(["identifier", "title", source])); + expect(match.score).toBeGreaterThanOrEqual(2000); + expect(match.score).toBeLessThan(3000); + expect(match.snippets).toHaveLength(2); + expect(match.snippets[0]).toMatchObject({ field: source, text: expect.stringContaining("quasar") }); + expect(match.snippet).toContain("quasar"); + expect(match.href).toContain(source === "comment" ? `#comment-${sourceId}` : "#document-plan"); + }); + + it("reflects edits, deleted comments and document updates immediately", async () => { + const companyId = await createCompany(); + const task = await createIssue(companyId, { title: "Old uniquequartz title" }); + const find = () => svc.search(companyId, companySearchQuerySchema.parse({ q: '"uniquequartz"' })); + expect((await find()).results.map((row) => row.id)).toEqual([task]); + await db.update(issues).set({ title: "New title" }).where(sql`${issues.id} = ${task}`); + expect((await find()).results).toEqual([]); + const comment = randomUUID(); + await db.insert(issueComments).values({ id: comment, companyId, issueId: task, body: "uniquequartz" }); + expect((await find()).results.map((row) => row.id)).toEqual([task]); + await db.update(issueComments).set({ deletedAt: new Date() }).where(sql`${issueComments.id} = ${comment}`); + expect((await find()).results).toEqual([]); + const doc = randomUUID(); + await db.insert(documents).values({ id: doc, companyId, title: "Findings", latestBody: "uniquequartz", format: "markdown" }); + await db.insert(issueDocuments).values({ companyId, issueId: task, documentId: doc, key: "plan" }); + expect((await find()).results.map((row) => row.id)).toEqual([task]); + await db.update(documents).set({ latestBody: "Updated findings" }).where(sql`${documents.id} = ${doc}`); + expect((await find()).results).toEqual([]); + }); + it("uses pg_trgm for conservative fuzzy title matches", async () => { const companyId = await createCompany(); const issueId = await createIssue(companyId, { diff --git a/server/src/__tests__/fixtures/task-search-corpus.ts b/server/src/__tests__/fixtures/task-search-corpus.ts new file mode 100644 index 0000000000..dec251c52d --- /dev/null +++ b/server/src/__tests__/fixtures/task-search-corpus.ts @@ -0,0 +1,77 @@ +// Authored relevance judgments, independent of the ranker's scoring constants. +export const taskSearchCorpus = [ + { key: "id", identifier: "PAP-42", title: "Repair callback state", status: "done" }, + { key: "id-mention", title: "PAP-42 follow-up discussion" }, + { key: "id-neighbor", identifier: "PAP-420", title: "Repair callback state later" }, + { key: "oauth", title: "Fix GitHub OAuth callback", status: "done" }, + { key: "oauth-noise", title: "GitHub release checklist", comments: ["OAuth is mentioned in an unrelated weekly update."] }, + { key: "oauth-partial", title: "GitHub repository badges" }, + { key: "oauth-body", title: "Repair the connection flow", description: "GitHub OAuth callback loses state on redirect." }, + { key: "oauth-comment", title: "Investigate sign-in", comments: ["The GitHub OAuth callback loses state on redirect."] }, + { key: "oauth-doc", title: "Connection investigation", document: { title: "GitHub OAuth callback findings", body: "State is lost on redirect." } }, + { key: "search", title: "Improve search performance" }, + { key: "search-noise", title: "Improve exports", description: "A search for performance numbers was included in the meeting." }, + { key: "search-partial", title: "Search typography" }, + { key: "mobile", title: "Polish mobile navigation" }, + { key: "mobile-api", title: "Build mobile API" }, + { key: "mobile-ui", title: "Build mobile UI" }, + { key: "onboarding", title: "Onboarding wizard polish" }, + { key: "quoted", title: "Repair connection timeout handling" }, + { key: "quoted-scattered", title: "Connection retry after timeout" }, + { key: "cross", title: "Checkout ownership", comments: ["A concurrency race needs a regression test."] }, + { key: "cross-partial", title: "Checkout style guide" }, + { key: "document", title: "Adapter investigation", document: { title: "Hermes parser plan", body: "Discover plugins from their package manifest." } }, + { key: "percentage", title: "Release 100% checklist" }, + { key: "percentage-decoy", title: "Release 1000 checklist" }, + { key: "path", title: "Fix foo_bar lookup" }, + { key: "path-decoy", title: "Fix fooXbar lookup" }, + { key: "unicode", title: "Réparer navigation mobile" }, + { key: "identifier-code", title: "Document heartbeat_run_events retention" }, + { key: "word", title: "Fix API authentication" }, + { key: "word-decoy", title: "Capistrano migration", description: "API details appear here.", comments: ["API is an incidental mention."] }, + { key: "freshness", title: "Reconcile billing ledger", status: "done" }, + { key: "freshness-noise", title: "Weekly financial update", comments: ["Reconcile billing ledger was one of many completed projects."] }, +] as const; + +export type TaskSearchCase = { + name: string; + q: string; + relevant: Record; // 3 = intended task; 2 = useful; 1 = incidental; absent = irrelevant + first?: string; + absent?: string[]; + scope?: "all" | "issues" | "comments" | "documents"; +}; +export const taskSearchCases: TaskSearchCase[] = [ + { name: "exact identifier", q: "PAP-42", relevant: { id: 3, "id-mention": 1, "id-neighbor": 1 }, first: "id" }, + { name: "identifier case", q: "pap-42", relevant: { id: 3, "id-mention": 1, "id-neighbor": 1 }, first: "id" }, + { name: "compact identifier", q: "pap42", relevant: { id: 3, "id-neighbor": 1 }, first: "id" }, + { name: "spaced identifier", q: "PAP 42", relevant: { id: 3, "id-mention": 1, "id-neighbor": 1 }, first: "id" }, + { name: "title phrase", q: "GitHub OAuth", relevant: { oauth: 3, "oauth-body": 2, "oauth-comment": 2, "oauth-doc": 2, "oauth-noise": 1 }, first: "oauth", absent: ["oauth-partial"] }, + { name: "reordered title words", q: "OAuth GitHub callback", relevant: { oauth: 3, "oauth-body": 2, "oauth-comment": 2, "oauth-doc": 2 }, first: "oauth", absent: ["oauth-partial", "oauth-noise"] }, + { name: "title beats body chatter", q: "performance search", relevant: { search: 3, "search-noise": 1 }, first: "search", absent: ["search-partial"] }, + { name: "title beats incidental comment", q: "billing ledger", relevant: { freshness: 3, "freshness-noise": 1 }, first: "freshness" }, + { name: "filler words", q: "the GitHub OAuth callback", relevant: { oauth: 3, "oauth-body": 2, "oauth-comment": 2, "oauth-doc": 2 }, first: "oauth", absent: ["oauth-noise"] }, + { name: "quoted phrase", q: '"connection timeout"', relevant: { quoted: 3 }, first: "quoted", absent: ["quoted-scattered"] }, + { name: "quoted phrase plus term", q: 'repair "connection timeout"', relevant: { quoted: 3 }, first: "quoted", absent: ["quoted-scattered"] }, + { name: "transposition", q: "serach", relevant: { search: 3, "search-partial": 3 } }, + { name: "substitution", q: "mibile navigation", relevant: { mobile: 3, unicode: 3 }, absent: ["mobile-api", "mobile-ui"] }, + { name: "two missing letters", q: "onbordng wizard", relevant: { onboarding: 3 }, first: "onboarding" }, + { name: "short token constrains typo", q: "mibile api", relevant: { "mobile-api": 3 }, first: "mobile-api", absent: ["mobile", "mobile-ui"] }, + { name: "cross-field thread", q: "checkout concurrency", relevant: { cross: 3 }, first: "cross", absent: ["cross-partial"] }, + { name: "document title", q: "Hermes parser", relevant: { document: 3 }, first: "document" }, + { name: "document body", q: "plugins manifest", relevant: { document: 3 }, first: "document" }, + { name: "literal percent", q: "100%", relevant: { percentage: 3 }, first: "percentage", absent: ["percentage-decoy"] }, + { name: "literal underscore", q: "foo_bar", relevant: { path: 3 }, first: "path", absent: ["path-decoy"] }, + { name: "code identifier", q: "heartbeat_run_events", relevant: { "identifier-code": 3 }, first: "identifier-code" }, + { name: "unicode", q: "réparer mobile", relevant: { unicode: 3 }, first: "unicode" }, + { name: "whole word title", q: "api", relevant: { word: 3, "mobile-api": 3, "word-decoy": 1 } }, + { name: "no result", q: "quasarxylophone", relevant: {} }, +]; + +export function searchQualityMetrics(keys: string[], relevant: Record) { + const gain = (grade: number, index: number) => (2 ** grade - 1) / Math.log2(index + 2); + const dcg = keys.slice(0, 5).reduce((sum, key, index) => sum + gain(relevant[key] ?? 0, index), 0); + const ideal = Object.values(relevant).sort((a, b) => b - a).slice(0, 5).reduce((sum, grade, index) => sum + gain(grade, index), 0); + const rank = keys.findIndex((key) => relevant[key] === 3); + return { ndcg5: ideal === 0 ? Number(keys.length === 0) : dcg / ideal, reciprocalRank: rank < 0 ? 0 : 1 / (rank + 1) }; +} diff --git a/server/src/__tests__/issues-service.test.ts b/server/src/__tests__/issues-service.test.ts index 3b91c8856c..9309731a6d 100644 --- a/server/src/__tests__/issues-service.test.ts +++ b/server/src/__tests__/issues-service.test.ts @@ -1466,7 +1466,7 @@ describeEmbeddedPostgres("issueService.list participantAgentId", () => { expect(result.map((issue) => issue.id)).toEqual([recentMediumIssueId]); }); - it("ranks comment matches ahead of description-only matches", async () => { + it("ranks direct description matches ahead of comment-only matches", async () => { const companyId = randomUUID(); const commentMatchId = randomUUID(); const descriptionMatchId = randomUUID(); @@ -1508,7 +1508,7 @@ describeEmbeddedPostgres("issueService.list participantAgentId", () => { includeRoutineExecutions: true, }); - expect(result.map((issue) => issue.id)).toEqual([commentMatchId, descriptionMatchId]); + expect(result.map((issue) => issue.id)).toEqual([descriptionMatchId, commentMatchId]); }); it("filters issue lists to the full descendant tree for a root issue", async () => { diff --git a/server/src/__tests__/task-search-quality.test.ts b/server/src/__tests__/task-search-quality.test.ts new file mode 100644 index 0000000000..de84c1de66 --- /dev/null +++ b/server/src/__tests__/task-search-quality.test.ts @@ -0,0 +1,147 @@ +import { randomUUID } from "node:crypto"; +import { performance } from "node:perf_hooks"; +import { writeFile } from "node:fs/promises"; +import { cpus, platform, release, totalmem } from "node:os"; +import { sql } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { companies, createDb, documents, issueComments, issueDocuments, issues, getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "@paperclipai/db"; +import { companySearchQuerySchema } from "@paperclipai/shared"; +import { companySearchService } from "../services/company-search.js"; +import { parseTaskSearch, taskSearchCtes, taskSearchScore } from "../services/task-search.js"; +import { issueService } from "../services/issues.js"; +import { searchQualityMetrics, taskSearchCases, taskSearchCorpus } from "./fixtures/task-search-corpus.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const baseline = process.env.SEARCH_EVAL_BASELINE === "1"; + +describe.skipIf(!support.supported)("task search relevance rubric (real PostgreSQL)", () => { + let tempDb: Awaited>; + let db: ReturnType; + const companyId = randomUUID(); + const keys = new Map(); + const plans: Record = {}; + const latency: Array<{ engine: string; q: string; firstMs: number; p95Ms: number; warmMs: number[] }> = []; + let postgresVersion = ""; + const report: Array<{ engine: string; name: string; q: string; keys: string[]; ndcg5: number; reciprocalRank: number; ms: number }> = []; + + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-search-quality-"); + db = createDb(tempDb.connectionString); + postgresVersion = String((await db.execute(sql`SELECT version()`))[0]!.version); + await db.insert(companies).values({ id: companyId, name: "Search benchmark", issuePrefix: "EVAL" }); + for (const [index, entry] of taskSearchCorpus.entries()) { + const id = randomUUID(); + keys.set(id, entry.key); + const updatedAt = new Date(Date.UTC(2026, 0, 1) + index * 60_000); + await db.insert(issues).values({ id, companyId, title: entry.title, + identifier: "identifier" in entry ? entry.identifier : `EVAL-${index + 1}`, + description: "description" in entry ? entry.description : null, + status: "status" in entry ? entry.status : "todo", updatedAt, createdAt: updatedAt }); + if ("comments" in entry) { + for (const body of entry.comments) await db.insert(issueComments).values({ companyId, issueId: id, body, updatedAt }); + } + if ("document" in entry) { + const documentId = randomUUID(); + await db.insert(documents).values({ id: documentId, companyId, title: entry.document.title, latestBody: entry.document.body, format: "markdown" }); + await db.insert(issueDocuments).values({ companyId, issueId: id, documentId, key: "plan" }); + } + } + }); + + afterAll(async () => { + const summary = ["full", "quick"].map((engine) => { + const rows = report.filter((row) => row.engine === engine); + const known = rows.filter((row) => Object.keys(taskSearchCases.find((item) => item.name === row.name)!.relevant).length > 0); + return { engine, queries: rows.length, ndcg5: rows.reduce((sum, row) => sum + row.ndcg5, 0) / rows.length, + mrr: known.reduce((sum, row) => sum + row.reciprocalRank, 0) / known.length }; + }); + console.log("SEARCH QUALITY", JSON.stringify(summary)); + if (process.env.SEARCH_EVAL_REPORT) await writeFile(process.env.SEARCH_EVAL_REPORT, JSON.stringify({ + environment: { postgresVersion, platform: platform(), release: release(), cpu: cpus()[0]?.model, memoryBytes: totalmem(), warmSamples: 20 }, + summary, queries: report, latency, plans, + }, null, 2)); + await tempDb?.cleanup(); + }); + + for (const engine of ["full", "quick"] as const) { + for (const testCase of taskSearchCases) { + it(`${engine}: ${testCase.name}`, async () => { + const start = performance.now(); + const rows = engine === "full" + ? (await companySearchService(db).search(companyId, companySearchQuerySchema.parse({ q: testCase.q }))).results.filter((row) => row.type === "issue") + : await issueService(db).list(companyId, { q: testCase.q, limit: 50 }); + const resultKeys = rows.map((row) => keys.get(row.id)!); + report.push({ engine, name: testCase.name, q: testCase.q, keys: resultKeys, ...searchQualityMetrics(resultKeys, testCase.relevant), ms: performance.now() - start }); + if (baseline) return; + if (testCase.first) expect(resultKeys[0], JSON.stringify(resultKeys)).toBe(testCase.first); + for (const absent of testCase.absent ?? []) expect(resultKeys).not.toContain(absent); + for (const [key, grade] of Object.entries(testCase.relevant)) if (grade === 3) expect(resultKeys.slice(0, 5)).toContain(key); + if (Object.keys(testCase.relevant).length === 0) expect(resultKeys).toEqual([]); + }); + } + } + it("meets the aggregate relevance gates", () => { + if (baseline) return; + for (const engine of ["full", "quick"]) { + const rows = report.filter((row) => row.engine === engine); + const known = rows.filter((row) => taskSearchCases.find((entry) => entry.name === row.name)!.q !== "quasarxylophone"); + expect(known.reduce((sum, row) => sum + row.reciprocalRank, 0) / known.length).toBeGreaterThanOrEqual(0.95); + expect(rows.reduce((sum, row) => sum + row.ndcg5, 0) / rows.length).toBeGreaterThanOrEqual(0.90); + } + }); + + it("handles empty quotes, literal punctuation, and oversized title words without errors", async () => { + if (baseline) return; + await db.insert(issues).values({ companyId, title: "x".repeat(300) }); + for (const q of ['""', "%", "_", "\\", "z".repeat(200)]) { + const full = await companySearchService(db).search(companyId, companySearchQuerySchema.parse({ q })); + const quick = await issueService(db).list(companyId, { q, limit: 50 }); + if (q === '""' || q.startsWith("z")) { + expect(full.results).toEqual([]); + expect(quick).toEqual([]); + } else { + for (const row of quick) expect(row.title).toContain(q); + } + } + }); + + it.runIf(process.env.SEARCH_EVAL_SCALE === "1")("measures 10k tasks / 30k comments", async () => { + await db.execute(sql` + INSERT INTO issues (company_id, title, description, identifier) + SELECT ${companyId}, 'Routine deployment checkpoint ' || n, + repeat('Review the build output and update the deployment checklist. ', 5), 'SCALE-' || n + FROM generate_series(1, 10000) n + `); + await db.execute(sql` + INSERT INTO issue_comments (company_id, issue_id, body) + SELECT ${companyId}, id, repeat('Routine progress report: verified the output and recorded the findings. ', 5) + FROM issues CROSS JOIN generate_series(1, 3) n + WHERE company_id = ${companyId} AND identifier LIKE 'SCALE-%' + `); + await db.execute(sql`ANALYZE issues`); + await db.execute(sql`ANALYZE issue_comments`); + for (const engine of ["full", "quick"] as const) { + for (const q of ["GitHub OAuth", "OAuth callback GitHub", "mibile api", "search", "quasarxylophone", "routine"]) { + const durations: number[] = []; + for (let i = 0; i < 21; i++) { + const start = performance.now(); + if (engine === "full") await companySearchService(db).search(companyId, companySearchQuerySchema.parse({ q })); + else await issueService(db).list(companyId, { q, limit: 20 }); + durations.push(performance.now() - start); + } + const warmMs = durations.slice(1); + latency.push({ engine, q, firstMs: durations[0]!, p95Ms: [...warmMs].sort((a, b) => a - b)[18]!, warmMs }); + } + } + for (const q of ["GitHub OAuth", "mibile api", "routine"]) { + const search = parseTaskSearch(q); + if (!baseline) plans[q] = await db.execute(sql` + EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) + ${taskSearchCtes(companyId, search)} + SELECT m.id, ${taskSearchScore(search)} AS score FROM matched m ORDER BY score DESC LIMIT 20 + `); + } + console.log("SEARCH LATENCY", JSON.stringify(latency.map(({ warmMs: _warmMs, ...row }) => row))); + }, 120_000); + +}); diff --git a/server/src/__tests__/task-search-query.test.ts b/server/src/__tests__/task-search-query.test.ts new file mode 100644 index 0000000000..4bc93dfa97 --- /dev/null +++ b/server/src/__tests__/task-search-query.test.ts @@ -0,0 +1,19 @@ +import { describe, expect, it } from "vitest"; +import { parseTaskSearch } from "../services/task-search.js"; + +describe("task search query intent", () => { + it("keeps negation, short domain terms and quoted filler", () => { + expect(parseTaskSearch('the API is not "in the UI"').tokens).toEqual(["api", "is", "not", "in the ui"]); + expect(parseTaskSearch("the and").tokens).toEqual(["the", "and"]); + }); + + it("retains the strictest intent for repeated terms", () => { + expect(parseTaskSearch('callback "callback"').terms).toEqual([{ text: "callback", quoted: true }]); + }); + + it("normalizes task identifiers without guessing their numbers", () => { + for (const q of ["PAP-42", "pap42", "PAP 42"]) expect(parseTaskSearch(q).identifierQuery).toBe("pap-42"); + expect(parseTaskSearch("T123-42").identifierQuery).toBe("t123-42"); + expect(parseTaskSearch("PAP-420").identifierQuery).toBe("pap-420"); + }); +}); diff --git a/server/src/services/company-search.ts b/server/src/services/company-search.ts index 4de7430cc9..96af85b6aa 100644 --- a/server/src/services/company-search.ts +++ b/server/src/services/company-search.ts @@ -15,7 +15,6 @@ import { import { COMPANY_SEARCH_MAX_LIMIT, COMPANY_SEARCH_MAX_OFFSET, - COMPANY_SEARCH_MAX_TOKENS, COMPANY_SEARCH_UPDATED_WITHIN_OPTIONS, COMPANY_ARTIFACTS_MAX_LIMIT, COMPANY_ARTIFACTS_MAX_QUERY_LENGTH, @@ -39,18 +38,8 @@ import { import { companyArtifactsService } from "./company-artifacts.js"; import { companySearchExtractService } from "./company-search-extract.js"; import { visibleIssueCondition } from "./issue-visibility.js"; +import { parseTaskSearch, taskSearchCtes, taskSearchScore, taskSearchFieldMatch, taskSearchTermMatch } from "./task-search.js"; -const MIN_TOKEN_LENGTH = 2; -const MIN_FUZZY_QUERY_LENGTH = 4; -const MIN_FUZZY_TOKEN_LENGTH = 4; -// Cap fuzzy edits using the shorter of (query token, title word) so common -// 4–5 letter English words don't sweep in noise (e.g. "serach" vs "each"). -const FUZZY_PAIR_LONG_LENGTH = 6; -const FUZZY_PAIR_LONG_MAX_EDITS = 2; -const FUZZY_PAIR_MEDIUM_LENGTH = 5; -const FUZZY_PAIR_MEDIUM_MAX_EDITS = 1; -const FUZZY_PAIR_SHORT_MAX_EDITS = 0; -const FUZZY_IDENTIFIER_SIMILARITY_THRESHOLD = 0.45; const SNIPPET_MAX_CHARS = 240; export const COMPANY_SEARCH_BRANCH_FETCH_LIMIT = COMPANY_SEARCH_MAX_OFFSET + COMPANY_SEARCH_MAX_LIMIT + 1; @@ -95,30 +84,6 @@ type SearchAggregateRow = { count: number | string; }; -function normalizeQuery(query: string) { - return query.trim().replace(/\s+/g, " ").toLowerCase(); -} - -function escapeLikePattern(value: string): string { - return value.replace(/[\\%_]/g, "\\$&"); -} - -function tokenizeQuery(normalizedQuery: string) { - const matches = normalizedQuery.match(/"[^"]+"|[^\s]+/g) ?? []; - const tokens: string[] = []; - for (const match of matches) { - const token = match.replace(/^"|"$/g, "").replace(/^[^\p{L}\p{N}%_\\-]+|[^\p{L}\p{N}%_\\-]+$/gu, ""); - if (token.length < MIN_TOKEN_LENGTH) continue; - if (!tokens.includes(token)) tokens.push(token); - if (tokens.length >= COMPANY_SEARCH_MAX_TOKENS) break; - } - return tokens; -} - -function fuzzyEligibleTokens(tokens: string[]): string[] { - return tokens.filter((token) => token.length >= MIN_FUZZY_TOKEN_LENGTH); -} - function sqlTextArray(values: string[]) { if (values.length === 0) return sql`ARRAY[]::text[]`; return sql`ARRAY[${sql.join(values.map((value) => sql`${value}`), sql`, `)}]::text[]`; @@ -138,7 +103,7 @@ function plainText(value: string | null | undefined) { .replace(/```[\s\S]*?```/g, " ") .replace(/`([^`]+)`/g, "$1") .replace(/\[([^\]]+)\]\([^)]+\)/g, "$1") - .replace(/[#>*_~|]+/g, " ") + .replace(/(^|\s)[#>*_~|]+|[#>*_~|]+(?=\s|$)/g, " ") .replace(/\s+/g, " ") .trim(); } @@ -431,6 +396,8 @@ function scopeIncludesProjects(scope: CompanySearchScope) { function selectPrimarySnippets(row: IssueSearchRow, normalizedQuery: string, tokens: string[]) { const terms = matchTerms(normalizedQuery, tokens); + const identifierQuery = parseTaskSearch(normalizedQuery).identifierQuery; + if (!terms.includes(identifierQuery)) terms.push(identifierQuery); const matchedFields = new Set(row.matchedFields ?? []); const candidates: Array = []; if (matchedFields.has("identifier")) { @@ -439,15 +406,22 @@ function selectPrimarySnippets(row: IssueSearchRow, normalizedQuery: string, tok if (matchedFields.has("title")) { candidates.push(createSnippet("title", "Title", row.title, terms)); } - if (matchedFields.has("comment")) { - candidates.push(createSnippet("comment", "Comment", row.commentSnippet, terms)); - } - if (matchedFields.has("document")) { - candidates.push(createSnippet("document", row.documentTitle || "Document", row.documentSnippet, terms)); - } - if (matchedFields.has("description")) { - candidates.push(createSnippet("description", "Description", row.description, terms)); - } + const description = matchedFields.has("description") + ? createSnippet("description", "Description", row.description, terms) : null; + const directMatch = Number(row.score) >= 3000 || Number(row.score) < 2000; + if (directMatch) candidates.push(description); + const context = [ + { field: "comment", label: "Comment", text: row.commentSnippet }, + { field: "document", label: row.documentTitle || "Document", text: [row.documentTitle, row.documentSnippet].filter(Boolean).join(" ") }, + ].filter((source) => matchedFields.has(source.field)); + const coverage = (text: string | null) => tokens.filter((term) => (text ?? "").toLowerCase().includes(term)).length; + context.sort((left, right) => coverage(right.text) - coverage(left.text)); + const contextSnippets = context.map((source) => createSnippet(source.field, source.label, source.text, terms)); + // The title and identifier are already visible in the row. For thread-only + // coverage, preserve the evidence before applying the two-snippet limit. + if (directMatch) candidates.push(...contextSnippets); + else candidates.unshift(...contextSnippets); + if (!directMatch) candidates.push(description); return candidates.filter((snippet): snippet is CompanySearchSnippet => Boolean(snippet)).slice(0, 2); } @@ -456,7 +430,11 @@ function issueResult(row: IssueSearchRow, prefix: string, normalizedQuery: strin const sourceLabel = snippets[0]?.label ?? null; const documentSuffix = row.documentKey ? `#document-${encodeURIComponent(row.documentKey)}` : ""; const commentSuffix = row.commentId ? `#comment-${encodeURIComponent(row.commentId)}` : ""; - const suffix = row.commentId ? commentSuffix : documentSuffix; + // Direct task matches open the task; context matches open the evidence shown. + const directMatch = Number(row.score) >= 3000 || Number(row.score) < 2000; + const evidence = snippets.find((snippet) => snippet.field === "comment" || snippet.field === "document"); + const suffix = directMatch ? "" : evidence?.field === "comment" ? commentSuffix + : evidence?.field === "document" ? documentSuffix : ""; const issue: CompanySearchIssueSummary = { id: row.id, identifier: row.identifier, @@ -495,7 +473,9 @@ function scoreSimpleRow(row: SimpleSearchRow, normalizedQuery: string, tokens: s if (haystack.includes(token)) score += 20; } if (row.title.toLowerCase().startsWith(normalizedQuery)) score += 80; - return score; + // Keep other entity types on the same scale as the task relevance bands: + // an exact agent/project name must still outrank a speculative task typo. + return score * 10; } function artifactResult(artifact: CompanyArtifact, normalizedQuery: string, tokens: string[]): CompanySearchResult { @@ -559,9 +539,10 @@ export function companySearchService(db: Db) { return { extract: extractService.extract, search: async (companyId: string, query: CompanySearchQuery): Promise => { - const normalizedQuery = normalizeQuery(query.q); + const taskSearch = parseTaskSearch(query.q); + const normalizedQuery = taskSearch.normalizedQuery; const hasSearchText = normalizedQuery.length > 0; - const tokens = tokenizeQuery(normalizedQuery); + const tokens = taskSearch.tokens; const scope = query.scope; const sort = query.sort; const limit = query.limit; @@ -583,147 +564,20 @@ export function companySearchService(db: Db) { } const fetchLimit = companySearchBranchFetchLimit(limit, offset); - const escapedTokens = tokens.map(escapeLikePattern); - // LIKE/ILIKE both treat backslash as the default escape character, so the - // escaped tokens stay literal inside ILIKE ANY(...) patterns too. - const tokenPatterns = escapedTokens.map((token) => `%${token}%`); - const tokenPatternArray = sqlTextArray(tokenPatterns); - const fuzzyTokens = fuzzyEligibleTokens(tokens); - const fuzzyTokenArray = sqlTextArray(fuzzyTokens); - const escapedQuery = escapeLikePattern(normalizedQuery); - const containsPattern = hasSearchText ? `%${escapedQuery}%` : "__paperclip_no_match__"; - const startsWithPattern = hasSearchText ? `${escapedQuery}%` : "__paperclip_no_match__"; - const fuzzyEnabled = hasSearchText && normalizedQuery.length >= MIN_FUZZY_QUERY_LENGTH && !/[\\%_]/.test(normalizedQuery); - const fuzzyTokensEnabled = fuzzyEnabled && fuzzyTokens.length > 0; + const tokenPatternArray = sqlTextArray(taskSearch.patterns); + const containsPattern = hasSearchText && tokens.length > 0 ? taskSearch.containsPattern : "__paperclip_no_match__"; const tokenCount = tokens.length; - // --- shared match expressions against the `issues` table ------------- - // Raw-column ILIKE keeps the predicates compatible with the existing - // pg_trgm GIN indexes (lower(col) LIKE expressions cannot use them). - const titlePhraseMatch = hasSearchText ? sql`issues.title ILIKE ${containsPattern}` : noMatchSql(); - const titleStartsWith = hasSearchText ? sql`issues.title ILIKE ${startsWithPattern}` : noMatchSql(); - const titleExactMatch = hasSearchText ? sql`lower(issues.title) = ${normalizedQuery}` : noMatchSql(); - const identifierPhraseMatch = hasSearchText ? sql`coalesce(issues.identifier, '') ILIKE ${containsPattern}` : noMatchSql(); - const identifierStartsWith = hasSearchText ? sql`coalesce(issues.identifier, '') ILIKE ${startsWithPattern}` : noMatchSql(); - const identifierExactMatch = hasSearchText ? sql`lower(coalesce(issues.identifier, '')) = ${normalizedQuery}` : noMatchSql(); - const descriptionPhraseMatch = hasSearchText ? sql`coalesce(issues.description, '') ILIKE ${containsPattern}` : noMatchSql(); - const titleTokenMatch = tokenCount > 0 ? sql`issues.title ILIKE ANY(${tokenPatternArray})` : noMatchSql(); - const identifierTokenMatch = tokenCount > 0 ? sql`coalesce(issues.identifier, '') ILIKE ANY(${tokenPatternArray})` : noMatchSql(); - const descriptionTokenMatch = tokenCount > 0 ? sql`coalesce(issues.description, '') ILIKE ANY(${tokenPatternArray})` : noMatchSql(); - // Comment/document matches are computed once per request into tagged - // CTEs (issue_id, ord) where ord 1 is the phrase pattern and ord k+1 is - // token k. Flags and per-token coverage become cheap hashed IN probes - // against those sets instead of per-issue-row correlated subqueries. - // Single-pattern queries stay a bare `col ILIKE pattern` so the pg_trgm - // GIN indexes can bitmap-scan them; multi-pattern queries use one tagged - // pass over the table (an OR/ANY form would seq-scan anyway). - const matchPatterns = hasSearchText - ? [containsPattern, ...tokenPatterns.filter((pattern) => pattern !== containsPattern)] - : []; - const matchPatternOrdinal = (pattern: string) => matchPatterns.indexOf(pattern) + 1; - const matchPatternArray = sqlTextArray(matchPatterns); - const commentMatchesCte = !hasSearchText - ? sql`SELECT NULL::uuid AS issue_id, 0 AS ord WHERE false` - : matchPatterns.length === 1 - ? sql` - SELECT search_comments.issue_id, 1 AS ord - FROM issue_comments search_comments - WHERE search_comments.company_id = ${companyId} - AND search_comments.deleted_at IS NULL - AND search_comments.body ILIKE ${matchPatterns[0]!} - GROUP BY 1, 2 - ` - : sql` - SELECT search_comments.issue_id, pat.ord::int AS ord - FROM issue_comments search_comments - INNER JOIN unnest(${matchPatternArray}) WITH ORDINALITY AS pat(pattern, ord) - ON search_comments.body ILIKE pat.pattern - WHERE search_comments.company_id = ${companyId} - AND search_comments.deleted_at IS NULL - GROUP BY 1, 2 - `; - // Documents get one UNION ALL arm per pattern (each arm a bare - // `col ILIKE pattern`) so the planner can pick a pg_trgm bitmap scan per - // pattern; latest_body is large enough that skipping the seq scan for - // selective patterns dwarfs the duplicate-recheck cost on common ones. - const documentMatchesCte = !hasSearchText - ? sql`SELECT NULL::uuid AS issue_id, 0 AS ord WHERE false` - : sql.join(matchPatterns.map((pattern, index) => sql` - SELECT search_issue_documents.issue_id, ${index + 1}::int AS ord - FROM issue_documents search_issue_documents - INNER JOIN documents search_documents - ON search_documents.id = search_issue_documents.document_id - AND search_documents.company_id = search_issue_documents.company_id - WHERE search_issue_documents.company_id = ${companyId} - AND ( - search_documents.title ILIKE ${pattern} - OR search_documents.latest_body ILIKE ${pattern} - ) - GROUP BY 1, 2 - `), sql` UNION ALL `); - const commentMatch = hasSearchText - ? sql`issues.id IN (SELECT comment_matches.issue_id FROM comment_matches)` - : noMatchSql(); - const documentMatch = hasSearchText - ? sql`issues.id IN (SELECT document_matches.issue_id FROM document_matches)` - : noMatchSql(); - // Each query token (length >= MIN_FUZZY_TOKEN_LENGTH) must have at least - // one title word within Levenshtein edit distance. This handles typos - // like "serach" -> "search" (transposition) and "mibile" -> "mobile" - // (substitution) without the trigram noise that drop-character variants - // produced (e.g. "serac" matching "service"). Edit budget is gated on - // the SHORTER of the two strings so 4–5 letter English words don't get - // swept in by lev=2 collisions. - const fuzzyMaxEditsExpr = sql.raw( - `CASE - WHEN least(length(qt.value), length(title_word.value)) >= ${FUZZY_PAIR_LONG_LENGTH} THEN ${FUZZY_PAIR_LONG_MAX_EDITS} - WHEN least(length(qt.value), length(title_word.value)) >= ${FUZZY_PAIR_MEDIUM_LENGTH} THEN ${FUZZY_PAIR_MEDIUM_MAX_EDITS} - ELSE ${FUZZY_PAIR_SHORT_MAX_EDITS} - END`, - ); - const fuzzyMinTitleWordLengthExpr = sql.raw(`${MIN_FUZZY_TOKEN_LENGTH}`); - const fuzzyTokenTitleMatch = fuzzyTokensEnabled - ? sql` - coalesce(( - SELECT bool_and( - EXISTS ( - SELECT 1 - FROM regexp_split_to_table(lower(issues.title), '[^a-z0-9]+') AS title_word(value) - WHERE length(title_word.value) >= ${fuzzyMinTitleWordLengthExpr} - AND levenshtein_less_equal(qt.value, title_word.value, ${fuzzyMaxEditsExpr}) <= ${fuzzyMaxEditsExpr} - ) - ) - FROM unnest(${fuzzyTokenArray}) AS qt(value) - ), false) - ` - : noMatchSql(); - const fuzzyIdentifierMatch = fuzzyEnabled - ? sql`similarity(lower(coalesce(issues.identifier, '')), ${normalizedQuery}) >= ${FUZZY_IDENTIFIER_SIMILARITY_THRESHOLD}` - : noMatchSql(); - - const issueTextMatch = sql`( - ${titlePhraseMatch} - OR ${identifierPhraseMatch} - OR ${descriptionPhraseMatch} - OR ${titleTokenMatch} - OR ${identifierTokenMatch} - OR ${descriptionTokenMatch} - )`; - const fuzzyMatch = sql`(${fuzzyTokenTitleMatch} OR ${fuzzyIdentifierMatch})`; - const anySearchMatch = sql`(${issueTextMatch} OR ${commentMatch} OR ${documentMatch} OR ${fuzzyMatch})`; - const issueFilters = issueFilterConditions(companyId, query); const hasIssueOnlyFilters = issueOnlyFiltersActive(query); // Scope conditions over precomputed flag columns (alias-qualified). function flagTextMatch(alias: string) { - return sql`( - ${sql.raw(alias)}.title_phrase OR ${sql.raw(alias)}.ident_phrase OR ${sql.raw(alias)}.desc_phrase - OR ${sql.raw(alias)}.title_token OR ${sql.raw(alias)}.ident_token OR ${sql.raw(alias)}.desc_token - )`; + return sql`(${sql.raw(alias)}.issue_coverage = ${tokenCount} + OR ${sql.raw(alias)}.ident_exact OR ${sql.raw(alias)}.ident_starts)`; } function flagFuzzyMatch(alias: string) { - return sql`(${sql.raw(alias)}.fuzzy_title OR ${sql.raw(alias)}.fuzzy_ident)`; + return sql`${sql.raw(alias)}.fuzzy_title`; } function flagScopeCondition(alias: string, forScope: CompanySearchScope): SQL { if (!hasSearchText) { @@ -732,7 +586,7 @@ export function companySearchService(db: Db) { if (forScope === "comments") return sql`${sql.raw(alias)}.comment_match`; if (forScope === "documents") return sql`${sql.raw(alias)}.document_match`; if (forScope === "issues") return sql`(${flagTextMatch(alias)} OR ${flagFuzzyMatch(alias)})`; - return sql`(${flagTextMatch(alias)} OR ${sql.raw(alias)}.comment_match OR ${sql.raw(alias)}.document_match OR ${flagFuzzyMatch(alias)})`; + return sql`true`; } // --- combined issue results + aggregates statement --------------------- @@ -764,24 +618,7 @@ export function companySearchService(db: Db) { const wantResultRows = scopeIncludesIssues(scope) && !(!hasSearchText && (scope === "comments" || scope === "documents")); if (wantResultRows) { - const allTokensBonus = tokenCount > 0 - ? sql`CASE WHEN m.token_coverage = ${tokenCount} THEN 260 ELSE 0 END` - : sql`0`; - const scoreSql = sql`( - CASE WHEN m.ident_exact THEN 1200 ELSE 0 END - + CASE WHEN m.ident_starts THEN 700 ELSE 0 END - + CASE WHEN m.title_exact THEN 900 ELSE 0 END - + CASE WHEN m.title_starts THEN 550 ELSE 0 END - + CASE WHEN m.title_phrase THEN 350 ELSE 0 END - + CASE WHEN m.ident_phrase THEN 320 ELSE 0 END - + CASE WHEN m.comment_match THEN 180 ELSE 0 END - + CASE WHEN m.document_match THEN 170 ELSE 0 END - + CASE WHEN m.desc_phrase THEN 120 ELSE 0 END - + ${allTokensBonus} - + (m.token_coverage * 70) - + CASE WHEN (m.fuzzy_title OR m.fuzzy_ident) THEN 110 ELSE 0 END - + CASE m.status WHEN 'done' THEN 0 WHEN 'cancelled' THEN -30 ELSE 20 END - )::double precision`; + const scoreSql = taskSearchScore(taskSearch); const priorityOrderSql = sql`CASE m.priority WHEN 'critical' THEN 0 WHEN 'high' THEN 1 WHEN 'medium' THEN 2 WHEN 'low' THEN 3 ELSE 4 END`; const orderBySql = sort === "updated" ? sql`m.updated_at DESC, score DESC, m.id DESC` @@ -798,7 +635,8 @@ export function companySearchService(db: Db) { m.id, m.identifier, m.title, - m.description, + (SELECT issue_text.description FROM issues issue_text + WHERE issue_text.id = m.id AND issue_text.company_id = ${companyId}) AS description, m.status, m.priority, m.assignee_agent_id AS "assigneeAgentId", @@ -808,9 +646,9 @@ export function companySearchService(db: Db) { m.updated_at AS "updatedAt", ${scoreSql} AS score, array_remove(ARRAY[ - CASE WHEN m.ident_phrase OR m.ident_token OR m.fuzzy_ident THEN 'identifier' END, - CASE WHEN m.title_phrase OR m.title_token OR m.fuzzy_title THEN 'title' END, - CASE WHEN m.desc_phrase OR m.desc_token THEN 'description' END, + CASE WHEN m.ident_exact OR m.ident_starts OR m.ident_phrase OR m.ident_token THEN 'identifier' END, + CASE WHEN m.title_token OR m.fuzzy_title THEN 'title' END, + CASE WHEN m.desc_token THEN 'description' END, CASE WHEN m.comment_match THEN 'comment' END, CASE WHEN m.document_match THEN 'document' END ], NULL)::text[] AS "matchedFields" @@ -825,10 +663,10 @@ export function companySearchService(db: Db) { branches.push(sql`SELECT 'type:issue' AS kind, NULL::text AS value, count(*)::int AS count ${countTail} FROM matched m ${branchWhere([...facetsAll, titleCond])}`); } if (hasSearchText && (scope === "all" || scope === "comments")) { - branches.push(sql`SELECT 'type:comment' AS kind, NULL::text AS value, count(*)::int AS count ${countTail} FROM matched m ${branchWhere([...facetsAll, sql`m.comment_match`])}`); + branches.push(sql`SELECT 'type:comment' AS kind, NULL::text AS value, count(*)::int AS count ${countTail} FROM matched m ${branchWhere([...facetsAll, flagScopeCondition("m", "comments")])}`); } if (hasSearchText && (scope === "all" || scope === "documents")) { - branches.push(sql`SELECT 'type:document' AS kind, NULL::text AS value, count(*)::int AS count ${countTail} FROM matched m ${branchWhere([...facetsAll, sql`m.document_match`])}`); + branches.push(sql`SELECT 'type:document' AS kind, NULL::text AS value, count(*)::int AS count ${countTail} FROM matched m ${branchWhere([...facetsAll, flagScopeCondition("m", "documents")])}`); } const facetBranch = (kind: string, valueSql: SQL, omit: CompanySearchIssueFilterKey, extra: SQL[] = []) => sql` @@ -878,59 +716,8 @@ export function companySearchService(db: Db) { } } - // Per-token coverage counts matches across issue text and the tagged - // comment/document match sets (hashed IN probes, one set per token). - const coverageSql = tokenCount > 0 - ? sql`(${sql.join(tokens.map((_, index) => { - const pattern = tokenPatterns[index]!; - const ord = matchPatternOrdinal(pattern); - return sql`(CASE WHEN - issues.title ILIKE ${pattern} - OR coalesce(issues.identifier, '') ILIKE ${pattern} - OR coalesce(issues.description, '') ILIKE ${pattern} - OR issues.id IN (SELECT comment_matches.issue_id FROM comment_matches WHERE comment_matches.ord = ${ord}) - OR issues.id IN (SELECT document_matches.issue_id FROM document_matches WHERE document_matches.ord = ${ord}) - THEN 1 ELSE 0 END)`; - }), sql` + `)})` - : sql`0`; - - const matchedWhere = hasSearchText ? sql` AND ${anySearchMatch}` : sql``; const resultRows = await db.execute(sql` - WITH comment_matches AS MATERIALIZED (${commentMatchesCte}), - document_matches AS MATERIALIZED (${documentMatchesCte}), - matched AS MATERIALIZED ( - SELECT - issues.id, - issues.identifier, - issues.title, - issues.description, - issues.status, - issues.priority, - issues.assignee_agent_id, - issues.assignee_user_id, - issues.project_id, - issues.created_at, - issues.updated_at, - ${titlePhraseMatch} AS title_phrase, - ${titleStartsWith} AS title_starts, - ${titleExactMatch} AS title_exact, - ${identifierPhraseMatch} AS ident_phrase, - ${identifierStartsWith} AS ident_starts, - ${identifierExactMatch} AS ident_exact, - ${descriptionPhraseMatch} AS desc_phrase, - ${titleTokenMatch} AS title_token, - ${identifierTokenMatch} AS ident_token, - ${descriptionTokenMatch} AS desc_token, - ${commentMatch} AS comment_match, - ${documentMatch} AS document_match, - ${fuzzyTokenTitleMatch} AS fuzzy_title, - ${fuzzyIdentifierMatch} AS fuzzy_ident, - ${coverageSql} AS token_coverage - FROM issues - WHERE issues.company_id = ${companyId} - AND ${visibleIssueCondition()} - ${matchedWhere} - ) + ${taskSearchCtes(companyId, taskSearch, scope !== "issues", and(...issueFilters))} ${sql.join(branches, sql` UNION ALL `)} `) as unknown as Array>; @@ -1014,11 +801,11 @@ export function companySearchService(db: Db) { AND search_comments.issue_id = target.id AND search_comments.deleted_at IS NULL AND ( - search_comments.body ILIKE ${containsPattern} - OR search_comments.body ILIKE ANY(${tokenPatternArray}) + ${taskSearchFieldMatch(sql`search_comments.body`, taskSearch)} ) ORDER BY CASE WHEN search_comments.body ILIKE ${containsPattern} THEN 0 ELSE 1 END, + ${sql.join(tokens.map((_, index) => sql`CASE WHEN ${taskSearchTermMatch(sql`search_comments.body`, taskSearch, index)} THEN 1 ELSE 0 END`), sql` + `)} DESC, search_comments.updated_at DESC, search_comments.id DESC LIMIT 1 @@ -1032,10 +819,8 @@ export function companySearchService(db: Db) { WHERE search_issue_documents.company_id = ${companyId} AND search_issue_documents.issue_id = target.id AND ( - coalesce(search_documents.title, '') ILIKE ${containsPattern} - OR search_documents.latest_body ILIKE ${containsPattern} - OR coalesce(search_documents.title, '') ILIKE ANY(${tokenPatternArray}) - OR search_documents.latest_body ILIKE ANY(${tokenPatternArray}) + ${taskSearchFieldMatch(sql`search_documents.title`, taskSearch)} + OR ${taskSearchFieldMatch(sql`search_documents.latest_body`, taskSearch)} ) ORDER BY CASE @@ -1043,6 +828,7 @@ export function companySearchService(db: Db) { WHEN search_documents.latest_body ILIKE ${containsPattern} THEN 1 ELSE 2 END, + ${sql.join(tokens.map((_, index) => sql`CASE WHEN ${taskSearchTermMatch(sql`search_documents.title`, taskSearch, index)} OR ${taskSearchTermMatch(sql`search_documents.latest_body`, taskSearch, index)} THEN 1 ELSE 0 END`), sql` + `)} DESC, search_documents.updated_at DESC, search_documents.id DESC LIMIT 1 diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index d15fb00a59..c74483e250 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,4 +1,5 @@ import { documentService } from "./documents.js"; +import { parseTaskSearch, taskSearchCtes, taskSearchScore } from "./task-search.js"; import { createdFromIssueCondition } from "./issue-creation-origin.js"; import { executionProjectionsForRuns } from "./execution-projection.js"; import type { ExecutionProjection } from "@paperclipai/shared"; @@ -7810,24 +7811,7 @@ export function issueService(db: Db) { filters?.includeLiveDescendantSummary === true; const rawSearch = filters?.q?.trim() ?? ""; const hasSearch = rawSearch.length > 0; - const escapedSearch = hasSearch ? escapeLikePattern(rawSearch) : ""; - const startsWithPattern = `${escapedSearch}%`; - const containsPattern = `%${escapedSearch}%`; - const titleStartsWithMatch = sql`${issues.title} ILIKE ${startsWithPattern} ESCAPE '\\'`; - const titleContainsMatch = sql`${issues.title} ILIKE ${containsPattern} ESCAPE '\\'`; - const identifierStartsWithMatch = sql`${issues.identifier} ILIKE ${startsWithPattern} ESCAPE '\\'`; - const identifierContainsMatch = sql`${issues.identifier} ILIKE ${containsPattern} ESCAPE '\\'`; - const descriptionContainsMatch = sql`${issues.description} ILIKE ${containsPattern} ESCAPE '\\'`; - const commentContainsMatch = sql` - EXISTS ( - SELECT 1 - FROM ${issueComments} - WHERE ${issueComments.issueId} = ${issues.id} - AND ${issueComments.companyId} = ${companyId} - AND ${issueComments.deletedAt} IS NULL - AND ${issueComments.body} ILIKE ${containsPattern} ESCAPE '\\' - ) - `; + const taskSearch = parseTaskSearch(rawSearch); if (filters?.createdFromIssueId) { conditions.push(createdFromIssueCondition(companyId, filters.createdFromIssueId)); } @@ -7935,16 +7919,6 @@ export function issueService(db: Db) { ), ); } - if (hasSearch) { - conditions.push( - or( - titleContainsMatch, - identifierContainsMatch, - descriptionContainsMatch, - commentContainsMatch, - )!, - ); - } if (filters?.updatedSince) { const since = new Date(filters.updatedSince); if (Number.isFinite(since.getTime())) { @@ -7959,20 +7933,15 @@ export function issueService(db: Db) { conditions.push(ne(issues.originKind, "routine_execution")); } const priorityOrder = sql`CASE ${issues.priority} WHEN 'critical' THEN 0 WHEN 'high' THEN 1 WHEN 'medium' THEN 2 WHEN 'low' THEN 3 ELSE 4 END`; - const searchOrder = sql` - CASE - WHEN ${titleStartsWithMatch} THEN 0 - WHEN ${titleContainsMatch} THEN 1 - WHEN ${identifierStartsWithMatch} THEN 2 - WHEN ${identifierContainsMatch} THEN 3 - WHEN ${commentContainsMatch} THEN 4 - WHEN ${descriptionContainsMatch} THEN 5 - ELSE 6 - END - `; - const baseQuery = db - .select(issueListSelect) - .from(issues) + const searchOrder = sql`-task_search.score`; + const issueSource = db.select(issueListSelect).from(issues); + const searchedSource = hasSearch + ? issueSource.innerJoin(sql`( + ${taskSearchCtes(companyId, taskSearch, true, and(...conditions))} + SELECT m.id, ${taskSearchScore(taskSearch)} AS score FROM matched m + ) task_search`, sql`task_search.id = ${issues.id}`) + : issueSource; + const baseQuery = searchedSource .where(and(...conditions)) .orderBy( ...issueListOrderBy(companyId, { diff --git a/server/src/services/task-search.ts b/server/src/services/task-search.ts new file mode 100644 index 0000000000..ac5044719a --- /dev/null +++ b/server/src/services/task-search.ts @@ -0,0 +1,196 @@ +import { sql, type SQL } from "drizzle-orm"; +import { COMPANY_SEARCH_MAX_QUERY_LENGTH, COMPANY_SEARCH_MAX_TOKENS } from "@paperclipai/shared"; +import { visibleIssueCondition } from "./issue-visibility.js"; + +// Only grammatical filler is ignored, only in multi-term queries, and never +// inside quotes. Keep negation and domain words (API, UI, PR, etc.) meaningful. +const FILLER = new Set(["a", "an", "the", "and", "of", "to", "for", "in", "on", "with"]); +export function escapeTaskSearchPattern(value: string) { + return value.replace(/[\\%_]/g, "\\$&"); +} + +export function parseTaskSearch(text: string) { + const normalizedQuery = text.slice(0, COMPANY_SEARCH_MAX_QUERY_LENGTH).trim().replace(/\s+/g, " ").toLowerCase(); + const parsed = Array.from(normalizedQuery.matchAll(/"([^"]+)"|([^\s"]+)/g), (match) => ({ + text: match[1] ?? match[2]!, quoted: match[1] !== undefined, + })); + const meaningful = parsed.filter((term) => term.quoted || !FILLER.has(term.text)); + const uniqueTerms = new Map(); + for (const term of meaningful.length > 0 ? meaningful : parsed) { + uniqueTerms.set(term.text, { ...term, quoted: term.quoted || uniqueTerms.get(term.text)?.quoted === true }); + } + const terms = [...uniqueTerms.values()].slice(0, COMPANY_SEARCH_MAX_TOKENS); + const tokens = terms.map((term) => term.text); + const phrase = tokens.join(" "); + // A copied/typed task identifier is navigation, never a fuzzy number match. + const identifier = /^([a-z][a-z0-9]*)[- ](\d+)$/i.exec(normalizedQuery) ?? /^([a-z]+)(\d+)$/i.exec(normalizedQuery); + const identifierQuery = identifier ? `${identifier[1]}-${identifier[2]}` : normalizedQuery; + const patterns = tokens.map((token) => `%${escapeTaskSearchPattern(token)}%`); + const containsPattern = `%${escapeTaskSearchPattern(phrase)}%`; + const startsWithPattern = `${escapeTaskSearchPattern(phrase)}%`; + return { normalizedQuery, terms, tokens, phrase, identifierQuery, patterns, containsPattern, startsWithPattern }; +} +export type TaskSearch = ReturnType; + +function taskSearchAny(field: SQL, search: TaskSearch): SQL { + return search.patterns.length === 0 ? sql`false` + : sql`(${sql.join(search.patterns.map((pattern) => sql`${field} ILIKE ${pattern}`), sql` OR `)})`; +} +// Short typeahead terms must start a word: UI must not match "build", and +// API must not match "Capistrano". Keep an indexable literal precondition. +export function taskSearchTermMatch(field: SQL, search: TaskSearch, index: number): SQL { + const term = search.tokens[index]!; + const literal = sql`${field} ILIKE ${search.patterns[index]!}`; + return /^[\p{L}]{1,3}$/u.test(term) + ? sql`(${literal} AND ${field} ~* ${`(^|[^[:alnum:]])${term}`})` + : literal; +} +export function taskSearchFieldMatch(field: SQL, search: TaskSearch): SQL { + return search.tokens.length === 0 ? sql`false` + : sql`(${sql.join(search.tokens.map((_, index) => taskSearchTermMatch(field, search, index)), sql` OR `)})`; +} +function coverage(matches: SQL[]): SQL { + return matches.length === 0 ? sql`0` + : sql`(${sql.join(matches.map((match) => sql`CASE WHEN ${match} THEN 1 ELSE 0 END`), sql` + `)})`; +} + +// Score bands are deliberately disjoint. Incidental comments, repeated terms, +// status and recency cannot outweigh a stronger kind of match. +export function taskSearchScore(search: TaskSearch): SQL { + const n = search.tokens.length; + if (n === 0) return sql`0`; + return sql`( + CASE + WHEN m.ident_exact THEN 8000 + WHEN m.ident_starts THEN 7000 + WHEN m.title_exact THEN 6000 + WHEN m.title_phrase AND m.title_coverage = ${n} THEN 5000 + WHEN m.title_coverage = ${n} THEN 4000 + WHEN m.issue_coverage = ${n} THEN 3000 + WHEN m.token_coverage = ${n} THEN 2000 + WHEN m.fuzzy_title THEN 1000 + ELSE 0 + END + + m.title_word_coverage * 10 + + CASE WHEN m.title_starts THEN 30 ELSE 0 END + + CASE m.status WHEN 'done' THEN 0 WHEN 'cancelled' THEN 0 ELSE 10 END + )::double precision`; +} + +/** Shared task retrieval for company search and issue-list/command-palette search. + * Uses existing pg_trgm indexes and current rows: no derived corpus or worker. + * The tagged comment/document sets are evaluated once, not once per task. + */ +export function taskSearchCtes(companyId: string, search: TaskSearch, includeContext = true, fallbackFilters?: SQL): SQL { + const n = search.tokens.length; + const comments = n === 0 || !includeContext ? sql`SELECT NULL::uuid AS issue_id, 0 AS ord WHERE false` + : sql.join(search.patterns.map((_, index) => sql` + SELECT c.issue_id, ${index}::int AS ord FROM issue_comments c + WHERE c.company_id = ${companyId} AND c.deleted_at IS NULL AND ${taskSearchTermMatch(sql`c.body`, search, index)} + GROUP BY c.issue_id + `), sql` UNION ALL `); + const documents = n === 0 || !includeContext ? sql`SELECT NULL::uuid AS issue_id, 0 AS ord WHERE false` + : sql.join(search.patterns.map((_, index) => sql` + SELECT d.issue_id, ${index}::int AS ord FROM issue_documents d + JOIN documents body ON body.id = d.document_id AND body.company_id = d.company_id + WHERE d.company_id = ${companyId} AND (${taskSearchTermMatch(sql`body.title`, search, index)} OR ${taskSearchTermMatch(sql`body.latest_body`, search, index)}) + GROUP BY d.issue_id + `), sql` UNION ALL `); + const titleTerms = search.patterns.map((_, index) => taskSearchTermMatch(sql`issues.title`, search, index)); + const issueTerms = search.patterns.map((_, index) => sql`( + ${titleTerms[index]!} OR ${taskSearchTermMatch(sql`issues.identifier`, search, index)} + OR ${taskSearchTermMatch(sql`issues.description`, search, index)} + )`); + const commentTerms = search.patterns.map((_, index) => sql`issues.id IN (SELECT issue_id FROM comment_matches WHERE ord = ${index})`); + const documentTerms = search.patterns.map((_, index) => sql`issues.id IN (SELECT issue_id FROM document_matches WHERE ord = ${index})`); + const allTerms = issueTerms.map((term, index) => sql`(${term} OR ${commentTerms[index]!} OR ${documentTerms[index]!})`); + const phraseMatch = (field: SQL) => n > 0 ? sql`coalesce(${field} ILIKE ${search.containsPattern}, false)` : sql`false`; + const identExact = n > 0 ? sql`lower(issues.identifier) = ${search.identifierQuery}` : sql`false`; + const identStarts = n > 0 ? sql`issues.identifier ILIKE ${escapeTaskSearchPattern(search.identifierQuery) + "%"}` : sql`false`; + const fuzzyAllowed = !search.terms.some((term) => term.quoted) + && search.terms.some((term) => /^[\p{L}]{4,255}$/u.test(term.text)) + && !/^[a-z][a-z0-9]*[- ]?\d+$/i.test(search.normalizedQuery); + const fuzzyTerms = search.terms.map((term, index) => { + if (!/^[\p{L}]{4,255}$/u.test(term.text)) return titleTerms[index]!; + // Bound both arguments before calling fuzzystrmatch (255-character limit). + // Cheap length checks prune word pairs before bounded edit-distance work. + const edits = sql`CASE WHEN least(char_length(word), ${Array.from(term.text).length}) >= 6 THEN 2 + WHEN least(char_length(word), ${Array.from(term.text).length}) >= 5 THEN 1 ELSE 0 END`; + return sql`(${titleTerms[index]!} OR EXISTS ( + SELECT 1 FROM regexp_split_to_table(lower(issues.title), '[^[:alnum:]]+') AS word + WHERE CASE WHEN char_length(word) BETWEEN 4 AND 255 + AND abs(char_length(word) - ${Array.from(term.text).length}) <= ${edits} + THEN levenshtein_less_equal(${term.text}, word, ${edits}) <= ${edits} + ELSE false END + ))`; + }); + const fuzzy = fuzzyAllowed ? sql`CASE WHEN ${coverage(titleTerms)} = ${n} THEN false + ELSE (${sql.join(fuzzyTerms, sql` AND `)}) END` : sql`false`; + const wordTerms = search.tokens.map((token) => { + const escaped = token.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + return sql`issues.title ~* ${`(^|[^[:alnum:]_])${escaped}($|[^[:alnum:]_])`}`; + }); + // Carry flags, not potentially large bodies, through materialized stages. + // The search page fetches descriptions only for its result window. + const flags = (fuzzyMatch: SQL) => sql` + SELECT issues.id, issues.identifier, issues.title, + issues.status, issues.priority, issues.assignee_agent_id, issues.assignee_user_id, + issues.project_id, issues.created_at, issues.updated_at, + ${identExact} AS ident_exact, ${identStarts} AS ident_starts, + ${phraseMatch(sql`issues.identifier`)} AS ident_phrase, + ${taskSearchFieldMatch(sql`issues.identifier`, search)} AS ident_token, + ${n > 0 ? sql`lower(issues.title) = ${search.phrase}` : sql`false`} AS title_exact, + ${n > 0 ? sql`issues.title ILIKE ${search.startsWithPattern}` : sql`false`} AS title_starts, + ${phraseMatch(sql`issues.title`)} AS title_phrase, + ${taskSearchFieldMatch(sql`issues.title`, search)} AS title_token, + ${phraseMatch(sql`issues.description`)} AS desc_phrase, + ${taskSearchFieldMatch(sql`issues.description`, search)} AS desc_token, + ${coverage(titleTerms)} AS title_coverage, + ${coverage(wordTerms)} AS title_word_coverage, + ${coverage(issueTerms)} AS issue_coverage, + ${coverage(commentTerms)} AS comment_coverage, + ${coverage(documentTerms)} AS document_coverage, + ${coverage(allTerms)} AS token_coverage, + ${fuzzyMatch} AS fuzzy_title, + issues.id IN (SELECT issue_id FROM comment_matches) AS comment_match, + issues.id IN (SELECT issue_id FROM document_matches) AS document_match + FROM issues + `; + return sql` + WITH comment_matches AS MATERIALIZED (${comments}), + document_matches AS MATERIALIZED (${documents}), + literal_candidates AS MATERIALIZED ( + SELECT issues.id FROM issues + WHERE issues.company_id = ${companyId} AND ${visibleIssueCondition()} + AND ${n === 0 ? sql`${search.normalizedQuery.length === 0}` : sql`( + ${taskSearchAny(sql`issues.title`, search)} + OR ${taskSearchAny(sql`issues.identifier`, search)} + OR ${taskSearchAny(sql`issues.description`, search)} + OR ${identStarts} + )`} + UNION SELECT issue_id FROM comment_matches + UNION SELECT issue_id FROM document_matches + ), search_flags AS MATERIALIZED ( + ${flags(sql`false`)} + WHERE issues.company_id = ${companyId} AND ${visibleIssueCondition()} + AND issues.id IN (SELECT id FROM literal_candidates) + ), literal_matches AS MATERIALIZED ( + SELECT * FROM search_flags + WHERE ${n === 0 ? sql`true` : sql`token_coverage = ${n} OR ident_exact OR ident_starts`} + ), fuzzy_candidates AS MATERIALIZED ( + SELECT issues.id FROM issues + WHERE NOT EXISTS ( + SELECT 1 FROM literal_matches literal + JOIN issues ON issues.id = literal.id + ${fallbackFilters ? sql`WHERE ${fallbackFilters}` : sql``} + ) + AND issues.company_id = ${companyId} AND ${visibleIssueCondition()} + AND ${fuzzy} + ), matched AS MATERIALIZED ( + SELECT * FROM literal_matches + UNION ALL + ${flags(sql`true`)} + WHERE issues.id IN (SELECT id FROM fuzzy_candidates) + ) + `; +} diff --git a/ui/src/pages/Search.test.tsx b/ui/src/pages/Search.test.tsx index 2618e93322..1fa6715be6 100644 --- a/ui/src/pages/Search.test.tsx +++ b/ui/src/pages/Search.test.tsx @@ -263,6 +263,30 @@ describe("Search page", () => { }); }); + it.each(["relevance", "updated"])("preserves server %s order across result sources", async (sort) => { + const results = ["document", "title", "comment"].map((field, index) => ({ + id: `rank-${index}`, type: "issue", score: 300 - index, + title: `Rank ${index}`, href: `/PAP/issues/rank-${index}`, + matchedFields: [field], sourceLabel: field, snippet: field, + snippets: [{ field, label: field, text: field, highlights: [] }], + updatedAt: "2026-01-01T00:00:00.000Z", + issue: { id: `rank-${index}`, identifier: `RANK-${index}`, title: `Rank ${index}`, + status: "todo", priority: "medium", assigneeAgentId: null, assigneeUserId: null, + projectId: null, updatedAt: "2026-01-01T00:00:00.000Z" }, + })); + searchApiMock.search.mockResolvedValue({ query: "rank", normalizedQuery: "rank", scope: "all", + sort, limit: 20, offset: 0, hasMore: false, zeroResults: null, results, + countsByType: { issue: 1, comment: 1, document: 1, artifact: 0, agent: 0, project: 0 }, + filterOptionCounts: { status: {}, priority: {}, assigneeAgentId: {}, assigneeUserId: {}, projectId: {}, labelId: {}, updatedWithin: {} }, + }); + const { root } = renderSearch(`/search?q=rank&sort=${sort}`, container); + await waitForAssertion(() => { + const links = Array.from(container.querySelectorAll('[data-testid="search-results"] a[data-result-type]')); + expect(links.map((link) => link.getAttribute("href"))).toEqual(results.map((result) => result.href)); + }); + flushSync(() => root.unmount()); + }); + it("renders artifact search results in the company search surface", async () => { searchApiMock.search.mockResolvedValueOnce({ query: "launch brief", diff --git a/ui/src/pages/Search.tsx b/ui/src/pages/Search.tsx index 3f528a50f9..b4cc5a4c59 100644 --- a/ui/src/pages/Search.tsx +++ b/ui/src/pages/Search.tsx @@ -15,7 +15,6 @@ import { Input } from "@/components/ui/input"; import { Button } from "@/components/ui/button"; import { Skeleton } from "@/components/ui/skeleton"; import { Badge } from "@/components/ui/badge"; -import { cn } from "@/lib/utils"; import { useNavigate, useSearchParams } from "@/lib/router"; import { useCompany } from "../context/CompanyContext"; import { useBreadcrumbs } from "../context/BreadcrumbContext"; @@ -39,7 +38,6 @@ import { type ParsedSearchQuery, type SearchQueryParserContext, } from "../lib/search-query-parser"; -import { IssueGroupHeader } from "../components/IssueGroupHeader"; import { SearchResultRow } from "../components/search/SearchResultRow"; import { SearchFilterBar, type SearchFilterDataProps } from "../components/search/SearchFilterBar"; import { SearchFilterChips } from "../components/search/SearchFilterChips"; @@ -69,44 +67,6 @@ const SCOPE_LABELS: Record = { projects: "Projects", }; -type SubGroupKey = "issues" | "comments" | "documents" | "artifacts" | "agents" | "projects"; - -const SUBGROUP_ORDER: SubGroupKey[] = ["issues", "comments", "documents", "artifacts", "agents", "projects"]; - -const SUBGROUP_LABELS: Record = { - issues: "Tasks", - comments: "Comments", - documents: "Documents", - artifacts: "Artifacts", - agents: "Agents", - projects: "Projects", -}; - -function classifyResult(result: CompanySearchResult): SubGroupKey { - if (result.type === "artifact") return "artifacts"; - if (result.type === "agent") return "agents"; - if (result.type === "project") return "projects"; - const matched = new Set(result.matchedFields); - if (matched.has("title") || matched.has("identifier") || matched.has("description")) return "issues"; - if (matched.has("comment")) return "comments"; - if (matched.has("document")) return "documents"; - return "issues"; -} - -function buildSubgroups(results: CompanySearchResult[]): Array<{ key: SubGroupKey; results: CompanySearchResult[] }> { - const buckets = new Map(); - for (const result of results) { - const key = classifyResult(result); - const list = buckets.get(key) ?? []; - list.push(result); - buckets.set(key, list); - } - return SUBGROUP_ORDER.filter((key) => (buckets.get(key)?.length ?? 0) > 0).map((key) => ({ - key, - results: buckets.get(key) ?? [], - })); -} - function isCompanySearchScope(value: string | null): value is CompanySearchScope { return Boolean(value) && (COMPANY_SEARCH_SCOPES as readonly string[]).includes(value as string); } @@ -533,8 +493,6 @@ export function Search() { }); }, [counts, data, filtersActive]); - const subgroups = useMemo(() => buildSubgroups(data?.results ?? []), [data?.results]); - const operatorPills = useMemo(() => searchFilterPills(draftFilters, parserContext), [draftFilters, parserContext]); const operatorSuggestions = useMemo( () => (inputFocused ? searchOperatorSuggestions(draftQuery, 4) : []), @@ -721,7 +679,7 @@ export function Search() { refetch={() => void refetch()} recentSearches={recentSearches} onRecentClick={handleRecentClick} - subgroups={subgroups} + results={data?.results ?? []} totalResults={totalResults} allMatchTotal={allMatchTotal} activeFilterCount={activeFilterCount} @@ -767,7 +725,7 @@ interface SearchTabContentProps { refetch: () => void; recentSearches: string[]; onRecentClick: (query: string) => void; - subgroups: Array<{ key: SubGroupKey; results: CompanySearchResult[] }>; + results: CompanySearchResult[]; totalResults: number; allMatchTotal: number; activeFilterCount: number; @@ -792,7 +750,7 @@ function SearchTabContent({ refetch, recentSearches, onRecentClick, - subgroups, + results, totalResults, allMatchTotal, activeFilterCount, @@ -950,47 +908,14 @@ function SearchTabContent({ {isFetching ? Updating… : null} -
- {scope === "all" ? ( - subgroups.map((group, groupIndex) => ( -
0 && "mt-6")} - > - - {group.results.length} - - } - className="pt-2 pb-1 text-(length:--text-micro) tracking-wider text-muted-foreground" - /> -
- {group.results.map((result) => ( - - ))} -
-
- )) - ) : ( -
- {subgroups - .flatMap((group) => group.results) - .map((result) => ( - - ))} -
- )} +
+ {results.map((result) => ( + + ))}
); diff --git a/ui/storybook/stories/search.stories.tsx b/ui/storybook/stories/search.stories.tsx index 605c152685..4a3291df9a 100644 --- a/ui/storybook/stories/search.stories.tsx +++ b/ui/storybook/stories/search.stories.tsx @@ -6,7 +6,6 @@ import type { CompanySearchZeroResults, } from "@paperclipai/shared"; import { Badge } from "@/components/ui/badge"; -import { IssueGroupHeader } from "@/components/IssueGroupHeader"; import { Input } from "@/components/ui/input"; import { PageTabBar, type PageTabItem } from "@/components/PageTabBar"; import { MatchSourceChip } from "@/components/search/MatchSourceChip"; @@ -296,58 +295,11 @@ function SearchPagePreview({
{response.results.length} results · sorted by relevance
-
- - {fixtureResults.length} - - } - className="pt-2 pb-1 text-[11px] tracking-wider text-muted-foreground" - /> -
- {fixtureResults.map((result) => ( - - ))} -
-
-
- - {fixtureAgents.length} - - } - className="pt-2 pb-1 text-[11px] tracking-wider text-muted-foreground" - /> -
- {fixtureAgents.map((result) => ( - - ))} -
-
-
- - {fixtureProjects.length} - - } - className="pt-2 pb-1 text-[11px] tracking-wider text-muted-foreground" - /> -
- {fixtureProjects.map((result) => ( - - ))} -
-
+
+ {response.results.map((result) => ( + + ))} +
) : null} @@ -770,7 +722,7 @@ const meta = { docs: { description: { component: - "Full search page surfaces and Command K Search-all handoff. Reuses StatusIcon, StatusBadge, Identity, IssueGroupHeader, and PageTabBar; adds MatchSourceChip + SearchResultRow.", + "Full search page surfaces and Command K Search-all handoff. Reuses StatusIcon, StatusBadge, Identity and PageTabBar; adds MatchSourceChip + SearchResultRow.", }, }, },