# Agent Session Shell Specification

## Purpose

Defines the client-side orchestration boundary between the UI layer, the API proxy, Zustand stores, and the tool registry. The route delegates to a config-selected provider adapter for real LLM execution, and the orchestrator drives a multi-turn tool-call loop with iteration guard and argument validation.

## Requirements

### Requirement: Chat Store

The system SHALL maintain a `chatStore` (Zustand) with state: `messages: ChatMessage[]`, `loading: boolean`, `error: string | null`. The store MUST provide `addMessage(msg)`, `setLoading(v)`, `setError(e)`, and `clearMessages()` actions. The store MUST follow the existing `create<State>()(persist(...))` pattern used by other stores in the project.

#### Scenario: Message lifecycle

- GIVEN `chatStore` is initialized with empty messages
- WHEN `addMessage({ role: "user", content: "Hello" })` is called
- THEN `chatStore.messages` contains one entry with the correct role and content

#### Scenario: Error state persists until cleared

- GIVEN `setError("API failed")` was called
- WHEN the UI reads `chatStore.error`
- THEN it returns `"API failed"`
- AND `chatStore.loading` is `false`

### Requirement: Agent Store

The system SHALL maintain an `agentStore` (Zustand) with state: `status: ExecutionStatus`, `activeTool: string | null`, `context: AgentContext | null`. The store MUST provide `setStatus`, `setActiveTool`, `setContext`, and `reset` actions.

#### Scenario: Tool execution tracking

- GIVEN the orchestrator begins executing tool `echo_context`
- WHEN `setActiveTool("echo_context")` and `setStatus("processing")` are called
- THEN `agentStore.activeTool` is `"echo_context"` and `agentStore.status` is `"processing"`

#### Scenario: Reset after completion

- GIVEN `agentStore` has `status: "processing"` and `activeTool: "echo_context"`
- WHEN `reset()` is called
- THEN `status` is `"idle"` and `activeTool` is `null`

### Requirement: API Proxy Route

The system SHALL expose `POST /api/agent/chat` that accepts `{ messages: ChatMessage[], context?: AgentContext }` and returns `{ content: string, toolCalls?: ToolCall[] }`. The route MUST resolve the provider adapter from configuration, assemble a system prompt via `buildSystemPrompt(context)`, and pass internal tool schemas from the registry to the adapter. The adapter handles all provider-specific wire-format conversion. The request body MUST be validated via zod; invalid input returns 400. Provider errors return 502.

#### Scenario: Valid request returns provider response

- GIVEN provider config is set and a POST with valid `{ messages, context }` arrives
- WHEN the handler calls through the provider adapter
- THEN it returns 200 with `{ content: "...", toolCalls?: [...] }`

#### Scenario: Invalid request returns 400

- GIVEN a POST request with `{}` (missing `messages`)
- WHEN the handler processes the request
- THEN it returns 400 with an error description

#### Scenario: Provider error returns 502

- GIVEN the provider adapter call throws a network or auth error
- WHEN the handler catches it
- THEN it returns 502 with `{ error: "LLM provider error" }`

### Requirement: Orchestrator Shell

The system SHALL provide a client-side `orchestrator.run(messages, context)` function that iterates: (1) sets `agentStore.status` to `"sending"`, (2) POSTs `{ messages, context }` to `/api/agent/chat`, (3) if the response contains `toolCalls`, validates args, resolves each via the tool registry (if a single tool throws, the error MUST be captured in a `ToolResult` and the loop MUST continue so the LLM can recover), appends tool results to the message history, and loops back to step 2 with the updated history, (4) when the response has no `toolCalls`, appends the final assistant message to `chatStore`, sets status to `"idle"`. The loop MUST NOT exceed the max iteration guard. On API or network errors (not per-tool errors), the orchestrator MUST set `chatStore.error` and `agentStore.status` to `"error"`.

#### Scenario: Happy path with no tool calls

- GIVEN `chatStore` has one user message and the API returns `{ content: "Hello!", toolCalls: [] }`
- WHEN `orchestrator.run()` is invoked
- THEN `chatStore.messages` contains the assistant reply "Hello!"
- AND `agentStore.status` is `"idle"`

#### Scenario: Tool call then final answer

- GIVEN iteration 1 returns `{ content: "", toolCalls: [{ name: "transcribe_video", args: {} }] }`
- AND the tool executes successfully
- AND iteration 2 returns `{ content: "Here is the transcription...", toolCalls: [] }`
- WHEN the orchestrator loop completes
- THEN `chatStore.messages` contains the tool result AND the final assistant message
- AND `agentStore.status` is `"idle"`

#### Scenario: Error during tool execution — per-tool recovery

- GIVEN iteration 1 returns `{ content: "", toolCalls: [{ name: "transcribe_video", args: {} }] }` and the tool throws
- WHEN the orchestrator catches the error inside `resolveToolCalls`
- THEN a `ToolResult` with the error description is appended to the message history
- AND the loop continues, sending tool-result back to the LLM for a final answer

### Requirement: Tool Registry Shell

The system SHALL provide a `ToolRegistry` extending `DefinitionRegistry<string, ToolDefinition>` where `ToolDefinition` contains `{ name, description, parameters, execute }`. The registry MUST support `register`, `get`, `has`, `getAll`, and a `toToolSchemas()` method that exports every registered tool's parameters in a provider-agnostic internal schema format. The provider adapter is responsible for converting these internal schemas to its specific wire format. In this phase `echo_context` MUST NOT be registered; exactly one tool — `transcribe_video` — MUST be registered.

#### Scenario: Schema export produces internal format

- GIVEN `transcribe_video` is registered with parameters `[{ key: "assetId", type: "string", required: false }, ...]`
- WHEN `toToolSchemas()` is called
- THEN it returns an array with one entry containing `{ name: "transcribe_video", description, parameters: [...] }` in the internal format

### Requirement: Shared Foundational Types

The system SHALL define types in `apps/web/src/agent/types.ts`: `ChatMessage { id, role, content, toolCalls?, timestamp }`, `ToolCall { id, name, args }`, `ToolResult { toolCallId, name, result, error? }`, `ExecutionState = "idle" | "sending" | "processing" | "responding" | "error"`, `AgentContext { projectId, activeSceneId, mediaAssets, playbackTimeMs }`, `ToolDefinition { name, description, parameters, execute }`. These types MUST be the single source of truth shared across stores, orchestrator, registry, and API route.

#### Scenario: Types are importable everywhere

- GIVEN `types.ts` exports `ChatMessage`, `ToolCall`, `ToolResult`, `ExecutionState`, `AgentContext`, `ToolDefinition`
- WHEN any module in `agent/`, `stores/`, or `app/api/agent/` imports from it
- THEN TypeScript compiles without errors

### Requirement: Provider Adapter Interface

The system SHALL define a server-side provider adapter interface with a single method `chat(params: ProviderChatParams): Promise<ProviderChatResponse>`. `ProviderChatParams` MUST include `messages`, `systemPrompt`, and `toolSchemas` in a provider-agnostic internal format. `ProviderChatResponse` MUST include `content: string` and `toolCalls?: ProviderToolCall[]` where `ProviderToolCall { id, name, args }`. Each concrete adapter converts internal tool schemas to its own wire format. The factory MUST support at least `openai-compatible` and `gemini` providers. Adapters whose native API does not assign IDs to tool calls MUST synthesize unique IDs so every returned `ToolCall.id` is a non-empty string.

#### Scenario: Adapter translates tool schemas to wire format

- GIVEN `transcribe_video` has internal parameters `[{ key: "assetId", type: "string", required: false }]`
- WHEN the adapter receives tool schemas before calling the provider
- THEN it converts them to its provider-specific wire format

#### Scenario: Adapter selected by config — OpenAI-compatible

- GIVEN `LLM_PROVIDER` is set to `"openai-compatible"`
- WHEN the route resolves the adapter
- THEN the OpenAI-compatible adapter is returned

#### Scenario: Adapter selected by config — Gemini

- GIVEN `LLM_PROVIDER` is set to `"gemini"`
- WHEN the route resolves the adapter
- THEN the Gemini adapter is returned

#### Scenario: Gemini adapter synthesizes tool-call IDs

- GIVEN the Gemini API returns a function call response with no call ID
- WHEN the Gemini adapter maps the response to `ProviderResponse`
- THEN each `ToolCall` in `toolCalls` has a non-empty `id` field generated by the adapter

### Requirement: Provider Configuration

Server-only environment variables `LLM_PROVIDER` (required), `LLM_MODEL` (required), `LLM_API_KEY` (required), and `LLM_BASE_URL` (optional) MUST determine which adapter and model the route uses. `LLM_PROVIDER` MUST accept at least `"openai-compatible"` and `"gemini"`. When `LLM_PROVIDER` is `"gemini"`, `LLM_BASE_URL` MUST be ignored. Configuration MUST be read server-side only — never exposed to the client.

#### Scenario: Missing required configuration

- GIVEN `LLM_API_KEY` is not set in the environment
- WHEN the route attempts to initialize the provider
- THEN the call returns 502 with a missing-configuration error

#### Scenario: Unknown provider

- GIVEN `LLM_PROVIDER` is set to `"unknown-provider"`
- WHEN the route resolves the adapter
- THEN the factory throws `"Unknown LLM provider: unknown-provider"`

#### Scenario: Gemini config ignores base URL

- GIVEN `LLM_PROVIDER` is `"gemini"` and `LLM_BASE_URL` is set
- WHEN the Gemini adapter is created
- THEN `LLM_BASE_URL` is not used in the API call

### Requirement: Tool Argument Validation

Before executing any tool call, the orchestrator MUST validate that each argument key declared `required: true` in the tool's `parameters` is present in the call args and is of the declared type. If validation fails, the tool MUST NOT execute; instead the orchestrator MUST produce a `ToolResult` with `error` describing the missing or malformed argument.

#### Scenario: Missing required argument

- GIVEN `transcribe_video` declares no required parameters and the LLM returns `toolCalls: [{ name: "transcribe_video", args: {} }]`
- WHEN the orchestrator validates args
- THEN validation passes and the tool executes

#### Scenario: Wrong argument type

- GIVEN a tool declares `{ key: "assetId", type: "string", required: true }`
- AND the LLM returns `args: { assetId: 123 }`
- WHEN the orchestrator validates args
- THEN the tool does NOT execute and a `ToolResult` with `error` containing "assetId" is returned

### Requirement: Max Iteration Guard

The orchestrator loop MUST enforce a hard iteration cap of **8**. An iteration is one round of "call provider → resolve tool calls → feed results back." If the cap is reached, the orchestrator MUST stop, append a final assistant message indicating the limit was reached, and set `agentStore.status` to `"idle"`.

#### Scenario: Cap reached mid-loop

- GIVEN the provider has returned tool calls on 7 consecutive iterations
- WHEN iteration 8 also returns tool calls
- THEN the orchestrator stops, appends "Reached maximum iteration limit" to `chatStore`, and sets status to `"idle"`

#### Scenario: Loop completes within cap

- GIVEN the provider returns a final text response after 2 tool-call iterations
- WHEN the orchestrator processes the response
- THEN no cap message is appended and status is `"idle"`

### Requirement: Gemini Free-Form Chat

When the Gemini adapter receives a request with no tool schemas (or tools array is empty), it MUST send messages to the Gemini `generateContent` API and return a `ProviderResponse` with `content` set to the assistant text and `toolCalls` undefined.

#### Scenario: Plain chat through Gemini

- GIVEN `LLM_PROVIDER` is `"gemini"` and messages contain `[{ role: "user", content: "Hello" }]`
- WHEN the adapter calls Gemini and receives a text-only response
- THEN `ProviderResponse.content` is the assistant text and `toolCalls` is undefined

#### Scenario: System prompt mapping

- GIVEN the adapter receives `systemPrompt: "You are a video editor"`
- WHEN it builds the Gemini request
- THEN the system prompt is mapped to Gemini's system instruction field, not as a user message

### Requirement: Gemini Tool Calling

When the Gemini adapter receives a request with tool schemas, it MUST convert internal `ToolSchema[]` to Gemini function declarations, call `generateContent` with declarations attached, and if Gemini returns function call parts, map each to a `ToolCall { id, name, args }` with a synthesized non-empty `id`.

#### Scenario: Gemini returns function calls

- GIVEN `transcribe_video` is in tool schemas and Gemini responds with a function call part `{ name: "transcribe_video", args: {} }`
- WHEN the adapter maps the response
- THEN `toolCalls` contains one `ToolCall` with `name: "transcribe_video"`, `args: {}`, and a synthesized `id`

#### Scenario: Gemini returns text despite tools

- GIVEN tool schemas are provided but Gemini responds with only text
- WHEN the adapter maps the response
- THEN `toolCalls` is undefined and `content` holds the text

#### Scenario: Tool schema type mapping

- GIVEN an internal tool schema has parameters `[{ key: "assetId", type: "string", required: true }]`
- WHEN the adapter converts to Gemini function declarations
- THEN the parameter appears in `FunctionDeclaration.parameters.properties` with type `"string"`