19 KiB

Raw Blame History

Agent Tool Specs — NeuralCut

Este documento define las primitives iniciales del agente. Cada tool debe ser pequeña, accionable y componible. El agente puede resolver features complejas combinando estas tools, pero ninguna tool debe intentar “hacer todo”.

Principios generales

Las tools reciben datos estructurados, no instrucciones libres.
Las tools deben validar inputs y devolver errores claros.
Las tools que modifican el editor deben usar los comandos/managers existentes para preservar undo/redo cuando aplique.
El agente debe usar list_project_assets y list_timeline antes de editar cuando no tenga IDs concretos.
Gemini se usa para comprensión multimodal; la edición real se ejecuta con tools determinísticas.

1. `list_project_assets`

Propósito

Listar assets disponibles en el proyecto para que el agente sepa qué archivos existen y cuáles están usados en el timeline.

Input

{
  filter?: "all" | "used" | "unused";
  type?: "all" | "video" | "audio" | "image";
}

Output

{
  assets: Array<{
    id: string;
    name: string;
    type: "video" | "audio" | "image";
    duration?: number;
    usedInTimeline: boolean;
  }>;
}

Requirements

MUST return all loaded project assets by default.
MUST support filtering by usage: used, unused, all.
MUST support filtering by asset type.
MUST include stable internal id values.
MUST NOT mutate editor state.

Errors

If no project is loaded, return { error: "No active project" }.

2. `list_timeline`

Propósito

Listar el estado actual del timeline para que el agente pueda referenciar clips, textos, stickers y otros elementos por trackId/elementId.

Input

{}

Output

{
  tracks: Array<{
    trackId: string;
    type: "main" | "overlay" | "audio" | "text" | "effect";
    elements: Array<{
      elementId: string;
      type: string;
      assetId?: string;
      name?: string;
      start: number;
      end: number;
    }>;
  }>;
}

Requirements

MUST return a structured timeline summary.
MUST include trackId and elementId for editable elements.
MUST include timing in seconds or a clearly documented unit.
MUST NOT mutate editor state.

Errors

If no active scene/timeline exists, return { error: "No active timeline" }.

3. `load_asset_context`

Propósito

Cargar un asset en el contexto multimodal del agente usando Gemini. Esta es la base para que el agente entienda video/audio/imagen antes de editar.

Input

{
  assetId: string;
}

Output

{
  assetId: string;
  status: "loaded" | "processing";
  cached: boolean;
  provider: "gemini";
  fileUri?: string;
  summary?: string;
}

Requirements

MUST resolve the asset by internal assetId.
MAY support fallback by filename if unambiguous, but internal ID is preferred.
MUST upload/process the asset with Gemini when not cached.
MUST cache the provider file reference by assetId to avoid repeated uploads.
MUST keep API keys server-side.
MUST support video first; audio/image support may be added if Gemini path supports it cleanly.
MUST NOT edit the timeline.

Cache behavior

Cache is application/orchestrator infrastructure, not necessarily a visible LLM tool.

Cache should store at least:

{
  assetId: string;
  provider: "gemini";
  fileUri: string;
  status: "active" | "processing" | "failed";
  createdAt: number;
}

Errors

Unknown asset: { error: "Asset not found" }.
Unsupported type: { error: "Unsupported asset type" }.
Upload/processing failure: { error: "Failed to load asset context" } with safe details.

Non-goals

No editing actions.
No @asset UI autocomplete.
No streaming progress in the first implementation.

4. `split`

Propósito

Dividir elementos del timeline en uno o más puntos de tiempo, sin borrar contenido. Esta primitive equivale a hacer cortes/splits puntuales en el editor; eliminar material debe ser una tool separada y puede componerse después de hacer splits.

Input

{
  times: number[]; // timeline seconds
}

Output

{
  success: boolean;
  affectedElements: string[];
}

Requirements

MUST validate times contains at least one finite number.
MUST accept timeline times in seconds and convert to the editor’s canonical time unit internally.
MUST use existing timeline/command infrastructure when possible.
MUST split every timeline element intersecting each requested time when the split point falls strictly inside the element.
MUST NOT delete, trim away, or move timeline content.
MUST be idempotent at existing boundaries: if a requested time already equals an element boundary, it MUST NOT create a duplicate split there.
MUST preserve undo/redo behavior if the editor supports it for the operation.

Errors

Invalid times: { error: "Invalid split times" }.
Empty timeline: { error: "No timeline content" }.

5. `delete_timeline_elements`

Propósito

Eliminar uno o más elementos concretos del timeline por elementId. Para eliminar un rango de tiempo, el agente debe primero usar split({ times: [start, end] }) y luego borrar los elementos aislados.

Input

{
  elementIds: string[];
}

Output

{
  success: boolean;
  deletedElements: string[];
}

Requirements

MUST validate elementIds contains at least one non-empty string.
MUST use list_timeline first when exact elementId values are unknown.
MUST resolve elementId values against the active timeline before mutating.
MUST fail without mutating if any requested element is missing.
MUST remove only the requested elements.
MUST preserve undo/redo behavior if supported.

Errors

Invalid ids: { error: "Invalid element ids" }.
Missing element: { error: "Timeline elements not found: <ids>" }.
Empty timeline: { error: "No timeline content" }.

6. `move_timeline_elements`

Propósito

Mover uno o más elementos existentes del timeline a un nuevo tiempo inicial. Si se mueven varios elementos, el elemento más temprano queda en start y los demás conservan sus offsets relativos. Opcionalmente puede moverse el grupo a otra pista compatible con targetTrackId.

Input

{
  elementIds: string[];
  start: number; // timeline seconds
  targetTrackId?: string;
}

Output

{
  success: boolean;
  movedElements: Array<{
    elementId: string;
    trackId: string;
    start: number;
    end: number;
  }>;
}

Requirements

MUST validate elementIds contains at least one non-empty string.
MUST validate start is a finite non-negative timeline time in seconds.
MUST use list_timeline first when exact elementId or targetTrackId values are unknown.
MUST preserve relative offsets when moving multiple elements.
MUST keep elements on their current tracks when targetTrackId is omitted.
MUST fail without mutating if any requested element is missing or the target track is incompatible.
MUST preserve undo/redo behavior if supported.

Errors

Invalid ids: { error: "Invalid element ids" }.
Invalid start: { error: "Invalid start time" }.
Missing element: { error: "Timeline elements not found: <ids>" }.
Missing target track: { error: "Target track not found: <id>" }.
Empty timeline: { error: "No timeline content" }.

7. `add_text`

Propósito

Agregar texto visual al timeline. Esta primitive cubre títulos, hooks, labels y subtítulos básicos.

Input

{
  text: string;
  start: number;
  end: number;
  position: "top" | "center" | "bottom";
  style?: "plain" | "subtitle" | "hook" | "label";
}

Output

{
  elementId: string;
  trackId: string;
}

Requirements

MUST validate non-empty text.
MUST validate start < end.
MUST create a text element using existing timeline APIs.
MUST map position to a sensible default placement.
SHOULD provide simple style presets, but implementation may start with defaults.
MUST NOT generate full subtitles automatically; that is a flow using transcript/context + repeated add_text calls.

Errors

Empty text: { error: "Text is required" }.
Invalid range: { error: "Invalid time range" }.

8. `update_text`

Propósito

Editar un texto existente en el timeline.

Input

{
  trackId: string;
  elementId: string;
  text?: string;
  start?: number;
  end?: number;
  position?: "top" | "center" | "bottom";
}

Output

{
  success: boolean;
  elementId: string;
}

Requirements

MUST find an existing text element by trackId + elementId.
MUST only update provided fields.
MUST validate timing if start/end are provided.
MUST preserve undo/redo behavior if supported.

Errors

Missing element: { error: "Text element not found" }.
Wrong element type: { error: "Element is not text" }.
Invalid range: { error: "Invalid time range" }.

9. `add_media_to_timeline`

Propósito

Agregar un asset existente al timeline.

Input

{
  assetId: string;
  startTime: number;
  trackType: "main" | "overlay" | "audio";
  duration?: number; // timeline seconds
}

Output

{
  elementId: string;
  trackId: string;
}

Requirements

MUST resolve asset by assetId.
MUST validate asset type compatibility with trackType.
MUST validate optional duration as a positive timeline duration in seconds.
MUST insert the element using existing timeline APIs.
MUST use full source duration for video/audio by default.
MUST use editor default duration for images when duration is omitted.
MUST reject requested video/audio duration beyond known source duration.
SHOULD choose a sensible track when one is not obvious, but first version requires explicit trackType.

Errors

Asset not found: { error: "Asset not found" }.
Invalid track type: { error: "Invalid track type for asset" }.
Invalid duration: { error: "Invalid duration" }.
Duration too long: { error: "Duration exceeds source duration" }.

10. `update_timeline_element_timing`

Propósito

Actualizar el inicio, final o duración de un elemento existente del timeline sin cambiar el asset original. Esta primitive cubre pedidos como “hacé que esta foto dure 10 segundos” o “que este clip termine en 00:15”.

Input

{
  elementId: string;
  start?: number; // timeline seconds
  end?: number; // timeline seconds
  duration?: number; // timeline seconds
}

Output

{
  success: boolean;
  elementId: string;
  trackId: string;
  start: number;
  end: number;
  duration: number;
}

Requirements

MUST validate elementId as a non-empty string.
MUST require at least one of start, end, or duration.
MUST accept all timing values in seconds and convert to the editor’s canonical unit internally.
MUST use list_timeline first when exact elementId is unknown.
MUST update only the timeline element, never the project asset.
MUST preserve undo/redo behavior if supported.
MUST reject conflicting end and duration values when both are provided.
MUST reject video/audio duration beyond known source duration.

Errors

Invalid id: { error: "Invalid element id" }.
Invalid update: { error: "Invalid timing update" }.
Invalid start: { error: "Invalid start time" }.
Invalid end: { error: "Invalid end time" }.
Invalid duration: { error: "Invalid duration" }.
Invalid range: { error: "Invalid time range" }.
Conflicting values: { error: "Conflicting timing values" }.
Missing element: { error: "Timeline element not found: <id>" }.
Duration too long: { error: "Duration exceeds source duration" }.

11. `delete_element` (deprecated in favor of `delete_timeline_elements`)

Propósito

Eliminar un elemento específico del timeline. La implementación actual debe preferir delete_timeline_elements porque soporta borrado en lote y permite componer rangos después de split.

Input

{
  trackId: string;
  elementId: string;
}

Output

{
  success: boolean;
}

Requirements

MUST find element by trackId + elementId.
MUST remove only that element.
MUST preserve undo/redo behavior if supported.

Errors

Missing element: { error: "Element not found" }.

12. `set_volume`

Propósito

Ajustar volumen de un elemento de audio o video.

Input

{
  trackId: string;
  elementId: string;
  volume: number;
}

Output

{
  success: boolean;
  volume: number;
}

Requirements

MUST validate volume in range 0..1.
MUST only apply to elements that support audio volume.
MUST preserve undo/redo behavior if supported.

Errors

Invalid volume: { error: "Volume must be between 0 and 1" }.
Unsupported element: { error: "Element does not support volume" }.

13. `add_sticker`

Propósito

Agregar un sticker existente al timeline.

Input

{
  stickerId: string;
  start: number;
  end: number;
  position: "top-left" | "top-right" | "center" | "bottom-left" | "bottom-right";
}

Output

{
  elementId: string;
  trackId: string;
}

Requirements

MUST resolve sticker by stickerId.
MUST validate start < end.
MUST create a timeline element using existing sticker/timeline APIs.
MUST map position to sensible coordinates.

Errors

Sticker not found: { error: "Sticker not found" }.
Invalid range: { error: "Invalid time range" }.

14. `list_effects`

Propósito

Listar todos los efectos disponibles para que el agente descubra qué efectos puede aplicar a elementos visuales del timeline. Esta es la primera tool en el flujo de efectos: list_effects → get_effect → apply_effect.

Input

{
  query?: string;
}

Output

{
  effects: Array<{
    id: string;
    name: string;
    description: string;
  }>;
}

Requirements

MUST return all registered effects by default.
MUST support optional query filtering by name or keywords (case-insensitive).
MUST NOT mutate editor state.
MUST use the effects registry directly (static data, not project-dependent).

Errors

None (always returns a list, possibly empty).

15. `get_effect`

Propósito

Obtener metadata detallada de un efecto específico, incluyendo todos los parámetros configurables con sus tipos, rangos, valores default y descripciones. El agente usa esto después de list_effects para saber cómo configurar un efecto antes de llamar apply_effect.

Input

{
  effectType: string;
}

Output

{
  id: string;
  name: string;
  description: string;
  params: Array<{
    key: string;
    label: string;
    type: "number" | "boolean" | "color" | "select";
    default: number | string | boolean;
    min?: number;
    max?: number;
    step?: number;
    options?: Array<{ value: string; label: string }>;
    description: string;
  }>;
}

Requirements

MUST resolve the effect by effectType from the effects registry.
MUST include full parameter metadata for each effect parameter.
MUST generate a human-readable description for each parameter (e.g. "Number between 0 and 100, step 1. Default: 15").
MUST NOT mutate editor state.
MUST NOT include renderer/shader details (internal only).

Errors

Invalid type: { error: "Invalid effect type" }.
Effect not found: { error: "Effect not found: <type>" }.

16. `apply_effect`

Propósito

Agregar un efecto como elemento standalone al timeline en una pista de efectos, equivalente a drag & dropear un efecto desde el panel de efectos. Soporta los 7 efectos registrados: blur, brightness-contrast, grayscale, saturation, sepia, invert, vignette.

Input

{
  effectType: string;
  start: number;
  end: number;
  params?: Record<string, number | string | boolean>;
}

Output

{
  elementId: string;
  trackId: string;
  appliedParams: Record<string, number | string | boolean>;
}

Requirements

MUST validate effectType exists in the effects registry.
MUST validate start and end as valid timeline seconds with start < end.
MUST validate params against the effect's parameter definitions (types, ranges).
MUST create an EffectElement via buildEffectElement() with the requested time range.
MUST merge custom params into the default effect instance before insertion.
MUST use InsertElementCommand with { mode: "auto", trackType: "effect" } to place on an effect track (creating one if needed).
MUST preserve undo/redo behavior.
MUST return the final applied parameter values.

Errors

Invalid effect type: { error: "Invalid effect type" }.
Invalid start time: { error: "Invalid start time" }.
Invalid end time: { error: "Invalid end time" }.
Invalid params: { error: "Invalid effect parameters" }.
Effect not found: { error: "Effect not found: <type>" }.
Invalid time range: { error: "Invalid time range" }.
No active timeline: { error: "No active timeline" }.
Failed placement: { error: "Failed to place effect element" }.
Unknown parameter: { error: "Unknown parameter: <key>" }.
Out of range: { error: "Parameter '<key>' must be >= <min>" }.

17. `update_effect`

Propósito

Actualizar los parámetros de un elemento de efecto existente en el timeline. Solo se actualizan los parámetros proporcionados; el resto mantiene sus valores actuales.

Input

{
  elementId: string;
  params: Record<string, number | string | boolean>;
}

Output

{
  success: boolean;
  elementId: string;
  appliedParams: Record<string, number | string | boolean>;
}

Requirements

MUST validate elementId exists in the active timeline.
MUST validate the target element is an effect element (type: "effect").
MUST validate params against the effect's parameter definitions (types, ranges).
MUST merge provided params with existing params (only override specified keys).
MUST use updateElements to apply the param patch.
MUST preserve undo/redo behavior.
MUST return the full merged params after update.

Errors

Invalid element id: { error: "Invalid element id" }.
Missing params: { error: "params is required and must be a non-empty object" }.
No active timeline: { error: "No active timeline" }.
Missing element: { error: "Timeline element not found: <id>" }.
Wrong type: { error: "Element is not an effect" }.
Effect not found: { error: "Effect not found: <type>" }.
Unknown parameter: { error: "Unknown parameter: <key>" }.
Out of range: { error: "Parameter '<key>' must be >= <min>" }.

Existing/secondary tool: `transcribe_video`

Propósito

Transcribir audio usando Whisper local. Es útil para captions y edición por texto, pero no debe ser la primitive principal de comprensión si load_asset_context con Gemini está disponible.

Input

{
  assetId?: string;
  language?: string;
  modelId?: string;
}

Estado

Implementada.
Debe mantenerse como secundaria.
Puede ser usada por flujos de subtítulos.

19 KiB Raw Blame History Unescape Escape

Agent Tool Specs — NeuralCut

Principios generales

1. list_project_assets

Propósito

Input

Output

Requirements

Errors

2. list_timeline

Propósito

Input

Output

Requirements

Errors

3. load_asset_context

Propósito

Input

Output

Requirements

Cache behavior

Errors

Non-goals

4. split

Propósito

Input

Output

Requirements

Errors

5. delete_timeline_elements

Propósito

Input

Output

Requirements

Errors

6. move_timeline_elements

Propósito

Input

Output

Requirements

Errors

7. add_text

Propósito

Input

Output

Requirements

Errors

8. update_text

Propósito

Input

Output

Requirements

Errors

9. add_media_to_timeline

Propósito

Input

Output

Requirements

Errors

10. update_timeline_element_timing

Propósito

Input

Output

Requirements

Errors

11. delete_element (deprecated in favor of delete_timeline_elements)

Propósito

Input

Output

Requirements

Errors

12. set_volume

Propósito

Input

Output

Requirements

Errors

13. add_sticker

Propósito

Input

19 KiB

Raw Blame History

1. `list_project_assets`

2. `list_timeline`

3. `load_asset_context`

4. `split`

5. `delete_timeline_elements`

6. `move_timeline_elements`

7. `add_text`

8. `update_text`

9. `add_media_to_timeline`

10. `update_timeline_element_timing`

11. `delete_element` (deprecated in favor of `delete_timeline_elements`)

12. `set_volume`

13. `add_sticker`

14. `list_effects`

15. `get_effect`

16. `apply_effect`

17. `update_effect`

Existing/secondary tool: `transcribe_video`