From 03e8ae8d7e6718a6efd731e0fa5537d1cbd2a6fe Mon Sep 17 00:00:00 2001 From: Francesco Balassone <77972200+duel0@users.noreply.github.com> Date: Mon, 1 Dec 2025 17:42:07 +0100 Subject: [PATCH] Added Council Docs (#355) Co-authored-by: Francesco Balassone <77972200+balassone@users.noreply.github.com> --- docs/council.md | 295 ++++++++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 296 insertions(+) create mode 100644 docs/council.md diff --git a/docs/council.md b/docs/council.md new file mode 100644 index 00000000..7252b460 --- /dev/null +++ b/docs/council.md @@ -0,0 +1,295 @@ +# LLM Council + +The LLM Council is a multi-model consensus system that queries multiple AI models, has them evaluate each other's responses, and synthesizes a final answer through a chairman. This approach improves accuracy for complex decisions by leveraging diverse model perspectives. + +## How It Works + +The council operates in three stages: + +``` +User Query: "What's the best approach for this security task?" +│ +├─ Stage 1: Council Members (parallel) +│ ├─ Agent + gpt-4o → Text response only +│ ├─ Agent + gpt-5 → Text response only +│ └─ Agent + claude-sonnet-4-5 → Text response only +│ +├─ Stage 2: Rankings (parallel) +│ ├─ Agent + gpt-4o → Ranks anonymized responses +│ ├─ Agent + gpt-5 → Ranks anonymized responses +│ └─ Agent + claude-sonnet-4-5 → Ranks anonymized responses +│ +└─ Stage 3: Chairman + └─ Active Agent + TOOLS + ├─ Synthesizes best answer + └─ Can execute operations if requested +``` + +**Key Points:** + +- **Stage 1 & 2**: Council members provide text-only responses (no tool execution) +- **Stage 3**: The chairman (your active agent) can use all available tools +- All members use the current agent's instructions and context + +## Quick Start + +```bash +# Configure council models +export CAI_COUNCIL="gpt-4o,gpt-5,claude-sonnet-4-20250514" + +# In CAI REPL, load an agent first +CAI> /agent redteam + +# Use the council command +CAI> /council What are the best practices for API security? + +# Or use the short alias +CAI> /c How should I approach this vulnerability assessment? +``` + +## Configuration + +### Environment Variables + +| Variable | Description | Default | +|----------|-------------|---------| +| `CAI_COUNCIL` | Comma-separated list of council member models | `gpt-4o,gpt-4o-mini` | +| `CAI_COUNCIL_AUTO` | Auto-convene setting: `false`, `true`/`1`, or interval number | `false` | +| `CAI_COUNCIL_PROMPT` | Custom prompt for auto-council reviews | See below | +| `CAI_COUNCIL_DEBUG` | Enable debug output (`1`, `true`, `yes`) | `false` | + +```bash +# Example configuration +export CAI_COUNCIL="gpt-4o,gpt-5,claude-sonnet-4-20250514" +export CAI_COUNCIL_AUTO="5" +export CAI_COUNCIL_PROMPT="Review the current progress and recommend the best approach." +export CAI_COUNCIL_DEBUG="1" +``` + +### API Keys + +Ensure you have the appropriate API keys set for your council models: + +```bash +export OPENAI_API_KEY="sk-..." +export ANTHROPIC_API_KEY="sk-ant-..." +export ALIAS_API_KEY="..." +``` + +### Verified Model Names + +Use exact model names as shown in the `/model` command: + +| Provider | Models | +|----------|--------| +| OpenAI | `gpt-5`, `gpt-4o`, `gpt-4o-mini`, `o3-mini` | +| Anthropic | `claude-sonnet-4-20250514`, `claude-3-5-sonnet-20240620` | +| Alias | `alias1` | +| DeepSeek | `deepseek-v3`, `deepseek-r1` | + +## Manual Usage + +The `/council` command (alias `/c`) invokes the council manually: + +```bash +# Load an agent +CAI> /agent redteam + +# Ask the council +CAI> /council What vulnerabilities should I look for in this web application? +``` + +The council uses the active agent's: + +- Instructions/system prompt +- Available tools (chairman only) +- Guardrails + +## Auto-Council Mode + +When `CAI_COUNCIL_AUTO` is enabled, the council convenes automatically at specified intervals during agent execution. + +### Configuration Options + +- `false` - Never auto-convene (use `/council` manually) +- `true` or `1` - Convene at every agent interaction +- `5`, `10`, etc. - Convene every N interactions + +### Example: Every Interaction + +```bash +export CAI_COUNCIL_AUTO="1" + +CAI> run ps aux, then analyze the results, then check for vulnerabilities + +🏛️ COUNCIL (auto-invoked at interaction [1]) +[Stage 1, 2, 3 run...] +[1] Agent: "I'll run ps aux" → executes command + +🏛️ COUNCIL (auto-invoked at interaction [2]) +[Stage 1, 2, 3 run...] +[2] Agent: "Analyzing results..." → analyzes output + +🏛️ COUNCIL (auto-invoked at interaction [3]) +[Stage 1, 2, 3 run...] +[3] Agent: "Checking for vulnerabilities..." → performs check +``` + +### Example: Every 5 Interactions + +```bash +export CAI_COUNCIL_AUTO="5" + +CAI> perform a comprehensive security audit + +[1] Agent executes first task +[2] Agent executes second task +[3] Agent executes third task +[4] Agent executes fourth task + +🏛️ COUNCIL (auto-invoked at interaction [5]) +[Stage 1, 2, 3 run...] +[5] Agent executes fifth task + +[6] Agent continues... +[7] Agent continues... +[8] Agent continues... +[9] Agent continues... + +🏛️ COUNCIL (auto-invoked at interaction [10]) +[Stage 1, 2, 3 run...] +[10] Agent executes tenth task +``` + +## Programmatic Usage + +You can use the council directly in Python code: + +```python +from cai.council import run_full_council_agents, CouncilAgentConfig +from cai.sdk.agents import Agent + +# With an existing agent +stage1, stage2, stage3, metadata = await run_full_council_agents( + base_agent=my_agent, + user_query="Your question here", +) + +# Access results +print(stage3["response"]) # Final answer +print(metadata["aggregate_rankings"]) # Model rankings +print(metadata["council_cost"]) # Total cost +print(metadata["council_input_tokens"]) # Input tokens +print(metadata["council_output_tokens"]) # Output tokens +``` + +### Return Values + +```python +stage1_results: List[Dict] # Individual responses from each model +stage2_results: List[Dict] # Rankings from each model +stage3_result: Dict # Final synthesized answer +metadata: Dict # Rankings, cost, tokens +``` + +### Metadata Structure + +```python +metadata = { + "aggregate_rankings": [ + {"model": "gpt-4o", "average_rank": 1.33, "rankings_count": 3}, + {"model": "gpt-5", "average_rank": 2.0, "rankings_count": 3}, + ], + "council_cost": 0.032, + "council_input_tokens": 5000, + "council_output_tokens": 2500, +} +``` + +## Visual Display + +During execution, the council shows an animated panel with progress: + +``` +╭─────────────────────── Alias Council ────────────────────────╮ +│ │ +│ 👑 Chairman: Red Team Agent (gpt-4o) │ +│ │ +│ ⠋ Stage 1: Collecting responses from council members │ +│ ██████████░░░░░░░░░░ 2/3 │ +│ ✓ gpt-4o │ +│ ✓ gpt-5 │ +│ ⠋ alias1 │ +│ │ +│ ○ Stage 2: Waiting... │ +│ │ +│ 💰 $0.012 (1.2k in / 800 out) ⏱ 15.2s │ +│ │ +╰─────────────────────────────────────────────────────────────────╯ +``` + +## Performance Considerations + +| Metric | Single Query | Council (3 models) | +|--------|--------------|-------------------| +| API Calls | 1 | ~7 (2N + 1) | +| Cost | 1x | 3-4x | +| Latency | 1x | 2-3x | +| Accuracy | Base | Improved | + +!!! tip "When to Use Council" + Use the council when accuracy matters more than speed or cost. It's particularly valuable for: + + - Complex security decisions + - Architecture recommendations + - Vulnerability assessments + - Strategic planning tasks + +## Troubleshooting + +### Debug Mode + +Enable detailed logging to diagnose issues: + +```bash +export CAI_COUNCIL_DEBUG=1 +``` + +### Common Issues + +**"All models failed to respond"** + +- Verify API keys are set correctly +- Check model names with `/model` command +- Check for rate limiting + +**Council hangs on Stage 1** + +- Model name might be incorrect (verify with `/model`) +- API key invalid or missing +- Network connectivity issues + +**"Temperature not supported"** + +- Handled automatically for GPT-5/O1/O3 models (temperature set to 1) + +### Test Individual Models + +Before using council, verify each model works independently: + +```bash +CAI> /model gpt-4o +CAI> What is 2+2? +``` + +### Minimal Configuration + +If experiencing issues, try a minimal setup: + +```bash +export CAI_COUNCIL="gpt-4o,gpt-4o-mini" +``` + +## Credits + +Inspired by [llm-council](https://github.com/karpathy/llm-council) by Andrej Karpathy. diff --git a/mkdocs.yml b/mkdocs.yml index 0bc610ff..010c01bb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -100,6 +100,7 @@ nav: - Tracing & Debugging: tracing.md - Context Management: context.md - Guardrails & Security: guardrails.md + - LLM Council: council.md - Environment Variables: environment_variables.md # ========================================