24 KiB
24 KiB
Agent Review Protocol
Trust but verify. Every agent's work must be reviewed before acceptance.
Related: AGENT_GUARDRAILS.md | ADVERSARIAL_TESTING.md | CODE_REVIEW.md
Overview
This document establishes the mandatory review protocol for AI agent work. No agent's output should be accepted without verification by either another agent, a different LLM, or systematic automated checks.
Core Principle: The agent that builds should NOT be the only agent that validates.
WHY AGENT REVIEW IS MANDATORY
The Hallucination Problem
PROBLEM: AI agents confidently produce incorrect output.
SYMPTOMS:
- Code that "looks right" but has subtle bugs
- Tests that pass but don't actually test the right thing
- Logic that works for happy path but fails edge cases
- Confident assertions about code that doesn't exist
SOLUTION: Independent verification breaks the hallucination loop.
The Context Contamination Problem
PROBLEM: Long sessions accumulate errors in context.
SYMPTOMS:
- Agent references earlier mistakes as "facts"
- Corrections don't fully override original errors
- Agent defends wrong approaches because they're in context
- Quality degrades as session length increases
SOLUTION: Fresh agent with clean context reviews the work.
REVIEW MODELS
Model 1: Dual-Agent Review (Recommended)
┌─────────────────────────────────────────────────────────────┐
│ DUAL-AGENT REVIEW │
├─────────────────────────────────────────────────────────────┤
│ │
│ BUILDER AGENT REVIEWER AGENT │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ │ │ │ │
│ │ Writes code │ ───Output───► │ Reviews code │ │
│ │ Writes tests│ │ Finds issues │ │
│ │ │ ◄───Feedback── │ │ │
│ └──────────────┘ └──────────────┘ │
│ │
│ RULES: │
│ - Different session (clean context) │
│ - Can be same model or different model │
│ - Reviewer has NO knowledge of builder's reasoning │
│ - Reviewer ONLY sees the output, not the process │
└─────────────────────────────────────────────────────────────┘
Model 2: Cross-Model Review
┌─────────────────────────────────────────────────────────────┐
│ CROSS-MODEL REVIEW │
├─────────────────────────────────────────────────────────────┤
│ │
│ MODEL A (Builder) MODEL B (Reviewer) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Claude │ ───Output───► │ GPT-4 │ │
│ │ (builds) │ │ (reviews) │ │
│ └──────────────┘ └──────────────┘ │
│ │
│ BENEFITS: │
│ - Different training data = different blind spots │
│ - Catches model-specific hallucinations │
│ - Diverse perspective on code quality │
│ │
│ GOOD COMBINATIONS: │
│ - Claude builds → GPT-4 reviews │
│ - GPT-4 builds → Claude reviews │
│ - Either builds → Gemini reviews │
└─────────────────────────────────────────────────────────────┘
Model 3: Specialized Agent Review
┌─────────────────────────────────────────────────────────────┐
│ SPECIALIZED AGENT REVIEW │
├─────────────────────────────────────────────────────────────┤
│ │
│ BUILDER SECURITY QA AGENT ARCHITECT │
│ AGENT AGENT AGENT │
│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │ │───────►│ │────►│ │───►│ │ │
│ │ Writes │ │Security│ │ Tests │ │Pattern │ │
│ │ code │ │ review │ │ review │ │ review │ │
│ └────────┘ └────────┘ └────────┘ └────────┘ │
│ │
│ SPECIALIST PERSONAS: │
│ - Security Agent: XSS, SQLi, auth vulnerabilities │
│ - QA Agent: Edge cases, error handling, test coverage │
│ - Architect Agent: Patterns, dependencies, scalability │
│ - Performance Agent: Complexity, memory, efficiency │
└─────────────────────────────────────────────────────────────┘
Model 4: Automated + Agent Hybrid
┌─────────────────────────────────────────────────────────────┐
│ HYBRID REVIEW PIPELINE │
├─────────────────────────────────────────────────────────────┤
│ │
│ BUILDER AUTOMATED AGENT HUMAN │
│ AGENT CHECKS REVIEW REVIEW │
│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐│
│ │ │────►│ Lint │──────►│ │─────►│ ││
│ │ Writes │ │ Test │ │ Semantic│ │ Final ││
│ │ code │ │ Types │ │ review │ │ approval││
│ └────────┘ └────────┘ └────────┘ └────────┘│
│ │
│ AUTOMATED GATES (Must Pass): │
│ - Linting (ESLint, Pylint, etc.) │
│ - Type checking (TypeScript, MyPy, etc.) │
│ - Unit tests (all pass) │
│ - Security scan (no critical issues) │
│ │
│ AGENT REVIEW (After Automated): │
│ - Logic review │
│ - Architecture review │
│ - Edge case analysis │
│ │
│ HUMAN REVIEW (For Critical Code): │
│ - Security-sensitive code │
│ - Database migrations │
│ - Authentication changes │
└─────────────────────────────────────────────────────────────┘
REVIEWER AGENT PROMPTS
General Code Reviewer Prompt
ROLE: Senior Code Reviewer
CONTEXT: You are reviewing code written by another AI agent.
You have NO knowledge of the original requirements or the
builder's reasoning. You only see the code output.
TASK: Review the following code for:
1. CORRECTNESS
- Does the logic appear correct?
- Are there obvious bugs?
- Are edge cases handled?
2. SECURITY
- Any injection vulnerabilities?
- Any hardcoded secrets?
- Any authentication issues?
3. QUALITY
- Is the code readable?
- Are names descriptive?
- Is complexity appropriate?
4. COMPLETENESS
- Are error cases handled?
- Is validation present?
- Are tests comprehensive?
CODE TO REVIEW:
[paste code here]
OUTPUT FORMAT:
## Review Summary
- **Verdict:** APPROVE / REQUEST_CHANGES / REJECT
- **Confidence:** HIGH / MEDIUM / LOW
- **Critical Issues:** [count]
- **Warnings:** [count]
## Critical Issues (Must Fix)
1. [Issue description, file, line]
## Warnings (Should Fix)
1. [Warning description, file, line]
## Suggestions (Optional)
1. [Suggestion]
## Questions for Builder
1. [Clarifying question]
Security-Focused Reviewer Prompt
ROLE: Security Engineer (Penetration Tester mindset)
CONTEXT: Review code for security vulnerabilities.
Assume the code will be attacked. Find every weakness.
CHECKLIST:
[ ] Input validation on all user inputs
[ ] Output encoding for XSS prevention
[ ] Parameterized queries for SQL
[ ] Authentication on protected routes
[ ] Authorization checks before actions
[ ] No hardcoded secrets
[ ] Secure session handling
[ ] Rate limiting present
[ ] Error messages don't leak info
[ ] Dependencies have no known CVEs
CODE TO REVIEW:
[paste code here]
REPORT VULNERABILITIES BY SEVERITY:
- CRITICAL: Immediate exploit possible
- HIGH: Significant security risk
- MEDIUM: Should be fixed soon
- LOW: Minor security concern
Test Quality Reviewer Prompt
ROLE: QA Engineer (Test Specialist)
CONTEXT: Review tests written by another agent.
Verify tests actually test what they claim to test.
CHECKLIST:
[ ] Tests cover happy path
[ ] Tests cover error cases
[ ] Tests cover edge cases
[ ] Tests are independent (no shared state)
[ ] Tests are deterministic (no random failures)
[ ] Assertions are meaningful (not just "doesn't crash")
[ ] Mocks are appropriate (not mocking the thing being tested)
[ ] Test names describe what they test
CODE TO REVIEW:
[paste test code here]
PRODUCTION CODE BEING TESTED:
[paste production code here]
IDENTIFY:
1. Missing test cases
2. Weak assertions
3. Tests that would pass even if code is broken
4. Tests that test implementation, not behavior
Architecture Reviewer Prompt
ROLE: Software Architect
CONTEXT: Review code for architectural quality.
Ensure it fits the project patterns and is maintainable.
CHECKLIST:
[ ] Follows established patterns (see PROJECT_CONTEXT.md)
[ ] Appropriate separation of concerns
[ ] No circular dependencies
[ ] Appropriate abstraction level
[ ] Scalable design
[ ] Testable design
[ ] Error handling strategy
[ ] Logging strategy
CODE TO REVIEW:
[paste code here]
PROJECT PATTERNS (from PROJECT_CONTEXT.md):
[paste relevant patterns]
EVALUATE:
1. Pattern compliance
2. Coupling assessment
3. Cohesion assessment
4. Suggestions for improvement
REVIEW WORKFLOW
Standard Review Flow
┌─────────────────────────────────────────────────────────────┐
│ REVIEW WORKFLOW │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. BUILDER COMPLETES WORK │
│ └── Code written, tests pass, lint passes │
│ │
│ 2. AUTOMATED CHECKS (Blocking) │
│ ├── [ ] TypeScript/Lint passes │
│ ├── [ ] All tests pass │
│ ├── [ ] Security scan clean │
│ ├── [ ] Coverage threshold met │
│ ├── [ ] Regression check passes │
│ └── [ ] No failure registry patterns matched │
│ │
│ 3. PREPARE REVIEW PACKAGE │
│ ├── Git diff of changes │
│ ├── List of files modified │
│ ├── Test results summary │
│ ├── Builder's description of changes │
│ └── Regression check report │
│ │
│ 4. AGENT REVIEW (New session/different agent) │
│ ├── [ ] Logic review │
│ ├── [ ] Security review │
│ ├── [ ] Test quality review │
│ ├── [ ] Architecture review │
│ ├── [ ] Regression pattern review │
│ └── [ ] Previous fixes intact review │
│ │
│ 5. REVIEW OUTCOME │
│ ├── APPROVE → Proceed to merge │
│ ├── REQUEST_CHANGES → Return to builder │
│ └── REJECT → Escalate to human │
│ │
│ 6. IF REQUEST_CHANGES: │
│ ├── Builder addresses feedback │
│ ├── Re-run automated checks │
│ └── Re-submit for review (max 3 cycles) │
│ │
│ 7. FINAL APPROVAL │
│ └── Commit and merge │
└─────────────────────────────────────────────────────────────┘
Regression Prevention Review Checklist
REGRESSION PREVENTION CHECKLIST (for Reviewer):
Before Review:
[ ] Run: python scripts/regression_check.py --staged
[ ] Review any warnings or matches
[ ] Check if files have known bug history
During Review:
[ ] Check if changes touch files with known bugs (see failure-registry.jsonl)
[ ] Verify no regression patterns present in diff
[ ] Verify no known bad patterns are being introduced
[ ] Confirm previous fixes remain intact
[ ] Check for null checks, error handling patterns from previous bugs
For Bug Fixes:
[ ] Confirm regression test exists
[ ] Verify test naming: test_<module>_regression_<failure_id>
[ ] Verify test would fail with old code, pass with fix
[ ] Check if prevention rule should be added/updated
Post Review:
[ ] If new bug found: Ensure builder will log it to registry
[ ] If pattern emerges: Suggest prevention rule addition
Review Package Template
# Review Package: [Task Name]
## Summary
- **Builder Agent:** [Agent/Model used]
- **Session ID:** [If applicable]
- **Task:** [Brief description]
- **Files Changed:** [Count]
## Changes Overview
[Brief description of what was done]
## Files Modified
| File | Type | Lines Changed |
|------|------|---------------|
| src/components/Button.tsx | Modified | +45, -12 |
| src/components/Button.test.tsx | Created | +78 |
## Git Diff
```diff
[Paste git diff here]
Test Results
- Total tests: X
- Passed: X
- Failed: 0
- Coverage: X%
Builder Notes
[Any context the builder wants to provide]
Review Request
Please review for:
- Correctness
- Security
- Test quality
- Architecture compliance
---
## REVIEW DECISION MATRIX
### When to APPROVE
APPROVE if ALL of the following are true:
[ ] No critical issues found [ ] No security vulnerabilities found [ ] Code is readable and maintainable [ ] Tests adequately cover the changes [ ] Architecture patterns are followed [ ] No obvious bugs or logic errors
APPROVAL MESSAGE: "Reviewed and approved. No critical issues found. [Optional: Minor suggestions for future consideration]"
### When to REQUEST_CHANGES
REQUEST_CHANGES if ANY of the following are true:
[ ] Logic error that could cause incorrect behavior [ ] Missing error handling for likely scenarios [ ] Tests don't cover important cases [ ] Code violates project patterns [ ] Security concern (non-critical) [ ] Poor naming or readability issues
REQUEST_CHANGES MESSAGE: "Changes requested. Please address the following before re-review:
- [Specific issue and suggested fix]
- [Specific issue and suggested fix]"
### When to REJECT
REJECT if ANY of the following are true:
[ ] Critical security vulnerability [ ] Obvious data loss risk [ ] Fundamental misunderstanding of requirements [ ] Code is unreviewable (too complex, no structure) [ ] Three review cycles without resolution
REJECT MESSAGE: "Review rejected. This requires human intervention because: [Reason for rejection]
Recommended action: [What should happen next]"
---
## REVIEW CYCLE LIMITS
### Three Strikes Rule
REVIEW CYCLE LIMITS:
MAX_REVIEW_CYCLES = 3
Cycle 1: Initial review └── If REQUEST_CHANGES → Builder fixes and resubmits
Cycle 2: Second review └── If REQUEST_CHANGES → Builder fixes and resubmits
Cycle 3: Final review └── If REQUEST_CHANGES → ESCALATE to human └── Do NOT allow further agent-only cycles
RATIONALE:
- Prevents infinite loops of fixes and reviews
- Forces escalation when agent can't resolve issues
- Protects against hallucination loops
### Context Reset Between Cycles
CONTEXT RESET PROTOCOL:
After each review cycle:
- Start NEW reviewer session (fresh context)
- Do NOT carry forward previous review comments
- Reviewer should see ONLY:
- Current code state
- Test results
- Original requirements
- Prevents bias from previous review attempts
---
## AUTOMATION INTEGRATION
### GitHub Actions Review Gate
```yaml
# .github/workflows/agent-review.yml
name: Agent Review Gate
on:
pull_request:
types: [opened, synchronize]
jobs:
automated-checks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Lint
run: npm run lint
- name: Type Check
run: npm run typecheck
- name: Test
run: npm test
- name: Security Scan
run: npm audit --audit-level=high
agent-review:
needs: automated-checks
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Prepare Review Package
run: |
echo "## Review Package" > review-package.md
echo "### Files Changed" >> review-package.md
git diff --name-only ${{ github.event.pull_request.base.sha }} >> review-package.md
echo "### Diff" >> review-package.md
git diff ${{ github.event.pull_request.base.sha }} >> review-package.md
- name: Request Agent Review
# This would call your agent review API/service
run: |
# Example: Send to review agent endpoint
curl -X POST ${{ secrets.AGENT_REVIEW_URL }} \
-H "Authorization: Bearer ${{ secrets.AGENT_REVIEW_TOKEN }}" \
-d @review-package.md
- name: Check Review Result
run: |
# Check if review passed
if [ "$REVIEW_RESULT" != "APPROVED" ]; then
echo "Agent review did not approve changes"
exit 1
fi
QUICK REFERENCE
+------------------------------------------------------------------+
| AGENT REVIEW PROTOCOL QUICK REFERENCE |
+------------------------------------------------------------------+
| MODELS: |
| 1. Dual-Agent: Builder → Reviewer (different session) |
| 2. Cross-Model: Claude → GPT-4 (or vice versa) |
| 3. Specialized: Builder → Security → QA → Architect |
| 4. Hybrid: Automated → Agent → Human |
+------------------------------------------------------------------+
| REVIEW FLOW: |
| 1. Builder completes work |
| 2. Automated checks pass (blocking) |
| 3. Prepare review package (diff, tests, description) |
| 4. Agent review (new session) |
| 5. APPROVE / REQUEST_CHANGES / REJECT |
+------------------------------------------------------------------+
| LIMITS: |
| - MAX_REVIEW_CYCLES = 3 |
| - After 3 cycles → Escalate to human |
| - Fresh context for each review cycle |
+------------------------------------------------------------------+
| VERDICTS: |
| APPROVE: No critical issues, proceed to merge |
| REQUEST_CHANGES: Fixable issues, return to builder |
| REJECT: Critical issues, escalate to human |
+------------------------------------------------------------------+
| KEY RULE: The agent that builds ≠ the agent that validates |
+------------------------------------------------------------------+
Authored by: TheArchitectit Document Owner: Project Maintainers Last Updated: 2026-01-21 Line Count: ~450