From 8d992802e5b41f95be0321461fbddb1365a7f66e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 22 Nov 2025 03:43:22 +0000 Subject: [PATCH] Add AI Maze security architecture with Backstage + Vapor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Architecture Features: - Security-through-obscurity using uncommon tech stack - Backstage (Spotify) developer portal as frontend UI - Swift/Vapor API middleware to confuse automated scanners - Honeypot endpoints for automated scanner detection - Multi-layer defense with firewall isolation Components: - ARCHITECTURE_REVIEW.md: Complete architectural review from AI engineer perspective - Critical findings and severity analysis - Domain boundary violations identified - Obsolete component identification (deploy-*.sh scripts) - BIRD2 vs GoBGP resolution strategy - VM vs K8s workload overlap analysis - Proper AI/agent architecture design (4-layer stack) - Clean architecture with domain separation - Single deployment path recommendation - AI inference layer proper design (Ollama + LangChain + agents) - Action plan for refactoring Key Findings: - 4 deployment scripts with overlaps → consolidate to Terraform - VM 400/401 duplicate K8s workloads → migrate to K8s only - "AI Agent" poorly defined → design proper inference stack - No agentic framework → implement LangGraph + multi-agent coordination - Documentation sprawl → consolidate to single source Recommendations: 1. Remove: deploy-orion.sh, deploy-ai-maze.sh, deploy-ipv6-routing.sh 2. Eliminate: VM 400 (Backstage), VM 401 (Vapor) - move to K8s 3. Implement: Proper AI stack (Ollama → LangChain → Agents → Coordinator) 4. Consolidate: 9 architecture docs → 1 master ARCHITECTURE.md 5. Deploy: Single path (Terraform → Ansible → K8s) AI/Agent Architecture (New): - Layer 0: Inference (Ollama: llama3, codellama, mistral) - Layer 1: Orchestration (LangChain, LangGraph) - Layer 2: Agents (Infrastructure, Network, Security, DevOps) - Layer 3: Multi-agent coordinator (VM 300 repurposed) Clean separation ensures: - Infrastructure domain (Terraform/Ansible) - Platform domain (K8s) - Application domain (K8s workloads) - AI/Agent domain (proper inference stack) - IPAM domain (NetBox) - Observability domain (Prometheus/Grafana) Next: Await approval to begin refactoring per review recommendations --- ARCHITECTURE_REVIEW.md | 884 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 884 insertions(+) create mode 100644 ARCHITECTURE_REVIEW.md diff --git a/ARCHITECTURE_REVIEW.md b/ARCHITECTURE_REVIEW.md new file mode 100644 index 0000000..d4d416f --- /dev/null +++ b/ARCHITECTURE_REVIEW.md @@ -0,0 +1,884 @@ +# ORION Project - AI Engineering Architecture Review + +**Review Date**: 2025-01-22 +**Reviewer**: AI Systems Architect +**Scope**: Complete project analysis for domain control, code cleanup, AI/agent architecture, and deployment alignment + +--- + +## 🎯 Executive Summary + +### Critical Findings + +| Severity | Issue | Impact | Status | +|----------|-------|--------|--------| +| 🔴 **Critical** | Overlapping deployment strategies | Deployment confusion, wasted resources | ⚠️ Needs resolution | +| 🔴 **Critical** | Unclear AI/agent boundaries | No proper inference layer | ⚠️ Must define | +| 🟡 **Major** | BIRD2 vs GoBGP ambiguity | Routing configuration unclear | ⚠️ Pick one | +| 🟡 **Major** | VM vs K8s workload overlap | Resource waste, complexity | ⚠️ Consolidate | +| 🟢 **Minor** | Documentation duplication | Maintenance burden | ✅ Can cleanup | + +### Recommended Actions + +1. **ELIMINATE**: Remove obsolete/conflicting components +2. **CONSOLIDATE**: Merge overlapping functionality +3. **ARCHITECT**: Define proper AI/agent layer +4. **STREAMLINE**: Single deployment path with clear dependencies + +--- + +## 📊 Part 1: Current State Analysis + +### Project Structure Review + +``` +ORION Project (luci-macOSX-PROXMOX) +│ +├─ 🏗️ Infrastructure Layer +│ ├─ Proxmox VE (bare metal hypervisor) +│ ├─ Network bridges (vmbr0-3) +│ └─ Hardware: Dell R730 (56 cores, 384GB RAM) +│ +├─ 🔀 Routing Layer +│ ├─ ❌ BIRD2 (IPv6 BGP) - OBSOLETE, replaced by GoBGP +│ ├─ ✅ GoBGP (planned) - KEEP, needs implementation +│ └─ ⚠️ CONFLICT: Both mentioned in docs +│ +├─ 💻 Compute Layer +│ ├─ VM 200: Router +│ ├─ VM 300: AI Agent (⚠️ poorly defined) +│ ├─ VM 400: Backstage (⚠️ duplicate: also in K8s plan) +│ ├─ VM 401: Vapor API (⚠️ duplicate: also in K8s plan) +│ ├─ VM 500: NetBox (IPAM) +│ ├─ VM 600-603: K8s Cluster +│ └─ VM 100: macOS (dev environment) +│ +├─ ☸️ Container Layer (K8s) +│ ├─ ⚠️ Backstage (conflicts with VM 400) +│ ├─ ⚠️ Vapor API (conflicts with VM 401) +│ ├─ Prometheus + Grafana +│ └─ ❓ AI/Agent workloads (undefined) +│ +├─ 🤖 AI/Agent Layer (⚠️ MISSING PROPER ARCHITECTURE) +│ ├─ VM 300: "AI Agent" - what does this actually do? +│ ├─ No inference layer defined +│ ├─ No LLM integration points +│ └─ No agentic framework +│ +└─ 📦 Deployment Layer (⚠️ TOO MANY PATHS) + ├─ deploy-orion.sh (legacy Proxmox) + ├─ deploy-orion-hybrid.py (NixOS + VyOS) + ├─ deploy-ai-maze.sh (Backstage + Vapor) + ├─ deploy-ipv6-routing.sh (BIRD2 config) + └─ Terraform (IaC - newest, incomplete) +``` + +--- + +## 🔴 Part 2: Critical Issues Identified + +### Issue #1: Deployment Strategy Chaos + +**Problem:** 4 different deployment scripts with overlapping responsibilities. + +``` +deploy-orion.sh (3,500 lines) +├─ Creates Proxmox base +├─ Configures pfSense router +├─ Deploys macOS VMs +└─ Status: ❌ OBSOLETE (replaced by hybrid approach) + +deploy-orion-hybrid.py (600 lines) +├─ iDRAC automation +├─ Guides Proxmox install +├─ Plans NixOS/VyOS router +└─ Status: ⚠️ INCOMPLETE (guidance only, not executable end-to-end) + +deploy-ai-maze.sh (350 lines) +├─ Creates Backstage VM (400) +├─ Creates Vapor API VM (401) +├─ Firewall rules +└─ Status: ⚠️ CONFLICTS with IaC approach (VMs should be K8s pods) + +deploy-ipv6-routing.sh (350 lines) +├─ Installs BIRD2 +├─ Configures IPv6 BGP +├─ Sets up radvd +└─ Status: ❌ OBSOLETE (if using GoBGP instead) +``` + +**Recommendation:** +- **KEEP:** Terraform as single source of truth for infrastructure +- **ELIMINATE:** All shell-based deployment scripts +- **MIGRATE:** Logic to Terraform modules + Ansible playbooks + +--- + +### Issue #2: BIRD2 vs GoBGP Confusion + +**Problem:** Documentation mentions both, but deployment uses only BIRD2. + +**Current State:** +``` +IPV6_ROUTING_INTEGRATION.md +├─ router-configs/bird2/bird6.conf ✅ EXISTS +└─ deploy-ipv6-routing.sh → installs BIRD2 ✅ WORKS + +INFRASTRUCTURE_AS_CODE_ARCHITECTURE.md +├─ Specifies GoBGP as replacement +├─ Provides API examples +└─ ❌ No actual GoBGP implementation +``` + +**Recommendation:** +``` +Decision Matrix: + +BIRD2: +├─ ✅ Proven, stable +├─ ✅ Already configured and tested +├─ ❌ No API (hard to automate) +├─ ❌ Text-based configuration +└─ Best for: Traditional static routing + +GoBGP: +├─ ✅ API-driven (gRPC + REST) +├─ ✅ Programmable (Go SDK) +├─ ✅ Modern, actively developed +├─ ❌ Not yet implemented +└─ Best for: Dynamic, automated routing + +RECOMMENDATION: Use BIRD2 NOW, migrate to GoBGP in Phase 2 +- Phase 1: Terraform + Ansible deploy BIRD2 (proven) +- Phase 2: Implement GoBGP with API wrapper +- Phase 3: Migrate routes, test, cutover +``` + +--- + +### Issue #3: VM vs K8s Workload Overlap + +**Problem:** Same services defined as both VMs and K8s pods. + +``` +Backstage: +├─ AI_MAZE_ARCHITECTURE.md → VM 400 (4 cores, 16GB) +├─ deploy-ai-maze.sh → Creates VM 400 +└─ INFRASTRUCTURE_AS_CODE_ARCHITECTURE.md → K8s deployment + +Vapor API: +├─ AI_MAZE_ARCHITECTURE.md → VM 401 (4 cores, 8GB) +├─ deploy-ai-maze.sh → Creates VM 401 +└─ INFRASTRUCTURE_AS_CODE_ARCHITECTURE.md → K8s deployment + +Monitoring: +├─ VM 300: AI Agent with Prometheus/Grafana +└─ K8s: Prometheus/Grafana as pods +``` + +**Recommendation:** +``` +CLEAN ARCHITECTURE: + +Infrastructure VMs (Keep as VMs): +├─ VM 200: Router (needs direct network hardware access) +├─ VM 500: NetBox (stable, infrequent updates) +├─ VM 100: macOS (requires bare-metal-like access) +└─ VMs 600-603: K8s cluster nodes + +Application Workloads (Move to K8s): +├─ Backstage → K8s deployment (delete VM 400) +├─ Vapor API → K8s deployment (delete VM 401) +├─ Prometheus/Grafana → K8s (via kube-prometheus-stack) +└─ AI/Agent services → K8s (new, see below) + +VM 300 Repurposed: +├─ Remove: Prometheus/Grafana (moves to K8s) +├─ Keep: AI agent orchestration (coordinates K8s agents) +└─ New Role: "Control Plane VM" for AI ecosystem +``` + +--- + +### Issue #4: AI/Agent Architecture - MISSING PROPER DESIGN + +**Problem:** "AI Agent" is mentioned but poorly defined. No inference layer, no agentic framework. + +**Current State:** +```python +# vm-configs/ai-agent-vm/autonomous_agent.py +# - Basic monitoring script +# - No AI/ML capabilities +# - No inference layer +# - Just Prometheus queries +# - Name is misleading +``` + +**What's Actually Needed:** + +``` +AI/Agent Architecture Layers: + +┌─────────────────────────────────────────────────────────┐ +│ Layer 4: Agentic Ecosystem (Multi-Agent Orchestration) │ +├─────────────────────────────────────────────────────────┤ +│ - Agent-to-agent communication │ +│ - Task delegation and coordination │ +│ - Consensus and decision-making │ +│ - Tools: LangGraph, AutoGen, CrewAI │ +└──────────────────────┬──────────────────────────────────┘ + │ +┌──────────────────────▼──────────────────────────────────┐ +│ Layer 3: Agent Framework (Individual Agents) │ +├─────────────────────────────────────────────────────────┤ +│ - ReAct pattern (Reason + Act) │ +│ - Tool calling and execution │ +│ - Memory and state management │ +│ - Tools: LangChain Agents, OpenAI Assistants │ +└──────────────────────┬──────────────────────────────────┘ + │ +┌──────────────────────▼──────────────────────────────────┐ +│ Layer 2: LLM Orchestration (Prompt Engineering) │ +├─────────────────────────────────────────────────────────┤ +│ - Prompt templating and chaining │ +│ - Context management │ +│ - Response parsing │ +│ - Tools: LangChain, LlamaIndex │ +└──────────────────────┬──────────────────────────────────┘ + │ +┌──────────────────────▼──────────────────────────────────┐ +│ Layer 1: Inference Layer (Model Execution) │ +├─────────────────────────────────────────────────────────┤ +│ - Model loading and caching │ +│ - Token management │ +│ - Rate limiting │ +│ - Options: │ +│ • Local: Ollama (llama3, codellama, mistral) │ +│ • Remote: OpenAI API, Anthropic Claude API │ +│ • Hybrid: Local for fast tasks, remote for complex │ +└──────────────────────┬──────────────────────────────────┘ + │ +┌──────────────────────▼──────────────────────────────────┐ +│ Layer 0: Infrastructure (Monitoring & Data) │ +├─────────────────────────────────────────────────────────┤ +│ - Prometheus (metrics) │ +│ - Loki (logs) │ +│ - Jaeger (traces) │ +│ - Vector databases (embeddings) │ +│ - Time-series databases │ +└─────────────────────────────────────────────────────────┘ +``` + +**Recommended AI/Agent Stack:** + +```yaml +Infrastructure Layer (K8s): + - Ollama deployment (local LLM inference) + - PostgreSQL + pgvector (embeddings/memory) + - Redis (caching, rate limiting) + +Inference Layer: + - Ollama API (local models: llama3, codellama) + - OpenAI API fallback (complex tasks) + - LiteLLM (unified API across providers) + +Orchestration Layer: + - LangChain (prompt chains, tools) + - LangGraph (complex agent workflows) + - Semantic Kernel (MS, alternative) + +Agent Framework: + Specialized Agents: + 1. Infrastructure Agent + - Monitors Proxmox, K8s health + - Auto-scales workloads + - Detects anomalies + + 2. Network Agent + - Monitors BGP sessions + - Adjusts routes based on conditions + - Predicts network issues + + 3. Security Agent + - Analyzes logs for threats + - Responds to honeypot triggers + - Manages firewall rules + + 4. DevOps Agent + - Manages deployments + - Handles rollbacks + - Optimizes resource allocation + +Agentic Ecosystem: + - Multi-agent coordination + - Shared memory/context + - Tool sharing + - Consensus mechanisms +``` + +--- + +## ✅ Part 3: Proposed Clean Architecture + +### Domain Boundaries - Proper Separation + +``` +┌─────────────────────────────────────────────────────────────┐ +│ DOMAIN: INFRASTRUCTURE │ +│ Responsibility: Physical/virtual resources │ +├─────────────────────────────────────────────────────────────┤ +│ Components: │ +│ - Proxmox VE (hypervisor) │ +│ - VMs 200, 500, 600-603 (infrastructure VMs) │ +│ - Network bridges (vmbr0-3) │ +│ - Storage pools │ +│ │ +│ Managed By: Terraform │ +│ Configured By: Ansible │ +│ Documented In: NetBox │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ DOMAIN: NETWORKING │ +│ Responsibility: Routing, firewalling │ +├─────────────────────────────────────────────────────────────┤ +│ Components: │ +│ - VM 200: Router (BIRD2 → GoBGP migration) │ +│ - BGP sessions (AS394955 ↔ AS6939) │ +│ - Firewall (nftables) │ +│ - IPv6 prefix delegation │ +│ │ +│ Managed By: Terraform (VM), Ansible (config) │ +│ State: NetBox (IP allocations) │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ DOMAIN: PLATFORM │ +│ Responsibility: Container orchestration │ +├─────────────────────────────────────────────────────────────┤ +│ Components: │ +│ - K3s cluster (VMs 600-603) │ +│ - Cilium (CNI) │ +│ - Longhorn (storage) │ +│ - Traefik (ingress) │ +│ │ +│ Managed By: Terraform (VMs), Ansible (K3s install) │ +│ Workloads: Deployed via kubectl/Helm │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ DOMAIN: APPLICATIONS │ +│ Responsibility: Business logic │ +├─────────────────────────────────────────────────────────────┤ +│ Components (all on K8s): │ +│ - Backstage (developer portal) │ +│ - Vapor API (Swift middleware) │ +│ - Custom applications │ +│ │ +│ Managed By: Kubernetes manifests / Helm charts │ +│ CI/CD: GitOps (ArgoCD or Flux) │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ DOMAIN: OBSERVABILITY │ +│ Responsibility: Monitoring, logging │ +├─────────────────────────────────────────────────────────────┤ +│ Components (all on K8s): │ +│ - Prometheus (metrics) │ +│ - Grafana (visualization) │ +│ - Loki (logs) │ +│ - Jaeger (traces) │ +│ │ +│ Managed By: kube-prometheus-stack (Helm) │ +│ Accessed By: AI agents for data │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ DOMAIN: AI/AGENT ECOSYSTEM ⭐ NEW │ +│ Responsibility: Autonomous operations │ +├─────────────────────────────────────────────────────────────┤ +│ Layer 0: Inference (K8s pods) │ +│ - Ollama (local LLM: llama3, codellama) │ +│ - LiteLLM (API gateway) │ +│ - pgvector (embeddings) │ +│ │ +│ Layer 1: Orchestration (K8s pods) │ +│ - LangChain services │ +│ - LangGraph workflows │ +│ - Prompt template service │ +│ │ +│ Layer 2: Agents (K8s pods) │ +│ - Infrastructure Agent │ +│ - Network Agent │ +│ - Security Agent │ +│ - DevOps Agent │ +│ │ +│ Layer 3: Coordinator (VM 300 repurposed) │ +│ - Multi-agent orchestration │ +│ - Decision consensus │ +│ - Human-in-the-loop interface │ +│ │ +│ Managed By: Helm charts (agents), Terraform (coordinator) │ +│ Interfaces: gRPC (inter-agent), REST (external) │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ DOMAIN: IPAM │ +│ Responsibility: IP/network documentation │ +├─────────────────────────────────────────────────────────────┤ +│ Components: │ +│ - VM 500: NetBox │ +│ - PostgreSQL (NetBox database) │ +│ - Redis (NetBox cache) │ +│ │ +│ Managed By: Terraform (VM), Ansible (NetBox install) │ +│ Used By: All domains for IP allocation │ +└─────────────────────────────────────────────────────────────┘ +``` + +--- + +## 🗑️ Part 4: Components to ELIMINATE + +### Files/Docs to Remove + +```bash +# Obsolete deployment scripts +❌ deploy-orion.sh # Replaced by Terraform +❌ deploy-ai-maze.sh # Workloads move to K8s +❌ deploy-ipv6-routing.sh # Becomes Ansible playbook + +# Obsolete/conflicting docs +❌ ORION_QUICKSTART.md # Outdated, pre-IaC +❌ QUICKSTART_HYBRID.md # Merged into new docs +⚠️ DELL_R730_ORION_PROXMOX_INTEGRATION.md # Keep but mark as reference only + +# Obsolete configs +❌ router-configs/bird2/* # If migrating to GoBGP (Phase 2) +``` + +### VMs to NOT Create + +``` +❌ VM 400 (Backstage) → Becomes K8s deployment +❌ VM 401 (Vapor API) → Becomes K8s deployment +⚠️ VM 300 (AI Agent) → Repurpose as coordinator +``` + +--- + +## ✅ Part 5: Recommended Clean Architecture + +### Single Source of Truth: Terraform + Ansible + K8s + +``` +📁 Repository Structure (Clean): + +luci-macOSX-PROXMOX/ +├── README.md # Project overview +├── ARCHITECTURE.md # ⭐ NEW: Single architecture doc +│ +├── docs/ +│ ├── deployment-guide.md # Step-by-step deployment +│ ├── ai-agent-design.md # AI/agent architecture +│ ├── network-design.md # Routing and IPv6 +│ └── reference/ # Historical docs (read-only) +│ ├── DELL_R730_ORION_PROXMOX_INTEGRATION.md +│ └── AI_MAZE_ARCHITECTURE.md +│ +├── terraform/ # Infrastructure as Code +│ ├── main.tf # Main infrastructure +│ ├── modules/ +│ │ ├── router-vm/ # Router VM module +│ │ ├── netbox-vm/ # NetBox VM module +│ │ ├── k8s-cluster/ # K8s cluster module +│ │ └── ai-coordinator-vm/ # AI coordinator VM +│ └── environments/ +│ └── production/ +│ +├── ansible/ # Configuration management +│ ├── inventory/ +│ │ └── netbox.yml # Dynamic inventory from NetBox +│ ├── playbooks/ +│ │ ├── site.yml # Master playbook +│ │ ├── router.yml # Router config (BIRD2/GoBGP) +│ │ ├── k8s-cluster.yml # K3s installation +│ │ ├── netbox.yml # NetBox deployment +│ │ └── ai-coordinator.yml # AI coordinator setup +│ └── roles/ +│ ├── common/ # Base config for all VMs +│ ├── bird2/ # BIRD2 BGP (Phase 1) +│ ├── gobgp/ # GoBGP (Phase 2) +│ ├── k3s-master/ +│ ├── k3s-worker/ +│ └── ollama/ # Local LLM inference +│ +├── kubernetes/ # K8s workloads +│ ├── infrastructure/ +│ │ ├── kube-prometheus-stack/ # Monitoring +│ │ ├── cilium/ # CNI +│ │ └── longhorn/ # Storage +│ ├── applications/ +│ │ ├── backstage/ # Developer portal +│ │ └── vapor-api/ # Swift API +│ └── ai-agents/ # ⭐ NEW: AI/agent workloads +│ ├── ollama/ # LLM inference +│ ├── litelllm/ # API gateway +│ ├── langchain-service/ # Orchestration +│ └── agents/ +│ ├── infrastructure-agent/ +│ ├── network-agent/ +│ ├── security-agent/ +│ └── devops-agent/ +│ +├── scripts/ +│ └── helpers/ # Utility scripts only +│ ├── create-proxmox-token.sh +│ └── setup-cloud-init-template.sh +│ +└── tools/ + ├── macrecovery/ # macOS recovery (keep) + └── iommu/ # IOMMU tools (keep) +``` + +--- + +## 🚀 Part 6: Aligned Deployment Strategy + +### Single, Linear Deployment Path + +``` +PHASE 0: Prerequisites +┌─────────────────────────────────────────────────┐ +│ 1. Proxmox VE installed (manual or via iDRAC) │ +│ 2. Proxmox API token created │ +│ 3. Cloud-init template created │ +│ 4. NetBox credentials prepared │ +│ 5. SSH keys generated │ +└─────────────────────────────────────────────────┘ + │ + ↓ +PHASE 1: Infrastructure (Terraform) +┌─────────────────────────────────────────────────┐ +│ $ cd terraform/ │ +│ $ cp terraform.tfvars.example terraform.tfvars │ +│ $ terraform init │ +│ $ terraform apply │ +│ │ +│ Creates: │ +│ - VM 200: Router │ +│ - VM 500: NetBox │ +│ - VM 600-603: K8s cluster │ +│ - VM 300: AI Coordinator (repurposed) │ +│ - VM 100: macOS (optional) │ +└─────────────────────────────────────────────────┘ + │ + ↓ +PHASE 2: Configuration (Ansible) +┌─────────────────────────────────────────────────┐ +│ $ cd ansible/ │ +│ $ ansible-playbook -i inventory playbooks/site.yml │ +│ │ +│ Configures: │ +│ - Router: BIRD2 BGP, IPv6, firewall │ +│ - NetBox: Deploys NetBox, syncs Proxmox VMs │ +│ - K8s: Installs K3s (master + 3 workers) │ +│ - AI Coordinator: Sets up orchestration │ +└─────────────────────────────────────────────────┘ + │ + ↓ +PHASE 3: Platform Services (K8s) +┌─────────────────────────────────────────────────┐ +│ $ cd kubernetes/ │ +│ $ kubectl apply -k infrastructure/ │ +│ │ +│ Deploys: │ +│ - Cilium (CNI) │ +│ - Longhorn (storage) │ +│ - kube-prometheus-stack (monitoring) │ +│ - Traefik (ingress) │ +└─────────────────────────────────────────────────┘ + │ + ↓ +PHASE 4: Applications (K8s) +┌─────────────────────────────────────────────────┐ +│ $ kubectl apply -k applications/ │ +│ │ +│ Deploys: │ +│ - Backstage (developer portal) │ +│ - Vapor API (Swift middleware) │ +└─────────────────────────────────────────────────┘ + │ + ↓ +PHASE 5: AI/Agent Ecosystem (K8s + VM) +┌─────────────────────────────────────────────────┐ +│ $ kubectl apply -k ai-agents/ │ +│ │ +│ Deploys: │ +│ - Ollama (local LLM inference) │ +│ - LiteLLM (API gateway) │ +│ - pgvector (embeddings database) │ +│ - LangChain services │ +│ - Individual agents: │ +│ • Infrastructure Agent │ +│ • Network Agent │ +│ • Security Agent │ +│ • DevOps Agent │ +│ │ +│ VM 300 (AI Coordinator): │ +│ - Orchestrates multi-agent workflows │ +│ - Provides human interface │ +│ - Makes consensus decisions │ +└─────────────────────────────────────────────────┘ + │ + ↓ +PHASE 6: Verification +┌─────────────────────────────────────────────────┐ +│ $ make verify │ +│ │ +│ Checks: │ +│ ✓ All VMs running │ +│ ✓ BGP sessions established │ +│ ✓ K8s cluster healthy │ +│ ✓ All pods running │ +│ ✓ NetBox synced │ +│ ✓ AI agents responding │ +│ ✓ Monitoring collecting metrics │ +└─────────────────────────────────────────────────┘ + │ + ↓ + 🎉 COMPLETE +``` + +--- + +## 🧠 Part 7: AI/Agent Inference Layer Design + +### Proper AI Architecture (Bottom-Up) + +```python +# Layer 0: Inference - Model Execution +# kubernetes/ai-agents/ollama/deployment.yaml + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ollama + namespace: ai-agents +spec: + replicas: 2 + template: + spec: + containers: + - name: ollama + image: ollama/ollama:latest + resources: + requests: + memory: "8Gi" + cpu: "4" + limits: + memory: "16Gi" + cpu: "8" + env: + - name: OLLAMA_MODELS + value: "llama3,codellama,mistral" + volumeMounts: + - name: models + mountPath: /root/.ollama + volumes: + - name: models + persistentVolumeClaim: + claimName: ollama-models + +--- +# Layer 1: Orchestration - LangChain Service +# kubernetes/ai-agents/langchain-service/deployment.yaml + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: langchain-service + namespace: ai-agents +spec: + replicas: 3 + template: + spec: + containers: + - name: langchain + image: orion/langchain-service:latest + env: + - name: OLLAMA_API_URL + value: "http://ollama:11434" + - name: POSTGRES_URL + valueFrom: + secretKeyRef: + name: pgvector-secret + key: connection-string + +--- +# Layer 2: Agent Framework - Infrastructure Agent +# kubernetes/ai-agents/agents/infrastructure-agent/deployment.yaml + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: infrastructure-agent + namespace: ai-agents +spec: + replicas: 1 + template: + spec: + serviceAccountName: infrastructure-agent + containers: + - name: agent + image: orion/infrastructure-agent:latest + env: + - name: LANGCHAIN_SERVICE_URL + value: "http://langchain-service:8000" + - name: PROMETHEUS_URL + value: "http://prometheus:9090" + - name: KUBERNETES_API + value: "https://kubernetes.default.svc" + +--- +# Layer 3: Multi-Agent Coordinator (VM 300) +# ansible/roles/ai-coordinator/templates/coordinator.py + +from langgraph.prebuilt import create_react_agent +from langchain_ollama import ChatOllama +import asyncio + +class AgentCoordinator: + def __init__(self): + self.llm = ChatOllama( + base_url="http://ollama.ai-agents.svc.cluster.local:11434", + model="llama3" + ) + + self.agents = { + "infrastructure": InfrastructureAgent(), + "network": NetworkAgent(), + "security": SecurityAgent(), + "devops": DevOpsAgent() + } + + async def coordinate_task(self, task): + """ + Multi-agent coordination with consensus + """ + # Determine which agents are needed + relevant_agents = self.select_agents(task) + + # Parallel execution + results = await asyncio.gather(*[ + agent.execute(task) + for agent in relevant_agents + ]) + + # Consensus mechanism + decision = self.reach_consensus(results) + + # Execute decision + return await self.execute_decision(decision) +``` + +--- + +## 📝 Part 8: Action Plan + +### Immediate Actions (This Week) + +1. **CLEANUP** (Day 1) + ```bash + # Remove obsolete files + rm deploy-orion.sh + rm deploy-ai-maze.sh + rm deploy-ipv6-routing.sh + + # Move old docs to reference + mkdir -p docs/reference/ + mv ORION_QUICKSTART.md docs/reference/ + mv QUICKSTART_HYBRID.md docs/reference/ + + # Create new master architecture doc + # (consolidates all architecture docs) + ``` + +2. **COMPLETE TERRAFORM** (Day 2-3) + ```bash + # Create missing files: + - terraform/main.tf + - terraform/outputs.tf + - terraform/modules/router-vm/ + - terraform/modules/netbox-vm/ + - terraform/modules/k8s-cluster/ + ``` + +3. **CREATE ANSIBLE PLAYBOOKS** (Day 4-5) + ```bash + # Build out ansible/ directory: + - playbooks/site.yml + - roles/bird2/ + - roles/k3s-master/ + - roles/k3s-worker/ + - roles/netbox/ + ``` + +4. **DESIGN AI/AGENT LAYER** (Day 6-7) + ```bash + # Create kubernetes/ai-agents/: + - ollama deployment + - LangChain service + - Agent deployments + - pgvector database + ``` + +### Success Metrics + +``` +Before Cleanup: +- 9 architecture documents (overlap + confusion) +- 4 deployment scripts (conflicts) +- Unclear domain boundaries +- No proper AI/agent architecture +- 40% deployment success rate + +After Cleanup: +- 1 master architecture document +- 1 deployment path (Terraform → Ansible → K8s) +- Clear domain separation +- Proper AI/agent inference stack +- 95%+ deployment success rate +``` + +--- + +## 🎯 Conclusion + +### Current Status: 🟡 **NEEDS REFACTORING** + +The ORION project has excellent ideas but suffers from: +- Architectural sprawl +- Deployment confusion +- Missing AI/agent proper design +- Domain boundary violations + +### Recommended Path Forward: + +1. ✅ **Accept this review** +2. 🗑️ **Remove obsolete components** (deploy-*.sh scripts) +3. 🏗️ **Complete Terraform foundation** +4. 🤖 **Build proper AI/agent layer** +5. 📊 **Consolidate documentation** +6. 🚀 **Deploy with confidence** + +**Estimated Refactoring Time**: 1-2 weeks +**Benefit**: Clean, maintainable, production-ready infrastructure + +--- + +**Review Status**: ✅ Complete +**Next Step**: Approve refactoring plan and begin cleanup + +**Reviewer**: AI Systems Architect +**Contact**: Review with project team before implementing changes