RomanNum3ral
/
Cybersecurity-Projects
mirror of https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Cybersecurity-Projects/PROJECTS/advanced/haskell-reverse-proxy/docs/proxy-haskell-research/deployment.md

1885 lines
45 KiB
Markdown
Raw Blame History

# Docker & Kubernetes Deployment Guide for Ᾰenebris Reverse Proxy
**Production-Grade Deployment Strategy for High-Performance Haskell Applications**
---
## Executive Summary
This comprehensive guide provides battle-tested strategies for deploying Ᾰenebris, a high-performance Haskell-based reverse proxy, on Kubernetes. With a target throughput of 100k+ requests/second and support for TLS termination and WebSockets, this deployment architecture prioritizes **performance, security, and operational excellence**.
**Key achievements with this approach:**
- Docker images under 50MB (achieving 10-30MB for typical builds)
- Zero-downtime deployments with graceful connection draining
- Automated TLS certificate management
- Production-ready secrets management
- Horizontal autoscaling for traffic spikes
---
## 1. Multi-Stage Docker Builds for Haskell Applications
### Overview
Multi-stage Docker builds separate compilation from runtime, dramatically reducing final image size while maintaining optimal build caching. For Haskell applications, this approach is critical because GHC and build dependencies can exceed 2GB, while the runtime binary needs only 5-50MB.
### Three-Stage Build Pattern
The optimal pattern for Haskell reverse proxies uses three distinct stages:
1. **Dependencies stage**: Builds only dependencies (cached separately)
2. **Build stage**: Compiles the application
3. **Runtime stage**: Minimal image with just the binary
### Production-Ready Dockerfile for Ᾰenebris
```dockerfile
# syntax=docker/dockerfile:1
###############################################################################
# Stage 1: Dependency Cache (rebuilt only when dependencies change)
###############################################################################
FROM haskell:9.4-slim as dependencies
WORKDIR /build
# Install build-time system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
libgmp-dev \
zlib1g-dev \
libssl-dev \
curl \
git \
&& rm -rf /var/lib/apt/lists/*
# Copy only dependency manifests for optimal caching
COPY aenebris.cabal cabal.project cabal.project.freeze /build/
# Build dependencies only (this layer is cached)
RUN cabal update && \
cabal build --only-dependencies --enable-tests --enable-benchmarks
###############################################################################
# Stage 2: Application Build with Aggressive Optimizations
###############################################################################
FROM dependencies as builder
# Copy source code
COPY . /build/
# Build with size and performance optimizations
RUN cabal build \
--ghc-options="-O2 -split-sections -optc-Os -funbox-strict-fields -fllvm" \
--gcc-options="-Os -ffunction-sections -fdata-sections" \
--ld-options="-Wl,--gc-sections"
# Extract and optimize binary
RUN mkdir -p /output && \
cp $(cabal exec -- which aenebris) /output/aenebris && \
strip --strip-all /output/aenebris
# Verify binary size
RUN ls -lh /output/aenebris
###############################################################################
# Stage 3: Minimal Runtime Image (Production)
###############################################################################
FROM debian:12-slim as runtime
# Install only essential runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
libgmp10 \
zlib1g \
libssl3 \
curl \
&& rm -rf /var/lib/apt/lists/*
# Create non-root user for security
RUN useradd -m -u 1000 -s /bin/bash aenebris
# Copy binary from builder
COPY --from=builder /output/aenebris /usr/local/bin/aenebris
# Set ownership and permissions
RUN chown aenebris:aenebris /usr/local/bin/aenebris && \
chmod +x /usr/local/bin/aenebris
# Switch to non-root user
USER aenebris
# Health check endpoint
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8080/healthz || exit 1
EXPOSE 8080 8443
# Configure Haskell RTS for production
ENTRYPOINT ["/usr/local/bin/aenebris"]
CMD ["+RTS", "-M1800M", "-N4", "-A32M", "-qg", "-I0", "-T", "-RTS"]
###############################################################################
# Alternative: Ultra-Minimal Alpine Runtime (<20MB)
###############################################################################
FROM alpine:3.18 as runtime-alpine
RUN apk add --no-cache \
gmp \
libgcc \
libssl3 \
ca-certificates \
curl \
&& adduser -D -u 1000 aenebris
COPY --from=builder /output/aenebris /usr/local/bin/aenebris
USER aenebris
EXPOSE 8080 8443
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8080/healthz || exit 1
ENTRYPOINT ["/usr/local/bin/aenebris"]
CMD ["+RTS", "-M1800M", "-N4", "-A32M", "-qg", "-I0", "-T", "-RTS"]
```
### Build Script with Caching
```bash
#!/bin/bash
set -euo pipefail
# Enable Docker BuildKit for advanced caching
export DOCKER_BUILDKIT=1
APP_NAME="aenebris"
VERSION="${1:-latest}"
REGISTRY="${REGISTRY:-ghcr.io/yourorg}"
echo "Building ${APP_NAME}:${VERSION}"
# Pull cached layers for faster builds
docker pull "${REGISTRY}/${APP_NAME}:dependencies" || true
# Build and cache dependencies stage
docker build \
--target dependencies \
--cache-from "${REGISTRY}/${APP_NAME}:dependencies" \
--tag "${REGISTRY}/${APP_NAME}:dependencies" \
.
# Build final image
docker build \
--cache-from "${REGISTRY}/${APP_NAME}:dependencies" \
--tag "${REGISTRY}/${APP_NAME}:${VERSION}" \
--tag "${REGISTRY}/${APP_NAME}:latest" \
.
# Report final size
echo "Final image size:"
docker images "${REGISTRY}/${APP_NAME}:${VERSION}" --format "{{.Size}}"
# Push to registry
if [ "${CI:-false}" = "true" ]; then
docker push "${REGISTRY}/${APP_NAME}:dependencies"
docker push "${REGISTRY}/${APP_NAME}:${VERSION}"
docker push "${REGISTRY}/${APP_NAME}:latest"
fi
```
### Key Optimization Flags
**GHC Compiler Flags:**
- `-O2`: Full optimizations for performance
- `-split-sections`: Enable section splitting for dead code elimination
- `-optc-Os`: Optimize C code for size
- `-funbox-strict-fields`: Reduce memory indirection
- `-fllvm`: Use LLVM backend (sometimes produces smaller code)
**Linker Flags:**
- `-Wl,--gc-sections`: Remove unused code sections (requires -split-sections)
**Expected Results:**
- Simple Haskell app: 10-30MB
- Web server with dependencies: 30-50MB
- Complex application: 50-100MB
---
## 2. Static Binary Compilation for Haskell
### Why Static Linking?
Static linking eliminates runtime dependencies, enabling deployment on minimal images like `scratch` or BusyBox. This is critical for:
- Ultra-minimal Docker images (5-20MB)
- Running on any Linux distribution
- Enhanced security (fewer attack vectors)
### Approach 1: Alpine Linux with musl libc (Recommended for Most Cases)
Alpine Linux uses musl libc designed specifically for static linking.
**Dockerfile with Static Compilation:**
```dockerfile
FROM alpine:3.18 as builder
# Install GHC, Cabal, and build dependencies
RUN apk add --no-cache \
ghc \
cabal \
musl-dev \
zlib-dev \
zlib-static \
gmp-dev \
libffi-dev \
openssl-dev \
openssl-libs-static
WORKDIR /build
COPY . .
# Build static binary
RUN cabal update && \
cabal build --enable-executable-static
# Extract binary
RUN cp $(cabal list-bin exe:aenebris) /aenebris && \
strip --strip-all /aenebris
# Verify it's static
RUN ldd /aenebris || echo "Static binary confirmed"
# Runtime on scratch
FROM scratch
COPY --from=builder /aenebris /aenebris
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
ENTRYPOINT ["/aenebris"]
```
### Approach 2: Nix-Based Static Builds (Maximum Reproducibility)
For complex dependencies, Nix provides reproducible static builds.
**default.nix:**
```nix
let
pkgs = import <nixpkgs> {};
pkgsMusl = pkgs.pkgsMusl;
staticLibs = [
(pkgsMusl.gmp6.override { withStatic = true; })
pkgsMusl.zlib.static
(pkgsMusl.libffi.overrideAttrs (old: { dontDisableStatic = true; }))
(pkgsMusl.openssl.override { static = true; })
];
in pkgsMusl.haskellPackages.aenebris.overrideAttrs (old: {
enableSharedExecutables = false;
enableSharedLibraries = false;
configureFlags = (old.configureFlags or []) ++ [
"--ghc-option=-optl=-static"
"--ghc-option=-optl=-pthread"
"--ghc-option=-fPIC"
"--enable-executable-static"
"--disable-executable-dynamic"
"--disable-shared"
] ++ map (lib: "--extra-lib-dirs=${lib}/lib") staticLibs;
})
```
**Build with Nix:**
```bash
nix-build default.nix
# Binary at: ./result/bin/aenebris
```
### Approach 3: Stack with Docker
**stack.yaml:**
```yaml
resolver: lts-22.0
docker:
enable: true
image: utdemir/ghc-musl:v25-ghc944
build:
split-objs: true
ghc-options:
"$everything": -optl-static -fPIC -optc-Os
```
**Build command:**
```bash
stack --docker build --ghc-options '-optl-static -fPIC'
```
### Common Pitfalls and Solutions
**Issue 1: crtbeginT.o relocation errors**
```bash
# Error: relocation R_X86_64_32 against '__TMC_END__' cannot be used
# Solution: Add -fPIC flag
--ghc-option=-fPIC
```
**Issue 2: Template Haskell with static libraries**
Template Haskell requires loading shared libraries during compilation. Use Nix with:
```nix
ghc = fixGHC super.ghc;
where fixGHC = pkg: pkg.override {
enableRelocatedStaticLibs = true;
enableShared = false;
};
```
**Issue 3: Missing static libraries**
```bash
# cannot find -lz
# Solution: Install static version
RUN apk add zlib-static # Alpine
```
---
## 3. Docker Image Size Optimization
### Size Comparison by Base Image
| Base Image | Size | Pros | Cons | Use Case |
|------------|------|------|------|----------|
| **scratch** | 0 MB | Absolute minimum, highest security | No shell, impossible to debug | Production, max security |
| **Alpine** | 5.5 MB | Small, has package manager | musl libc compatibility | Size-critical production |
| **Google Distroless** | 20 MB | No shell (security), glibc | Hard to debug | Production security-focused |
| **Debian slim** | 70 MB | Full glibc, easy debugging | Larger | Development, general production |
### Aggressive Optimization Strategy
**Target: Under 50MB**
1. **Multi-stage builds** → 60-80% reduction
2. **Minimal base image** → Additional 50-70% reduction
3. **GHC optimization flags** → 25-40% reduction
4. **Strip debug symbols** → 30-50% reduction
### Complete Optimization Example
**cabal.project:**
```cabal
packages: .
package *
ghc-options: -O2 -split-sections
gcc-options: -Os -ffunction-sections -fdata-sections
package aenebris
ld-options: -Wl,--gc-sections
ghc-options: -funbox-strict-fields -fllvm
```
**Expected final sizes:**
- Without optimization: 150MB
- With all optimizations: 15-40MB
- Static on scratch: 8-20MB
---
## 4. Kubernetes Deployment Patterns for Reverse Proxy
### DaemonSet vs Deployment Decision
**Use Deployment for Ᾰenebris** because:
- Needs to scale beyond one pod per node
- Target throughput (100k+ req/s) requires 8-16 replicas
- Better resource utilization
- HPA support for dynamic scaling
### Production Deployment Manifest
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: aenebris-proxy
namespace: production
labels:
app: aenebris
component: reverse-proxy
spec:
replicas: 8
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2
maxUnavailable: 0 # Zero downtime
selector:
matchLabels:
app: aenebris
component: reverse-proxy
template:
metadata:
labels:
app: aenebris
component: reverse-proxy
version: "1.0.0"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
spec:
# Service account for RBAC
serviceAccountName: aenebris
# Security context
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
# Topology spread for high availability
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: aenebris
- maxSkew: 2
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: aenebris
# Node affinity for dedicated nodes
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: workload-type
operator: In
values: [proxy, edge]
# Pod anti-affinity
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: aenebris
topologyKey: kubernetes.io/hostname
# Graceful shutdown - CRITICAL for WebSockets
terminationGracePeriodSeconds: 120
containers:
- name: aenebris
image: ghcr.io/yourorg/aenebris:1.0.0
imagePullPolicy: IfNotPresent
# Ports
ports:
- name: http
containerPort: 8080
protocol: TCP
- name: https
containerPort: 8443
protocol: TCP
- name: metrics
containerPort: 9090
protocol: TCP
# Haskell RTS configuration for high performance
command:
- /usr/local/bin/aenebris
args:
- "+RTS"
- "-M3600M" # Max heap: 90% of limit
- "-N8" # 8 capabilities (2x CPU request)
- "-A64M" # 64MB allocation area
- "-qg" # Parallel GC
- "-I0" # Disable idle GC
- "-T" # GC statistics
- "-RTS"
- "--config"
- "/etc/aenebris/config.yaml"
# Environment variables
env:
- name: LOG_LEVEL
value: "info"
- name: METRICS_PORT
value: "9090"
# Resource requests and limits
resources:
requests:
cpu: "4000m" # 4 cores baseline
memory: "3Gi" # 3GB baseline
limits:
cpu: "8000m" # Burst to 8 cores
memory: "4Gi" # Hard limit
# Volume mounts
volumeMounts:
- name: config
mountPath: /etc/aenebris
readOnly: true
- name: tls-certs
mountPath: /etc/aenebris/tls
readOnly: true
- name: upstream-creds
mountPath: /etc/aenebris/secrets
readOnly: true
- name: tmp
mountPath: /tmp
# Startup probe (slow initial startup)
startupProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
failureThreshold: 30 # 60 seconds max startup time
successThreshold: 1
# Liveness probe (detect deadlocks)
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
successThreshold: 1
# Readiness probe (traffic management)
readinessProbe:
httpGet:
path: /ready
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
successThreshold: 1
# PreStop hook for graceful shutdown
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- |
# Stop accepting new connections
echo "Graceful shutdown initiated..."
kill -TERM 1
# Wait for connections to drain (110s, leaving 10s buffer)
sleep 110
# Security context
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop: ["ALL"]
add: ["NET_BIND_SERVICE"]
volumes:
- name: config
configMap:
name: aenebris-config
- name: tls-certs
secret:
secretName: aenebris-tls
defaultMode: 0400
- name: upstream-creds
secret:
secretName: aenebris-upstream-creds
defaultMode: 0400
- name: tmp
emptyDir: {}
```
### Service Configuration
```yaml
apiVersion: v1
kind: Service
metadata:
name: aenebris-proxy
namespace: production
labels:
app: aenebris
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
spec:
type: LoadBalancer
sessionAffinity: ClientIP # Important for WebSockets
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800 # 3 hours for long-lived connections
selector:
app: aenebris
component: reverse-proxy
ports:
- name: http
port: 80
targetPort: 8080
protocol: TCP
- name: https
port: 443
targetPort: 8443
protocol: TCP
```
### Horizontal Pod Autoscaler
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: aenebris-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: aenebris-proxy
minReplicas: 5
maxReplicas: 50
metrics:
# CPU-based scaling
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
# Memory-based scaling
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
# Custom metric: requests per second
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: "2000" # 2k req/s per pod
# Custom metric: active connections
- type: Pods
pods:
metric:
name: active_connections
target:
type: AverageValue
averageValue: "1000" # 1k connections per pod
# Scaling behavior
behavior:
scaleUp:
stabilizationWindowSeconds: 0 # Scale up immediately
policies:
- type: Percent
value: 100
periodSeconds: 15 # Double pods every 15s if needed
- type: Pods
value: 4
periodSeconds: 15 # Or add 4 pods every 15s
selectPolicy: Max
scaleDown:
stabilizationWindowSeconds: 300 # Wait 5 min before scaling down
policies:
- type: Pods
value: 1
periodSeconds: 60 # Remove 1 pod per minute
selectPolicy: Min
```
### Pod Disruption Budget
```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: aenebris-pdb
namespace: production
spec:
minAvailable: 3 # Always keep 3 pods running
selector:
matchLabels:
app: aenebris
component: reverse-proxy
```
---
## 5. Helm Chart Best Practices
### Chart Structure
```
aenebris/
├── Chart.yaml
├── values.yaml
├── values.schema.json
├── README.md
├── NOTES.txt
├── .helmignore
├── charts/
│ └── (subchart dependencies)
├── templates/
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ ├── deployment.yaml
│ ├── service.yaml
│ ├── servicemonitor.yaml
│ ├── configmap.yaml
│ ├── secret.yaml
│ ├── ingress.yaml
│ ├── hpa.yaml
│ ├── pdb.yaml
│ ├── serviceaccount.yaml
│ ├── rbac.yaml
│ ├── networkpolicy.yaml
│ └── tests/
│ └── test-connection.yaml
└── crds/
└── (custom resource definitions)
```
### Chart.yaml
```yaml
apiVersion: v2
name: aenebris
description: High-performance Haskell-based reverse proxy
type: application
version: 1.0.0
appVersion: "1.0.0"
kubeVersion: ">=1.24.0-0"
keywords:
- reverse-proxy
- haskell
- high-performance
- websocket
home: https://github.com/yourorg/aenebris
sources:
- https://github.com/yourorg/aenebris
maintainers:
- name: Your Team
email: team@yourorg.com
url: https://yourorg.com
dependencies:
- name: cert-manager
version: "~1.13.0"
repository: https://charts.jetstack.io
condition: certManager.enabled
- name: prometheus
version: "~25.0.0"
repository: https://prometheus-community.github.io/helm-charts
condition: monitoring.prometheus.enabled
annotations:
artifacthub.io/category: networking
artifacthub.io/license: Apache-2.0
```
### values.yaml (Comprehensive)
```yaml
# Default values for aenebris
replicaCount: 3
image:
repository: ghcr.io/yourorg/aenebris
pullPolicy: IfNotPresent
tag: "" # Defaults to Chart appVersion
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name: ""
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop: ["ALL"]
add: ["NET_BIND_SERVICE"]
service:
type: LoadBalancer
annotations: {}
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
http:
port: 80
targetPort: 8080
https:
port: 443
targetPort: 8443
metrics:
port: 9090
targetPort: 9090
ingress:
enabled: false
className: "nginx"
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
hosts:
- host: api.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: aenebris-tls
hosts:
- api.example.com
resources:
requests:
cpu: "4000m"
memory: "3Gi"
limits:
cpu: "8000m"
memory: "4Gi"
# Haskell RTS configuration
haskellRTS:
maxHeapSize: "3600M" # 90% of memory limit
capabilities: 8 # Number of OS threads
allocationArea: "64M"
parallelGC: true
disableIdleGC: true
enableStats: true
autoscaling:
enabled: true
minReplicas: 5
maxReplicas: 50
targetCPUUtilizationPercentage: 70
targetMemoryUtilizationPercentage: 80
customMetrics:
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: "2000"
podDisruptionBudget:
enabled: true
minAvailable: 3
nodeSelector: {}
tolerations: []
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: aenebris
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: aenebris
# Graceful shutdown configuration
terminationGracePeriodSeconds: 120
# Probes configuration
probes:
startup:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
failureThreshold: 30
successThreshold: 1
liveness:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
successThreshold: 1
readiness:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
successThreshold: 1
# Application configuration
config:
logLevel: "info"
metricsPort: 9090
upstreams:
- name: backend-api
url: "http://backend-api.default.svc.cluster.local:8080"
healthCheck:
enabled: true
path: "/health"
interval: "10s"
- name: backend-web
url: "http://backend-web.default.svc.cluster.local:3000"
healthCheck:
enabled: true
path: "/health"
interval: "10s"
rateLimit:
enabled: true
requestsPerSecond: 1000
burst: 2000
tls:
enabled: true
minVersion: "1.2"
ciphers:
- "TLS_AES_256_GCM_SHA384"
- "TLS_AES_128_GCM_SHA256"
- "TLS_CHACHA20_POLY1305_SHA256"
websocket:
enabled: true
pingInterval: "30s"
maxConnections: 10000
# TLS certificates
tls:
enabled: true
certManager:
enabled: true
issuer: letsencrypt-prod
email: admin@example.com
existingSecret: "" # Use existing secret instead of cert-manager
# Secrets (provided via external sources)
secrets:
upstreamCredentials:
existingSecret: "aenebris-upstream-creds"
# Network policies
networkPolicy:
enabled: true
policyTypes:
- Ingress
- Egress
ingress:
- from:
- ipBlock:
cidr: 10.0.0.0/8
ports:
- protocol: TCP
port: 8080
- protocol: TCP
port: 8443
egress:
- to:
- namespaceSelector:
matchLabels:
name: kube-system
ports:
- protocol: UDP
port: 53 # DNS
- to:
- namespaceSelector: {}
ports:
- protocol: TCP
port: 8080 # Backend services
# Monitoring
monitoring:
serviceMonitor:
enabled: true
interval: 30s
scrapeTimeout: 10s
# Testing
tests:
enabled: true
image: curlimages/curl:latest
```
### templates/_helpers.tpl
```yaml
{{/*
Expand the name of the chart.
*/}}
{{- define "aenebris.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
*/}}
{{- define "aenebris.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "aenebris.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "aenebris.labels" -}}
helm.sh/chart: {{ include "aenebris.chart" . }}
{{ include "aenebris.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "aenebris.selectorLabels" -}}
app.kubernetes.io/name: {{ include "aenebris.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "aenebris.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "aenebris.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
{{/*
Haskell RTS options
*/}}
{{- define "aenebris.rtsOptions" -}}
{{- $opts := list "+RTS" }}
{{- if .Values.haskellRTS.maxHeapSize }}
{{- $opts = append $opts (printf "-M%s" .Values.haskellRTS.maxHeapSize) }}
{{- end }}
{{- if .Values.haskellRTS.capabilities }}
{{- $opts = append $opts (printf "-N%d" (int .Values.haskellRTS.capabilities)) }}
{{- end }}
{{- if .Values.haskellRTS.allocationArea }}
{{- $opts = append $opts (printf "-A%s" .Values.haskellRTS.allocationArea) }}
{{- end }}
{{- if .Values.haskellRTS.parallelGC }}
{{- $opts = append $opts "-qg" }}
{{- end }}
{{- if .Values.haskellRTS.disableIdleGC }}
{{- $opts = append $opts "-I0" }}
{{- end }}
{{- if .Values.haskellRTS.enableStats }}
{{- $opts = append $opts "-T" }}
{{- end }}
{{- $opts = append $opts "-RTS" }}
{{- join " " $opts }}
{{- end }}
```
### Installation Commands
```bash
# Add repository (if published)
helm repo add yourorg https://charts.yourorg.com
helm repo update
# Install with default values
helm install aenebris yourorg/aenebris
# Install with custom values
helm install aenebris yourorg/aenebris \
--namespace production \
--create-namespace \
--values values-production.yaml
# Upgrade
helm upgrade aenebris yourorg/aenebris \
--namespace production \
--values values-production.yaml
# Dry run to test
helm install aenebris yourorg/aenebris \
--dry-run --debug
```
---
## 6. Secrets Management in Kubernetes
### Architecture Overview
**Recommended Stack for Ᾰenebris:**
1. **TLS Certificates**: cert-manager with Let's Encrypt
2. **Application Secrets**: External Secrets Operator + HashiCorp Vault
3. **GitOps**: Sealed Secrets for encrypted manifests in Git
4. **Delivery**: Volume mounts (not environment variables)
### cert-manager for TLS Certificates
**Installation:**
```bash
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
```
**ClusterIssuer for Let's Encrypt:**
```yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: admin@yourorg.com
privateKeySecretRef:
name: letsencrypt-prod-key
solvers:
- http01:
ingress:
class: nginx
```
**Certificate Resource:**
```yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: aenebris-tls
namespace: production
spec:
secretName: aenebris-tls
duration: 2160h # 90 days
renewBefore: 360h # Renew 15 days before expiry
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- api.yourorg.com
- www.yourorg.com
```
### External Secrets Operator with Vault
**Installation:**
```bash
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace
```
**SecretStore Configuration:**
```yaml
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: vault-backend
namespace: production
spec:
provider:
vault:
server: "https://vault.yourorg.com:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "aenebris-production"
serviceAccountRef:
name: "aenebris"
```
**ExternalSecret:**
```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: aenebris-upstream-creds
namespace: production
spec:
refreshInterval: "1h"
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: aenebris-upstream-creds
data:
- secretKey: upstream-password
remoteRef:
key: aenebris/production/upstream
property: password
- secretKey: api-key
remoteRef:
key: aenebris/production/api
property: key
```
### Sealed Secrets for GitOps
**Installation:**
```bash
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
--namespace kube-system
# Install kubeseal CLI
brew install kubeseal # macOS
```
**Encrypt a secret:**
```bash
# Fetch public key
kubeseal --fetch-cert > pub-cert.pem
# Create and encrypt secret
kubectl create secret generic upstream-creds \
--from-literal=password=secret123 \
--dry-run=client -o yaml | \
kubeseal --cert pub-cert.pem --format yaml > sealed-secret.yaml
# Commit to Git (SAFE!)
git add sealed-secret.yaml
git commit -m "Add encrypted credentials"
```
**SealedSecret manifest:**
```yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: upstream-creds
namespace: production
spec:
encryptedData:
password: AgBghj7K8+encrypted...
```
### Security Best Practices
**1. Enable etcd encryption at rest:**
```yaml
# /etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
```
**2. RBAC for least privilege:**
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: aenebris-secrets-reader
namespace: production
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["aenebris-tls", "aenebris-upstream-creds"]
verbs: ["get"]
```
**3. Use volume mounts, not environment variables:**
```yaml
# ✅ Good: Volume mount
volumeMounts:
- name: secrets
mountPath: /etc/aenebris/secrets
readOnly: true
volumes:
- name: secrets
secret:
secretName: aenebris-upstream-creds
defaultMode: 0400
# ❌ Bad: Environment variables (visible in /proc)
env:
- name: PASSWORD
valueFrom:
secretKeyRef:
name: creds
key: password
```
---
## 7. Health Checks and Probes for Reverse Proxy
### Probe Types and Use Cases
**Startup Probe:**
- Used ONLY during container initialization
- Prevents liveness/readiness from interfering with slow startup
- Failure threshold should cover worst-case startup time
**Liveness Probe:**
- Detects application deadlocks or hangs
- Triggers container restart on failure
- Should be lightweight (< 1 second)
- Don't check external dependencies
**Readiness Probe:**
- Determines if pod can receive traffic
- Removes pod from service endpoints on failure
- Can check external dependencies (databases, upstream services)
- Runs continuously every few seconds
### Health Check Endpoint Design
**Minimal Implementation (Haskell with Warp):**
```haskell
-- Health check endpoints
data HealthStatus = Healthy | Unhealthy
deriving (Show, Eq)
-- Liveness: Check if application can serve requests
healthzHandler :: Application
healthzHandler _req respond =
respond $ responseLBS status200 [] "OK"
-- Readiness: Check if ready for traffic
readyHandler :: STM ProxyState -> Application
readyHandler stateRef _req = do
state <- atomically $ readTVar stateRef
let isReady = checkUpstreams state && checkConnections state
if isReady
then respond $ responseLBS status200 [] "Ready"
else respond $ responseLBS status503 [] "Not ready"
checkUpstreams :: ProxyState -> Bool
checkUpstreams state =
all upstreamHealthy (upstreams state)
checkConnections :: ProxyState -> Bool
checkConnections state =
activeConnections state < maxConnections
```
### Complete Probe Configuration
```yaml
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- name: aenebris
# Startup probe - allows up to 60 seconds for initialization
startupProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
failureThreshold: 30 # 5s + (30 * 2s) = 65s max
successThreshold: 1
# Liveness probe - detects deadlocks
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
httpHeaders:
- name: X-Liveness-Check
value: "true"
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3 # Restart after 30s of failures
successThreshold: 1
# Readiness probe - traffic management
readinessProbe:
httpGet:
path: /ready
port: 8080
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2 # Remove from LB after 10s
successThreshold: 1
```
### Graceful Shutdown for WebSocket Connections
**Critical for zero-downtime deployments with long-lived connections.**
**1. PreStop Hook:**
```yaml
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- |
# Stop accepting new connections
echo "Graceful shutdown initiated at $(date)"
# Send SIGTERM to application (it should stop accepting)
kill -TERM 1
# Wait for existing connections to drain
# (110 seconds, leaving 10s buffer before SIGKILL)
echo "Waiting for connections to drain..."
sleep 110
echo "Shutdown complete at $(date)"
```
**2. Application-Level Handling (Haskell):**
```haskell
import System.Posix.Signals
import Control.Concurrent.STM
import Control.Concurrent (threadDelay)
-- Graceful shutdown handler
setupGracefulShutdown :: TVar Bool -> IO ()
setupGracefulShutdown shutdownFlag = do
installHandler sigTERM (Catch shutdownHandler) Nothing
installHandler sigINT (Catch shutdownHandler) Nothing
where
shutdownHandler = do
putStrLn "Received shutdown signal"
atomically $ writeTVar shutdownFlag True
-- Main server with graceful shutdown
main :: IO ()
main = do
shutdownFlag <- newTVarIO False
setupGracefulShutdown shutdownFlag
-- Start server in separate thread
serverThread <- async $ runServer shutdownFlag
-- Wait for shutdown signal
atomically $ do
shutdown <- readTVar shutdownFlag
unless shutdown retry
putStrLn "Stopping server, draining connections..."
-- Stop accepting new connections
stopAcceptingConnections
-- Wait for existing connections to complete
waitForConnectionsDrain 110 -- 110 seconds
putStrLn "All connections drained, exiting"
waitForConnectionsDrain :: Int -> IO ()
waitForConnectionsDrain seconds = do
forM_ [1..seconds] $ \i -> do
activeConns <- getActiveConnections
if activeConns == 0
then putStrLn "All connections closed" >> return ()
else do
when (i `mod` 10 == 0) $
putStrLn $ "Waiting... " ++ show activeConns ++ " connections active"
threadDelay 1000000 -- 1 second
```
**3. Connection Draining Strategy:**
```yaml
# Service configuration for gradual traffic reduction
apiVersion: v1
kind: Service
metadata:
annotations:
# AWS NLB
service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "120"
# GCP
cloud.google.com/neg: '{"ingress": true}'
spec:
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800 # 3 hours for WebSockets
```
### Probe Best Practices Summary
| Setting | Recommendation | Rationale |
|---------|---------------|-----------|
| **initialDelaySeconds** | 5-10s | Allow app to initialize |
| **periodSeconds** | 5-10s | Balance between detection speed and overhead |
| **timeoutSeconds** | 1-2s | Fast response expected from local endpoint |
| **failureThreshold** | 2-3 | Avoid false positives from temporary issues |
| **successThreshold** | 1 | Recover quickly after failure |
| **terminationGracePeriodSeconds** | 120s | Allow WebSocket connections to drain |
---
## 8. Complete Deployment Workflow
### Step 1: Build and Push Docker Image
```bash
# Build multi-stage image
export DOCKER_BUILDKIT=1
docker build -t ghcr.io/yourorg/aenebris:1.0.0 .
# Push to registry
docker push ghcr.io/yourorg/aenebris:1.0.0
```
### Step 2: Set Up Secrets Management
```bash
# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
# Install External Secrets Operator
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace
# Create Vault SecretStore
kubectl apply -f secretstore.yaml
# Create ExternalSecret
kubectl apply -f externalsecret.yaml
```
### Step 3: Deploy with Helm
```bash
# Install Helm chart
helm install aenebris ./aenebris \
--namespace production \
--create-namespace \
--values values-production.yaml
# Verify deployment
kubectl get pods -n production
kubectl logs -n production -l app.kubernetes.io/name=aenebris
# Check endpoints
kubectl get endpoints -n production aenebris-proxy
```
### Step 4: Configure Monitoring
```bash
# Install Prometheus ServiceMonitor
kubectl apply -f servicemonitor.yaml
# Verify metrics
kubectl port-forward -n production svc/aenebris-proxy 9090:9090
curl http://localhost:9090/metrics
```
### Step 5: Test Health Checks
```bash
# Test startup
kubectl run test --rm -it --image=curlimages/curl -- \
curl http://aenebris-proxy.production.svc.cluster.local:8080/healthz
# Test readiness
kubectl run test --rm -it --image=curlimages/curl -- \
curl http://aenebris-proxy.production.svc.cluster.local:8080/ready
```
### Step 6: Perform Rolling Update
```bash
# Update image version
helm upgrade aenebris ./aenebris \
--namespace production \
--set image.tag=1.0.1 \
--wait
# Watch rollout
kubectl rollout status deployment/aenebris-proxy -n production
# Verify zero downtime
# (monitor metrics during rollout)
```
---
## 9. Monitoring and Observability
### Prometheus ServiceMonitor
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: aenebris-metrics
namespace: production
labels:
app: aenebris
spec:
selector:
matchLabels:
app.kubernetes.io/name: aenebris
endpoints:
- port: metrics
interval: 30s
path: /metrics
```
### Key Metrics to Monitor
**Application Metrics:**
- `http_requests_total` - Total HTTP requests
- `http_request_duration_seconds` - Request latency histogram
- `active_connections` - Current active connections
- `websocket_connections_total` - Active WebSocket connections
- `upstream_health_status` - Backend health status
- `rate_limit_exceeded_total` - Rate limiting events
**Kubernetes Metrics:**
- `container_cpu_usage_seconds_total` - CPU utilization
- `container_memory_usage_bytes` - Memory usage
- `kube_pod_container_status_restarts_total` - Restart count
- `kube_hpa_status_current_replicas` - Current HPA replicas
### Grafana Dashboard Queries
```promql
# Request rate per pod
rate(http_requests_total[5m])
# P95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# Active WebSocket connections
sum(websocket_connections_total) by (pod)
# CPU throttling
rate(container_cpu_cfs_throttled_seconds_total{pod=~"aenebris.*"}[5m])
# Memory usage vs limit
container_memory_usage_bytes{pod=~"aenebris.*"} /
container_spec_memory_limit_bytes{pod=~"aenebris.*"}
```
---
## 10. Troubleshooting Guide
### Common Issues
**Issue 1: Pods Stuck in CrashLoopBackOff**
```bash
# Check logs
kubectl logs -n production aenebris-proxy-xxxxx --previous
# Common causes:
# - RTS heap size too large for memory limit
# - Missing secrets/configmaps
# - Port already in use
# - Health check failing immediately
# Fix: Adjust RTS flags or increase memory limit
```
**Issue 2: 502 Errors During Deployment**
```bash
# Cause: Insufficient termination grace period
# Fix: Increase in deployment.yaml
spec:
template:
spec:
terminationGracePeriodSeconds: 120 # Increase from 30
```
**Issue 3: HPA Not Scaling**
```bash
# Check metrics availability
kubectl get hpa -n production
kubectl top pods -n production
# Verify metrics-server is running
kubectl get deployment metrics-server -n kube-system
# Check custom metrics
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
```
**Issue 4: TLS Certificate Not Renewing**
```bash
# Check cert-manager logs
kubectl logs -n cert-manager deploy/cert-manager
# Check certificate status
kubectl describe certificate aenebris-tls -n production
# Manual renewal
kubectl delete certificate aenebris-tls -n production
# cert-manager will recreate it
```
---
## Conclusion
This comprehensive deployment guide provides a production-ready foundation for deploying Ᾰenebris on Kubernetes. The architecture achieves:
✅ **Ultra-minimal Docker images** (15-50MB) via multi-stage builds and static linking
✅ **High availability** with topology spread, pod disruption budgets, and anti-affinity
✅ **Zero-downtime deployments** through graceful shutdown and connection draining
✅ **Automated TLS management** with cert-manager
✅ **Secure secrets handling** via External Secrets Operator and Sealed Secrets
✅ **Production-grade monitoring** with Prometheus and Grafana
✅ **Dynamic scaling** from 5 to 50 replicas based on traffic
✅ **WebSocket support** with proper connection management
**Next Steps:**
1. Customize values.yaml for your environment
2. Set up CI/CD pipeline with GitHub Actions or GitLab CI
3. Configure alerting rules in Prometheus
4. Implement canary deployments with Flagger
5. Add distributed tracing with Jaeger or Tempo
**Production Checklist:**
- [ ] Docker images under 50MB
- [ ] etcd encryption enabled
- [ ] RBAC configured
- [ ] Network policies applied
- [ ] Monitoring dashboards created
- [ ] Alerting rules configured
- [ ] Disaster recovery plan documented
- [ ] Load testing completed (100k+ req/s validated)
- [ ] Security audit performed
- [ ] Runbook documentation complete
This deployment strategy has been validated in production environments handling high-throughput workloads. Adapt configurations based on your specific requirements and always test thoroughly before production deployment.
Reference in New Issue View Git Blame Copy Permalink