45 KiB

Raw Blame History

Docker & Kubernetes Deployment Guide for Ᾰenebris Reverse Proxy

Production-Grade Deployment Strategy for High-Performance Haskell Applications

Executive Summary

This comprehensive guide provides battle-tested strategies for deploying Ᾰenebris, a high-performance Haskell-based reverse proxy, on Kubernetes. With a target throughput of 100k+ requests/second and support for TLS termination and WebSockets, this deployment architecture prioritizes performance, security, and operational excellence.

Key achievements with this approach:

Docker images under 50MB (achieving 10-30MB for typical builds)
Zero-downtime deployments with graceful connection draining
Automated TLS certificate management
Production-ready secrets management
Horizontal autoscaling for traffic spikes

1. Multi-Stage Docker Builds for Haskell Applications

Overview

Multi-stage Docker builds separate compilation from runtime, dramatically reducing final image size while maintaining optimal build caching. For Haskell applications, this approach is critical because GHC and build dependencies can exceed 2GB, while the runtime binary needs only 5-50MB.

Three-Stage Build Pattern

The optimal pattern for Haskell reverse proxies uses three distinct stages:

Dependencies stage: Builds only dependencies (cached separately)
Build stage: Compiles the application
Runtime stage: Minimal image with just the binary

Production-Ready Dockerfile for Ᾰenebris

# syntax=docker/dockerfile:1

###############################################################################
# Stage 1: Dependency Cache (rebuilt only when dependencies change)
###############################################################################
FROM haskell:9.4-slim as dependencies

WORKDIR /build

# Install build-time system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    ca-certificates \
    libgmp-dev \
    zlib1g-dev \
    libssl-dev \
    curl \
    git \
    && rm -rf /var/lib/apt/lists/*

# Copy only dependency manifests for optimal caching
COPY aenebris.cabal cabal.project cabal.project.freeze /build/

# Build dependencies only (this layer is cached)
RUN cabal update && \
    cabal build --only-dependencies --enable-tests --enable-benchmarks

###############################################################################
# Stage 2: Application Build with Aggressive Optimizations
###############################################################################
FROM dependencies as builder

# Copy source code
COPY . /build/

# Build with size and performance optimizations
RUN cabal build \
    --ghc-options="-O2 -split-sections -optc-Os -funbox-strict-fields -fllvm" \
    --gcc-options="-Os -ffunction-sections -fdata-sections" \
    --ld-options="-Wl,--gc-sections"

# Extract and optimize binary
RUN mkdir -p /output && \
    cp $(cabal exec -- which aenebris) /output/aenebris && \
    strip --strip-all /output/aenebris

# Verify binary size
RUN ls -lh /output/aenebris

###############################################################################
# Stage 3: Minimal Runtime Image (Production)
###############################################################################
FROM debian:12-slim as runtime

# Install only essential runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    libgmp10 \
    zlib1g \
    libssl3 \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Create non-root user for security
RUN useradd -m -u 1000 -s /bin/bash aenebris

# Copy binary from builder
COPY --from=builder /output/aenebris /usr/local/bin/aenebris

# Set ownership and permissions
RUN chown aenebris:aenebris /usr/local/bin/aenebris && \
    chmod +x /usr/local/bin/aenebris

# Switch to non-root user
USER aenebris

# Health check endpoint
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:8080/healthz || exit 1

EXPOSE 8080 8443

# Configure Haskell RTS for production
ENTRYPOINT ["/usr/local/bin/aenebris"]
CMD ["+RTS", "-M1800M", "-N4", "-A32M", "-qg", "-I0", "-T", "-RTS"]

###############################################################################
# Alternative: Ultra-Minimal Alpine Runtime (<20MB)
###############################################################################
FROM alpine:3.18 as runtime-alpine

RUN apk add --no-cache \
    gmp \
    libgcc \
    libssl3 \
    ca-certificates \
    curl \
    && adduser -D -u 1000 aenebris

COPY --from=builder /output/aenebris /usr/local/bin/aenebris

USER aenebris
EXPOSE 8080 8443

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:8080/healthz || exit 1

ENTRYPOINT ["/usr/local/bin/aenebris"]
CMD ["+RTS", "-M1800M", "-N4", "-A32M", "-qg", "-I0", "-T", "-RTS"]

Build Script with Caching

#!/bin/bash
set -euo pipefail

# Enable Docker BuildKit for advanced caching
export DOCKER_BUILDKIT=1

APP_NAME="aenebris"
VERSION="${1:-latest}"
REGISTRY="${REGISTRY:-ghcr.io/yourorg}"

echo "Building ${APP_NAME}:${VERSION}"

# Pull cached layers for faster builds
docker pull "${REGISTRY}/${APP_NAME}:dependencies" || true

# Build and cache dependencies stage
docker build \
  --target dependencies \
  --cache-from "${REGISTRY}/${APP_NAME}:dependencies" \
  --tag "${REGISTRY}/${APP_NAME}:dependencies" \
  .

# Build final image
docker build \
  --cache-from "${REGISTRY}/${APP_NAME}:dependencies" \
  --tag "${REGISTRY}/${APP_NAME}:${VERSION}" \
  --tag "${REGISTRY}/${APP_NAME}:latest" \
  .

# Report final size
echo "Final image size:"
docker images "${REGISTRY}/${APP_NAME}:${VERSION}" --format "{{.Size}}"

# Push to registry
if [ "${CI:-false}" = "true" ]; then
  docker push "${REGISTRY}/${APP_NAME}:dependencies"
  docker push "${REGISTRY}/${APP_NAME}:${VERSION}"
  docker push "${REGISTRY}/${APP_NAME}:latest"
fi

Key Optimization Flags

GHC Compiler Flags:

-O2: Full optimizations for performance
-split-sections: Enable section splitting for dead code elimination
-optc-Os: Optimize C code for size
-funbox-strict-fields: Reduce memory indirection
-fllvm: Use LLVM backend (sometimes produces smaller code)

Linker Flags:

-Wl,--gc-sections: Remove unused code sections (requires -split-sections)

Expected Results:

Simple Haskell app: 10-30MB
Web server with dependencies: 30-50MB
Complex application: 50-100MB

2. Static Binary Compilation for Haskell

Why Static Linking?

Static linking eliminates runtime dependencies, enabling deployment on minimal images like scratch or BusyBox. This is critical for:

Ultra-minimal Docker images (5-20MB)
Running on any Linux distribution
Enhanced security (fewer attack vectors)

Approach 1: Alpine Linux with musl libc (Recommended for Most Cases)

Alpine Linux uses musl libc designed specifically for static linking.

Dockerfile with Static Compilation:

FROM alpine:3.18 as builder

# Install GHC, Cabal, and build dependencies
RUN apk add --no-cache \
    ghc \
    cabal \
    musl-dev \
    zlib-dev \
    zlib-static \
    gmp-dev \
    libffi-dev \
    openssl-dev \
    openssl-libs-static

WORKDIR /build
COPY . .

# Build static binary
RUN cabal update && \
    cabal build --enable-executable-static

# Extract binary
RUN cp $(cabal list-bin exe:aenebris) /aenebris && \
    strip --strip-all /aenebris

# Verify it's static
RUN ldd /aenebris || echo "Static binary confirmed"

# Runtime on scratch
FROM scratch
COPY --from=builder /aenebris /aenebris
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
ENTRYPOINT ["/aenebris"]

Approach 2: Nix-Based Static Builds (Maximum Reproducibility)

For complex dependencies, Nix provides reproducible static builds.

default.nix:

let
  pkgs = import <nixpkgs> {};
  pkgsMusl = pkgs.pkgsMusl;

  staticLibs = [
    (pkgsMusl.gmp6.override { withStatic = true; })
    pkgsMusl.zlib.static
    (pkgsMusl.libffi.overrideAttrs (old: { dontDisableStatic = true; }))
    (pkgsMusl.openssl.override { static = true; })
  ];

in pkgsMusl.haskellPackages.aenebris.overrideAttrs (old: {
  enableSharedExecutables = false;
  enableSharedLibraries = false;
  configureFlags = (old.configureFlags or []) ++ [
    "--ghc-option=-optl=-static"
    "--ghc-option=-optl=-pthread"
    "--ghc-option=-fPIC"
    "--enable-executable-static"
    "--disable-executable-dynamic"
    "--disable-shared"
  ] ++ map (lib: "--extra-lib-dirs=${lib}/lib") staticLibs;
})

Build with Nix:

nix-build default.nix
# Binary at: ./result/bin/aenebris

Approach 3: Stack with Docker

stack.yaml:

resolver: lts-22.0

docker:
  enable: true
  image: utdemir/ghc-musl:v25-ghc944

build:
  split-objs: true

ghc-options:
  "$everything": -optl-static -fPIC -optc-Os

Build command:

stack --docker build --ghc-options '-optl-static -fPIC'

Common Pitfalls and Solutions

Issue 1: crtbeginT.o relocation errors

# Error: relocation R_X86_64_32 against '__TMC_END__' cannot be used
# Solution: Add -fPIC flag
--ghc-option=-fPIC

Issue 2: Template Haskell with static libraries

Template Haskell requires loading shared libraries during compilation. Use Nix with:

ghc = fixGHC super.ghc;
  where fixGHC = pkg: pkg.override {
    enableRelocatedStaticLibs = true;
    enableShared = false;
  };

Issue 3: Missing static libraries

# cannot find -lz
# Solution: Install static version
RUN apk add zlib-static  # Alpine

3. Docker Image Size Optimization

Size Comparison by Base Image

Base Image	Size	Pros	Cons	Use Case
scratch	0 MB	Absolute minimum, highest security	No shell, impossible to debug	Production, max security
Alpine	5.5 MB	Small, has package manager	musl libc compatibility	Size-critical production
Google Distroless	20 MB	No shell (security), glibc	Hard to debug	Production security-focused
Debian slim	70 MB	Full glibc, easy debugging	Larger	Development, general production

Aggressive Optimization Strategy

Target: Under 50MB

Multi-stage builds → 60-80% reduction
Minimal base image → Additional 50-70% reduction
GHC optimization flags → 25-40% reduction
Strip debug symbols → 30-50% reduction

Complete Optimization Example

cabal.project:

packages: .

package *
  ghc-options: -O2 -split-sections
  gcc-options: -Os -ffunction-sections -fdata-sections

package aenebris
  ld-options: -Wl,--gc-sections
  ghc-options: -funbox-strict-fields -fllvm

Expected final sizes:

Without optimization: 150MB
With all optimizations: 15-40MB
Static on scratch: 8-20MB

4. Kubernetes Deployment Patterns for Reverse Proxy

DaemonSet vs Deployment Decision

Use Deployment for Ᾰenebris because:

Needs to scale beyond one pod per node
Target throughput (100k+ req/s) requires 8-16 replicas
Better resource utilization
HPA support for dynamic scaling

Production Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aenebris-proxy
  namespace: production
  labels:
    app: aenebris
    component: reverse-proxy
spec:
  replicas: 8
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0  # Zero downtime
  selector:
    matchLabels:
      app: aenebris
      component: reverse-proxy
  template:
    metadata:
      labels:
        app: aenebris
        component: reverse-proxy
        version: "1.0.0"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/metrics"
    spec:
      # Service account for RBAC
      serviceAccountName: aenebris

      # Security context
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000

      # Topology spread for high availability
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: aenebris
      - maxSkew: 2
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: aenebris

      # Node affinity for dedicated nodes
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: workload-type
                operator: In
                values: [proxy, edge]

        # Pod anti-affinity
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: aenebris
              topologyKey: kubernetes.io/hostname

      # Graceful shutdown - CRITICAL for WebSockets
      terminationGracePeriodSeconds: 120

      containers:
      - name: aenebris
        image: ghcr.io/yourorg/aenebris:1.0.0
        imagePullPolicy: IfNotPresent

        # Ports
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        - name: https
          containerPort: 8443
          protocol: TCP
        - name: metrics
          containerPort: 9090
          protocol: TCP

        # Haskell RTS configuration for high performance
        command:
          - /usr/local/bin/aenebris
        args:
          - "+RTS"
          - "-M3600M"        # Max heap: 90% of limit
          - "-N8"            # 8 capabilities (2x CPU request)
          - "-A64M"          # 64MB allocation area
          - "-qg"            # Parallel GC
          - "-I0"            # Disable idle GC
          - "-T"             # GC statistics
          - "-RTS"
          - "--config"
          - "/etc/aenebris/config.yaml"

        # Environment variables
        env:
        - name: LOG_LEVEL
          value: "info"
        - name: METRICS_PORT
          value: "9090"

        # Resource requests and limits
        resources:
          requests:
            cpu: "4000m"      # 4 cores baseline
            memory: "3Gi"     # 3GB baseline
          limits:
            cpu: "8000m"      # Burst to 8 cores
            memory: "4Gi"     # Hard limit

        # Volume mounts
        volumeMounts:
        - name: config
          mountPath: /etc/aenebris
          readOnly: true
        - name: tls-certs
          mountPath: /etc/aenebris/tls
          readOnly: true
        - name: upstream-creds
          mountPath: /etc/aenebris/secrets
          readOnly: true
        - name: tmp
          mountPath: /tmp

        # Startup probe (slow initial startup)
        startupProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 2
          timeoutSeconds: 1
          failureThreshold: 30  # 60 seconds max startup time
          successThreshold: 1

        # Liveness probe (detect deadlocks)
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3
          successThreshold: 1

        # Readiness probe (traffic management)
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 2
          failureThreshold: 2
          successThreshold: 1

        # PreStop hook for graceful shutdown
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - |
                # Stop accepting new connections
                echo "Graceful shutdown initiated..."
                kill -TERM 1
                # Wait for connections to drain (110s, leaving 10s buffer)
                sleep 110                

        # Security context
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          capabilities:
            drop: ["ALL"]
            add: ["NET_BIND_SERVICE"]

      volumes:
      - name: config
        configMap:
          name: aenebris-config
      - name: tls-certs
        secret:
          secretName: aenebris-tls
          defaultMode: 0400
      - name: upstream-creds
        secret:
          secretName: aenebris-upstream-creds
          defaultMode: 0400
      - name: tmp
        emptyDir: {}

Service Configuration

apiVersion: v1
kind: Service
metadata:
  name: aenebris-proxy
  namespace: production
  labels:
    app: aenebris
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
spec:
  type: LoadBalancer
  sessionAffinity: ClientIP  # Important for WebSockets
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800  # 3 hours for long-lived connections
  selector:
    app: aenebris
    component: reverse-proxy
  ports:
  - name: http
    port: 80
    targetPort: 8080
    protocol: TCP
  - name: https
    port: 443
    targetPort: 8443
    protocol: TCP

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: aenebris-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: aenebris-proxy
  minReplicas: 5
  maxReplicas: 50
  metrics:
  # CPU-based scaling
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

  # Memory-based scaling
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

  # Custom metric: requests per second
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "2000"  # 2k req/s per pod

  # Custom metric: active connections
  - type: Pods
    pods:
      metric:
        name: active_connections
      target:
        type: AverageValue
        averageValue: "1000"  # 1k connections per pod

  # Scaling behavior
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0  # Scale up immediately
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15  # Double pods every 15s if needed
      - type: Pods
        value: 4
        periodSeconds: 15  # Or add 4 pods every 15s
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60  # Remove 1 pod per minute
      selectPolicy: Min

Pod Disruption Budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: aenebris-pdb
  namespace: production
spec:
  minAvailable: 3  # Always keep 3 pods running
  selector:
    matchLabels:
      app: aenebris
      component: reverse-proxy

5. Helm Chart Best Practices

Chart Structure

aenebris/
├── Chart.yaml
├── values.yaml
├── values.schema.json
├── README.md
├── NOTES.txt
├── .helmignore
├── charts/
│   └── (subchart dependencies)
├── templates/
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── servicemonitor.yaml
│   ├── configmap.yaml
│   ├── secret.yaml
│   ├── ingress.yaml
│   ├── hpa.yaml
│   ├── pdb.yaml
│   ├── serviceaccount.yaml
│   ├── rbac.yaml
│   ├── networkpolicy.yaml
│   └── tests/
│       └── test-connection.yaml
└── crds/
    └── (custom resource definitions)

Chart.yaml

apiVersion: v2
name: aenebris
description: High-performance Haskell-based reverse proxy
type: application
version: 1.0.0
appVersion: "1.0.0"
kubeVersion: ">=1.24.0-0"

keywords:
  - reverse-proxy
  - haskell
  - high-performance
  - websocket

home: https://github.com/yourorg/aenebris
sources:
  - https://github.com/yourorg/aenebris

maintainers:
  - name: Your Team
    email: team@yourorg.com
    url: https://yourorg.com

dependencies:
  - name: cert-manager
    version: "~1.13.0"
    repository: https://charts.jetstack.io
    condition: certManager.enabled
  - name: prometheus
    version: "~25.0.0"
    repository: https://prometheus-community.github.io/helm-charts
    condition: monitoring.prometheus.enabled

annotations:
  artifacthub.io/category: networking
  artifacthub.io/license: Apache-2.0

values.yaml (Comprehensive)

# Default values for aenebris

replicaCount: 3

image:
  repository: ghcr.io/yourorg/aenebris
  pullPolicy: IfNotPresent
  tag: ""  # Defaults to Chart appVersion

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  create: true
  annotations: {}
  name: ""

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9090"
  prometheus.io/path: "/metrics"

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]

service:
  type: LoadBalancer
  annotations: {}
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  http:
    port: 80
    targetPort: 8080
  https:
    port: 443
    targetPort: 8443
  metrics:
    port: 9090
    targetPort: 9090

ingress:
  enabled: false
  className: "nginx"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  hosts:
    - host: api.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: aenebris-tls
      hosts:
        - api.example.com

resources:
  requests:
    cpu: "4000m"
    memory: "3Gi"
  limits:
    cpu: "8000m"
    memory: "4Gi"

# Haskell RTS configuration
haskellRTS:
  maxHeapSize: "3600M"  # 90% of memory limit
  capabilities: 8       # Number of OS threads
  allocationArea: "64M"
  parallelGC: true
  disableIdleGC: true
  enableStats: true

autoscaling:
  enabled: true
  minReplicas: 5
  maxReplicas: 50
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80
  customMetrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "2000"

podDisruptionBudget:
  enabled: true
  minAvailable: 3

nodeSelector: {}

tolerations: []

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: aenebris
          topologyKey: kubernetes.io/hostname

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: aenebris

# Graceful shutdown configuration
terminationGracePeriodSeconds: 120

# Probes configuration
probes:
  startup:
    httpGet:
      path: /healthz
      port: 8080
    initialDelaySeconds: 5
    periodSeconds: 2
    timeoutSeconds: 1
    failureThreshold: 30
    successThreshold: 1

  liveness:
    httpGet:
      path: /healthz
      port: 8080
    initialDelaySeconds: 10
    periodSeconds: 10
    timeoutSeconds: 2
    failureThreshold: 3
    successThreshold: 1

  readiness:
    httpGet:
      path: /ready
      port: 8080
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 2
    failureThreshold: 2
    successThreshold: 1

# Application configuration
config:
  logLevel: "info"
  metricsPort: 9090
  upstreams:
    - name: backend-api
      url: "http://backend-api.default.svc.cluster.local:8080"
      healthCheck:
        enabled: true
        path: "/health"
        interval: "10s"
    - name: backend-web
      url: "http://backend-web.default.svc.cluster.local:3000"
      healthCheck:
        enabled: true
        path: "/health"
        interval: "10s"

  rateLimit:
    enabled: true
    requestsPerSecond: 1000
    burst: 2000

  tls:
    enabled: true
    minVersion: "1.2"
    ciphers:
      - "TLS_AES_256_GCM_SHA384"
      - "TLS_AES_128_GCM_SHA256"
      - "TLS_CHACHA20_POLY1305_SHA256"

  websocket:
    enabled: true
    pingInterval: "30s"
    maxConnections: 10000

# TLS certificates
tls:
  enabled: true
  certManager:
    enabled: true
    issuer: letsencrypt-prod
    email: admin@example.com
  existingSecret: ""  # Use existing secret instead of cert-manager

# Secrets (provided via external sources)
secrets:
  upstreamCredentials:
    existingSecret: "aenebris-upstream-creds"

# Network policies
networkPolicy:
  enabled: true
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
      - ipBlock:
          cidr: 10.0.0.0/8
      ports:
      - protocol: TCP
        port: 8080
      - protocol: TCP
        port: 8443
  egress:
    - to:
      - namespaceSelector:
          matchLabels:
            name: kube-system
      ports:
      - protocol: UDP
        port: 53  # DNS
    - to:
      - namespaceSelector: {}
      ports:
      - protocol: TCP
        port: 8080  # Backend services

# Monitoring
monitoring:
  serviceMonitor:
    enabled: true
    interval: 30s
    scrapeTimeout: 10s

# Testing
tests:
  enabled: true
  image: curlimages/curl:latest

templates/_helpers.tpl

{{/*
Expand the name of the chart.
*/}}
{{- define "aenebris.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
*/}}
{{- define "aenebris.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "aenebris.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "aenebris.labels" -}}
helm.sh/chart: {{ include "aenebris.chart" . }}
{{ include "aenebris.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "aenebris.selectorLabels" -}}
app.kubernetes.io/name: {{ include "aenebris.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/*
Create the name of the service account to use
*/}}
{{- define "aenebris.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "aenebris.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

{{/*
Haskell RTS options
*/}}
{{- define "aenebris.rtsOptions" -}}
{{- $opts := list "+RTS" }}
{{- if .Values.haskellRTS.maxHeapSize }}
{{- $opts = append $opts (printf "-M%s" .Values.haskellRTS.maxHeapSize) }}
{{- end }}
{{- if .Values.haskellRTS.capabilities }}
{{- $opts = append $opts (printf "-N%d" (int .Values.haskellRTS.capabilities)) }}
{{- end }}
{{- if .Values.haskellRTS.allocationArea }}
{{- $opts = append $opts (printf "-A%s" .Values.haskellRTS.allocationArea) }}
{{- end }}
{{- if .Values.haskellRTS.parallelGC }}
{{- $opts = append $opts "-qg" }}
{{- end }}
{{- if .Values.haskellRTS.disableIdleGC }}
{{- $opts = append $opts "-I0" }}
{{- end }}
{{- if .Values.haskellRTS.enableStats }}
{{- $opts = append $opts "-T" }}
{{- end }}
{{- $opts = append $opts "-RTS" }}
{{- join " " $opts }}
{{- end }}

Installation Commands

# Add repository (if published)
helm repo add yourorg https://charts.yourorg.com
helm repo update

# Install with default values
helm install aenebris yourorg/aenebris

# Install with custom values
helm install aenebris yourorg/aenebris \
  --namespace production \
  --create-namespace \
  --values values-production.yaml

# Upgrade
helm upgrade aenebris yourorg/aenebris \
  --namespace production \
  --values values-production.yaml

# Dry run to test
helm install aenebris yourorg/aenebris \
  --dry-run --debug

6. Secrets Management in Kubernetes

Architecture Overview

Recommended Stack for Ᾰenebris:

TLS Certificates: cert-manager with Let's Encrypt
Application Secrets: External Secrets Operator + HashiCorp Vault
GitOps: Sealed Secrets for encrypted manifests in Git
Delivery: Volume mounts (not environment variables)

cert-manager for TLS Certificates

Installation:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml

ClusterIssuer for Let's Encrypt:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@yourorg.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - http01:
        ingress:
          class: nginx

Certificate Resource:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: aenebris-tls
  namespace: production
spec:
  secretName: aenebris-tls
  duration: 2160h  # 90 days
  renewBefore: 360h  # Renew 15 days before expiry
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - api.yourorg.com
  - www.yourorg.com

External Secrets Operator with Vault

Installation:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

SecretStore Configuration:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.yourorg.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "aenebris-production"
          serviceAccountRef:
            name: "aenebris"

ExternalSecret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: aenebris-upstream-creds
  namespace: production
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: aenebris-upstream-creds
  data:
    - secretKey: upstream-password
      remoteRef:
        key: aenebris/production/upstream
        property: password
    - secretKey: api-key
      remoteRef:
        key: aenebris/production/api
        property: key

Sealed Secrets for GitOps

Installation:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Install kubeseal CLI
brew install kubeseal  # macOS

Encrypt a secret:

# Fetch public key
kubeseal --fetch-cert > pub-cert.pem

# Create and encrypt secret
kubectl create secret generic upstream-creds \
  --from-literal=password=secret123 \
  --dry-run=client -o yaml | \
kubeseal --cert pub-cert.pem --format yaml > sealed-secret.yaml

# Commit to Git (SAFE!)
git add sealed-secret.yaml
git commit -m "Add encrypted credentials"

SealedSecret manifest:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: upstream-creds
  namespace: production
spec:
  encryptedData:
    password: AgBghj7K8+encrypted...

Security Best Practices

1. Enable etcd encryption at rest:

# /etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

2. RBAC for least privilege:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: aenebris-secrets-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["aenebris-tls", "aenebris-upstream-creds"]
  verbs: ["get"]

3. Use volume mounts, not environment variables:

# ✅ Good: Volume mount
volumeMounts:
- name: secrets
  mountPath: /etc/aenebris/secrets
  readOnly: true
volumes:
- name: secrets
  secret:
    secretName: aenebris-upstream-creds
    defaultMode: 0400

# ❌ Bad: Environment variables (visible in /proc)
env:
- name: PASSWORD
  valueFrom:
    secretKeyRef:
      name: creds
      key: password

7. Health Checks and Probes for Reverse Proxy

Probe Types and Use Cases

Startup Probe:

Used ONLY during container initialization
Prevents liveness/readiness from interfering with slow startup
Failure threshold should cover worst-case startup time

Liveness Probe:

Detects application deadlocks or hangs
Triggers container restart on failure
Should be lightweight (< 1 second)
Don't check external dependencies

Readiness Probe:

Determines if pod can receive traffic
Removes pod from service endpoints on failure
Can check external dependencies (databases, upstream services)
Runs continuously every few seconds

Health Check Endpoint Design

Minimal Implementation (Haskell with Warp):

-- Health check endpoints
data HealthStatus = Healthy | Unhealthy
  deriving (Show, Eq)

-- Liveness: Check if application can serve requests
healthzHandler :: Application
healthzHandler _req respond =
  respond $ responseLBS status200 [] "OK"

-- Readiness: Check if ready for traffic
readyHandler :: STM ProxyState -> Application
readyHandler stateRef _req = do
  state <- atomically $ readTVar stateRef
  let isReady = checkUpstreams state && checkConnections state
  if isReady
    then respond $ responseLBS status200 [] "Ready"
    else respond $ responseLBS status503 [] "Not ready"

checkUpstreams :: ProxyState -> Bool
checkUpstreams state =
  all upstreamHealthy (upstreams state)

checkConnections :: ProxyState -> Bool
checkConnections state =
  activeConnections state < maxConnections

Complete Probe Configuration

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: aenebris
        # Startup probe - allows up to 60 seconds for initialization
        startupProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 2
          timeoutSeconds: 1
          failureThreshold: 30  # 5s + (30 * 2s) = 65s max
          successThreshold: 1

        # Liveness probe - detects deadlocks
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
            httpHeaders:
            - name: X-Liveness-Check
              value: "true"
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3  # Restart after 30s of failures
          successThreshold: 1

        # Readiness probe - traffic management
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 2
          failureThreshold: 2  # Remove from LB after 10s
          successThreshold: 1

Graceful Shutdown for WebSocket Connections

Critical for zero-downtime deployments with long-lived connections.

1. PreStop Hook:

lifecycle:
  preStop:
    exec:
      command:
      - /bin/sh
      - -c
      - |
        # Stop accepting new connections
        echo "Graceful shutdown initiated at $(date)"

        # Send SIGTERM to application (it should stop accepting)
        kill -TERM 1

        # Wait for existing connections to drain
        # (110 seconds, leaving 10s buffer before SIGKILL)
        echo "Waiting for connections to drain..."
        sleep 110

        echo "Shutdown complete at $(date)"

2. Application-Level Handling (Haskell):

import System.Posix.Signals
import Control.Concurrent.STM
import Control.Concurrent (threadDelay)

-- Graceful shutdown handler
setupGracefulShutdown :: TVar Bool -> IO ()
setupGracefulShutdown shutdownFlag = do
  installHandler sigTERM (Catch shutdownHandler) Nothing
  installHandler sigINT (Catch shutdownHandler) Nothing
  where
    shutdownHandler = do
      putStrLn "Received shutdown signal"
      atomically $ writeTVar shutdownFlag True

-- Main server with graceful shutdown
main :: IO ()
main = do
  shutdownFlag <- newTVarIO False
  setupGracefulShutdown shutdownFlag

  -- Start server in separate thread
  serverThread <- async $ runServer shutdownFlag

  -- Wait for shutdown signal
  atomically $ do
    shutdown <- readTVar shutdownFlag
    unless shutdown retry

  putStrLn "Stopping server, draining connections..."

  -- Stop accepting new connections
  stopAcceptingConnections

  -- Wait for existing connections to complete
  waitForConnectionsDrain 110  -- 110 seconds

  putStrLn "All connections drained, exiting"

waitForConnectionsDrain :: Int -> IO ()
waitForConnectionsDrain seconds = do
  forM_ [1..seconds] $ \i -> do
    activeConns <- getActiveConnections
    if activeConns == 0
      then putStrLn "All connections closed" >> return ()
      else do
        when (i `mod` 10 == 0) $
          putStrLn $ "Waiting... " ++ show activeConns ++ " connections active"
        threadDelay 1000000  -- 1 second

3. Connection Draining Strategy:

# Service configuration for gradual traffic reduction
apiVersion: v1
kind: Service
metadata:
  annotations:
    # AWS NLB
    service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "120"

    # GCP
    cloud.google.com/neg: '{"ingress": true}'

spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800  # 3 hours for WebSockets

Probe Best Practices Summary

Setting	Recommendation	Rationale
initialDelaySeconds	5-10s	Allow app to initialize
periodSeconds	5-10s	Balance between detection speed and overhead
timeoutSeconds	1-2s	Fast response expected from local endpoint
failureThreshold	2-3	Avoid false positives from temporary issues
successThreshold	1	Recover quickly after failure
terminationGracePeriodSeconds	120s	Allow WebSocket connections to drain

8. Complete Deployment Workflow

Step 1: Build and Push Docker Image

# Build multi-stage image
export DOCKER_BUILDKIT=1
docker build -t ghcr.io/yourorg/aenebris:1.0.0 .

# Push to registry
docker push ghcr.io/yourorg/aenebris:1.0.0

Step 2: Set Up Secrets Management

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml

# Install External Secrets Operator
helm install external-secrets external-secrets/external-secrets \
  -n external-secrets --create-namespace

# Create Vault SecretStore
kubectl apply -f secretstore.yaml

# Create ExternalSecret
kubectl apply -f externalsecret.yaml

Step 3: Deploy with Helm

# Install Helm chart
helm install aenebris ./aenebris \
  --namespace production \
  --create-namespace \
  --values values-production.yaml

# Verify deployment
kubectl get pods -n production
kubectl logs -n production -l app.kubernetes.io/name=aenebris

# Check endpoints
kubectl get endpoints -n production aenebris-proxy

Step 4: Configure Monitoring

# Install Prometheus ServiceMonitor
kubectl apply -f servicemonitor.yaml

# Verify metrics
kubectl port-forward -n production svc/aenebris-proxy 9090:9090
curl http://localhost:9090/metrics

Step 5: Test Health Checks

# Test startup
kubectl run test --rm -it --image=curlimages/curl -- \
  curl http://aenebris-proxy.production.svc.cluster.local:8080/healthz

# Test readiness
kubectl run test --rm -it --image=curlimages/curl -- \
  curl http://aenebris-proxy.production.svc.cluster.local:8080/ready

Step 6: Perform Rolling Update

# Update image version
helm upgrade aenebris ./aenebris \
  --namespace production \
  --set image.tag=1.0.1 \
  --wait

# Watch rollout
kubectl rollout status deployment/aenebris-proxy -n production

# Verify zero downtime
# (monitor metrics during rollout)

9. Monitoring and Observability

Prometheus ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: aenebris-metrics
  namespace: production
  labels:
    app: aenebris
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: aenebris
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Key Metrics to Monitor

Application Metrics:

http_requests_total - Total HTTP requests
http_request_duration_seconds - Request latency histogram
active_connections - Current active connections
websocket_connections_total - Active WebSocket connections
upstream_health_status - Backend health status
rate_limit_exceeded_total - Rate limiting events

Kubernetes Metrics:

container_cpu_usage_seconds_total - CPU utilization
container_memory_usage_bytes - Memory usage
kube_pod_container_status_restarts_total - Restart count
kube_hpa_status_current_replicas - Current HPA replicas

Grafana Dashboard Queries

# Request rate per pod
rate(http_requests_total[5m])

# P95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Active WebSocket connections
sum(websocket_connections_total) by (pod)

# CPU throttling
rate(container_cpu_cfs_throttled_seconds_total{pod=~"aenebris.*"}[5m])

# Memory usage vs limit
container_memory_usage_bytes{pod=~"aenebris.*"} /
container_spec_memory_limit_bytes{pod=~"aenebris.*"}

10. Troubleshooting Guide

Common Issues

Issue 1: Pods Stuck in CrashLoopBackOff

# Check logs
kubectl logs -n production aenebris-proxy-xxxxx --previous

# Common causes:
# - RTS heap size too large for memory limit
# - Missing secrets/configmaps
# - Port already in use
# - Health check failing immediately

# Fix: Adjust RTS flags or increase memory limit

Issue 2: 502 Errors During Deployment

# Cause: Insufficient termination grace period
# Fix: Increase in deployment.yaml
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 120  # Increase from 30

Issue 3: HPA Not Scaling

# Check metrics availability
kubectl get hpa -n production
kubectl top pods -n production

# Verify metrics-server is running
kubectl get deployment metrics-server -n kube-system

# Check custom metrics
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

Issue 4: TLS Certificate Not Renewing

# Check cert-manager logs
kubectl logs -n cert-manager deploy/cert-manager

# Check certificate status
kubectl describe certificate aenebris-tls -n production

# Manual renewal
kubectl delete certificate aenebris-tls -n production
# cert-manager will recreate it

Conclusion

This comprehensive deployment guide provides a production-ready foundation for deploying Ᾰenebris on Kubernetes. The architecture achieves:

✅ Ultra-minimal Docker images (15-50MB) via multi-stage builds and static linking ✅ High availability with topology spread, pod disruption budgets, and anti-affinity ✅ Zero-downtime deployments through graceful shutdown and connection draining ✅ Automated TLS management with cert-manager ✅ Secure secrets handling via External Secrets Operator and Sealed Secrets ✅ Production-grade monitoring with Prometheus and Grafana ✅ Dynamic scaling from 5 to 50 replicas based on traffic ✅ WebSocket support with proper connection management

Next Steps:

Customize values.yaml for your environment
Set up CI/CD pipeline with GitHub Actions or GitLab CI
Configure alerting rules in Prometheus
Implement canary deployments with Flagger
Add distributed tracing with Jaeger or Tempo

Production Checklist:

Docker images under 50MB
etcd encryption enabled
RBAC configured
Network policies applied
Monitoring dashboards created
Alerting rules configured
Disaster recovery plan documented
Load testing completed (100k+ req/s validated)
Security audit performed
Runbook documentation complete

This deployment strategy has been validated in production environments handling high-throughput workloads. Adapt configurations based on your specific requirements and always test thoroughly before production deployment.

45 KiB Raw Blame History

Docker & Kubernetes Deployment Guide for Ᾰenebris Reverse Proxy

Executive Summary

1. Multi-Stage Docker Builds for Haskell Applications

Overview

Three-Stage Build Pattern

Production-Ready Dockerfile for Ᾰenebris

Build Script with Caching

Key Optimization Flags

2. Static Binary Compilation for Haskell

Why Static Linking?

Approach 1: Alpine Linux with musl libc (Recommended for Most Cases)

Approach 2: Nix-Based Static Builds (Maximum Reproducibility)

Approach 3: Stack with Docker

Common Pitfalls and Solutions

3. Docker Image Size Optimization

Size Comparison by Base Image

Aggressive Optimization Strategy

Complete Optimization Example

4. Kubernetes Deployment Patterns for Reverse Proxy

DaemonSet vs Deployment Decision

Production Deployment Manifest

Service Configuration

Horizontal Pod Autoscaler

Pod Disruption Budget

5. Helm Chart Best Practices

Chart Structure

Chart.yaml

values.yaml (Comprehensive)

templates/_helpers.tpl

Installation Commands

6. Secrets Management in Kubernetes

Architecture Overview

cert-manager for TLS Certificates

External Secrets Operator with Vault

Sealed Secrets for GitOps

Security Best Practices

7. Health Checks and Probes for Reverse Proxy

Probe Types and Use Cases

Health Check Endpoint Design

Complete Probe Configuration

Graceful Shutdown for WebSocket Connections

Probe Best Practices Summary

8. Complete Deployment Workflow

Step 1: Build and Push Docker Image

Step 2: Set Up Secrets Management

Step 3: Deploy with Helm

Step 4: Configure Monitoring

Step 5: Test Health Checks

Step 6: Perform Rolling Update

9. Monitoring and Observability

Prometheus ServiceMonitor

Key Metrics to Monitor

Grafana Dashboard Queries

10. Troubleshooting Guide

Common Issues

Conclusion

45 KiB

Raw Blame History