🚀 Multiple Bouncer Deployments

Guide for deploying, scaling, and configuring multiple Control Core bouncers for high-availability production environments.

📌 Overview

Control Core bouncers enforce authorization policies at the network layer. Each bouncer:

Intercepts traffic to protected applications
Evaluates policies in real time
Receives policy updates from the Control Plane automatically
Can optionally use Work ID for cryptographic workload identity

Deploy multiple bouncers to scale throughput, improve availability, and distribute load.

🚀 Deployment Patterns

1. Gateway Pattern (Centralized)

All traffic flows through a centralized bouncer fleet:

┌─────────┐      ┌─────────────────────────────────┐      ┌─────────┐
│ Clients │─────►│  Bouncer Fleet (Load Balanced)  │─────►│  Apps   │
└─────────┘      │  ┌─────┐ ┌─────┐ ┌─────┐        │      └─────────┘
                 │  │ B1  │ │ B2  │ │ B-N │        │
                 │  └─────┘ └─────┘ └─────┘        │
                 └─────────────────────────────────┘

Use cases: API Gateway, centralized authorization Advantages: Simple topology, centralized logging Trade-offs: Single point of failure (mitigated with HA)

2. Sidecar Pattern (Distributed)

Each application has a dedicated bouncer:

┌─────────┐      ┌──────────────┐      ┌──────────────┐
│ Clients │─────►│ App 1        │      │ App 2        │
└─────────┘      │ ┌──────────┐ │      │ ┌──────────┐ │
                 │ │ Bouncer  │ │      │ │ Bouncer  │ │
                 │ └────┬─────┘ │      │ └────┬─────┘ │
                 │      ▼       │      │      ▼       │
                 │ ┌──────────┐ │      │ ┌──────────┐ │
                 │ │ Backend  │ │      │ │ Backend  │ │
                 │ └──────────┘ │      │ └──────────┘ │
                 └──────────────┘      └──────────────┘

Use cases: Microservices, containerized applications Advantages: Isolated failures, per-app policies Trade-offs: Higher resource usage

3. Hybrid Pattern

Centralized gateway for external traffic, sidecars for internal services:

┌─────────┐      ┌───────────┐      ┌───────────────────┐
│External │─────►│  Gateway  │─────►│ Internal Services │
│Clients  │      │  Bouncer  │      │  (with sidecars)  │
└─────────┘      └───────────┘      └───────────────────┘

Use cases: Complex architectures, mixed trust zones Advantages: Flexible, defense in depth Trade-offs: Most complex to manage

🤖 Failure Semantics (Production)

Enterprise teams should define failure semantics as part of deployment design, not as an afterthought.

Primary objective: Preserve protected resource availability during bouncer instance failures.
Recommended HA path: Load Balancer -> multiple reverse-proxy bouncers -> protected apps.
Degraded policy mode: Maintain a policy-driven baseline route set for low-risk availability endpoints during partial control-plane or enforcement degradation.
Strict route classes: Keep high-risk routes fail-closed (admin, write, sensitive data) while allowing degraded behavior only where explicitly approved.

Pattern guidance by SLO

If uptime is the top priority: Prefer reverse-proxy fleet with active-active replicas.
If isolation is the top priority: Sidecar is strong, but validate listener failure behavior and provide resilient routing design.
If both are required: Use hybrid mode with clear route segmentation and documented failover expectations.

📌 Resource Sizing

Per-Bouncer Resources

Traffic Load	CPU	Memory	Connections
Low (<100 req/s)	0.5-1	1 GB	1,000
Medium (100-1,000 req/s)	1-2	2 GB	5,000
High (1,000-5,000 req/s)	2-4	4 GB	10,000
Very High (5,000+ req/s)	4-8	8 GB	20,000+

Fleet Sizing

Expected Load	Min Bouncers	Recommended	Max (HPA)
Development	1	2	3
Production (Small)	3	5	10
Production (Medium)	5	10	30
Production (Large)	10	20	100

📌 Configuration

Required Settings

Configure each bouncer with these required values:

Setting	Description	Example
Control Plane URL	API endpoint for registration and policy sync	`https://api.controlcore.yourcompany.com`
API Key	Credentials for Control Plane	From Settings → Environments
Bouncer ID	Unique identifier	`bouncer-us-east-1a-001`
Environment	`sandbox` or `production`	`production`
Target Host	Protected application host	`backend.yourapp.com`
Target Port	Protected application port	`8080`

Optional Settings

Setting	Description	Default
Policy sync interval	How often to check for policy updates	30 seconds
Decision cache TTL	How long to cache authorization decisions	60 seconds
Policy cache TTL	How long to cache compiled policies	300 seconds
Max concurrent requests	Connection limit per bouncer	10,000
Request timeout	Timeout for upstream requests	30 seconds

Work ID (Optional)

When Work ID is enabled, bouncers obtain cryptographic workload identity for zero-trust communications. See Work ID Configuration for setup.

Setting	Description
Work ID enabled	Enable workload identity
Trust domain	Organizational identity boundary
Join token	One-time token for bouncer attestation

🚀 Kubernetes Deployment

Basic Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bouncer
  namespace: control-core
spec:
  replicas: 10
  selector:
    matchLabels:
      app: bouncer
  template:
    metadata:
      labels:
        app: bouncer
    spec:
      containers:
        - name: bouncer
          image: controlcore/bouncer:2.0.0
          ports:
            - containerPort: 8080
              name: http
          env:
            - name: CONTROL_PLANE_URL
              value: "https://api.controlcore.yourcompany.com"
            - name: API_KEY
              valueFrom:
                secretKeyRef:
                  name: bouncer-secrets
                  key: api-key
            - name: BOUNCER_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: ENVIRONMENT
              value: "production"
            - name: TARGET_HOST
              value: "backend.yourapp.com"
            - name: TARGET_PORT
              value: "8080"
          resources:
            requests:
              memory: "2Gi"
              cpu: "1000m"
            limits:
              memory: "4Gi"
              cpu: "2000m"
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 5

High Availability

Spread bouncers across nodes and availability zones:

spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: bouncer
                topologyKey: kubernetes.io/hostname
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: bouncer

Auto-Scaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: bouncer-hpa
  namespace: control-core
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bouncer
  minReplicas: 10
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70

Pod Disruption Budget

Ensure minimum availability during upgrades:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: bouncer-pdb
  namespace: control-core
spec:
  minAvailable: 5
  selector:
    matchLabels:
      app: bouncer

📌 Load Balancing

Network Load Balancer (AWS)

apiVersion: v1
kind: Service
metadata:
  name: bouncer
  namespace: control-core
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: bouncer
  ports:
    - name: http
      port: 80
      targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

Ingress (Layer 7)

Use NGINX Ingress or your cloud provider's load balancer to route traffic to the bouncer service. Configure TLS termination at the ingress layer.

📌 Cache Tuning

Load Profile	Policy Cache TTL	Decision Cache TTL	Cache Size
Low	10 min	5 min	10,000
Medium	5 min	1 min	50,000
High	5 min	1 min	100,000
Very High	3 min	30 sec	500,000

Increase cache TTL for stable policies; decrease when policies change frequently.

📌 Monitoring

Health Endpoints

Liveness: GET /health — Returns 200 when bouncer is running
Readiness: GET /ready — Returns 200 when ready to accept traffic

Metrics

Bouncers expose Prometheus-compatible metrics. Key metrics:

Metric	Description
Requests total	Total requests processed
Request duration	Latency histogram
Policy decisions	Allow/deny counts
Control Plane latency	Response time to Control Plane
Cache hits/misses	Cache effectiveness

Alerts

Set up alerts for:

Bouncer unhealthy for > 2 minutes
Request error rate > 5%
P95 latency > 500ms
Policy sync lag > 5 minutes

⚡ Performance Benchmarks

Bouncer Count	Total Capacity	Latency (p95)	Availability
3	1,500 req/s	<50ms	99.5%
10	5,000 req/s	<20ms	99.9%
20	10,000 req/s	<15ms	99.95%
50	25,000 req/s	<10ms	99.99%

🛠️ Troubleshooting

Bouncer not connecting: Verify Control Plane URL is reachable, API key is valid, and firewall allows outbound HTTPS.

High latency: Enable and tune decision cache, increase timeout values, deploy bouncers closer to applications.

Policy not updating: Check policy sync interval, verify Control Plane connectivity, review bouncer logs for sync errors.

All requests denied: Verify policies are enabled and assigned to the correct environment and resource. Test in sandbox first.

Bouncer Deployment - Initial bouncer setup
Control Plane Scaling - Scale the Control Plane
Work ID Configuration - Workload identity setup
Security Best Practices - Hardening guidance