AI Pilot deployment modes
Audience: Platform engineers, AI governance architects
Time: ~12 min read
AI Pilot runs on the same universal bouncer binary. For AI governance, choose Gateway Mode or App-Embedded — enterprise labels for where the PEP sits on the path. Broader topology (including Shadow) is covered in Deployment modes.
What
| Mode | Enterprise label | Placement idea |
|---|---|---|
| Shared edge hop | Gateway Mode | Clients and apps send AI traffic to a shared bouncer VIP/DNS; the bouncer forwards to LLM/GenAI upstreams. |
| Co-located with the app | App-Embedded | Bouncer runs next to one application; that app’s AI egress goes through the local bouncer. |
Both modes use the AI Inspection Engine for PII and safety, and the same controls from the Control Plane.
Why
- Gateway Mode — centralize governance for many apps and providers with one (or few) PEPs; simpler DNS cutover for LLM base URLs.
- App-Embedded — lowest hop latency per app, strong isolation per workload, natural fit for Kubernetes pods.
How — capability matrix
| Concern | Gateway Mode | App-Embedded |
|---|---|---|
| TLS termination | Often terminates client TLS at the bouncer (or LB in front); upstream TLS to providers as configured | Usually localhost/mesh to the app; egress TLS to providers from the bouncer |
| DNS / routing | Point LLM base URLs or edge DNS at the bouncer | App config points at local bouncer listen address |
| Placement | Shared network hop in front of many workloads | Same host/pod as one workload |
| Redis (token limits) | Strongly recommended for shared global counters across replicas | Recommended if you scale multiple replicas of the same embedded pattern |
| Blast radius | Misconfig can affect all apps behind that VIP | Limited to the co-located app |
| When to use | Enterprise AI edge, multi-app LLM gateway, shared MCP broker | Per-service AI governance, strict tenancy isolation, mesh-style pods |
Technical reference (env names only)
| Enterprise label | Typical BOUNCER_TYPE | Typical TARGET_HOST |
|---|---|---|
| Gateway Mode | reverse-proxy | Shared upstream (API gateway, LLM router, or provider-facing proxy) |
| App-Embedded | sidecar | Local application listen address |
Full install steps: Quick start · platform Deployment modes.
Verify
- In AI Pilot, select the bouncer — badge shows Gateway Mode or App-Embedded.
- Send traffic only via the intended path; confirm Activity rows.
- Attempt a direct provider call (bypass) — it must not appear in AI Pilot (proves governance coverage).
Troubleshooting: Badge and docs disagree? The badge reflects how that PEP was registered/deployed. Redeploy with the correct type; see Troubleshooting — By deployment mode.
Troubleshoot
| Symptom | Gateway Mode tip | App-Embedded tip |
|---|---|---|
| 502 / upstream errors | Check shared TARGET_HOST and provider reachability from the bouncer network | Check local app still listening; confirm loopback target |
| Some apps ungoverned | Those apps still call the provider DNS directly | Only the co-located app is in scope — deploy another embedded bouncer or move to Gateway |
| Token limits drift | Add Redis for the gateway replica set | Add Redis if multiple replicas share the same logical limit |