Environments

Audience: Administrators, platform operators, infrastructure engineers Time: ~15 min to read; ~5 min to add or rotate an environment Prerequisites: Control Plane is up at https://<your-host>/, you can reach Settings → Environments, and you have admin role.

TL;DR

• Control Core ships with two environments by default: sandbox (lowest, where controls are authored) and production (where promoted controls run live).
• A single Control Plane is the master for all environments; never stand up a second Control Plane.
• You may add up to 3 additional environments (5 total) — typically dev, staging, qa, etc. Each environment owns its own Bouncers, Resources, GitHub config, PIP data sources, API key, and (optionally) SPIRE workload identity.
• Each environment has one unified API Key: it is the secret a Bouncer uses to register with the Control Plane and the secret API/SDK integrations (Policies-as-Code, OEM SDK) use for that environment.
• The API Key is generated automatically at install — it never blocks deployment. You can rotate it from Settings → Environments any time.
• Controls can be created only in the lowest environment (sandbox). They are then promoted upward and modified in higher environments — never created there.

What "environment" means in Control Core

An environment is an isolation boundary inside one Control Plane:

Topology:

                 ┌──────────────────────────┐
                 │     Control Plane        │
                 │  (UI + API services)     │  ← one per customer
                 │  master for all envs     │
                 └──────────┬───────────────┘
                            │
        ┌───────────────────┼───────────────────────┐
        │                   │                       │
   ┌────▼────┐         ┌────▼────┐             ┌────▼────┐
   │ Sandbox │         │   Prod   │             │  Extra  │ (up to 3 more)
   │ env key │         │ env key  │             │ env key │
   │ bouncers│         │ bouncers │             │ bouncer │
   │ resources         │ resources│             │ resource│
   └─────────┘         └──────────┘             └─────────┘

Why one Control Plane? It is the single management surface for policy authoring, audit visibility, and license metering. Splitting it would fragment audit history, break the promotion flow, and double the licensing cost. Bouncers are the part that scales horizontally per environment.

The Environment API Key

A single key per environment serves both:

1. Bouncer ↔ Control Plane registration. When you start a Bouncer, you give it the API Key for the environment it belongs to. The Bouncer presents the key on its first call to POST /peps/register and on every subsequent heartbeat.
2. API/SDK integrations. Policies-as-Code, the OEM SDK, and any external automation that calls the Control Plane API for that environment authenticate with the same key.

Key prefixes make accidental cross-environment use obvious:

When is it generated?

• Auto-generated at install. The deployment scripts and Helm chart write a fresh, high-entropy key for each default environment into both the Control Plane database and the Bouncer's environment variable / Kubernetes Secret. Installation never blocks on you logging in to mint a key.
• You can re-generate it any time from Settings → Environments → <env> → Rotate API Key. The old key remains valid for a 60-minute grace window so Bouncers can re-roll without dropping traffic.

Is it a blocker for installation?

No. If the install process generates one for you, deployment proceeds end-to-end without the page being touched. You only return to the page if you want to:

• View whether a key is configured for an environment (status pill).
• Rotate the key (e.g. after a personnel change or as part of a scheduled rotation).
• Generate the first key for a newly added environment.

Lifecycle and rotation

install     ─►  key auto-generated, written to PAP DB + Bouncer secret store
operate     ─►  Bouncers register and heartbeat with the key; SDK integrations use it
rotate      ─►  admin clicks "Rotate API Key" → new plaintext shown ONCE
                old key remains valid for 60 min (grace window)
                ops rolls Bouncers (compose restart / kubectl rollout) onto new key
                old key revoked at end of grace window

Always copy the plaintext key the moment it is shown — the Control Plane only stores the hash.

Cheatsheet — Helm and Compose values

Hand this to the infrastructure team when they update deployment manifests.

Tip: When you add a 3rd / 4th / 5th environment from the UI, the Control Plane shows you a snippet pre-filled with the new slug and API Key. Drop that snippet into your Helm values or compose file and roll the Bouncer.

Adding more environments later

1. Settings → Environments → Add Environment (button is disabled at 5).
2. Pick a slug (lowercase, hyphenated, unique — e.g. staging).
3. Optionally set GitHub repo/branch/path for that env (otherwise inherits from the lowest env's repo on first publish).
4. Click Create. The page generates the API key once and shows it.
5. Copy the deployment cheatsheet snippet shown by the modal.
6. In your infrastructure tool of choice (Helm or compose), add the new environment block and the new Bouncer that points at it. Roll out.
7. The new Bouncer registers itself with the Control Plane on first start; you'll see it in Settings → Bouncers and its protected resource in Settings → Resources.

SPIRE workload identity (optional, per environment) [#spire]

SPIRE provides cryptographic workload identity (SPIFFE SVIDs) so that a Bouncer authenticates to the Control Plane (and, where supported, the Control Plane to Redis, Postgres, and the policy sync path) using mTLS with short-lived rotated certificates instead of a long-lived shared secret.

Posture matrix

Per the Control Plane bouncer registration policy:

• When the env row has spire_enabled=true, the Control Plane requires a valid SVID matching spiffe://<trust-domain>/bouncer/<bouncer-id> on every register/heartbeat. The API Key is not consulted.
• When SPIRE is disabled for that env, the API Key path is the only path. SVID headers are ignored.
• The global SPIRE_ATTESTATION_REQUIRED=true env-var override forces SPIRE on every environment regardless of UI state.

Wiring (Bouncer side)

The Bouncer ships with the SPIRE Workload API agent socket already wired. Set:

SPIRE_AGENT_SOCKET=/run/spire/sockets/agent.sock
SPIRE_TRUST_DOMAIN=<value matching the env row>     # only needed for legacy global config

In Kubernetes, mount the SPIRE agent socket as a hostPath volume on the Bouncer pod. In Docker Compose, mount the host socket via:

volumes:
  - /run/spire/sockets:/run/spire/sockets:ro

Envoy already has SDS configured to fetch SVIDs over this socket; no additional cert plumbing is required on the Bouncer side.

Wiring (Control Plane internal connections)

The Control Plane is a single-instance service so its internal connections (Postgres, Redis, and policy distribution) cannot be per-env in the strict sense. Instead, the internal SPIRE helper resolves the Control Plane posture as:

• SPIRE is on for internal traffic if any active env has spire_enabled=true or CC_SPIRE_INTERNAL=true is set.
• Trust domain comes from the lowest env row first, falling back to SPIRE_TRUST_DOMAIN.

When that returns enabled=true, your infra team should:

• Ensure the Control Plane Pod has the SPIRE agent socket mounted.
• Switch Postgres, Redis, and policy-distribution connection strings from cleartext to mTLS variants.
• Issue per-workload SPIFFE IDs of the form spiffe://<trust-domain>/control-plane/<service> (e.g. …/control-plane/api).

Cross-reference

For the full SPIRE installation walkthrough (registering workloads, rotating trust bundles, federation): Workload Identity (SPIRE/SPIFFE).

Runbook — Adding an environment [#add-env]

Use this when promoting a 2-env install to 3, 4, or 5 environments. Total cap: 5 active envs.

Goal

Add a new environment slug staging between sandbox and production with its own Bouncer fleet, GitHub config, and API Key.

Steps

1. Plan the slug. Lowercase, hyphenated, 3–32 chars, unique. The slug is immutable (used in deployment scripts, policy bundle paths, audit records).
2. From the UI: Settings → Environments → Add Environment button.
   • Slug: staging
   • Display name: Staging
   • Description: Pre-prod soak before promotion to production (optional)
   • Click Create & Mint Key. The API key plaintext is shown once. Copy it now.
3. From the CLI (alternative). If you bootstrap from a script, run inside the Control Plane API container:

   docker exec <control-plane-api-container> python -m app.scripts.seed_environment \
     --slug staging \
     --display-name Staging \
     --emit-key-to /tmp/staging.api-key
   docker cp <control-plane-api-container>:/tmp/staging.api-key ./staging.api-key
   

4. Update Helm values (or compose .env for non-K8s installs):

   controlcore:
     environments:
       - slug: sandbox
         isLowest: true
         ...
       - slug: staging
         displayName: "Staging"
         description: "Pre-prod soak"
         apiKey:
           secretRef: cc-env-staging-api-key
       - slug: production
         ...
   

5. Apply & roll the Bouncer. Helm:

   helm upgrade controlcore ./helm-chart/controlcore -f values.yaml
   kubectl rollout status deployment/cc-bouncer-staging
   

Compose / VM installs:

# Roll the Control Plane and Bouncers using your deployment runbook after updating environment configuration.

6. Verify.
   • Settings → Bouncers shows the new Bouncer registered.
   • Settings → Resources shows its protected resource.
   • Settings → GitHub lets you optionally set a per-env repo/branch/path.
   • Audit Logs filtered to staging shows the ENVIRONMENT_CREATED row.

Troubleshooting: 409 Cap exceeded You're at the 5-env cap. Either deactivate an unused environment (Settings → Environments → <env> → Delete) or consolidate two custom envs.

Promoting controls into the new environment

Promotion follows the Environment.order ladder. By default, Sandbox (100) → Staging (150) → Production (200). To promote a control:

curl -X POST "https://<host>/api/v1/policies/<policy-id>/promote" \
     -H "Authorization: Bearer <token>" \
     -H "Content-Type: application/json" \
     -d '{"target_environment": "staging"}'

If target_environment is omitted, the next environment in the ladder is used.

Runbook — Rotating an Environment API Key [#rotate-key]

This is a zero-downtime rotation. The old key is valid during a 60-minute grace window so Bouncers can re-roll without dropping traffic.

Steps

1. Pre-flight. Identify all consumers of the env's API Key:
   • All Bouncers tagged with ENVIRONMENT=<slug> (find them with Settings → Bouncers filtered by env).
   • Any external automation calling the Control Plane API for that env (Policies-as-Code CI jobs, OEM SDK clients, ChatOps integrations).
2. Rotate from the UI. Settings → Environments → <env> → Rotate API Key. The new plaintext is shown once in a yellow banner. Copy it immediately into your secret store.
3. Push the new key to your secret store. AWS Secrets Manager / Azure Key Vault / Hashicorp Vault / Kubernetes Secret. Tag with controlcore.io/environment=<slug> for traceability.
4. Roll the Bouncers within 60 minutes.
   • Kubernetes: kubectl rollout restart deployment/cc-bouncer-<slug> (the new Pod picks up the rotated cc-env-<slug>-api-key Secret on start).
   • Compose: docker compose up -d cc-bouncer-<slug> after updating the .env file.
5. Update external automation. Update CI variables, ChatOps secrets, OEM SDK config — anywhere the previous plaintext was used.
6. Verify.
   • Settings → Environments → <env> → Active Keys shows the new key with status active and the previous key with status grace. After 60 minutes the previous key flips to revoked.
   • Audit Logs filtered to the env shows two rows: API_KEY_ROTATED and (after 60 min) API_KEY_REVOKED.

Troubleshooting: a Bouncer fell off-fleet during rotation Symptom: a Bouncer logs 401 after the grace window, the heartbeat stops, the Bouncer goes red on Settings → Bouncers. Fix: copy the current key from the secret store (the rotated one), restart the Bouncer (kubectl rollout restart … or docker compose restart cc-bouncer-<slug>), confirm a heartbeat lands within 60s. Prevention: add a watchdog Job to your CI/CD that runs curl -fsS -H "Authorization: Bearer <new-key>" https://<host>/api/v1/peps/<bouncer>/heartbeat against every Bouncer immediately after a rotation.

Troubleshooting: revocation never happens Cause: the key-rotation watchdog job in the Control Plane API is paused or the container restarted during the grace window. Fix: Settings → Environments → <env> → Active Keys → <old-key> → Revoke. The old key is invalidated immediately; an API_KEY_REVOKED audit row is written.

Troubleshooting

Troubleshooting: 404 Not Found on /settings/environments Older Control Plane builds mounted the settings router only at /settings/... (no /v1 prefix). The UI calls /api/v1/settings/environments. Upgrade the Control Plane to a build that includes the /v1 alias, or check that your reverse proxy doesn't strip the path differently than the rest of /api/*. Verify: curl -fsS https://<host>/api/v1/settings/environments -H "Authorization: Bearer <token>" should return 200.

Troubleshooting: Bouncer cannot register Symptom: Bouncer container logs 401 Unauthorized from POST /peps/register. Causes: (1) Wrong BOUNCER_API_KEY for this environment; (2) using a cc_sandbox_ key but registering under production; (3) old key already past the rotation grace window. Fix: confirm the env slug and key match. From the Control Plane: Settings → Environments → <env> — the configured-status pill must show green.

Troubleshooting: Rotation broke a fleet of Bouncers Cause: Bouncers were not rolled within the 60-minute grace window. Fix: re-generate the key, re-distribute it to your secret store (Kubernetes Secret, AWS Secrets Manager, Vault), and roll the Bouncers immediately. Use kubectl rollout restart deployment/<bouncer> or docker compose up -d <bouncer>.

Troubleshooting: New environment is not visible to policy sync Cause: subscription directories are scoped per environment. After adding a new environment, the embedded policy distribution server must pick up the new scope record. Fix: a Sync Now from Settings → GitHub for that environment, or wait for the next polling tick (default 30 s).

Troubleshooting: I see two Bouncers for the same environment after a rename The Control Plane deduplicates per (environment, name). If you renamed a Bouncer mid-flight, the old row is automatically swept by the orphan-PEP watchdog (every 5 min). If you need to force the cleanup: restart the Control Plane API container to trigger the cold-boot heal pass.

Next steps

• Quick-Start: Deploy and Enforce in 60 Minutes
• Multiple Bouncers
• Bouncer Deployment Guide
• Workload Identity (SPIRE/SPIFFE)
• Administrator Guide