Troubleshooting Log Framework

Audience: DevOps / Platform engineers, Security architects
Time: ~20 min
Prerequisites:

Control Plane running (see Quick-Start)
Administrator role
Feature flag flags.observability.troubleshooting_logs_v2 enabled (Settings → Feature flags)

Troubleshooting logs capture system health — Policy Bridge sync traces, Bouncer/Bouncer execution, runtime exceptions — and are strictly separate from Compliance Audit Logs. By default, detailed logging is Level 0 (off) and no telemetry leaves the customer VPC.

TL;DR

Enable the feature flag, open Settings → Troubleshooting Logs
Raise only the subsystems you need (prefer Level 1–2 in production)
Reproduce the issue; use Live Tail or export a signed package
Leave sanitization ON before sharing with Control Core support
Revert Level 4 promptly (auto-TTL is 60 minutes)

Troubleshooting: If the Settings card is missing, confirm flags.observability.troubleshooting_logs_v2 is ON and your role is Admin. Full reference: Audit vs diagnostic logs.

1. Architecture & privacy overview

Surface	Purpose	UI
Decision Trace Audit Logs	Who accessed what; PBAC decisions; regulatory evidence	`/audit`, Settings → Audit Logs
System Troubleshooting Logs	Component health, Policy Bridge sync, proxy traces, exceptions	Settings → Troubleshooting Logs

Telemetry path (all inside the sovereign VPC):

Control Plane / Control Plane / Policy Bridge / Bouncer
        │ mTLS 1.3 / local socket
        ▼
Local OpenTelemetry Collector (memory_limiter 128MB, FIFO ≤5GB)
        ├─► Local UI WebSocket viewer
        └─► Optional OTLP → Grafana Loki / Datadog Agent / Splunk HEC

Zero egress by default. OTLP exporters are disabled until an Admin configures them in Helm.

2. Subsystem logging reference (Levels 0–4)

Level	Name	Use	Overhead target
0	Disabled / Silent	Production steady-state	0.0 ms
1	Error & Critical	Alerting & health	under 0.1 ms
2	Operational Info	Standard ops	under 0.3 ms
3	Verbose Debug	Active engineering	under 1.2 ms
4	Deep Forensic Trace	Incident analysis — 60-min auto-TTL	under 3.5 ms

Level 4 automatically reverts to Level 0 or 1 unless renewed. Prefer Diagnostic Logs capture windows for short DEBUG sessions when the v2.9 flag is off.

3. SIEM / observability setup (OTLP)

Exporters are Helm-gated under observability.otelCollector.exporters (default enabled: false):

Grafana Loki — set grafanaLoki.enabled=true and the in-cluster OTLP endpoint
Datadog Agent — OTLP/gRPC to datadog-agent.monitoring.svc:4317
Splunk HEC — provide existingSecret with hec-url / hec-token; sourcetype controlcore:troubleshooting

Troubleshooting: If the Collector pod is CrashLoopBackOff, check memory_limiter limit (128 MiB) and that exporters do not point outside the VPC without intentional Admin config.

4. Air-gapped forensic export playbook

Set needed subsystems to Level 1–3 (or Level 4 with TTL awareness)
Reproduce the issue once
Settings → Troubleshooting Logs → Export Signed Log Package with sanitization ON
Verify the companion .sha256 file: sha256sum -c LOGS_*.tar.gz.sha256
Transmit the package out-of-band to Control Core Enterprise Support

Archive naming: LOGS_[CUSTOMER_ID]_[YYYYMMDD_HHMMSS]_[SANITIZED|RAW].tar.gz