Incident Readiness

Kubernetes kubectl Trojan Defense Playbook

A practical response-and-hardening kit for workstation-to-cluster intrusion chains: trojanized tooling, token leakage, privileged pods, and CI/CD identity abuse. Includes 0-4h containment steps, 72h recovery actions, and a 30-day hardening plan.

Mar 11, 202620 min read

What This Playbook Covers

This is a field playbook for the workstation-to-cluster intrusion pattern documented in Google's Threat Horizons report: social engineering, tool masquerading, token theft, privileged pod abuse, and cloud persistence.

Goal: reduce blast radius in the first 4 hours, eliminate persistence in 72 hours, and harden controls in 30 days.

0-4 Hour Containment Runbook

containment-runbook.mdmarkdown
# 0-4h containment runbook (high priority)

1. Isolate suspected developer endpoints immediately
   - Remove network access from corporate/VPN segments
   - Preserve forensic evidence (disk/process/network telemetry)

2. Revoke exposed cloud credentials and CI/CD service accounts
   - Disable affected service accounts
   - Rotate keys/tokens and invalidate active sessions

3. Freeze risky Kubernetes execution paths
   - Block new privileged pods and hostPath mounts
   - Temporarily pause high-risk deployment automations

4. Hunt for persistence indicators
   - Unexpected changes to deployment manifests
   - New cronjobs, init containers, or sidecars
   - Suspicious node-level processes spawned from containers

5. Contain financial blast radius
   - Restrict wallet/payment automation pathways
   - Enforce out-of-band approvals on high-value transactions

30-Day Rollout Phases

Phase 0 (Day 0): Contain and Preserve Evidence

Quarantine compromised endpoints and revoke suspect identity artifacts first; no debate, no delay.

Capture process trees, command histories, and cloud audit logs before broad cleanup starts.

Run emergency admission controls to block privileged pod creation.

Phase 1 (Day 1-3): Eradicate Persistence

Rebuild developer workstation trust chains (EDR baseline, package attestations, signed binaries).

Diff cluster workloads against last known good manifests; remove unauthorized drift.

Rotate all CI/CD and runtime identities touched during incident window.

Phase 2 (Day 4-14): Harden Identity and Runtime

Disable token auto-mount where not explicitly required.

Constrain CI/CD service accounts to minimum verbs/resources per pipeline.

Enforce Pod Security Standards and admission policies for privilege boundaries.

Phase 3 (Day 15-30): Operationalize and Validate

Run a tabletop replay of this exact kill chain with security + platform + SRE.

Add continuous controls: kubectl integrity checks, log redaction tests, privileged pod dashboards.

Track MTTR, privileged workload count, and token-exposure findings as board-level metrics.

Guardrail Templates

Admission Policy

disallow-privileged-and-hostpath.yamlyaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-and-hostpath
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: no-privileged-containers
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "Privileged containers are not allowed."
        pattern:
          spec:
            =(containers):
              - securityContext:
                  =(privileged): false
            =(initContainers):
              - securityContext:
                  =(privileged): false

    - name: no-hostpath-volumes
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "hostPath volumes are not allowed."
        pattern:
          spec:
            =(volumes):
              - X(hostPath): null

kubectl Integrity Gate

verify-kubectl-integrity.shbash
#!/usr/bin/env bash
set -euo pipefail

# CI gate: fail build if kubectl checksum is not an approved value
BIN="$(command -v kubectl)"
ACTUAL="$(shasum -a 256 "$BIN" | awk '{print $1}')"

if ! grep -q "^$ACTUAL$" .security/approved-kubectl-sha256.txt; then
  echo "Unapproved kubectl hash: $ACTUAL"
  exit 1
fi

echo "kubectl hash approved"

Readiness Checklist

Developer workstation file-transfer policy defined and enforced (AirDrop/peer transfer controls).

kubectl binary integrity verification in developer bootstrap + CI checks.

No privileged pods in production namespaces unless documented and exception-approved.

No service account tokens, API keys, or private keys in CI/CD logs.

CI/CD identities follow least privilege and short-lived token patterns.

Runtime detection alerts on suspicious child-process execution from kubectl-like tooling.

Incident runbook tested quarterly with engineering and security teams.

Download the Playbook

Get the portable version for incident drills and engineering reviews.

Read companion breach analysis

Sources

One Rule

If a developer workstation can silently alter cluster-control tooling and your cluster still accepts privileged workload paths, your architecture is waiting for an incident. Fix that before the next sprint, not after the next outage.