DevOpsInterviews logo
← DevSecOps & Supply Chain Security / 17
hard★ EssentialCloudflareNetflixDatadog

Your images were scanned and signed, yet a pod starts behaving maliciously. How do you detect and respond?

Signing proves what shipped, not what runs. The strong answer covers syscall-level detection, the signals that matter, tuning against alert fatigue, and the response path once a sensor fires.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Scanning and signing cover the artifact, not the running process. Runtime security watches behaviour, syscalls, process trees, outbound connections and file writes, with sensors like Falco or Tetragon flagging what should never happen, such as a package manager spawned by a webserver. Tool choice matters less than rule quality and a rehearsed response.

How to approach it

Open by naming the gap between artifact guarantees and runtime behaviour; that framing is what the question tests. Then walk signals, tooling and honestly discuss the tuning burden, because everyone who has run Falco in production has an alert-fatigue story and pretending otherwise reads as never having run it.

A strong answer

A signed, scanned image can still be exploited at runtime: an RCE in the application executes inside a perfectly attested pod. Runtime detection is the last control plane, watching what processes actually do rather than what they promised to be.

Signals that earn alerts, drawn from how real compromises look: unexpected process ancestry, the nginx user spawning shells or piping curl into a package manager; outbound connections to addresses or ports the workload never legitimately uses, mining pools and command-and-control included, though careful operators hide inside TLS to ordinary-looking destinations; writes outside declared volumes, especially into /etc or /usr; capability or privilege transitions; ptrace against other processes; kernel module loads, which usually indicate node compromise rather than pod compromise.

Tooling: Falco is the open-source reference, an eBPF driver feeding syscall events enriched with container metadata into a rules engine. Sysdig extends it commercially; Tetragon adds eBPF-side enforcement and can kill offending processes synchronously; platforms such as Datadog CSM, Aqua and Prisma bundle sensors with response workflows. Draw the preventive sibling clearly: seccomp profiles, dropped capabilities and read-only root filesystems shrink the action space so exploits fail early and detections fire on narrower ground. Prevention reduces the space; detection catches what moves inside it.

Writing the rules is where the work actually is. Out of the box they fire constantly on a perfectly healthy cluster: cron reads as persistence, backup jobs touch strange paths, and operators exec into containers for entirely good reasons. Scope per workload and per namespace, baseline normal behaviour before tightening, tier severities so only anomalous events page a human, and suppress known-benign patterns with expiry dates. Alert-to-real-incident ratio is the health metric of the whole programme; a sensor that cries wolf gets muted, and a muted sensor is worse than none because it manufactures false confidence.

Response, rehearsed before it is needed:

rendering diagram…

Contain the workload through a tested isolation procedure and preserve evidence where that does not prolong active harm. Adding a deny-all NetworkPolicy does not cancel existing allow policies: Kubernetes NetworkPolicies combine allowed traffic. Remove matching allow paths or use an explicit-deny feature supported by the installed network implementation. Verify ingress, egress and established connections; host-networked workloads and node traffic need separate controls. If containment cannot be established promptly, stop the workload.

Rotate reachable credentials and revoke sessions where supported. For node compromise, isolate at a trusted network boundary and cordon to prevent new scheduling, then preserve required evidence and rebuild. Cordon alone only affects scheduling, and draining can destroy evidence or start replacement workloads before the compromised credentials have been revoked.

What interviewers probe next

"Why not rely on seccomp alone?" Prevention constrains which syscalls exist, but permitted syscalls still compose into attacks; detection observes composition. The two answer different questions and stack.

"How do you tune without blinding yourself?" Suppress narrowly with expiry dates attached, route by severity, review suppression lists monthly. The dangerous failure is broad suppression installed for quiet and never revisited.

"First five minutes after a high-severity firing?" Enrich and check it against known-benign patterns, quarantine networking, snapshot logs and events, page the workload owner, then choose between killing now and observing briefly for forensics based on containment urgency.

Common mistakes

Deploying Falco with stock rules forwarding everything to Slack, then calling the cluster monitored.

Treating scanning as runtime security because both sit under "container security" in vendor decks.

Skipping secret rotation after containment, leaving the attacker's credential access intact after cleanup.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.