TL;DR: The lifecycle has five stages: generate high entropy unique per consumer, store centrally, deliver as short leases, rotate automatically with a dual-validity window so consumers migrate without downtime, and revoke fast, with all of it rehearsed. Rotation that humans execute quarterly from a runbook is archaeology, not rotation.
How to approach it
Walk the stages in order and spend your time on rotation and revocation, where candidates are weakest. Mention tool names only in passing; this is usually asked after a tooling-comparison question, and the interviewer is testing lifecycle thinking, not product familiarity.
A strong answer
Generation: machines create credentials, never humans, with high entropy and uniqueness per consumer and per environment. One database password shared by three services and copied into staging turns rotation into a coordinated outage instead of a routine event.
Storage and delivery: a central manager such as Vault or AWS Secrets Manager holds them; workloads receive leases at runtime through a sidecar, CSI driver or startup fetch. Nothing bakes into git, images or Terraform state. Dynamic secrets are useful where the target system supports them: per-deployment database credentials issued with TTLs of hours, where expiry plus reissue replaces rotation entirely because there is nothing long-lived to leak.
The following cadences are examples to choose against provider limits, exposure and organizational policy, not universal requirements:
| Credential type | Rotation model | Practical cadence |
|---|---|---|
| Dynamic DB credentials | TTL expiry, automatic reissue | minutes to hours |
| Static DB credentials | dual-user switchover | 30 to 90 days |
| Cloud provider keys | overlapping pair, retire old | 90 days common |
| Third-party API keys | provider-dependent, worst case single-slot | quarterly |
| Certificates | automated issue and renew | 90 days or shorter |
Dual validity is the mechanism worth describing concretely: issue the new credential while the old stays valid, migrate consumers during the window, then retire the old one.
The window length is the part people guess at, and it is derivable from the slowest consumer rather than from a policy number. Suppose 200 workloads hold the credential. Those with a sidecar re-rendering every 5 minutes have all picked up the new value within about 15 minutes, allowing for stagger and a missed render. But if 3 of the 200 read the secret once at process start and cache it for the life of the pod, they do not refresh until they restart, so their pickup time is the deployment cadence, which might be a week.
So the overlap has to exceed the slowest path, not the average one: retiring the old credential after an hour leaves those 3 workloads authenticating with something that no longer works, and they will fail at the least convenient moment because nothing about their behaviour changed at rotation time. Either the window runs to the deploy cadence, or those three get a refresh path, and the second is the cheaper fix precisely because it collapses the window from days to minutes for everything. Applications caching secrets need an explicit refresh path: sidecar re-rendering with a reload signal, or fetching at request scope. Databases make this clean through two users and a switchover. Single-slot third-party APIs are the painful case since no overlap exists, which is exactly why consumers should support instant refetch, and you schedule the cutover in low traffic.
Revocation is tested, never assumed. Lease revocation asks the backend to revoke credentials and can fail or be delayed. Established database sessions may survive password/user changes, depending on backend behavior. A cached copy of a revoked static credential cannot authenticate if the target checks it and rejects it, but cached authorization or already-issued sessions may remain valid. Test those paths separately. Vault lease revocation. Run drills: revoke a live credential mid-week and confirm recovery without heroics. Incident-driven rotation follows one rule: if a credential could have been exposed, rotate everything reachable with it, not merely what you confirmed taken, because confirming exfiltration precisely is a luxury incidents do not grant.
Audit closes the loop. The manager logs every read, so anomalous access, a credential read from a subnet that never reads it, alarms immediately instead of surfacing in next quarter's review.
What interviewers probe next
"Consumers cache secrets for a day and rotation breaks them. Fix?" Either shorten cache life with a refresh signal, or extend dual validity beyond the longest cache TTL. Choose based on whether you can change consumer code at all.
"How do you rotate the secrets system's own credentials?" Vault root tokens are revoked after use and generated through a controlled procedure when needed. Rekeying unseal or recovery shares is a separate operation; root tokens are not reshared. Vault rekey. Rarely exercised anywhere, which is precisely why you drill it.
"Break-glass credentials?" Few, offline, MFA-wrapped, monitored, with any opening triggering an alert. They exist for the day rotation itself is broken.
Common mistakes
Rotating the store while consumers keep serving the cached old value, discovered mid-outage.
Rotation runbooks written for auditors, never rehearsed, failing at first real execution.
One logical credential cloned across environments, so a single leak rotates three worlds and blast radius triples.