TLS renewal is automated and has worked for months. One night every client starts failing certificate verification. What happened?

TL;DR: First verify whether the leaf renewed and whether clients receive that certificate. Read the actual bytes the server serves with openssl s_client before touching any configuration: then test the leaf dates and hostname, chain completeness and client trust.

How to approach it

Ask for the exact error string first, because it partitions the search space: certificate has expired points at dates, unable to verify or unknown authority points at the chain, hostname mismatch points at SANs. Ask which clients fail, all of them or only old ones, and whether the timing correlates with a renewal or a deploy. Those three answers eliminate half the causes before you open a terminal.

A strong answer

The mistake is treating "automated renewal" as one event. Renewal replaces the leaf certificate. Verification depends on the leaf, every intermediate above it, and the trust store on the client, and automation routinely updates only the first of those.

Look at what is actually served, not at what you believe is deployed:

openssl s_client -connect api.example.com:443 -servername api.example.com \
  -showcerts 2>/dev/null | openssl x509 -noout -dates -issuer -subject
openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null \
  | grep -E "Verify return|verify error"

Then branch on what you see.

The intermediate is missing from what the server sends. The ACME tool wrote the leaf into the certificate but the deployment only picked up cert.pem instead of fullchain.pem, or a Helm template mounted the wrong key of the secret. Clients that cache the intermediate work; fresh clients fail with unknown authority. Fix the served chain, not the certificate.

An intermediate or cross-sign above your CA expired. This is the midnight case with no deploy nearby. The September 2021 DST Root CA X3 expiry exposed verifier-specific path building. Let's Encrypt's cross-sign arrangement preserved older Android compatibility, while OpenSSL 1.0.0–1.0.2 rejected that compatibility chain. The server's own date math is happy; the client cannot build a path to a root it trusts. Old clients failing while new ones succeed is the signature.

The secret updated but nothing reloaded it. In Kubernetes, cert-manager dutifully renews the Secret, and the ingress controller keeps serving the old chain from memory until it reloads. Some proxies read the certificate per handshake, others cache it for hours. Restarting the controller is a diagnostic here: if the new chain appears afterwards, the bug is a missing reload trigger, and it can recur at a later renewal.

The client side moved. A base image bump swapped out the CA bundle, or a pinned root in an embedded device aged out. If one fleet fails and the browser works, compare verifier behavior and trust stores while checking whether the server chain is compatible with the supported clients.

Mitigation comes before elegance: restore a verified valid serving certificate and chain, using previous material only if it is still acceptable, restore service, then diagnose outside the critical path.

Test the served chain with the affected client's trust

A renewed Secret and one successful browser request leave several possibilities open. Test each serving endpoint with the expected server name, then reproduce with the affected client's trust bundle and verifier. For an authorized endpoint:

openssl s_client -connect api.example.com:443 -servername api.example.com   -verify_hostname api.example.com -verify_return_error -CAfile client-roots.pem </dev/null

Here client-roots.pem represents the intended client trust anchors, not a bundle copied from the server to make validation pass. Record the leaf fingerprint and verification result per endpoint. Different fingerprints can expose one proxy replica still serving the old certificate.

Before reverting a certificate, verify that it remains valid for the hostname and current time and has not been revoked or compromised. A previous file is not automatically a safe return path. After correction, test a fresh handshake without a cached intermediate and confirm every serving replica has loaded the intended material.

What interviewers probe next

"How would you monitor for this class of failure?" Alert on expiry of every certificate in the chain at 30, 14 and 7 days, not just leaves, and run a synthetic TLS handshake from a minimal client with the intended trusted roots but no cached intermediates, which catches missing intermediates that a browser silently forgives.

"Why did staging not catch it?" Staging usually shares the client fleet with production only in name. A staging curl uses the machine CA bundle; production mobile clients carry their own. Reproduce with the real client, not with curl.

"Doesn't restart always fix stale certificates?" A restart helps only if the correct certificate is available and the process loads it on startup. Hot-reload mechanisms and per-handshake certificate lookup are separate, implementation-specific behaviors. Knowing that difference for the stack you run is part of the answer.

Common mistakes

Saying "the certificate expired, renew it" without reading the served chain. The question states that automation worked previously; it does not prove this renewal succeeded or that the new chain was loaded.

Fixing production by disabling verification (--insecure, ssl_verify=false). That converts an outage into a silent MITM exposure, while hiding the verification failure rather than repairing it.

Treating the leaf's notAfter date as the whole story. Intermediates and roots have their own clocks, and they expire on five-to-ten-year cadences nobody remembers setting up.

References

Let’s Encrypt cross-sign history.