TL;DR: Model deploys fail differently: outputs stay well-formed while quality silently regresses, so error-rate canaries are blind. The rollout stack is an offline evaluation gate on frozen datasets, shadow traffic against production inputs, then a quality-scored canary with automatic rollback on metric regression, plus the operational discipline that weights are immutable artifacts with their own provenance.
How to approach it
Lead with the epistemic difference: both code and models can return plausible but wrong results; model evaluation often needs statistical task-quality measures in addition to deterministic checks. That reframes what "healthy" means and everything downstream (gates, canaries, rollback triggers) follows from it.
A strong answer
The failure signature inverts. Some broken code deploys announce themselves through 5xx errors or crash loops; others silently corrupt business behavior. A model can likewise serve fluent but incorrect answers. Latency and error dashboards stay green through a quality disaster. So the canary's primary metrics are task-quality scores (task-specific: answer relevance judged by evaluators or human review samples, extraction accuracy against labelled sets, safety-classifier flags), with latency as a secondary gate.
The rollout ladder.
- Offline evaluation gate. Every candidate model runs against frozen benchmark datasets before touching users. This is cheap, repeatable, and catches gross regressions; its weakness is drift between benchmarks and live traffic.
- Shadow traffic. The candidate receives mirrors of real production requests and generates answers nobody sees. You can measure representative latency and quality without returning candidate answers to users. Shadow traffic still requires authorized data handling and disabled or isolated tool side effects. Watch cost here: shadowing doubles inference spend for the duration, so sample rather than mirror when volume makes it matter.
- Quality-scored canary. Real traffic at low percentage with the quality metrics wired to automatic rollback. The rollback trigger is a statistically meaningful regression on the primary score or a guardrail breach (refusal-pattern shifts, safety flags), not a timeout threshold.
Weights change the artifact discipline too. A model version is an immutable, content-addressed artifact (weights plus tokenizer plus inference config) with recorded lineage: training data snapshot, hyperparameters, evaluation results. Rollback means repointing to the previous immutable version, never mutating files in place, because "the same model" with a silently replaced checkpoint file is undebuggable. The serving config travels separately: temperature, max tokens and system prompts are code-shaped knobs and belong in normal reviewed configuration.
| Concern | Code rollout | Model rollout |
|---|---|---|
| Failure visibility | Errors or silent business defects | Errors or silent quality defects |
| Primary gate | Tests, integration suites | Offline eval on frozen datasets |
| Canary signal | Error rate, latency | Task-quality scores, guardrail metrics |
| Rollback trigger | Threshold breach | Statistically significant score drop |
| Artifact | Immutable image digest | Immutable weight bundle plus lineage |
Interplay with traffic shape: model quality varies across segments (languages, query types, long-tail topics), so aggregate canary scores hide regressions concentrated in minorities. Slice canary metrics by segment, because a model that wins overall while failing your largest non-English segment is a failed rollout wearing good averages.
What interviewers probe next
"Who judges quality in the canary?" Layered: deterministic validators where tasks allow exact scoring, evaluator models for open-ended output (with their own known biases acknowledged), sampled human review for calibration. Say which layer gates automatically versus informs weekly reviews.
"How do you handle embeddings or retrieval components?" Same ladder, but quality surfaces indirectly through retrieval hit metrics and downstream answer quality, so the gate belongs at the pipeline level, not the component level alone.
"Rollback of prompts versus weights?" Prompt changes are config: fast, reversible, fully covered by the same scored-canary machinery. Weight changes carry the full ladder. Confusing the two tempos is a common operations mistake.
Common mistakes
Reusing code-rollout dashboards as model gates. Green error rates during a quality collapse show why task-quality checks are needed.
Canarying only aggregate metrics. Segment regressions hide inside healthy averages until support tickets arrive.
No lineage on artifacts. Without recorded training-data snapshots and eval scores per version, every rollback decision is archaeology under pressure.