TL;DR: Prometheus periodically scrapes configured HTTP targets and stores their samples. Discovery and relabeling select those targets; exporters translate systems that do not expose compatible metrics. A Pushgateway can retain results from short-lived service-level batch jobs, but its retained values need lifecycle management.
How to approach it
State the pull model in the first sentence and then derive the rest from it, because service discovery, the health signal and the short-lived-job problem are all consequences of it. If you are asked to compare pull and push, avoid the symmetric non-answer: say what pull buys and name the one case where it does not work.
A strong answer
Your service exposes an HTTP endpoint, conventionally /metrics, returning plain text:
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 48271
http_requests_total{method="GET",status="500"} 13
process_resident_memory_bytes 5.2428e+07
Prometheus requests that page from every target on an interval, parses it, and appends each sample with a timestamp. The application does not send anything and holds no connection. It just keeps counters in memory and renders them when asked.
Targets can come from static configuration or service discovery. Kubernetes discovery reads API objects; scrape configuration and relabeling decide which endpoints to collect. An annotation only has meaning when the configuration interprets it, and a ServiceMonitor requires a compatible operator setup. Static targets remain reasonable for a small, stable set of endpoints.
Exporters cover everything that cannot expose the format itself. node_exporter for host CPU, memory, disk and network. postgres_exporter for a database. blackbox_exporter for probing a URL from outside. An exporter is a small process that translates something's native statistics into the Prometheus format on an HTTP endpoint, so from Prometheus's point of view it is an ordinary target.
Pull lets Prometheus control scrape scheduling and report scrape success through up. It does not prevent a target from returning an excessive number of series: configure sample limits and review cardinality. Inspecting /metrics helps distinguish exporter output from scrape or query problems, but application health requires application-level signals.
Short-lived jobs can finish between scrapes. For a service-level batch job, a Pushgateway can retain the last published result for collection. Machine-bound jobs can instead expose results through the node-exporter textfile collector. Ordinary long-running services should usually be scraped directly; routing everything through a gateway obscures individual target health and adds retained-series cleanup work.
One detail on the data model: a metric is a name plus labels, and every distinct label combination is its own time series. That is why putting a user ID or a request ID in a label is the standard way to destroy a Prometheus instance, and it is behind most "Prometheus memory exploded" incidents.
A green scrape can hide a failed job
Suppose up{job="billing"} remains 1 while invoices stop appearing. That value establishes that Prometheus successfully scraped the target. It does not establish that the application completed a transaction. Check an application signal such as the age of the last successful invoice batch, and compare it with the completion deadline users depend on.
For a service-level batch that publishes through a Pushgateway, expose a last-success timestamp alongside duration and processed-item counts. If yesterday's success value remains unchanged, the gateway can still return HTTP 200 today. An alert should compare the timestamp with the expected schedule, with a separate check for missing telemetry. Assign ownership for removing obsolete groups when the job is retired; restarting Prometheus does not clean up the gateway.
During diagnosis, compare the target's current exposition with Prometheus's target status, scrape timestamp and query result. A value missing at the source is an instrumentation problem; a source value absent from storage calls for scrape, relabeling or ingestion checks. Keep the endpoint private and use configured authentication when inspecting it.
See Prometheus's Pushgateway guidance for service-level jobs and the node-exporter textfile alternative for machine-bound jobs.
What interviewers probe next
"What happens if a scrape is missed?" A failed scrape sets up to 0, and previously scraped target series are marked stale. Range queries still work with samples available in their windows. A later cumulative counter can include increments during the gap, but cannot recover the timing of those increments or intermediate gauge values.
"How does it scale past one server?" Shard by scrape target, then use remote write into a long-term store like Thanos, Mimir or Cortex for global query and retention. A single Prometheus is deliberately simple and holds everything in local storage with no clustering.
"Push or pull, which is better?" Pull for services you run, because of discovery and the free liveness signal. Push where the source is ephemeral or cannot be reached, which is why OpenTelemetry's collector uses push and why the two models coexist without contradiction.
Common mistakes
Routing normal service metrics through a pushgateway, which loses the up signal and leaves stale series behind.
Using a static target list for a rapidly changing fleet without an update mechanism, so monitoring drifts from the actual services.
High-cardinality labels such as user or request identifiers, which multiply series and exhaust memory.
Assuming missing gauge samples will be reconstructed, or interpreting scrape success as proof of successful user transactions.