DevOpsInterviews logo
← Observability, SLOs & Reliability / 15
mediumCloudflareWalmart Global TechRed Hat

One Grafana serves three hundred teams and half the dashboards are abandoned. How do you organise it?

Dashboard sprawl is an operating-model problem wearing a tooling costume. The scored answer moves definitions into git, attaches ownership metadata to every panel, and deletes without ceremony.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Treat dashboards as code: provisioned from git through the provisioning system or Terraform, filed in team-owned folders under RBAC, each carrying a team tag and an expiry, with a quarterly delete of anything nobody claims. Folders beat orgs for almost every estate; reserve separate orgs for genuine isolation boundaries.

How to approach it

Ask three things first: where authentication lives (SSO or local users), whether alerting has already accumulated inside Grafana, and how many people can currently save changes. The sprawl is usually a permissions story before it is a tidiness story.

A strong answer

Structure follows ownership. Top-level folder per team, nested folders for domains within big teams. RBAC grants the owning team edit rights on its folder and view on everyone else's. Datasources are defined centrally once, with stable UIDs that dashboard JSON references by UID, so a datasource rename never breaks four hundred panels. Organizations are the heavier unit: separate datasources, separate users, no cross-org views. They fit real isolation boundaries (production versus partner-facing, regulated versus not), not three hundred teams; one org per team multiplies administration and kills the cross-team dashboard nobody planned but everybody needs.

Provisioning as code makes review possible. Dashboards live as JSON in git, applied by the dashboard provisioning mechanism (a sidecar watching ConfigMaps in the kube-prometheus-stack pattern) or Terraform's grafana_dashboard resource. A pull request becomes the quality gate, and CI lints what manual saving never enforces: required owner tag, datasource referenced by UID rather than name, template defaults sane, title conventions followed. Estimate migration effort from a sampled dashboard inventory, including query repair, owner review and permissions; export/import time alone omits most coordination work.

Hygiene is scheduled, not aspirational. A job lists last-updated timestamps and ownership. Where the edition provides usage insights, include viewer counts; each quarter, unowned or untouched dashboards get archived out of the way. Deleting is the point: search that returns one authoritative latency dashboard beats search returning twelve stale copies with contradictory thresholds. Library panels give shared components a single versioned home so the canonical request-rate panel stops being forked.

Alerts follow the same discipline. Unified alerting routes notification policies by team label, so on-call mapping lives in one place instead of scattered contact points. Alert rules provision alongside dashboards, linted for a runbook_url annotation, because an alert without a runbook link is a pager message addressed to nobody.

Access closes the loop: SSO with group/team synchronization where the provider and Grafana edition support it, service accounts with rotating tokens for automation, no shared admin logins. After cutover, make the UI read-only for edits outside provisioned folders; otherwise hand-edits fight git reconciliation forever and drift wins.

The position worth stating: below twenty teams this much machinery is ceremony, folders plus export discipline suffice. Above roughly a thousand dashboards without lint gates and expiry enforcement, sprawl returns within months because nothing changed about incentives.

What interviewers probe next

"Orgs or folders for multi-tenancy?" Folder permissions protect dashboards, not all queries against shared datasources. Fine-grained RBAC, datasource permissions and team sync vary across OSS, Enterprise and Cloud. Use backend credentials/tenant isolation, datasource permissions where supported, or separate organizations/instances for the actual data boundary; Grafana folders do not enforce retention.

"Centralising alerts caused spam. Fix it." Route by labels to per-team contact points, require runbook links at lint time, and review new alert rules in PRs like code, which they are.

"Someone hand-edited production dashboards and now git fights Grafana." Remove the ability: viewers cannot save, editors work against a staging folder or branch preview, and GitOps reconciliation overwrites stragglers on its next sync rather than nightly drama.

Common mistakes

Creating an organization per team and discovering six months later that nobody can build the cross-team view.

Exporting JSON into git once, then letting manual edits resume, so the repo becomes a museum of stale definitions.

Treating cleanup as a big-bang purge that terrifies teams, instead of quiet quarterly expiry-plus-archive that nobody fights.

References

Grafana roles and permissions.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.