The default in most operations environments is binary: either an alert triggers a full investigation, or it doesn't. There is no middle ground. But not all signals carry the same weight — and treating them as if they do is expensive, noisy, and operationally wasteful.
The Cost of Investigating Everything Equally
When every deviation from normal triggers the same investigation pipeline, three things happen:
- Cost scales with signal volume, not incident count. A fleet of 200 services generating 8,000 health checks per cycle will burn through compute and analysis resources whether those checks find anomalies or not.
- Operators drown in noise. When every investigation produces a report — no matter how trivial the finding — teams learn to ignore them. The signal-to-noise ratio collapses.
- Critical incidents get the same treatment as blips. A memory spike on a staging service receives the same analytical rigor as a production database outage. That's not thoroughness. That's misallocated attention.
The fix isn't fewer investigations. It's proportional investigation — matching the depth of analysis to the severity and confidence of what was detected.
Three Tiers, Three Cost Profiles
A proportional response model uses distinct investigation tiers, each with a different cost profile and trigger condition:
Tier 0 — Health Sweep (Zero Analysis Cost)
The baseline layer. Every watched service gets a health check at a regular interval — query response times, error rates, throughput against learned normals. No deep analysis, no complex reasoning. Just: does this look normal?
Cost: Minimal. Deterministic queries, no AI/ML inference on the hot path.
Runs on: Every service, every interval.
Outcome: Either "normal" (nothing happens) or "deviation detected" (escalate to Tier 1).
This alone eliminates the vast majority of investigation costs. If 95% of health checks return normal, 95% of cycles cost nearly nothing.
Tier 1 — Quick Investigation (Moderate Cost)
When Tier 0 detects a meaningful deviation — something outside the learned baseline — Tier 1 runs a focused investigation. It gathers relevant data, checks for known patterns, and produces a verdict with a confidence score.
Cost: Moderate. One pass through the data, one analysis cycle.
Runs on: Deviations that cross a significance threshold.
Outcome: A verdict. If confidence is high, report and close. If confidence is low, escalate to Tier 2.
This is where most actual work gets done. A memory trend that drifted 15% above baseline on a non-production service doesn't need a deep investigation. It needs someone to check if it matters — and close it if it doesn't.
Tier 2 — Deep Forensic Investigation (Full Cost)
The heavy artillery. Reserved for incidents where the stakes are high and the evidence is ambiguous. Tier 2 analyzes findings from multiple angles, verifies conclusions against available data, and produces an evidence-backed verdict with an audit trail.
Cost: Full. Deep multi-source analysis with thorough evidence review.
Runs on: Critical severity incidents, low-confidence Tier 1 verdicts, patterns that repeat across investigation cycles, or explicit human requests.
Outcome: Forensic-grade root cause analysis with supporting evidence.
This is the tier you want on a production outage at 3 AM. It's also the tier you absolutely do not want running on a routine deviation that self-resolved three minutes later.
The Economics Work
The math is straightforward. In a typical environment:
| Tier | % of Signals | Cost per Investigation | Share of Total Spend |
|---|---|---|---|
| Tier 0 | ~92% | Near-zero | < 1% |
| Tier 1 | ~7% | Moderate | ~15% |
| Tier 2 | ~1% | Full | ~85% |
By routing 92% of signals through Tier 0 — deterministic checks that cost almost nothing — the total investigation budget drops by 60–80% compared to running full analysis on everything. The remaining spend concentrates on the 1% of signals that actually warrant deep investigation.
This isn't theoretical. It's how triage works in every other domain — emergency rooms don't run full-body MRIs on every patient who walks in the door. They use a tiered system: vitals check (Tier 0), focused exam (Tier 1), full diagnostic workup (Tier 2). Operations deserves the same discipline.
Trust Gates: Letting Operators Stay in Control
Proportional response also means respecting operator judgment. Not every team wants the platform making autonomous decisions about when to investigate — and the tiering model accommodates that.
A trust gate sits between tiers. Operators set autonomy levels per service:
- Read-only mode: The platform watches and reports deviations, but never investigates without human confirmation. Ideal for services where the team wants visibility without automation.
- Recommend mode: Tier 1 investigations run automatically, but Tier 2 escalations require operator approval.
- Full autonomy: The platform escalates through all three tiers based on severity and confidence, notifying operators but not waiting for them.
The key: trust is configurable per service, not global. A payments service can run at full autonomy while a new experimental service stays in read-only. The platform adapts to the team's comfort level.
What This Means for Operations
A tiered investigation model changes how teams think about monitoring:
- Coverage becomes cheap. When health checks cost near-zero, you can watch everything — all services, all environments — without a proportional cost increase.
- Attention becomes precious. By suppressing noise at Tier 0 and Tier 1, the alerts that reach operators are the ones that actually need human judgment.
- Confidence gates replace alert storms. Tier 1 escalation to Tier 2 creates a natural quality filter — only low-confidence or high-severity findings go deep.
- Cost becomes predictable. Per-investigation cost tracking means teams know exactly what their investigation spend looks like, by tier, by service, by time period.
The industry has spent years optimizing how fast we can investigate everything. The smarter move is investigating fewer things more carefully — and matching the depth of analysis to what the situation actually demands.
Further Reading
- Site Reliability Engineering: How Google Runs Production Systems — Chapter 6, "Monitoring Distributed Systems," on the relationship between signal volume and operational load.
- The PagerDuty Incident Response Guide — On severity classification and proportional response in incident management.
- Gartner: How to Build an AIOps Business Case — Cost modeling for autonomous operations platforms.