The Cost of Watching Everything
Engineering teams often assume more data means more insight. They instrument every function, collect every log level, and alert on every threshold. The result is a monitoring stack that costs more than the infrastructure it watches — and still fails to catch the outage before customers do.
Datadog and similar platforms can scale to astronomical bills when telemetry volume is unchecked. One mid-sized SaaS company we worked with was spending $14,000 per month on observability alone. Half of that data had never been queried. The alerts they did receive were mostly false positives. Their mean time to resolution (MTTR) was 47 minutes. After cutting telemetry volume by 60 percent and redesigning their alerting strategy, MTTR dropped to 12 minutes. Cost dropped by 55 percent.
The lesson is simple: observability is not a data hoarding exercise. It is a decision-support system. If the telemetry does not help someone make a faster, more confident decision during an incident, it is waste.
The Three Signals That Matter
Google’s Site Reliability Engineering book defines four "golden signals": latency, traffic, errors, and saturation. These are a good starting point, but in practice we have found that three categories cover most operational needs for cloud-native systems:
- Latency and error rates — How fast are requests completing, and how many are failing? These are the only metrics that directly reflect user experience.
- Resource saturation — CPU, memory, disk I/O, and network throughput. Not because individual thresholds matter, but because saturation predicts failure before it happens.
- Change events — Deployments, configuration changes, feature flag flips, and autoscaling actions. Most incidents follow a change. If you cannot correlate a spike with a change, you are debugging in the dark.
Everything else is optional until you have these three locked down. Database connection pool depth, queue length, cache hit ratio — these are useful, but they are secondary. They explain why latency spiked; they do not tell you that users are suffering.
Latency and error rates
Measure latency at the edge, not inside the service. A fast internal API mean nothing if the load balancer is queuing requests for three seconds. Track p50, p95, and p99. Alert on p95 latency crossing your service level objective (SLO), not on average response time. Averages hide outliers, and outliers are what damage customer trust.
Error rates should be measured as a percentage of total requests, not as an absolute count. One hundred errors per minute is catastrophic at low traffic and irrelevant at high traffic. Express it as a ratio, and alert when it breaches your error budget.
Resource saturation
Watch for trends, not thresholds. A CPU at 85 percent is not inherently bad if it is steady. A CPU climbing from 20 percent to 60 percent over two hours is a signal — even if 60 percent is below your alert threshold. Use baseline anomaly detection where possible. Static thresholds generate alert fatigue in autoscaling environments because "normal" shifts daily.
Metrics, Logs, and Traces — In That Order
The three pillars of observability are often presented as equals. They are not. They are a hierarchy:
- Metrics tell you that something is wrong. They are cheap to collect and fast to query. Start here.
- Logs tell you what happened. They are expensive to store and slow to search. Use them after metrics narrow the scope.
- Traces tell you where the time went. They are invaluable for distributed systems, but they carry the highest overhead. Instrument critical user journeys first. Everything else can wait.
Teams that start with distributed tracing often drown in span volume before they have a single meaningful dashboard. Build your metrics layer first. Define SLOs. Then add logs and traces where they answer specific questions that metrics cannot.
Alert on Symptoms, Not Vital Signs
The most common alerting mistake is paging an on-call engineer because a server reached 90 percent CPU. CPU is a vital sign. It is not a symptom. A user-facing symptom is "checkout latency exceeds two seconds" or "payment success rate drops below 99.5 percent." These are the conditions that matter.
Alert on what your users feel, not on what your servers feel. If an internal metric degrades but users notice nothing, log it. Do not page anyone.
Multi-condition alerts reduce noise further. A single high-error spike can be a transient network blip. High errors combined with elevated latency and sustained traffic is an incident. Combine conditions. Route alerts to the team that owns the affected service. Attach a runbook link. An alert without context is just a siren.
Start Small, Expand Deliberately
You do not need a unified observability platform on day one. You need three dashboards: one for latency, one for errors, and one for saturation. Define one SLO per critical user journey. Set two alerts. Everything else is decoration until those basics prove their value.
Once the foundation is solid, expand in this order:
- Add log aggregation for the services that fail most often.
- Instrument distributed tracing for your top three user journeys.
- Build service dependency maps so you can see blast radius during an incident.
- Introduce error budgets and review them in weekly operational meetings.
Each layer should justify itself with shorter MTTR or fewer customer-impacting incidents. If adding a tool does not measurably improve either, remove it.
When Observability Pays Off
Observability is not an insurance policy. It is operational infrastructure. The return on investment shows up in three places: faster incident resolution, fewer false alerts, and confident deployments. Teams with solid observability deploy twice as often and recover from failures in one-quarter the time of teams flying blind.
The goal is not perfect visibility. The goal is sufficient visibility to act with confidence. Most teams are closer to that goal than they think. They just need to stop measuring everything — and start measuring what matters.