Skip to content

Alerting Standards

Purpose

The purpose of alerting is to notify engineers when a service requires intervention to restore or protect the experience of its users.

Alerts are a tool for detecting operational problems, not for reporting every abnormal metric. Modern systems expose thousands of metrics, many of which fluctuate as part of normal operation. Alerting on every deviation creates noise, increases alert fatigue and makes genuine incidents harder to identify.

An alert should answer a simple question:

Does somebody need to take action now?

If the answer is no, it is probably not an alert.

Symptoms and Causes

One of the most important principles when designing alerts is distinguishing between symptoms and causes.

A symptom is an observable impact on a service. It describes what is happening from the perspective of the service or its users.

A cause is the underlying technical reason that the symptom exists.

Alerts should report symptoms. Engineers should use metrics, logs and traces to determine the cause.

Distributed systems rarely fail in a single predictable way. The same customer-facing problem may be caused by a failed deployment, a database outage, an exhausted connection pool, a networking issue, an external dependency or an application defect. Creating alerts for every possible cause produces multiple alerts for the same incident and obscures the real operational problem.

Alert on the symptom once, then use telemetry to identify the cause.

Alerting Philosophy

Services are made up of many components, each with its own dependencies, queues, databases, caches and external integrations. Individual components will occasionally experience transient latency, retries or minor failures without affecting the service that customers actually consume.

Alerting should therefore focus on service outcomes rather than component behaviour.

Consider a service with the following workflow:

Worker Service
      │
      ▼
 Message Queue
      │
      ▼
 Queue Listener
      │
      ▼
 External Email API
      │
      ▼
 Customer receives email

The objective of the platform is not to post messages to a queue quickly. It is to deliver emails successfully to customers.

If the worker service takes two seconds instead of 200 milliseconds to publish messages for a short period, that may be worth investigating later, but it is not necessarily an operational incident. If customers continue receiving emails within the expected time and success rate, there is no customer impact and no immediate action required.

The worker may be slow. The queue may be unavailable. The listener may have crashed. The email provider may be unavailable. A database dependency may be failing.

These are all different causes.

The symptom remains the same.

Customers are not receiving their emails.

That is the condition engineers need to respond to, so that is the condition that should generate the alert.

A symptom-based approach reduces operational noise, naturally correlates multiple technical failures into a single operational incident and keeps engineers focused on restoring service rather than chasing individual component alerts.

Component-level metrics remain valuable, but they are diagnostic signals. They explain why a service has degraded rather than whether engineers should be notified.

Cause-Based vs Symptom-Based Alerting

Suppose a production database becomes unavailable.

A cause-based strategy might generate alerts for:

  • Database unavailable
  • API timeout
  • Queue backlog increasing
  • Worker retry rate increasing
  • Application CPU increasing
  • Message processing latency
  • External API retries
  • Email delivery failures

One fault has generated multiple alerts.

A symptom-based strategy generates a single operational alert:

  • Customer notification delivery below the agreed service objective

Engineers investigate one incident and use telemetry to determine that the database outage is the root cause.

Actionable Alerts

Every alert should require a response from an engineer. If the expected response is simply to review the information during business hours, the condition should normally be exposed through dashboards, reports or informational notifications rather than operational alerting.

The severity of an alert should be determined by the urgency of the required response, not by how unusual the underlying metric appears.

Alerting Examples

For every alert ask two questions:

  1. Is this describing a symptom or a cause?
  2. Does someone need to take action immediately?

If the answer to the first question is "cause", or the answer to the second is "no", it is probably not an operational alert.

Example 1: Customer Notifications

Framing

A notification platform publishes messages to a queue before sending emails through an external provider.

Mindset

Customers care about receiving emails. They do not care how quickly messages are placed onto a queue.

Good alert

Notification delivery success rate below 98% for 10 minutes.

This reports a verified symptom. It remains correct whether the underlying cause is the worker, queue, database or email provider.

Bad alerts

  • Worker processing time exceeds two seconds.
  • Queue depth exceeds 10,000 messages.
  • Email provider returned five failed requests.

These are possible causes or diagnostic signals. None prove customers are being affected.

Example 2: Customer Web Application

Framing

A web application runs across multiple servers behind a load balancer.

Mindset

Customers care whether pages load and requests succeed, not whether one application instance has high CPU utilisation.

Good alert

HTTP 5xx responses exceed 2% across the service for five minutes.

This reports degraded customer experience without assuming the cause.

Bad alerts

  • Application server CPU exceeds 85%.
  • One application instance has increased response latency.
  • Garbage collection exceeds five seconds.

These metrics are valuable during investigation but do not necessarily require operational intervention.

Example 3: Customer Transactions

Framing

Customers complete purchases through several backend services and external providers.

Mindset

The important outcome is whether purchases succeed.

Good alert

Successful transaction rate falls below 95% for ten minutes.

This identifies a business-impacting symptom regardless of the technical cause.

Bad alerts

  • Payment API latency increased.
  • Database query duration exceeded one second.
  • Inventory service returned errors.

These may contribute to an incident but do not necessarily indicate customer impact.

Example 4: Platform Availability

Framing

A resilient platform spans multiple services and availability zones.

Mindset

Infrastructure components are expected to fail occasionally. The platform should continue delivering its service.

Good alert

Successful customer requests fall below the availability objective.

This reports that the platform is failing to deliver its intended outcome.

Bad alerts

  • Container restarted.
  • One availability zone has reduced capacity.
  • Node maintenance in progress.