Monitoring Standards
Purpose
This document defines the monitoring standards expected for services supported by the SRE practice.
Monitoring exists to answer one question:
Can we detect that users are experiencing a problem quickly enough to take effective action?
Monitoring should prioritise detecting service degradation from the user's perspective before identifying the underlying technical cause.
This document defines the principles that monitoring implementations should follow. Detailed standards for logging, metrics, tracing, dashboards and alerting are covered in their respective documents.
Scope
These standards apply to:
- Applications
- APIs
- Background services
- Worker processes
- Scheduled jobs
- Containerised workloads
- Virtual machine hosted services
- Platform services where telemetry is available
They apply regardless of monitoring platform.
Monitoring Objectives
Monitoring should enable engineers to answer the following questions with confidence:
- Is the service available?
- Is the service performing within expected limits?
- Are users successfully completing their requests?
- Has behaviour changed from normal?
- Which component is affected?
- What telemetry should be investigated next?
Monitoring should reduce detection time rather than simply collect large volumes of telemetry.
Monitoring Principles
Monitor user experience first
Monitoring should focus on what users experience rather than the internal implementation of a system.
Users do not care whether a database CPU reaches 90%. They care that pages fail to load, transactions timeout or requests become slow.
Monitoring should therefore begin with externally visible symptoms before considering infrastructure health.
Examples include:
- HTTP error rates
- API response latency
- Failed business transactions
- Queue processing delays
- Authentication failures
- Successful versus failed user journeys
Infrastructure monitoring remains valuable, but it should support diagnosis rather than provide the primary indication that a service is unhealthy.
Prefer symptom-based monitoring over cause-based monitoring
Many infrastructure events never affect users.
Examples include:
- High CPU utilisation
- Memory pressure
- Disk utilisation
- Replica failover
- Container restarts
- Individual node failures
Modern distributed platforms are often designed to tolerate these conditions automatically.
Monitoring should therefore detect measurable service degradation rather than assume infrastructure events always represent incidents.
For example:
| Avoid monitoring | Prefer monitoring |
|---|---|
| Database CPU > 90% | Database response time increasing |
| Pod restarted | API availability reduced |
| High memory usage | Increased request latency |
| Queue depth | Queue processing delay |
| Node unavailable | Customer transactions failing |
Cause-based monitoring frequently produces unnecessary alerts.
Symptom-based monitoring identifies when resilience mechanisms have failed to hide infrastructure problems from users.
Monitor services, not individual components
Modern services consist of many interconnected components.
Monitoring every component independently often creates duplicate alerts during a single incident.
Monitoring should reflect how the service behaves as a whole.
Component telemetry remains important for diagnosis but should generally not determine service health.
Every monitor should have a purpose
Each monitor should answer a specific operational question.
| Question | Suitable monitor |
|---|---|
| Can users access the service? | Availability |
| Are requests completing successfully? | Success rate |
| Are requests becoming slower? | Latency |
| Are background jobs completing? | Job completion rate |
| Is message processing keeping up? | Processing latency |
If a monitor cannot clearly answer an operational question, its value should be challenged.
Monitoring should drive action
A monitor that detects an issue but provides no useful next step creates operational noise.
Monitoring should provide sufficient context for responders to determine:
- what is affected
- when it began
- how severe the impact is
- which telemetry should be investigated next
The Three Pillars of Observability
Modern monitoring depends upon three complementary forms of telemetry. None replaces the others.
| Pillar | Answers |
|---|---|
| Metrics | What is happening? |
| Logs | What happened? |
| Traces | Where is the problem occurring? |
Together they provide the information required to detect, investigate and resolve incidents.
Metrics
Metrics provide numerical measurements describing system behaviour over time.
They are typically used for:
- alerting
- trend analysis
- capacity planning
- Service Level Objective (SLO) measurement
- dashboard visualisation
Metrics should be inexpensive to collect and available at sufficient resolution to detect changes quickly.
Further guidance is provided in the Metrics Standards document.
Logs
Logs provide detailed records of discrete events.
They provide context that metrics cannot, including:
- errors
- warnings
- business events
- application behaviour
- exception details
Logs should be structured, searchable and correlated with requests wherever possible.
Further guidance is provided in the Logging Standards document.
Traces
Distributed traces record how a request flows through a system.
Tracing is particularly valuable for:
- microservices
- event-driven systems
- asynchronous processing
- systems with many external dependencies
Tracing significantly reduces investigation time by identifying where latency or failures occur.
Further guidance is provided in the Tracing Standards document.
The Golden Signals
Google's Site Reliability Engineering guidance identifies four primary indicators of service health, commonly known as the Golden Signals.
These should form the foundation of monitoring for user-facing services.
Latency
Latency measures how long users wait for requests to complete.
Monitor:
- average latency
- median latency
- percentile latency (typically P95 and P99)
Average response times alone rarely reflect user experience because they hide slower requests.
Traffic
Traffic measures demand placed upon the service.
Examples include:
- requests per second
- transactions per minute
- queue throughput
- messages processed
- active users
Traffic provides operational context and helps distinguish between increased demand and degraded performance.
Errors
Errors measure unsuccessful operations.
Examples include:
- HTTP 5xx responses
- failed API requests
- failed database operations
- authentication failures
- failed business transactions
Where possible, business failures should be monitored separately from technical failures.
Saturation
Saturation measures how close a system is to its operational limits.
Examples include:
- CPU utilisation
- memory consumption
- thread pool exhaustion
- connection pool usage
- queue processing delay
- storage utilisation
Saturation should be interpreted alongside user-facing symptoms.
High resource utilisation without degraded service should rarely require immediate intervention.
Beyond the Golden Signals
The Golden Signals provide an excellent baseline but are not sufficient for every workload.
Many services require monitoring of business-specific indicators.
Examples include:
- Orders processed
- Payments authorised
- Customers logged in
- Messages consumed
- Reports generated
- Backups completed
- Scheduled jobs executed
Business telemetry often detects incidents that infrastructure monitoring cannot.
Monitoring Layers
Effective monitoring should exist at multiple layers.
| Layer | Purpose |
|---|---|
| User experience | Detect customer impact |
| Application | Detect service behaviour |
| Dependencies | Detect downstream failures |
| Platform | Detect hosting issues |
| Infrastructure | Detect underlying resource problems |
Monitoring should begin at the highest layer and progressively narrow investigation towards lower layers.
Avoid Common Anti-Patterns
Monitoring implementations should avoid:
- monitoring every available metric
- creating alerts for every warning
- relying solely on infrastructure metrics
- dashboards containing excessive charts
- duplicate monitors measuring the same condition
- thresholds without operational justification
- telemetry that is never reviewed
- monitors with no identified owner
Large volumes of telemetry do not necessarily improve observability.
Useful telemetry answers operational questions.
Characteristics of Good Monitoring
Good monitoring is:
- user-focused
- actionable
- reliable
- easy to understand
- proportional to operational risk
- consistent across services
- inexpensive to maintain
- resistant to alert fatigue
Engineers should be able to determine service health within minutes without interpreting dozens of unrelated charts.
Relationship to Other Standards
This document defines the overall monitoring philosophy.
Supporting standards provide implementation guidance for individual areas:
- Logging Standards
- Metrics Standards
- Tracing Standards
- Dashboard Standards
- Alerting Standards
- Service Level Objectives (SLO) Standards (where applicable)