Skip to content

Metrics Standards

Metrics provide a measurable view of system and application behaviour over time. They are a core part of observability because they allow teams to understand system health, identify trends, measure service performance and detect conditions that may impact users.

Effective metrics should focus on answering operational questions rather than collecting data simply because it is available. Poorly designed metrics create noise, increase storage costs and make it harder to identify meaningful signals.

This document defines the standards expected for collecting, storing, querying and using metrics within managed services.

Purpose of Metrics

Metrics should provide visibility into:

  • Service health.
  • User impact.
  • System performance.
  • Resource utilisation.
  • Capacity requirements.
  • Reliability trends.
  • Operational risks.

Metrics should help answer questions such as:

  • Is the service available?
  • Are users experiencing failures?
  • Is performance degrading?
  • Are dependencies operating correctly?
  • Is the system approaching capacity limits?
  • Are reliability objectives being met?

Metrics should not exist solely to describe technical implementation details. They should provide information that supports operational decisions.

Metric Design Principles

Focus on Meaningful Signals

Teams should prioritise metrics that represent service behaviour and user experience.

A large number of low-value metrics can reduce observability rather than improve it. Engineers should be able to understand why a metric exists and what decision it supports.

Good metrics:

  • Request success rate.
  • Request latency.
  • Number of failed transactions.
  • Queue depth.
  • Dependency availability.
  • Resource saturation.
  • Capacity consumption trends.

Poor metrics:

  • Number of internal function calls.
  • Every individual code path execution count.
  • Metrics that are collected but never queried or reviewed.
  • Metrics without an operational purpose.

Golden Signals

Application and service metrics should align with the four golden signals where applicable.

Signal Purpose Examples
Latency Measures time taken to complete operations API response time, database query duration
Traffic Measures demand on the service Requests per second, transactions per minute
Errors Measures failed operations HTTP failures, failed transactions, exceptions
Saturation Measures how constrained the system is CPU utilisation, memory pressure, queue depth

These metrics provide a consistent baseline for understanding service behaviour. Additional metrics may be required depending on the application architecture, but they should complement these signals rather than replace them.

Application Metrics

Applications should expose metrics that represent business and technical behaviour.

Examples include:

Availability Metrics

Measure whether the service is functioning.

Examples:

  • Successful request percentage.
  • Failed request percentage.
  • Health check results.
  • Dependency availability.

Performance Metrics

Measure the experience delivered to users.

Examples:

  • Request latency.
  • Response time percentiles.
  • Processing duration.
  • Queue processing time.

Latency metrics should generally use percentiles rather than averages.

For example:

  • p50 shows typical user experience.
  • p95 shows degraded experience affecting a smaller group of users.
  • p99 highlights severe performance issues.

An average response time can hide significant user impact. For example, a service may have an average response time of 200ms while 5% of users experience responses above 10 seconds.

Throughput Metrics

Measure workload and demand.

Examples:

  • Requests per second.
  • Messages processed.
  • Transactions completed.
  • Batch jobs executed.

Throughput metrics are important for understanding capacity requirements and identifying unusual activity.

Business Metrics

Where appropriate, teams should collect metrics that represent business outcomes.

Examples:

  • Orders completed.
  • Payments processed.
  • User registrations.
  • Successful customer actions.

Business metrics help identify whether technical issues are affecting business outcomes.

Infrastructure and Platform Metrics

Infrastructure metrics provide visibility into the resources supporting applications.

Common metrics include:

  • CPU utilisation.
  • Memory usage.
  • Disk usage.
  • Network throughput.
  • Container restarts.
  • Pod availability.
  • Database connections.
  • Storage capacity.
  • Queue depth.

Infrastructure metrics should be interpreted in the context of service impact.

High CPU usage alone does not necessarily indicate a problem. If the service remains healthy and latency remains within acceptable limits, the increased usage may represent normal demand.

Metric Naming and Labels

Metrics must use consistent naming conventions and include meaningful metadata.

A metric should clearly describe:

  • What is being measured.
  • The unit of measurement.
  • The scope of the metric.

Examples:

Good:

http_request_duration_seconds
payment_transaction_failures_total
database_connection_pool_active

Poor:

request_time
count1
metric_test

Metrics should use labels or dimensions to allow useful filtering.

Examples:

http_requests_total{
    service="customer-api",
    environment="production",
    status="500"
}

Useful dimensions include:

  • Service name.
  • Environment.
  • Region.
  • Availability zone.
  • Endpoint.
  • Response status.
  • Dependency.

Avoid excessive metric dimensions. High-cardinality labels, such as unique customer IDs or request IDs, can significantly increase storage requirements and reduce query performance.

Metrics Collection and Ingestion

Metrics should be collected through approved monitoring platforms using standard collection methods.

Collection should:

  • Capture metrics at appropriate intervals.
  • Preserve metric names and labels.
  • Include required metadata.
  • Handle temporary collection failures.
  • Avoid unnecessary duplicate collection.

Common collection approaches include:

  • Application instrumentation.
  • Exporters.
  • Platform agents.
  • Cloud provider integrations.
  • Infrastructure monitoring agents.

Collection intervals should balance freshness requirements with storage and processing overhead.

For example:

  • Short intervals may be required for critical user-facing services.
  • Longer intervals may be sufficient for capacity and trend analysis.

Metrics Storage and Retention

Metrics retention should support both operational troubleshooting and longer-term analysis.

Retention should consider:

  • Service criticality.
  • Incident investigation requirements.
  • Capacity planning needs.
  • Cost constraints.

High-resolution metrics are typically most valuable for recent operational analysis, while aggregated metrics may be retained longer for trend analysis.

Metric Type Typical Usage
High-resolution metrics Real-time monitoring and incident response
Aggregated metrics Capacity planning and reporting
Long-term trends Reliability and performance analysis

Metrics Querying

The metrics platform must allow engineers to efficiently query and analyse system behaviour.

Queries should support:

  • Filtering by service and environment.
  • Aggregating across instances.
  • Comparing trends over time.
  • Creating calculated indicators.
  • Correlating related metrics.

Examples of useful queries:

Request error rate:

failed_requests / total_requests

Latency percentile:

p95(request_duration)

Resource saturation:

memory_used / memory_available

Queries should be understandable and reusable. Complex queries should be documented where they are used for operational dashboards or alerts.

Metrics Visualisation and Dashboards

Dashboards should present the information required to understand service health quickly.

A dashboard should answer a specific operational question rather than display every available metric.

Good dashboards include:

  • Service overview dashboards.
  • Application health dashboards.
  • Dependency dashboards.
  • Capacity dashboards.
  • Platform health dashboards.

A service dashboard should typically include:

Service Health

  • Availability.
  • Error rate.
  • Latency.
  • Traffic.

Dependencies

  • External service health.
  • Database performance.
  • Queue status.

Capacity

  • Resource utilisation.
  • Scaling activity.
  • Growth trends.

Dashboards should avoid:

  • Large collections of unrelated graphs.
  • Metrics without clear purpose.
  • Showing every available metric.
  • Dashboards that require extensive interpretation.

Metrics and Alerting

Metrics are the primary source for most operational alerts because they provide measurable signals of service impact.

Alerts should generally be based on symptoms rather than causes.

Good alert:

Alert when API error rate exceeds 5% for 10 minutes.

Poor alert:

Alert when CPU exceeds 90%.

High CPU may be a contributing factor, but it does not necessarily indicate user impact.

A useful alert should:

  • Indicate a meaningful service issue.
  • Have a clear owner.
  • Include relevant context.
  • Provide actionable information.
  • Avoid triggering on normal variation.

Metrics used for alerting should be reviewed regularly to identify:

  • Alerts that never trigger.
  • Alerts that trigger too frequently.
  • Alerts that do not lead to action.
  • Missing coverage for important failure modes.

Service Level Indicators

Metrics should support the definition and measurement of Service Level Indicators (SLIs).

Examples:

Availability SLI:

Successful requests / total requests

Latency SLI:

Requests completed within target response time / total requests

SLIs provide a measurable view of service reliability and should be preferred over isolated infrastructure measurements when assessing customer impact.

Common Metrics Issues

Issue Impact
Collecting every available metric Increased cost and reduced signal quality
No agreed naming standards Difficult querying and dashboard creation
High-cardinality labels Poor performance and increased storage usage
Alerting on infrastructure symptoms only Increased noise and missed user impact
Dashboards without purpose Slow incident response
Missing application metrics Limited understanding of service health
Using averages for latency Hides degraded user experience

Metrics Standards Checklist

Requirement Expected Standard
Purpose Metrics support operational decisions
Focus Golden signals and service health
Naming Consistent, descriptive names
Labels Useful dimensions without excessive cardinality
Collection Centralised approved monitoring platform
Querying Supports filtering, aggregation and analysis
Dashboards Focused on operational questions
Alerting Based on meaningful service impact
Retention Defined based on operational needs
Review Regularly reviewed for usefulness

Effective metrics provide a reliable view of system behaviour. They allow teams to understand service health, detect issues early, respond to incidents and make informed operational decisions.