Skip to content

Logging Standards

Effective logging provides the information required to understand application behaviour, investigate incidents, identify trends and support operational decision making. Logs should be designed as operational data rather than as a record of application execution.

This document defines the minimum standards expected for application logging within managed services. It covers log structure, content, retention, ingestion, querying and visualisation requirements.

Structured Logging

All application logs must use a structured format that allows systems and people to reliably parse, search and analyse log data.

Plain text logs that require manual interpretation should be avoided because they limit the ability to automate analysis, create dashboards, correlate events and identify patterns across systems.

JSON is the preferred format for structured application logs because it provides consistent fields and integrates well with most modern log collection platforms.

Example structured log:

{
  "timestamp": "2026-07-17T12:30:15.123Z",
  "level": "ERROR",
  "service": "payment-api",
  "environment": "production",
  "message": "Payment request failed",
  "error_code": "PAYMENT_TIMEOUT",
  "transaction_id": "abc123",
  "customer_id": "customer-456",
  "duration_ms": 5000
}

A structured log should allow an engineer to answer key operational questions without needing to interpret free text:

  • What happened?
  • Where did it happen?
  • When did it happen?
  • Which service or component was involved?
  • Who or what was affected?
  • What was the impact?
  • What additional context is required for investigation?

Log Levels

Applications must use consistent log severity levels to indicate the importance and operational impact of an event.

Level Usage Example
DEBUG Detailed diagnostic information used during development or troubleshooting. Usually disabled in production. Request payload validation details
INFO Normal application events that describe expected behaviour. Service started successfully
WARN Unexpected conditions that do not prevent normal operation but may require investigation. Retry required due to temporary dependency failure
ERROR Failures that impact a request, transaction or application function. Database query failed
FATAL Severe failures that prevent the application from operating. Application cannot start due to missing configuration

Logging levels should represent the operational importance of an event, not the amount of information contained within the message.

Log Message Content

Each log entry must contain enough information to support investigation without requiring access to the application source code.

Field Purpose
Timestamp When the event occurred. Use UTC and an ISO 8601 format.
Log level Severity of the event.
Service name The application or component generating the log.
Environment The runtime environment, such as production, test or development.
Message A concise description of the event.
Correlation ID Identifier used to trace activity across multiple services.
Request ID Identifier for a single request where applicable.
User or transaction identifier Identifier for the affected operation where appropriate.
Error details Error type, code and relevant diagnostic information.
Duration Processing time for requests or operations where applicable.

Logs should describe the event and its context rather than only recording the failure.

Poor:

Database error

Better:

{
  "level": "ERROR",
  "message": "Unable to retrieve customer profile",
  "service": "customer-api",
  "database": "customer-db",
  "error_code": "CONNECTION_TIMEOUT",
  "duration_ms": 3000,
  "correlation_id": "12345"
}

Correlation and Traceability

Applications must provide identifiers that allow events to be correlated across services.

At a minimum, distributed applications should support:

  • Correlation IDs for end-to-end business transactions.
  • Request IDs for individual application requests.
  • Trace IDs where distributed tracing is implemented.

Correlation identifiers should be:

  • Generated at the start of a request if one does not already exist.
  • Passed between dependent services.
  • Included in all relevant log entries.

Without correlation identifiers, investigation across multiple services becomes significantly slower and may require manual analysis of unrelated events.

Logging Sensitive Data

Logs must not contain sensitive information unless there is a documented requirement and appropriate controls are in place.

The following data should not be logged:

  • Passwords.
  • Authentication tokens.
  • API keys.
  • Full payment card information.
  • Personal data that is not required for operational purposes.

Where sensitive data is required for investigation, values should be masked or anonymised.

Example:

Bad:

{
  "card_number": "1234567890123456"
}

Good:

{
  "card_number": "**** **** **** 3456"
}

Log Collection and Ingestion

Applications must send logs to an approved centralised logging platform rather than relying on local storage.

Centralised ingestion provides:

  • Consistent access across support teams.
  • Improved incident investigation.
  • Long-term retention.
  • Search and analysis capability.
  • Correlation across multiple applications and infrastructure components.

Log collection should:

  • Capture application stdout and stderr where appropriate.
  • Preserve structured fields during ingestion.
  • Include metadata such as environment, application name and host or container identity.
  • Handle log rotation and transmission failures.
  • Prevent excessive log generation from impacting application performance.

Log Retention

Retention periods should balance operational requirements, compliance obligations and storage costs.

The required retention period should be defined based on:

  • Business requirements.
  • Regulatory requirements.
  • Security requirements.
  • Incident investigation needs.
  • Application criticality.

Typical retention guidance:

Log Type Recommended Retention
Application operational logs 30-90 days
Security and audit logs Based on security and compliance requirements
Debug logs Short retention due to volume
Transactional audit logs Based on business requirements

Retention decisions should consider whether logs are required for:

  • Active incident investigation.
  • Post-incident review.
  • Trend analysis.
  • Capacity planning.
  • Security investigations.

Log Querying

The logging platform must support efficient querying of structured log data.

Engineers should be able to search logs using fields rather than relying only on free text searches.

Examples of useful queries:

Find all errors for a service:

service="payment-api" AND level="ERROR"

Find failed requests for a transaction:

correlation_id="abc123"

Identify increasing error rates:

service="customer-api" AND level="ERROR"
group by time interval

Queries should support:

  • Filtering by service, environment and severity.
  • Searching by correlation identifiers.
  • Aggregating events over time.
  • Identifying trends and patterns.
  • Exporting data for further analysis.

Log Visualisation

Logs should support visualisation where trends or patterns are operationally useful.

Examples include:

  • Error volume over time.
  • Error distribution by service.
  • Top recurring error messages.
  • Failed requests by endpoint.
  • Dependency failures.
  • Authentication failures.

Dashboards should focus on operational questions rather than displaying raw log volume.

Poor dashboard:

  • Number of logs generated per hour.

Useful dashboard:

  • Number of failed requests by service.
  • Top application errors in the last 24 hours.
  • Services experiencing increasing error rates.

Logging and Alerting

Logs may provide valuable diagnostic information, but they should not normally be the primary mechanism for alerting.

Alerting should generally be based on user-visible symptoms and service impact rather than individual log events.

Examples:

Poor alert:

Alert when "Database connection timeout" appears 10 times in logs.

Better alert:

Alert when customer transaction failure rate exceeds the defined threshold.

Logs should help explain why an issue occurred after an alert identifies that user impact exists.

Logging Standards Checklist

Applications should meet the following minimum requirements:

Requirement Expected Standard
Format Structured logs, preferably JSON
Timestamp UTC using ISO 8601 format
Severity Consistent log levels
Context Include service, environment and correlation identifiers
Errors Include meaningful error details
Sensitive data Mask or exclude confidential information
Collection Centralised ingestion into an approved platform
Querying Searchable structured fields
Visualisation Dashboards for useful operational trends
Retention Defined based on operational and compliance needs

Common Logging Issues

Issue Impact
Unstructured text logs Difficult to search and analyse
Missing correlation IDs Slower incident investigation
Excessive debug logging Increased storage cost and reduced signal-to-noise ratio
Logging sensitive data Security and compliance risk
Logging only errors Limited visibility into application behaviour
Alerts based directly on logs High risk of noisy and low-value alerts
Missing context Engineers cannot determine impact or cause

Good logging provides reliable operational evidence. It enables engineers to understand system behaviour, investigate incidents efficiently and identify trends before they become service-impacting issues.