Skip to content

Monitoring and Alerting

Collecting logs isn't monitoring. Monitoring means alerts fire for the problems that matter, reach someone who can act, and don't fire so often that people stop reading them. This page sets the alert baseline for platform resources, and how alerts are routed.

Status: planned

The central workspace log-platform-prod exists, but no operational alert rules or action groups are defined.

Alert baseline

Start with Azure Monitor Baseline Alerts (AMBA). It is Microsoft's policy-driven set of recommended metric and log alerts for Azure Landing Zones. It deploys alert rules automatically as resources are created, using DeployIfNotExists policies assigned per management group. It fits the platform's EPAC model.

Priority alerts for the platform itself:

Resource Alert Why
Terraform state storage Availability drop, unusual deletes No state means no deployments
log-platform-prod Ingestion stopped, daily cap reached, query failures The platform loses its visibility and audit trail without warning
Log ingestion by source No SigninLogs or AzureActivity rows for 6 hours A diagnostic setting was removed or broke. See Entra ID Diagnostic Settings.
Budgets Actual 80% and 100%, forecast 100% Already in place per subscription. See Budgets.
Firewall and gateways (when built) Health, SNAT port exhaustion, tunnel down The shared network is a single point of failure
Key Vault Availability, throttling Apps fail when they can't read secrets

Workload teams own alerts for their own applications. The platform provides the action group pattern and the baseline policies.

Routing

Action group Receives Channel
ag-platform-critical Outages and platform failures Email and phone or push. On-call once there is a rota.
ag-platform-warning Degradation, capacity, budget warnings Email and a chat channel
ag-security Security detections Kept separate. See Threat Detection.

Rules for every alert:

  • It must be actionable. Every alert names what to check first. If nobody would act on it, delete it.
  • Severity reflects urgency, not importance. Sev 0 or 1 wakes someone. Sev 2 to 4 waits for working hours.
  • Tune in the first month. Alerts that fire repeatedly without action get tuned or removed.

Managing alerts as code

  • Baseline alerts: AMBA policies, assigned through EPAC.
  • Platform-specific alerts, such as the log-silence checks: azurerm_monitor_scheduled_query_rules_alert_v2 in the azure-monitoring deployment, next to the workspace they query.
  • Action groups: in azure-monitoring, so every alert refers to one central set.