Skip to content

Incident Response

Incidents are easier to handle when the first steps are decided in advance. This page sets out severity levels and a short runbook for each incident most likely to hit this platform. Each runbook covers how to contain the incident, what to investigate, and how to recover.

Status: planned

These runbooks are the agreed starting point. They have not yet been tested in an exercise.

Severity

Severity Definition Examples Response
SEV1 Tier 0 compromise, or the tenant is unusable Stolen Global Administrator or pipeline identity with Tier 0 rights, tenant-wide lockout Immediately, around the clock
SEV2 Compromise contained to one subscription or workload, or loss of audit logging A workload pipeline identity is misused, a diagnostic setting is removed Same working day
SEV3 Suspicious activity with no confirmed impact Unusual sign-in, a single policy violation Next working day

For every incident: start a log with timestamps. Preserve evidence before you change anything. The workspace keeps sign-in, audit and Activity Log data for about 2 years, so export the relevant queries into the incident record. Then contain, investigate, recover, and hold a blameless review.

Runbooks

Compromised user account

  1. Contain: revoke sessions (Entra ID → user → Revoke sessions), reset the password, and remove MFA methods you don't recognise. Disable the account if in doubt.
  2. Check privilege: did the account have PIM eligibility or access package assignments? Remove them until the investigation ends.
  3. Investigate: SigninLogs for where and when it signed in, AuditLogs for directory changes it made, AzureActivity for resource changes, all filtered on the user.
  4. Recover: reverse unauthorised changes, re-enable the account with fresh MFA, then restore access through the normal request process.

Compromised pipeline identity (service principal)

  1. Contain: delete the service principal's federated identity credentials so no new tokens can be issued. Remove any client secrets or certificates, which platform identities should never have. If the source is unclear, disable the service principal.
  2. Stop the pipeline: pause the GitLab project's pipelines and review recent MRs, merges and CI variable changes.
  3. Investigate: ServicePrincipalSignInLogs for token use, and AzureActivity and AuditLogs filtered on the service principal's object ID. Look especially for new credentials, new app registrations or new role assignments.
  4. Recover: remove anything the identity created. Recreate the identity through service principal vending if needed, and update the dependent object IDs (see the stale object ID warning in Entra ID Diagnostic Settings).

Break-glass account used

  1. Confirm: was this a planned test or a declared emergency? If neither, treat it as SEV1.
  2. Contain: reset the password, check the FIDO2 key is still in the safe, and review every action taken in the session.
  3. Recover: rotate the credentials following Break-Glass Accounts, and record the use.

Tenant-wide lockout

Follow the Conditional Access lockout recovery steps using a break-glass account. Afterwards, bring Terraform state back in line with any changes made in the portal.

Audit logging removed

  1. Contain: restore the diagnostic setting by re-running the owning pipeline: azure-entra-diagnostics for Entra ID, or azure-monitoring or an EPAC remediation for a subscription.
  2. Investigate: who removed it, and what happened during the gap? The subscription's own Activity Log keeps 90 days even when export stops.

Secret leaked into a repository

  1. Contain: revoke or rotate the secret first. Removing it from git history comes second: once pushed, assume it is compromised.
  2. Investigate: check where the secret was used during the exposure window.
  3. Prevent: confirm secret detection is running on the repository (see Vulnerability Management).

Test the runbooks

Run a tabletop exercise at least once a year. Include a break-glass sign-in test each quarter (see Break-Glass Accounts). Update this page with what the exercise shows.