Skip to content

Lane A and Lane B Delivery Methodology

Every Terraform deployment is delivered through one of two lanes. Lane A is for Tier 0 (control plane): CI plans and saves the plan, and a PIM-elevated human applies exactly that saved plan — never local files. Lane B is for Tiers 1 and 2 (management and workload planes): full GitOps, where merging to main applies through a scoped OIDC identity. This keeps automation fast everywhere except the few places where a standing pipeline credential would put the whole tenant at risk.

Why two lanes

Pure GitOps, where every merge applies automatically, is the right default. It breaks down at Tier 0. To vend identities, manage directory roles or assign roles at the tenant root, a pipeline would need standing Privileged Role Administrator, Application Administrator or root Owner. Anyone who could merge to the repository, or compromise the runner, would then control the tenant.

So delivery is split by tier:

  • Lane A (Tier 0): CI runs checks and plan with a read-only identity, and on main saves the plan. A local plan/apply from local files is blocked. The saved plan goes live one of two ways, both behind a PIM-activated senior engineer:
    • Human apply (most Tier 0 projects): there is no apply job. The engineer applies the saved plan with the project's ./apply.ps1 (apply-merged in azure-tenant-bootstrap).
    • Just-in-time CI apply (t0-azure-management-groups): a manual apply job runs as a main-only write identity that holds no roles. The senior gives it one role for 30 minutes through PIM; Azure removes it by itself. See Tier 0 Just-in-Time CI Apply.
  • Lane B (Tiers 1 and 2): CI runs checks and plan on the MR, then apply automatically on merge, using an OIDC identity scoped to its own management group or subscription.

The detailed case against CI apply at Tier 0 is in Tier 0 Pipeline Design. Platform service principal vending uses the split-identity exception: see Platform Service Principal Vending.

The lanes at a glance

Lane A: Tier 0 Lane B: Tiers 1 and 2
Repositories azure-tenant-bootstrap (steps 002–009), t0-azure-service-principals, azure-privileged-access-grants, t0-azure-conditional-access, t0-azure-management-groups, azure-entra-diagnostics, subscription-vending-platform subscription-vending-workload (Tier 1 once its controls are in), azure-privileged-access-packages (restricted — see below), azure-monitoring, workload projects under deployments/workloads/
Scope Tenant root, mg-intermediate-root, Entra directory roles and app registrations, tenant root state mg-platform, mg-landingzones and their subscriptions; access packages and group PIM policies (restricted)
CI identity Read-only (sp-pla-tf-*-ro); with just-in-time CI apply, also a main-only write identity holding no standing roles Read-write, bounded to its scope (sp-pla-tf-*-rw, sp-sub-tf-*-rw)
CI stages checks → plan (on main, saved as the tfplan artifact); with just-in-time CI apply, then a manual apply checks → plan → apply
Who applies A senior engineer, locally, with ./apply.ps1 (apply-merged in azure-tenant-bootstrap) — only CI's saved plan of the merged commit. With just-in-time CI apply: CI, after a senior runs ./authorize-apply.ps1 GitLab CI, on merge to main
Gate MR review and merge, PIM activation, the apply script's checks (merged MR, saved plan for exactly that commit, matching Terraform version) MR review and approval
Privilege None standing. PIM activation, time-bound. Standing, least-privilege, bounded to the target scope
State Central platform containers in sa<tenant>tenantroot Dedicated per-subscription state account in rg-terraform-state
Blast radius The whole tenant One management group, subscription or resource group

Choosing a lane

Classify every new repository before its first pipeline runs. If any answer points to Lane A, it is Lane A.

flowchart TD
    Q1{"Does it manage Entra ID app registrations,<br/>service principals or Graph permissions?"}
    Q1 -->|Yes| LANE_A["LANE A<br/>Plan-only CI, human applies the saved plan"]
    Q1 -->|No| Q2{"Does it assign roles at the tenant root<br/>or mg-intermediate-root?"}
    Q2 -->|Yes| LANE_A
    Q2 -->|No| Q3{"Could a compromised pipeline credential<br/>escalate its own access or create new admin access?"}
    Q3 -->|Yes| LANE_A
    Q3 -->|No| Q4{"Is the blast radius bounded to one subscription<br/>or a non-identity management group?"}
    Q4 -->|Yes| LANE_B["LANE B<br/>Automated GitOps, scoped OIDC"]
    Q4 -->|No| LANE_A
Component Lane Why
Directory role management (Privileged Role Administrator, Global Administrator) A Changes the identity control plane directly, and could create a permanent backdoor.
Persona groups, their directory roles and Tier 0 RBAC (azure-privileged-access-grants) A Role-assignable groups holding Tier 0 roles; who may request and approve Tier 0 access.
Service principal and app registration creation A Needs admin consent and credential-creation rights (EAM control plane).
Role assignments at the tenant root A Inherited by every subscription in the tenant.
PIM for Groups policies and access packages (azure-privileged-access-packages) B, restricted The Azure CLI can never get these Graph scopes (AADSTS65002), so only a pipeline can manage them. Accepted as EX-002 with guardrails: the identity can't modify role-assignable groups or their membership; who may request and approve is Lane A; nobody is made eligible in code (eligibility only comes from an approved request); it owns a group only briefly, during catalog onboarding, and every plan warns if it still does.
Tenant root state storage A Holds state with sensitive root metadata.
Management group hierarchy (t0-azure-management-groups) A Restructuring it moves subscriptions out from under inherited policy and RBAC. Just-in-time CI apply: pag-t0-management-groups gives CI's write identity Management Group Contributor at the tenant root for 30 minutes. See Manage Management Groups.
Platform subscription vending (subscription-vending-platform) A Vends the platform subscriptions: rare, platform-owned, and Tier 0 by decision, so a PIM-elevated human applies.
Workload subscription vending (subscription-vending-workload) B Creates landing zones through billing APIs. Its Graph grants are owned-only (Application.ReadWrite.OwnedBy, Group.Create), so it can only manage the identities and groups it created. Tier 1 once its remaining Tier 0 routes are closed (GAP-011): its own state in the Tier 0 state account, and a group-wide GitLab token. Its Azure grants are already down at mg-landingzones and mg-sandboxes. See Subscription Vending.
Platform monitoring (azure-monitoring) B Bounded to its own resource group (rg-monitoring-prod), with no directory permissions. A write role on mg-management would reach Tier 0 state.
Workload resources (deployments/workloads/*) B Bounded to individual workload subscriptions.

Lane A: Tier 0

sequenceDiagram
    autonumber
    actor Engineer as Senior Engineer
    actor Reviewer as Peer Senior Engineer
    participant GitLab as GitLab CI
    participant PIM as Entra ID PIM
    participant Azure as Azure / Entra / State

    Engineer->>GitLab: Open MR
    GitLab->>Azure: Checks + terraform plan (read-only OIDC)
    Reviewer->>GitLab: Review plan, approve and merge
    GitLab->>Azure: main pipeline: plan again, save as tfplan
    Note over GitLab,Azure: CI stops here. There is no apply job.
    Engineer->>PIM: Activate the operator persona, with justification
    Reviewer->>PIM: Approve (when two-person approval is enabled)
    Engineer->>GitLab: ./apply.ps1 downloads main's saved plan
    Engineer->>Azure: Shows it, engineer types "apply", terraform apply tfplan
    Note over Azure: Audit logged in Entra ID, ARM and storage

The plan-only pipeline

  • CI signs in with a keyless OIDC identity (sp-pla-tf-*-ro) that can't change live infrastructure: Directory Readers in Entra ID (or read-only Graph permissions), Reader at the scopes it plans (up to the tenant root), and Storage Blob Data Reader on its state container. It can't take the state lease, so it plans with -lock=false; a saved plan still can't be applied over changed state.
  • The pipeline YAML leaves out the template's apply.yml: it includes the template's job files one by one. With human apply there is no apply job at all; with just-in-time CI apply there is a manual one whose identity holds nothing until a senior grants it. Either way, taking over GitLab or forcing a merge gives no credential that can change live infrastructure.
  • On main, the plan job saves the plan (tfplan, or <step>/tfplan in a multi-root repo) — that artifact is the only thing that can be applied.
  • Merged-only guard. Each Lane A root has a ci-only-guard.tf: a local terraform plan/apply fails immediately, because only CI sets TF_VAR_running_in_ci. So a human can't apply local, unreviewed files by accident; the only path is the project's apply script. It stops accidents, not a deliberate bypass — that's what the audit trail is for.
  • Terraform is pinned to the CI image's version (required_version), and provider lock files carry Linux and Windows hashes, so CI's saved plan applies on the engineer's machine.

apply.ps1 and apply-merged

Each Lane A project has its own ./apply.ps1 (azure-privileged-access-grants, t0-azure-conditional-access, t0-azure-service-principals, azure-entra-diagnostics, subscription-vending-platform; in t0-azure-management-groups it's only the fallback for when CI can't run). Run from the project's folder, it refuses unless:

  1. the project's operator group is active for you in PIM, and your Azure CLI sign-in is from after the activation;
  2. you're on an up-to-date, clean main;
  3. that commit came from a merged merge request (it shows the MR);
  4. the latest successful main pipeline saved a plan for exactly that commit.

It downloads the Terraform version pinned in versions.tf if yours differs, shows the saved plan and asks you to type apply. ./apply.ps1 -WhatIf stops after showing the plan.

azure-tenant-bootstrap is the exception: it has no apply.ps1 and uses the older apply-merged (from devops/ps-menu), run from each step's folder. It does the same checks except the PIM one, and initialises with the folder's backend.hcl. Its preflight.ps1 covers that gap: run it first to check the az login session and tenant, state data-plane access, and the roles the apply needs.

The full runbook, the checks and what to do when they fail are in Lane A Apply.

Operator runbook

  1. Review and merge. Checks pass, the plan from sp-pla-tf-*-ro is reviewed in the MR, and the MR is merged. Wait for the main pipeline to finish — it saves the plan.
  2. Elevate. In PIM → My roles → Groups, activate the project's Tier 0 group with a justification — for example, for azure-privileged-access-grants, (tier 0) Privileged Access Grants (pag-t0-privileged-access-grants). Some changes need a second group, and the project's guide says so. Deactivate anything extra straight after.

  3. Sign in again, so the token carries the activation:

    az account clear
az login

  4. Apply the merged commit's saved plan:

    git checkout main; git pull
./apply.ps1                  # from the project folder
# azure-tenant-bootstrap only, from the step's folder:
# pwsh -File ../preflight.ps1; apply-merged

  5. Close out. Confirm the apply completed, deactivate your PIM roles (or let them expire), and attach the apply summary to the change ticket.

The one planned exception is azure-tenant-bootstrap step 002's first-time bootstrap: it creates the CI identity itself, so no CI plan can exist yet. Its runbook documents the single guarded local run.

Lane B: Tiers 1 and 2

Workflow

  1. Open an MR. CI runs fmt, tflint, validate and checkov, then plan with the repository's scoped OIDC identity, and posts the plan to the MR.
  2. A peer reviews and approves. The MR plan is the only gate — read it for destroys you didn't intend (a renamed module or resource without a moved block is the usual cause). Merge to main.
  3. The main pipeline runs terraform apply with the same scoped identity.

Identity and state

  • No secrets. Each repository has a federated credential tied to its project path (project_path:<group>/<repo>). If a project is renamed or moved, its credential's subject must follow, or CI can't sign in.
  • Bounded scope. The identity only has rights on its own subscription or management group, for example Contributor plus User Access Administrator on one workload subscription. It has no access to parent management groups and no Entra directory roles.
  • Isolated state. Each vended subscription gets its own state storage account in rg-terraform-state. The pipeline identity has Storage Blob Data Contributor on its own container only, and no access to sa<tenant>tenantroot.

Break-glass

In an emergency, such as a lockout or security incident:

  1. Request Emergency Tenant Recovery through its access package where PIM still works; otherwise use a break-glass account (see Break-Glass Accounts). Sign-in alerting for those accounts is planned, not yet in place (GAP-006 in the Exceptions Register).
  2. Prefer Lane A mechanics even in an emergency: a merged MR and the project's apply script.
  3. If CI itself is unavailable, the merged-only guard can be lifted for a single, recorded local run — the same mechanism as step 002's bootstrap exception. Record the plan, the apply output and why, for review afterwards (NIST AU-6), and follow up with an MR that brings main in line.