Skip to content

Lane A and Lane B Delivery Methodology

Every Terraform deployment is delivered through one of two lanes. Lane A is for Tier 0 (control plane): CI plans and saves the plan, and a PIM-elevated human applies exactly that saved plan — never local files. Lane B is for Tiers 1 and 2 (management and workload planes): full GitOps, where merging to main applies through a scoped OIDC identity. This keeps automation fast everywhere except the few places where a standing pipeline credential would put the whole tenant at risk.

Why two lanes

Pure GitOps, where every merge applies automatically, is the right default. It breaks down at Tier 0. To vend identities, manage directory roles or assign roles at the tenant root, a pipeline would need standing Privileged Role Administrator, Application Administrator or root Owner. Anyone who could merge to the repository, or compromise the runner, would then control the tenant.

So delivery is split by tier:

  • Lane A (Tier 0): CI runs checks and plan with a read-only identity, and on main saves the plan. There is no apply job, and a local plan/apply from local files is blocked. A PIM-elevated engineer applies the saved plan of the merged commit with apply-merged.
  • Lane B (Tiers 1 and 2): CI runs checks and plan on the MR, then apply automatically on merge, using an OIDC identity scoped to its own management group or subscription.

The detailed case against CI apply at Tier 0 is in Tier 0 Pipeline Design and Platform Service Principal Vending.

The lanes at a glance

Lane A: Tier 0 Lane B: Tiers 1 and 2
Repositories azure-tenant-bootstrap (steps 002–007), azure-service-principals, azure-privileged-access-grants azure-management-groups, subscription-vending-platform, subscription-vending-workload, azure-privileged-access-packages (restricted — see below), azure-monitoring, workload projects under deployments/workloads/
Scope Tenant root, mg-intermediate-root, Entra directory roles and app registrations, tenant root state mg-platform, mg-landingzones and their subscriptions; access packages and group PIM policies (restricted)
CI identity Read-only (sp-pla-tf-*-ro) Read-write, bounded to its scope (sp-pla-tf-*-rw, sp-sub-tf-*-rw)
CI stages checks → plan (on main, saved as the tfplan artifact) checks → plan → apply
Who applies A senior engineer, locally, with apply-merged — only CI's saved plan of the merged commit GitLab CI, on merge to main
Gate MR review and merge, PIM activation, apply-merged's checks (merged MR, saved plan for exactly that commit, matching Terraform version) MR review and approval
Privilege None standing. PIM activation, time-bound. Standing, least-privilege, bounded to the target scope
State Central platform containers in sa<tenant>tenantroot Dedicated per-subscription state account in rg-terraform-state
Blast radius The whole tenant One management group, subscription or resource group

Choosing a lane

Classify every new repository before its first pipeline runs. If any answer points to Lane A, it is Lane A.

flowchart TD
    Q1{"Does it manage Entra ID app registrations,<br/>service principals or Graph permissions?"}
    Q1 -->|Yes| LANE_A["LANE A<br/>Plan-only CI, human applies the saved plan"]
    Q1 -->|No| Q2{"Does it assign roles at the tenant root<br/>or mg-intermediate-root?"}
    Q2 -->|Yes| LANE_A
    Q2 -->|No| Q3{"Could a compromised pipeline credential<br/>escalate its own access or create new admin access?"}
    Q3 -->|Yes| LANE_A
    Q3 -->|No| Q4{"Is the blast radius bounded to one subscription<br/>or a non-identity management group?"}
    Q4 -->|Yes| LANE_B["LANE B<br/>Automated GitOps, scoped OIDC"]
    Q4 -->|No| LANE_A
Component Lane Why
Directory role management (Privileged Role Administrator, Global Administrator) A Changes the identity control plane directly, and could create a permanent backdoor.
Persona groups, their directory roles and Tier 0 RBAC (azure-privileged-access-grants) A Role-assignable groups holding Tier 0 roles; who may request and approve Tier 0 access.
Service principal and app registration creation A Needs admin consent and credential-creation rights (EAM control plane).
Role assignments at the tenant root A Inherited by every subscription in the tenant.
PIM for Groups policies and access packages (azure-privileged-access-packages) B, restricted The Azure CLI can never get these Graph scopes (AADSTS65002), so only a pipeline can manage them. Accepted as EX-002 with guardrails: the identity can't modify role-assignable groups or their membership; who may request and approve is Lane A; nobody is made eligible in code (eligibility only comes from an approved request); it owns a group only briefly, during catalog onboarding, and every plan warns if it still does.
Tenant root state storage A Holds state with sensitive root metadata.
Management group hierarchy (azure-management-groups) B Creates and moves groups below the root, with Management Group Contributor and no directory roles.
Subscription vending (subscription-vending-platform, subscription-vending-workload) B Creates landing zones through billing APIs. It cannot change the identity architecture.
Platform monitoring (azure-monitoring) B Bounded to mg-management, with no directory permissions.
Workload resources (deployments/workloads/*) B Bounded to individual workload subscriptions.

Lane A: Tier 0

sequenceDiagram
    autonumber
    actor Engineer as Senior Engineer
    actor Reviewer as Peer Senior Engineer
    participant GitLab as GitLab CI
    participant PIM as Entra ID PIM
    participant Azure as Azure / Entra / State

    Engineer->>GitLab: Open MR
    GitLab->>Azure: Checks + terraform plan (read-only OIDC)
    Reviewer->>GitLab: Review plan, approve and merge
    GitLab->>Azure: main pipeline: plan again, save as tfplan
    Note over GitLab,Azure: CI stops here. There is no apply job.
    Engineer->>PIM: Activate the operator persona, with justification
    Reviewer->>PIM: Approve (when two-person approval is enabled)
    Engineer->>GitLab: apply-merged downloads main's saved plan
    Engineer->>Azure: Shows it, engineer types "apply", terraform apply tfplan
    Note over Azure: Audit logged in Entra ID, ARM and storage

The plan-only pipeline

  • CI signs in with a keyless OIDC identity (sp-pla-tf-*-ro) that can't change live infrastructure: Directory Readers in Entra ID, Reader at the scopes it plans (up to the tenant root), and Storage Blob Data Contributor on its state container — needed only to take the state lock, not to change anything it describes.
  • The pipeline YAML has no apply job. Even if an attacker takes over GitLab or forces a merge, there is no job and no credential that can change live infrastructure.
  • On main, the plan job saves the plan (tfplan, or <step>/tfplan in a multi-root repo) — that artifact is the only thing that can be applied.
  • Merged-only guard. Each Lane A root has a ci-only-guard.tf: a local terraform plan/apply fails immediately, because only CI sets TF_VAR_running_in_ci. So a human can't apply local, unreviewed files by accident; the only path is apply-merged. It stops accidents, not a deliberate bypass — that's what the audit trail is for.
  • Terraform is pinned to the CI image's version (required_version), and provider lock files carry Linux and Windows hashes, so CI's saved plan applies on the engineer's machine.

apply-merged

apply-merged (from devops/ps-menu) is the Lane A apply. Run in the project's folder — or, in azure-tenant-bootstrap, in the step's folder — it refuses unless:

  1. you're on an up-to-date, clean main;
  2. that commit came from a merged merge request (it shows the MR, author and approvers);
  3. the latest successful main pipeline saved a plan for exactly that commit (job plan, or plan:<step>).

Then it initialises (using the folder's backend.hcl if it has one), shows the saved plan and asks you to type apply. Terraform itself refuses a plan whose state changed since CI made it, or that was made by a different Terraform version — re-run the main pipeline and try again. apply-merged -WhatIf stops after showing the plan.

Preflight checks (preflight.ps1)

azure-service-principals and azure-tenant-bootstrap also ship a preflight.ps1. Run it before apply-merged there: it checks the az login session and tenant, state data-plane access, and the directory and management-plane roles the apply needs, so an apply never fails halfway.

Fail closed

If any check fails, the script stops with the missing role or mismatch and how to fix it. Never bypass or disable preflight checks.

Operator runbook

Use this for azure-privileged-access-grants, azure-service-principals and each azure-tenant-bootstrap step.

  1. Review and merge. Checks pass, the plan from sp-pla-tf-*-ro is reviewed in the MR, and the MR is merged. Wait for the main pipeline to finish — it saves the plan.
  2. Elevate. In PIM → My roles → Groups, activate the project's Tier 0 group — for azure-privileged-access-grants, Manage Azure Privileged Access Grants (pag-t0-manage-azure-privileged-access-grants: Privileged Role Administrator, conditioned RBAC Administrator at the tenant root, its state container), with a justification. Some changes need more, and the project's runbook says so — for example Emergency Tenant Recovery to assign an Azure role to a service principal, since the operator persona can only assign roles to groups. Deactivate anything extra straight after.

  3. Sign in again, so the token carries the activation:

    az account clear
az login

  4. Apply the merged commit's saved plan:

    git checkout main; git pull
pwsh -File ./preflight.ps1   # where the project has one
apply-merged                 # from the project folder, or the bootstrap step's folder

  5. Close out. Confirm the apply completed, deactivate your PIM roles (or let them expire), and attach the apply summary to the change ticket.

The one planned exception is azure-tenant-bootstrap step 002's first-time bootstrap: it creates the CI identity itself, so no CI plan can exist yet. Its runbook documents the single guarded local run.

Lane B: Tiers 1 and 2

Workflow

  1. Open an MR. CI runs fmt, tflint, validate and checkov, then plan with the repository's scoped OIDC identity, and posts the plan to the MR.
  2. A peer reviews and approves. The MR plan is the only gate — read it for destroys you didn't intend (a renamed module or resource without a moved block is the usual cause). Merge to main.
  3. The main pipeline runs terraform apply with the same scoped identity.

Identity and state

  • No secrets. Each repository has a federated credential tied to its project path (project_path:<group>/<repo>). If a project is renamed or moved, its credential's subject must follow, or CI can't sign in.
  • Bounded scope. The identity only has rights on its own subscription or management group, for example Contributor plus User Access Administrator on one workload subscription. It has no access to parent management groups and no Entra directory roles.
  • Isolated state. Each vended subscription gets its own state storage account in rg-terraform-state. The pipeline identity has Storage Blob Data Contributor on its own container only, and no access to sa<tenant>tenantroot.

Break-glass

In an emergency, such as a lockout or security incident:

  1. Request Emergency Tenant Recovery through its access package where PIM still works; otherwise use a break-glass account (see Break-Glass Accounts). Sign-in alerting for those accounts is planned, not yet in place (GAP-006 in the Exceptions Register).
  2. Prefer Lane A mechanics even in an emergency: a merged MR and apply-merged.
  3. If CI itself is unavailable, the merged-only guard can be lifted for a single, recorded local run — the same mechanism as step 002's bootstrap exception. Record the plan, the apply output and why, for review afterwards (NIST AU-6), and follow up with an MR that brings main in line.