Subscription Vending¶

Subscription vending is the end-to-end process of taking a billing scope and producing a fully configured, pipeline-ready Azure subscription. It is the primary product of the platform team — what workload teams receive when they are ready to build.

What a workload team receives¶

When a new subscription is vended, the workload team gets:

An Azure subscription placed in the correct management group, tagged to the ADR-0001 standard
A dedicated Terraform state storage account (isolated per subscription) with a private state container
A service principal (sp-sub-tf-{name}-rw) with a single OIDC federated credential — no passwords
Three Entra ID security groups for human access: grp-{name}-role-based-access-administrator and grp-{name}-contributor (both PIM-eligible), and grp-{name}-reader (standing)
A GitLab deployment project with ARM_CLIENT_ID and ARM_SUBSCRIPTION_ID set, and a starter scaffold committed once. The scaffold includes a resource lock the team can switch off and on themselves.
A monthly budget with actual and forecast spend alerts
Defender for Cloud configured (free posture layer + security contact, plus any paid plans requested)
A locked, empty rg-platform-{name} resource group, a baseline set of registered resource providers, and the Activity Log streamed to the central Log Analytics workspace

The workload team does not configure authentication, state storage, or RBAC. It is all done.

What gets created, resource by resource¶

Everything below is produced by one call to the solution-manage-azure-subscription module (Tier 2), which composes five pinned modules and adds a few solution-level resources. {name} is the subscription name, e.g. payments-prod-westeu-00.

Subscription and placement — `terraform-azurerm-subscription`¶

Resource	Name / identifier	What it is and why
`azurerm_subscription`	`{name}`	The MCA subscription itself, created against `billing_scope_id`. Given a deterministic alias equal to `{name}` so it can be looked up directly instead of by random GUID. Creation takes 10–30 min.
`azurerm_management_group_subscription_association`	→ `mg-{purpose}`	Moves the subscription out of the tenant root group into its target management group, so it inherits that group's policy and RBAC. Owned here, not by the bootstrap module.

Terraform state storage — `terraform-azurerm-storage-account`¶

Resource	Name / identifier	What it is and why
`azurerm_storage_account`	`sttfstate{8-hex}`	A storage account dedicated to this subscription's Terraform state, created in the shared `rg-terraform-state` resource group on the platform subscription. The 8 hex characters are derived from a stable `random_id` keyed on `{name}` — the name only changes if `{name}` changes. Baseline security: TLS 1.2 min, HTTPS only, no public blob access, no cross-tenant replication. Blob versioning on; 30-day delete retention on blobs and containers.

Shared Key auth is disabled by default (shared_access_key_enabled = false); a few pre-policy accounts still run with it enabled until they can be remediated.

Identity, state container, and RBAC — `terraform-azurerm-subscription-bootstrap`¶

Resource	Name / identifier	What it is and why
`azurerm_storage_container`	`{name}` (in `sttfstate…`)	Private blob container holding `terraform.tfstate` for the workload deployment. One container per subscription — RBAC is scoped to it.
`azuread_application`	`sp-sub-tf-{name}-rw`	App Registration for the workload deployment pipeline. `sp` = service principal, `sub` = a subscription's own deployment identity, `tf` = Terraform, `rw` = read/write. Owned by the vending identity (see Pipeline identity and tier).
`azuread_service_principal`	(for the app above)	The directory principal that role assignments and the federated credential attach to. Owned by the vending identity.
`azuread_application_federated_identity_credential`	`gitlab-project-path`	The OIDC trust. Subject is `project_path:{gitlab project path}` only — deliberately ref-agnostic, so one credential covers both `plan` on any MR branch and `apply` on `main`. Requires the GitLab project's `ci_id_token_sub_claim_components = ["project_path"]`. No client secret exists.
`azurerm_role_assignment` — Contributor	scope: target subscription	Lets Terraform create/modify/delete resources in the subscription. Scoped to this subscription only.
`azurerm_role_definition` — `Grinntec - Lock Contributor - {name}`	scope: target subscription	Custom role granting only `Microsoft.Authorization/locks/*`. Needed because Contributor's `NotActions` exclude lock writes, and deployments that create management locks would otherwise fail. Avoids handing the SP Owner / User Access Administrator.
`azurerm_role_assignment` — Lock Contributor	scope: target subscription	Assigns the custom role above to the deployment SP.
`azurerm_role_assignment` — Storage Blob Data Contributor	scope: the state container	Read/write access to the state blob — Entra ID auth, no storage keys. Scoped to this subscription's container only.
`azurerm_role_assignment` — Reader	scope: `rg-terraform-state`	`terraform init` reads storage account metadata in the state resource group.
`azuread_group` ×3	`grp-{name}-role-based-access-administrator`, `grp-{name}-contributor`, `grp-{name}-reader`	Security groups (mail-disabled) for human access to the subscription. Owned by the vending identity.
`azurerm_pim_eligible_role_assignment` ×2	RBAC Administrator / Contributor → their group, scope: target subscription	Eligible, not standing. A member must activate the role in PIM before it applies. Each eligibility has an expiry and is renewed on the next apply after it lapses.
`azurerm_role_assignment`	Reader → `grp-{name}-reader`, scope: target subscription	Standing. Nothing behind read-only access is worth the activation step.
`azuread_group_member` (0..n)	—	Adds the groups named in the `rbac` block (by display name) as members of the three groups. Groups only, never individual users, so membership is managed in one place. Empty lists make no changes.
Extra assignments (0..n)	extra RBAC Administrator / Contributor / Reader groups	Optional additional group object IDs via `role_based_access_administrator_group_ids` / `contributor_group_ids` / `reader_group_ids`, with the same eligible-or-standing split.

GitLab deployment project — `terraform-gitlab-deployment-project`¶

Resource	Name / identifier	What it is and why
`gitlab_project`	`{name}` under `deployments/workloads`	The private repo the workload team commits Terraform to. Merge (not rebase) method, shared runners on. Sets `ci_id_token_sub_claim_components = ["project_path"]` so pipeline ID tokens match the federated credential.
`gitlab_project_variable` — `ARM_CLIENT_ID`	masked, unprotected	The deployment SP's client ID. Unprotected on purpose — the `plan` job runs on unprotected MR branches and a protected variable would be withheld.
`gitlab_project_variable` — `ARM_SUBSCRIPTION_ID`	masked, unprotected	The target subscription ID.
`terraform_data.seed`	one commit: `feat: initial scaffold`	One-shot scaffold. When the project is created, one commit adds `.gitlab-ci.yml`, `versions.tf`, `main.tf` (a starter resource group), `resource-lock.tf` and `README.md`, so the repo is pipeline-ready on day one. It only ever commits into an empty repository, and vending never looks at the files again. See The starter scaffold.

Cost management — `terraform-azurerm-subscription-budget`¶

Resource	Name / identifier	What it is and why
`azurerm_consumption_budget_subscription`	`{name}`	Monthly budget at `budget_amount` (default 10). Three notifications: 80% actual, 100% actual, and 100% forecast — each emailing `budget_alert_emails`. `start_date` is frozen after first apply so month rollovers never force a replacement.

Solution-level resources (added directly by `solution-manage-azure-subscription`)¶

Resource	Name / identifier	What it is and why
`random_uuid.finops`	—	An immutable cost-allocation UUID. Generated once, never regenerated, `prevent_destroy` on. Surfaces only as the `FinOpsId` tag on the subscription, the platform RG and the state account.
`azapi_update_resource.foundational_cspm`	`FoundationalCspm` pricing	The free Defender for Cloud posture layer — Secure Score, recommendations, asset inventory, MCSB. On by default.
`azapi_resource.security_contact`	`default`	Defender for Cloud security contact — where high-severity alert and posture notifications go. Subscription Owners are notified too.
`azapi_update_resource.defender_plan` (0..n)	one per `defender_plans` entry	Paid Defender plans (e.g. `StorageAccounts = "on"` → Standard tier with the `DefenderForStorageV2` sub-plan). Default `{}` = no paid plans.
`azapi_resource.platform_resource_group`	`rg-platform-{name}`	An empty resource group inside the vended subscription — a known home for platform-owned bits (diagnostic targets, shared identities) so workloads don't scatter them into ad-hoc RGs. On by default. Tagged with the same ADR-0001 set as the subscription.
`azapi_resource.platform_resource_group_lock`	`CanNotDelete`	Delete lock on `rg-platform-{name}`. On by default. To remove the group, turn the lock off in one MR and delete in a separate one.
`azapi_resource_action.register_resource_provider` (n)	baseline provider list	Registers ~20 resource providers (Compute, Network, KeyVault, Insights, ManagedIdentity, …) so the first workload deploy doesn't hit `MissingSubscriptionRegistration`. Idempotent; removing an entry does not unregister it.
`azurerm_monitor_diagnostic_setting.central_activity_log`	`to-{workspace}`	Always on. Streams the subscription's Activity Log to the central Log Analytics workspace, which is where "who changed what, when" comes from.
`azurerm_pim_eligible_role_assignment` (0..n)	one per `pim_eligible_role_assignments` entry	Optional extra JIT-eligible roles at subscription scope. Default `{}` = none.

Tags applied to the subscription (and `rg-platform-{name}`)¶

Caller tags < always-on IaC provenance < FinOpsId (last wins). The provenance keys — which a caller cannot override — are:

ManagedBy = terraform, ResourceType = azure-subscription, IaCModule, IaCModuleVersion, IaCRepo, Environment, Workload (derived from {name}), WorkloadOwner, PlatformOwner = platform-team, FinOpsId.

Pipeline identity and tier¶

The two vending deployments are tiered differently:

Deployment	Identity	Tier	Lane
`subscription-vending-platform`	`sp-pla-tf-subscription-vending-platform-rw`	Tier 0, by decision	Lane A (being adopted): CI plans, a PIM-elevated human applies. Platform subscriptions are rare and platform-owned, so a human apply costs little.
`subscription-vending-workload`	`sp-pla-tf-subscription-vending-rw`	Tier 1 target. Tier 0 in effect until the controls below are in.	Lane B: CI applies on merge, which Tier 1 allows.

Both identities are declared in t0-azure-service-principals and today hold the same grants:

Grant	Scope	Why
`Application.ReadWrite.OwnedBy` (Graph)	apps and SPs it owns	Create and manage each `sp-sub-tf-{name}-rw` and its federated credential
`Group.Create` (Graph)	groups it owns	Create the three `grp-{name}-*` groups and manage their membership
`Directory.Read.All` (Graph)	tenant	Look up member groups by name
RBAC Administrator, Management Group Contributor	`mg-landingzones`, `mg-sandboxes`	Role and PIM assignments on vended subscriptions, and placing them in the right MG
Owner	each subscription it creates (automatic)	Resource providers, Defender, the platform RG, locks and budgets
Contributor, conditioned RBAC Administrator	`rg-terraform-state-workloads-prod`	The workload state accounts
Azure subscription creator	MCA invoice section (manual grant)	Create and read subscription aliases

Why owned-only. Tenant-wide Application.ReadWrite.All would let a vending pipeline add a credential to any app registration, including the Tier 0 apply identities. Group.ReadWrite.All would let it change the membership of any group that isn't role-assignable. Either one fails the Tier 0 test. The OwnedBy and Group.Create forms limit each identity to objects it created. The bootstrap module makes the vending identity owner of every app, SP and group it creates, so nothing else is needed. The Graph side therefore passes as Tier 1.

What still makes workload vending Tier 0 (GAP-011). Two of the original routes are closed: the Azure grants have moved from mg-intermediate-root down to mg-landingzones and mg-sandboxes, which hold no Tier 0 assets, and workload state accounts now live in rg-terraform-state-workloads-prod, where vending's rights stop. Two remain, and both must be closed before workload vending is Tier 1:

Its own state is in sa17452008tenantroot, beside Tier 0 state (test 4). It can write only its own container, but it shares the account with Tier 0. Fix: move its backend out of rg-terraform-state.
A group-wide GitLab token. If it can change the pipelines of Tier 0 repositories, it controls the identities they sign in as (test 2). Fix: a token scoped to deployments/workloads.

The starter scaffold¶

The scaffold gives the workload team a working repo from the first commit:

File	Purpose
`.gitlab-ci.yml`	Includes the standard Terraform deployment pipeline
`versions.tf`	Providers and this subscription's state backend, already filled in
`main.tf`	A starter resource group
`resource-lock.tf`	`CanNotDelete` lock on that resource group, behind an `enable_resource_lock` toggle
`README.md`	How the project and subscription were provisioned

After the first commit the files belong to the team. Vending never tracks them, so the team can edit, rename or delete any of them. Changes to the scaffold template only reach projects created afterwards.

Unlocking the starter resource group

To change or remove something the lock protects:

Set enable_resource_lock = false in resource-lock.tf and apply.
Make the destructive change.
Set it back to true and apply again.

Why the scaffold is seeded, not tracked

Each scaffold file used to be a Terraform resource that vending kept checking. When a team deleted one, vending tried to recreate it, the push to the protected main branch failed, and every vending apply for that repository broke. Seeding once, then forgetting, removes that coupling.

The vending model¶

Subscription vending is broken into discrete chunks. This table shows what is implemented and where.

#	Chunk	Status	Where
1	Subscription creation	✅ Done	`terraform-azurerm-subscription` (via the solution)
2	Management group placement	✅ Done	`terraform-azurerm-subscription` (via the solution)
3	IaC identity (OIDC)	✅ Done	`terraform-azurerm-subscription-bootstrap`
4	State storage	✅ Done	`terraform-azurerm-storage-account` (account) + bootstrap (container + RBAC)
5	Pipeline project	✅ Done	`terraform-gitlab-deployment-project`
6	Human access (RBAC)	✅ Done	`terraform-azurerm-subscription-bootstrap`
7	Cost management (budget)	✅ Done	`terraform-azurerm-subscription-budget`
8	Security baseline (Defender for Cloud)	✅ Done	solution `azapi` resources (CSPM, security contact, optional paid plans)
9	Platform resource group	✅ Done	solution `azapi` resources (`rg-platform-{name}` + delete lock)
10	Resource provider registration	✅ Done	solution `azapi` action
11	Policy assignments	➖ Inherited	From the target management group's EPAC assignments — none applied per-subscription
12	Networking (spoke VNet, hub peering)	❌ Not started	—
13	Private DNS zone links	❌ Not started	—
14	Monitoring (activity log export)	✅ Done	solution `azurerm_monitor_diagnostic_setting` to the central workspace

How a subscription is defined¶

Each subscription has its own <name>.tf file in a vending deployment: deployments/azure-platform/subscription-vending-workload for workloads, …/subscription-vending-platform for platform subscriptions. One module call vends the whole thing.

module "sub_payments_prod_westeu_00" {
  source = "git::https://gitlab.com/grinntec-cloud/terraform-modules/terraform-solutions/solution-manage-azure-subscription.git?ref=v8.1.13"

  providers = {
    azurerm = azurerm
    azapi   = azapi
    azuread = azuread
    gitlab  = gitlab
    random  = random
  }

  name             = "payments-prod-westeu-00"
  management_group = "mg-landingzones"
  environment      = "prod"

  tags = {}

  # Groups only, by display name. RBAC Administrator and Contributor are
  # PIM-eligible; Reader is standing.
  rbac = {
    role-based-access-administrator = { group_names = ["grp-payments-leads"] }
    contributor                     = { group_names = ["grp-payments-devs"] }
    reader                          = { group_names = [] }
  }

  budget_amount       = 10
  budget_alert_emails = ["neil@grinntec.net"]

  # Optional — shown with their defaults:
  # defender_plans                  = {}      # { StorageAccounts = "on" } to enable a paid plan
  # platform_resource_group_enabled = true    # locked, empty rg-platform-{name}
  # pim_eligible_role_assignments   = {}      # extra JIT-eligible roles at subscription scope
}

output "payments_prod_westeu_00" {
  description = "Bootstrap outputs for payments-prod-westeu-00."
  sensitive   = true
  value = {
    gitlab_ci_variables          = module.sub_payments_prod_westeu_00.gitlab_ci_variables
    backend_config               = module.sub_payments_prod_westeu_00.backend_config
    tags                         = module.sub_payments_prod_westeu_00.tags
    platform_resource_group_name = module.sub_payments_prod_westeu_00.platform_resource_group_name
  }
}

One file per subscription means adding or removing a subscription is a one-file change — no cross-file coordination.

Subscription naming¶

All subscriptions follow this pattern:

{workload}-{environment}-{region}-{sequence}

Component	Values	Example
Workload	Short workload name	`payments`, `networking`
Environment	`prod`, `dev`, `test`	`prod`
Region	Azure region abbreviation	`westeu`
Sequence	Two-digit number, from `00`	`00`, `01`

Examples: networking-prod-westeu-00, payments-dev-westeu-01

Current subscriptions¶

Subscription	Management Group	Vended by
`networking-prod-westeu-00`	`mg-connectivity`	`subscription-vending-platform`
`azgovviz-hosting-prod-westeu-00`	`mg-management`	`subscription-vending-workload`
`grinntec-website-prod-westeu-00`	`mg-landingzones`	`subscription-vending-workload`

The management subscription that holds rg-terraform-state was created during tenant bootstrap, not vended.

Adding a new subscription¶

In subscription-vending-workload, copy subscription-template.tf.example to <name>.tf. For a platform subscription, use subscription-vending-platform and copy an existing file.
Find and replace SUB-NAME (hyphenated) and SUB_NAME (underscored) throughout the file
Set management_group, the rbac block, and any non-default options
Open a Merge Request — the pipeline posts terraform plan as a comment
Review and merge — the pipeline applies on merge to main
Run terraform output <name> to retrieve the GitLab CI variable and backend values if needed manually

Note

Azure MCA subscription creation can take 10–30 minutes. If the pipeline times out, check whether the subscription was created before retrying:

az account list --all --query "[?name=='{name}'].{name:name,id:id}" -o table

If it exists, re-trigger the pipeline — Terraform will pick up the existing subscription via its deterministic alias.

What the bootstrap and solution create¶

graph TD
    SUB["azurerm_subscription<br/>{name}"] --> MG["Management group association<br/>mg-{purpose}"]

    SA["State Storage Account<br/>sttfstate{8-hex} (in rg-terraform-state)"]
    SA --> CONT["Blob Container<br/>name = {name}"]

    APP["App Registration<br/>sp-sub-tf-{name}-rw"]
    APP --> SP["Service Principal"]
    SP --> FC["Federated Credential: gitlab-project-path<br/>subject = project_path:{path} (plan + apply)"]
    SP --> RA1["Contributor — target subscription"]
    SP --> RA2["Lock Contributor (custom) — target subscription"]
    SP --> RA3["Storage Blob Data Contributor — state container"]
    SP --> RA4["Reader — rg-terraform-state"]

    GRP_A["grp-{name}-role-based-access-administrator"] --> RA_A["RBAC Administrator — target subscription (PIM-eligible)"]
    GRP_C["grp-{name}-contributor"] --> RA_C["Contributor — target subscription (PIM-eligible)"]
    GRP_R["grp-{name}-reader"] --> RA_R["Reader — target subscription (standing)"]

    GL["GitLab project<br/>deployments/workloads/{name}"] --> CIV["CI vars: ARM_CLIENT_ID, ARM_SUBSCRIPTION_ID"]
    GL --> SEED["One-shot scaffold commit<br/>(incl. resource-lock.tf)"]

    SOL["Solution extras"] --> BUD["Budget — monthly, 80/100% actual + 100% forecast"]
    SOL --> DEF["Defender for Cloud — CSPM + security contact (+ optional paid plans)"]
    SOL --> PRG["rg-platform-{name} (empty, delete-locked)"]
    SOL --> RPR["Resource provider registration (baseline set)"]
    SOL --> LOG["Activity Log → central Log Analytics"]
    SOL --> FIN["FinOpsId tag (immutable UUID)"]

The OIDC federated credential model means the pipeline authenticates to Azure using a short-lived JWT issued by GitLab. There are no client secrets. The credential subject is a specific GitLab project path — a stolen token from one project cannot authenticate as another subscription's service principal.

State storage design¶

Each subscription gets its own dedicated storage account in rg-terraform-state on the platform subscription. Isolation is deliberate:

A corrupted state file in one subscription cannot affect another
RBAC is scoped to a single blob container — the workload SP cannot read another subscription's state
Blob versioning is on, with 30-day delete retention on blobs and containers, so a bad write can be rolled back
Shared Key auth is disabled by default — access is via Entra ID; a few pre-policy accounts are exceptions until remediated

The storage account name is sttfstate{8-hex-chars}, where the hex is derived from the subscription name via a stable random_id. The name does not change unless the subscription name changes.

The state accounts are protected operationally rather than by a Terraform prevent_destroy guard — a calling module cannot inject lifecycle into a composed module. The shared rg-terraform-state resource group itself does carry prevent_destroy = true.