Skip to content

Latest commit

Β 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

☁️ Azure Machine Learning Blob Datastore Terraform Module

Registers an Azure Machine Learning datastore pointing at a blob container, defaulting to credential-free identity-based access, targeting hashicorp/azurerm ~> 4.0.

Terraform azurerm Module Type Resources Posture


🧩 Overview

  • βš™οΈ Creates one azurerm_machine_learning_datastore_blobstorage, named this.
  • πŸ” Defaults to identity-based access, where the provider defaults to credential-based. The provider's default caches a storage key in the workspace's Key Vault β€” retrievable by anyone with workspace Reader.
  • 🎯 Groups the three credential fields into one variable so both of Microsoft's documented rules become parse-time errors: the key/SAS conflict, and the credential-required-unless-identity rule.
  • πŸ”— Emits datastore_uri_prefix β€” the azureml:// string a notebook actually uses, which exists nowhere in the resource's own state.
  • πŸ“€ Emits storage_account_id, derived by truncating the container ID, ready to be a role-assignment scope.
  • ⚠️ Rejects a bare workspace GUID by name, because the workspace module's correctly-named workspace_id output is the wrong one.

πŸ’‘ Why it matters: a datastore looks like a two-line resource and is really a choice about where a storage key lives. Credential-based access puts an account key β€” full control of every container in the account β€” behind a workspace Reader role that people hand out for visibility. Identity-based access stores nothing. The provider defaults to the first; this module defaults to the second and is explicit about the one prerequisite that makes it work.


❀️ Support this project

If this module saves you time:


πŸ—ΊοΈ Where this fits in the family

flowchart TB
  RG["terraform-azurerm-resource-group"]
  SA["terraform-azurerm-storage-account -- the data this references. NEVER created or deleted by a datastore."]
  MLW["terraform-azurerm-machine-learning-workspace"]
  BLOB["terraform-azurerm-machine-learning-datastore-blobstorage"]
  FS["terraform-azurerm-machine-learning-datastore-fileshare"]
  ADLS["terraform-azurerm-machine-learning-datastore-datalake-gen2"]
  RA["terraform-azurerm-role-assignments -- Storage Blob Data Reader, required for identity-based access"]
  CC["terraform-azurerm-machine-learning-compute-cluster -- runs the jobs that read these datastores"]

  RG -->|"name as resource_group_name"| MLW
  MLW -->|"id as workspace_id -- NOT its workspace_id output, which is a GUID"| BLOB
  MLW -->|"id as workspace_id"| FS
  MLW -->|"id as workspace_id"| ADLS
  SA -->|"container id as storage_container_id"| BLOB
  SA -->|"share id as storage_fileshare_id -- A DIFFERENT ARGUMENT NAME"| FS
  SA -->|"container id as storage_container_id"| ADLS
  MLW -->|"identity_principal_id"| RA
  BLOB -->|"storage_account_id as the role scope"| RA
  BLOB -->|"datastore_uri_prefix"| CC

  style BLOB fill:#0078D4,stroke:#004578,color:#ffffff
  style MLW fill:#004578,stroke:#00335c,color:#ffffff
  style RG fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style SA fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style FS fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style ADLS fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style RA fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style CC fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
Loading

Two edges are the ones to get right. The workspace's id becomes workspace_id β€” not its workspace_id output, which is a GUID. And the role assignment is not optional decoration: identity-based access is the default here, and it does nothing until the workspace identity holds Storage Blob Data Reader on the account. This DAG is shared with the two sibling datastore modules, which differ in their storage target and their authentication.


🧬 What this module builds

flowchart TB
  REQ["required: name, workspace_id, storage_container_id"]
  CRED["credential -- ONE variable holding account_key, shared_access_signature and service_data_auth_identity, because TWO documented rules span them. Marked sensitive as a whole."]
  R1["RULE 1: account_key CONFLICTS WITH shared_access_signature. At most one."]
  R2["RULE 2: if service_data_auth_identity is None or omitted, one of the two credentials is REQUIRED"]
  FLIP["THIS MODULE DEFAULTS service_data_auth_identity TO WorkspaceSystemAssignedIdentity. The provider defaults it to None, which caches a storage key in the workspace Key Vault."]
  THIS["azurerm_machine_learning_datastore_blobstorage.this"]
  KV["workspace Key Vault -- caches the key or SAS when credential-based. ANY WORKSPACE READER CAN RETRIEVE IT."]
  ROLE["identity-based instead needs Storage Blob Data Reader on the storage account. Nothing here can check that."]
  URI["derived: datastore_uri_prefix, the azureml:// string a notebook actually references"]
  DERIVED["derived: authentication_mode, stores_no_credential_in_the_workspace_key_vault, storage_account_id, posture_summary"]

  REQ --> THIS
  CRED --> R1
  CRED --> R2
  CRED --> FLIP
  CRED --> THIS
  THIS -->|"credential-based only"| KV
  THIS -->|"identity-based"| ROLE
  THIS --> URI
  THIS --> DERIVED

  style THIS fill:#0078D4,stroke:#004578,color:#ffffff
  style KV fill:#004578,stroke:#00335c,color:#ffffff
  style REQ fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style CRED fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style R1 fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style R2 fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style FLIP fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style ROLE fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style URI fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
  style DERIVED fill:#f2f4f7,stroke:#98a2b3,color:#1d2939
Loading
Resource Count Notes
azurerm_machine_learning_datastore_blobstorage.this 1 The keystone.
nested timeouts block 0 or 1 All four operations (30m/5m/30m/30m).

βœ… Provider / Versions

Requirement Value
Terraform >= 1.12.0
hashicorp/azurerm ~> 4.0
Provider block None here. The caller configures the provider, its authentication, and the mandatory features {} block.
Resource azurerm_machine_learning_datastore_blobstorage
ARM type Microsoft.MachineLearningServices/workspaces/dataStores
API provider Microsoft.MachineLearningServices, version 2025-06-01

Schema notes that bite

  • πŸ”΄ The workspace module's workspace_id output is NOT what this argument wants. The workspace emits id (the ARM Resource ID) and workspace_id (an immutable GUID). This argument is named workspace_id, so workspace_id = module.mlw.workspace_id reads perfectly and is wrong. Pass module.mlw.id. A bare GUID is rejected here with that message.
  • πŸ”΄ account_key conflicts with shared_access_signature, and "if service_data_auth_identity is set to None or omitted, one of account_key or shared_access_signature must be specified." Two rules, three fields β€” hence one variable.
  • πŸ”΄ A credential-based datastore's key is readable by workspace Readers. Microsoft: the credential is cached in the workspace's Key Vault and "users with Reader workspace access can access the credentials."
  • πŸ”΄ tags and description are FORCE-NEW. Unusual enough to be worth repeating: a routine retagging sweep destroys and recreates every datastore it touches. Updatable in place: the credentials, the identity, is_default.
  • πŸ”΄ is_default can only be set to true on update. A first apply with true fails, and no validation {} can catch it β€” a variable validation cannot see create-versus-update.
  • ⚠️ The argument is service_data_auth_identity here and service_data_identity on both siblings. Two spellings of one concept across three resources.
  • ⚠️ is_default is settable here and read-only (computed) on both siblings.
  • ⚠️ The identity values carry a Workspace prefix β€” WorkspaceSystemAssignedIdentity, not the SystemAssigned of a normal ARM identity block. One prefix apart, different mechanism; the wrong vocabulary is rejected by name.
  • ⚠️ The ARM segment is dataStores, capital S.
  • ⚠️ A datastore URI uses lower-case resourcegroups where an ARM ID uses resourceGroups β€” which is why datastore_uri_prefix is emitted rather than assembled by hand.
  • ℹ️ A SAS expires and nothing in Terraform tracks it. It works until it doesn't, with no plan showing a change.

Force-new facts come from the provider's documentation markdown. The binary schema carries no force-new marker at all. Shapes, types and sensitivity marks come from the schema; the credential-visibility and role requirements come from Microsoft's platform documentation.


πŸ”‘ Required Azure RBAC Roles / Permissions

Role Scope Why
Contributor or AzureML Data Scientist the ML workspace Create, update, delete the datastore.
Reader the ML workspace Refresh and plan β€” but see the warning.
Storage Blob Data Reader the storage account Granted to the workspace's identity, not Terraform's. Required for identity-based access; unnecessary on the workspace's own default account.
Reader on the storage private endpoint that endpoint Only when the account is behind a private endpoint. Also the workspace identity.
(none) storage permission for Terraform β€” The Terraform principal never touches the data plane.

πŸ”΄ Reader on the workspace is not a read-only role when credential-based datastores exist. Microsoft names it as a limitation: the credential is cached in the workspace Key Vault and "other workspace users with sufficient permissions can retrieve those credentials."

The interesting direction is the reverse of how it sounds. Reader is the role granted freely for visibility β€” and on such a workspace it grants the storage account key, which controls every container in that account, not just this one. No storage-side permission limits it, because the holder is not using one. Put it in the workspace's access review. Or store no credential, which is the default here.


Azure Prerequisites

  • An existing Machine Learning workspace, and an existing storage account and blob container. A datastore is a reference; this module creates none of them.
  • Microsoft.MachineLearningServices registered on the subscription.
  • For identity-based access (the default): Storage Blob Data Reader for the workspace identity on the storage account β€” except on the workspace's own default account. The prerequisite most likely to be missed, because its absence produces no Terraform error at all.
  • Behind a private endpoint: the workspace identity also needs Reader on the endpoint. An account may have separate blob, file and dfs endpoints.
  • For user-identity passthrough on a credential-less blob datastore: soft delete must not be enabled. A published constraint, invisible from here.
  • To set is_default = true: an existing datastore, since the provider permits it only on update.

πŸ“ Module Structure

terraform-azurerm-machine-learning-datastore-blobstorage/
β”œβ”€β”€ providers.tf   # required_version + the pinned azurerm; no provider block
β”œβ”€β”€ variables.tf   # 8 inputs, 18 validations
β”œβ”€β”€ main.tf        # locals deriving the URI, the account ID and the auth posture
β”œβ”€β”€ outputs.tf     # 24 outputs: id first, then the derived posture, then the constants
β”œβ”€β”€ README.md      # this file
β”œβ”€β”€ SCOPE.md       # the cross-module contract
β”œβ”€β”€ LICENSE        # MIT
└── .gitignore     # the canonical library ignore file

βš™οΈ Quick Start

provider "azurerm" {
  features {}
}

module "training_data" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-machine-learning-datastore-blobstorage.git?ref=v1.0.0"

  name                 = "training_data"
  workspace_id         = module.mlw.id # NOT module.mlw.workspace_id
  storage_container_id = var.training_container_id
}

Three arguments. No credential β€” the module defaults to WorkspaceSystemAssignedIdentity, so nothing is stored in the workspace's Key Vault.

⚠️ That empty call needs one thing outside it: Storage Blob Data Reader for the workspace identity on the storage account. Without it the datastore is created cleanly and jobs fail later. See example 3.


πŸ”Œ Cross-Module Contract

Consumes

Input Type Typical source
workspace_id string module.mlw.id β€” not module.mlw.workspace_id
storage_container_id string var.container_id
name string caller
credential object, sensitive omit it; or a secret store, out of band
description, is_default, tags, timeouts β€” caller

Emits

Output Consumed by
id anything referencing the datastore
datastore_uri_prefix job definitions, data assets, notebooks
storage_account_id terraform-azurerm-role-assignments β†’ scope
uses_identity_based_access, stores_no_credential_in_the_workspace_key_vault check blocks, security review
posture_summary a composition's own output

πŸ“š Example Library

The examples below reference existing resources by ID or name rather than creating them; this module owns only its own resource. Those references are declared inputs:

variable "container_id" {
  description = "id of an existing container that these examples reference but do not create."
  type        = string
}

variable "training_container_id" {
  description = "id of an existing training container that these examples reference but do not create."
  type        = string
}
1 Β· The minimum call β€” credential-free
module "training_data" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-machine-learning-datastore-blobstorage.git?ref=v1.0.0"

  name                 = "training_data"
  workspace_id         = module.mlw.id
  storage_container_id = var.training_container_id
}

πŸ’‘ authentication_mode reads Identity-based, supplied_credential_count reads 0, and stores_no_credential_in_the_workspace_key_vault reads true. The provider would have given you None β€” meaning credential-based β€” and then rejected the call for having no credential. This module's default is both safer and the shorter thing to type.

2 Β· The mistake that reads perfectly
# FAILS AT PARSE TIME, and the message says why.
workspace_id = module.mlw.workspace_id

# Correct.
workspace_id = module.mlw.id

πŸ”΄ The workspace module emits both. id is the ARM Resource ID; workspace_id is the workspace's immutable GUID, a different thing entirely. Because this argument is also called workspace_id, the wrong one is the one whose name matches β€” the single most seductive wiring error in this family. The module rejects a bare GUID and names the output to use instead.

3 Β· πŸ”‘ The role assignment identity-based access needs
module "training_data" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-machine-learning-datastore-blobstorage.git?ref=v1.0.0"

  name                 = "training_data"
  workspace_id         = module.mlw.id
  storage_container_id = var.training_container_id
}

module "mlw_can_read_training_data" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-role-assignments.git?ref=v1.0.0"

  # Derived by truncating the container ID -- no need to restate the account.
  scope = module.training_data.storage_account_id

  role_assignments = {
    blob_read = {
      role_definition_name = "Storage Blob Data Reader"
      principal_id         = module.mlw.identity_principal_id
      principal_type       = "ServicePrincipal"
    }
  }
}

πŸ’‘ This is the failure mode worth internalising. Without the grant, the datastore is created successfully, the plan is clean, and a job reading azureml://.../datastores/training_data/paths/x.csv fails days later with an authorization error against the storage account. Nothing in Terraform ever mentions it.

ℹ️ Microsoft documents one exception: the grant is unnecessary when this is the workspace's own default storage account.

4 Β· Credential-based, when you must
data "azurerm_storage_account" "legacy" {
  name                = "legacydatastore"
  resource_group_name = "rg-legacy"
}

module "legacy_data" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-machine-learning-datastore-blobstorage.git?ref=v1.0.0"

  name                 = "legacy_data"
  workspace_id         = module.mlw.id
  storage_container_id = "${data.azurerm_storage_account.legacy.id}/blobServices/default/containers/exports"

  credential = {
    # BOTH fields are needed: the identity must be explicitly "None".
    service_data_auth_identity = "None"
    account_key                = data.azurerm_storage_account.legacy.primary_access_key
  }
}

πŸ”’ Three consequences. sensitive = true redacts plan output and does not encrypt state β€” that key is in your state file in plaintext. The key is also cached in the workspace's Key Vault, where any workspace Reader can retrieve it. And an account key grants full control of every container in that account, not just exports.

πŸ’‘ supplied_credential_count reads 1 and the module never emits which field it was, because account_key and shared_access_signature describe very different blast radii.

5 Β· A SAS instead of an account key
credential = {
  service_data_auth_identity = "None"
  shared_access_signature    = var.exports_container_sas
}

πŸ’‘ Narrower than an account key and still a stored credential β€” prefer it if a credential is unavoidable.

⚠️ A SAS expires and nothing in Terraform tracks it. The datastore keeps working until the expiry passes, then fails at job time with no plan showing any change. uses_sas_token reads true; put the expiry in a calendar.

6 Β· The two SAS formats the portal offers
# BOTH FAIL AT PARSE TIME.
credential = {
  service_data_auth_identity = "None"
  shared_access_signature    = "?sv=2022-11-02&ss=b&sig=REDACTED"
}
credential = {
  service_data_auth_identity = "None"
  shared_access_signature    = "https://acct.blob.core.windows.net/exports?sv=2022-11-02&sig=REDACTED"
}

# Correct -- the token alone.
credential = {
  service_data_auth_identity = "None"
  shared_access_signature    = "sv=2022-11-02&ss=b&srt=sco&sp=rl&sig=REDACTED"
}

ℹ️ The portal shows a SAS twice: as a token beginning ?sv= and as a full Blob SAS URL. This argument wants neither of those verbatim β€” it wants the token with the leading ? stripped. Both wrong forms are rejected by name, because copying from the portal is the normal way to obtain one.

7 Β· What you cannot do β€” both credentials
# FAILS AT PARSE TIME.
credential = {
  service_data_auth_identity = "None"
  account_key                = var.key
  shared_access_signature    = var.sas
}

⚠️ The provider documents them as conflicting. They are two alternative credentials for the same container, and supplying both leaves which one Azure uses undefined. This is one of the two rules that only became enforceable by putting the three fields in a single variable.

8 Β· And what you cannot do β€” no credential with `None`
# FAILS AT PARSE TIME.
credential = {
  service_data_auth_identity = "None"
}

⚠️ The provider's rule: "if service_data_auth_identity is set to None or omitted, one of account_key or shared_access_signature must be specified." So None is a commitment to supplying a credential.

πŸ’‘ The error message names the way out: set service_data_auth_identity to WorkspaceSystemAssignedIdentity β€” which is the module's default β€” and supply nothing.

9 Β· The wrong identity vocabulary
# FAILS -- this is the ARM identity-block vocabulary, a different mechanism.
credential = { service_data_auth_identity = "SystemAssigned" }

# FAILS -- recognised value, wrong case.
credential = { service_data_auth_identity = "workspacesystemassignedidentity" }

# Correct.
credential = { service_data_auth_identity = "WorkspaceSystemAssignedIdentity" }

πŸ’‘ SystemAssigned and UserAssigned are what an identity {} block takes on most Azure resources. This argument selects which of the workspace's identities retrieves data, so the values carry a Workspace prefix. One prefix apart, and a caller who typed SystemAssigned was not guessing β€” they were applying a pattern that holds everywhere else. The message maps it across.

10 Β· A user-assigned workspace identity
credential = {
  service_data_auth_identity = "WorkspaceUserAssignedIdentity"
}

ℹ️ Also credential-free. The workspace must already have a user-assigned identity attached β€” that is the workspace module's identity input, not this module's concern β€” and that identity is the one needing Storage Blob Data Reader. uses_identity_based_access reads true either way; the distinction is which principal to grant.

11 Β· A credential alongside an identity β€” legal, pointless, reported
credential = {
  service_data_auth_identity = "WorkspaceSystemAssignedIdentity"
  shared_access_signature    = var.exports_container_sas
}

⚠️ credential_supplied_but_identity_selected reads true. This is accepted β€” the provider documents no conflict, so rejecting it could refuse input the API accepts β€” and the SAS has no effect, because the identity is what Azure uses to retrieve data. The cost is a pointless credential sitting in the workspace's Key Vault and in your state file, retrievable by any workspace Reader. Remove it.

πŸ’‘ This is the difference between reject and report: a caller could plausibly have meant this as a transition step, so the module says so rather than refusing.

12 Β· The `is_default` rule Terraform cannot enforce
# Apply ONE: creates the datastore.
module "training_data" {
  # ...
  is_default = false
}

# Apply TWO: promotes it. Only now is `true` accepted.
module "training_data" {
  # ...
  is_default = true
}

⚠️ The provider documents it in one line: "is_default can only be set to true on update." A first apply with true fails at Azure.

πŸ’‘ No validation {} block can catch this, because a variable validation runs against the input value alone and cannot see whether Terraform is creating or updating. It is the only rule in this module that is both published and genuinely uncheckable, which is why it is a constant output instead: is_default_cannot_be_set_to_true_at_creation.

13 Β· Rejecting the wrong storage target
# FAILS -- a FILE SHARE id. The sibling module wants this, under a different argument name.
storage_container_id = "${data.azurerm_storage_account.x.id}/fileServices/default/shares/share1"

# FAILS -- a bare STORAGE ACCOUNT id. Nothing in it says which container.
storage_container_id = data.azurerm_storage_account.x.id

# Correct.
storage_container_id = "${data.azurerm_storage_account.x.id}/blobServices/default/containers/exports"

πŸ”’ The regex is anchored at both ends. A file share lives under /fileServices/default/shares/ and a blob container under /blobServices/default/containers/ β€” one path segment apart, and they belong to two different Terraform resources. The message names the sibling module and its differently-named argument.

14 Β· Several datastores on one workspace
locals {
  datastores = {
    training_data = "training-data"
    validation    = "validation-data"
    features      = "feature-store"
  }
}

module "datastores" {
  source   = "git::https://github.com/microsoftexpert/terraform-azurerm-machine-learning-datastore-blobstorage.git?ref=v1.0.0"
  for_each = local.datastores

  name                 = each.key
  workspace_id         = module.mlw.id
  storage_container_id = "${module.data_lake.id}/blobServices/default/containers/${each.value}"
}

output "datastore_uris" {
  value = { for k, m in module.datastores : k => m.datastore_uri_prefix }
}

πŸ’‘ for_each over a keyed map, never count β€” the key is the datastore name, so adding a fourth does not re-index the first three. And because all three point into one storage account, one Storage Blob Data Reader assignment at the account scope covers them all.

⚠️ Note name is force-new and appears inside every azureml:// URI. Changing a map key here recreates the datastore and breaks every notebook referencing the old name.

15 Β· πŸ—οΈ End-to-end composition
provider "azurerm" {
  features {}
}

locals {
  platform_tags = {
    environment = "production"
    owner       = "data-platform"
  }
}

module "ml_rg" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-resource-group.git?ref=v1.0.0"

  name     = "rg-ml-eastus"
  location = "eastus"

  tags = local.platform_tags
}

module "data_lake" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-storage-account.git?ref=v1.0.0"

  name                = "stmlplatformeastus"
  resource_group_name = module.ml_rg.name
  location            = module.ml_rg.location

  account_tier             = "Standard"
  account_replication_type = "ZRS"

  containers = {
    training-data = {}
  }

  tags = local.platform_tags
}

module "mlw" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-machine-learning-workspace.git?ref=v1.0.0"

  name                = "mlw-platform-eastus"
  resource_group_name = module.ml_rg.name
  location            = module.ml_rg.location

  application_insights_id = var.platform_app_insights_id
  key_vault_id            = var.platform_key_vault_id
  storage_account_id      = module.data_lake.id

  identity = { type = "SystemAssigned" }

  tags = local.platform_tags
}

module "training_data" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-machine-learning-datastore-blobstorage.git?ref=v1.0.0"

  name = "training_data"

  # The workspace's ARM id -- NOT its workspace_id output, which is a GUID.
  workspace_id = module.mlw.id

  storage_container_id = "${module.data_lake.id}/blobServices/default/containers/training-data"

  # Omitted entirely: the module defaults to WorkspaceSystemAssignedIdentity,
  # so no key is cached in the workspace Key Vault and none reaches state.

  description = "Curated training data for the platform models."

  # tags are FORCE-NEW on this resource -- set once, and keep changeable
  # metadata on the workspace or the storage account instead.
  tags = local.platform_tags
}

# WITHOUT THIS the datastore is created cleanly and every job reading it fails.
module "mlw_reads_training_data" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-role-assignments.git?ref=v1.0.0"

  scope = module.training_data.storage_account_id

  role_assignments = {
    blob_read = {
      role_definition_name = "Storage Blob Data Reader"
      principal_id         = module.mlw.identity_principal_id
      principal_type       = "ServicePrincipal"
    }
  }
}

check "datastore_posture" {
  assert {
    condition     = module.training_data.uses_identity_based_access
    error_message = "Datastores must use identity-based access, not a cached storage credential."
  }
  assert {
    condition     = module.training_data.stores_no_credential_in_the_workspace_key_vault
    error_message = "A credential in the workspace Key Vault is retrievable by any workspace Reader."
  }
  assert {
    condition     = !module.training_data.credential_supplied_but_identity_selected
    error_message = "A credential was supplied alongside an identity. It has no effect -- remove it."
  }
}

output "training_datastore" {
  value = {
    id      = module.training_data.id
    uri     = module.training_data.datastore_uri_prefix
    summary = module.training_data.posture_summary
    account = module.training_data.storage_account_name

    # Terraform confirms the reference exists. It cannot confirm the identity can read.
    read_access_confirmed_by_terraform = false
  }
}

πŸ’‘ Four things this composition gets right. The workspace's id β€” not its workspace_id β€” is wired in. The credential is omitted entirely rather than sourced and protected. The role assignment's scope is module.training_data.storage_account_id, derived from the container ID rather than restated. And the check block asserts the posture instead of assuming it.

⚠️ Note read_access_confirmed_by_terraform = false, hard-coded. A clean apply proves the datastore reference exists. It proves nothing about whether the workspace identity can actually read the container β€” that fails later, in a job, with nothing in Terraform to point at.


πŸ“₯ Inputs

Required (3): name, workspace_id, storage_container_id β€” all force-new.

The grouped credential (1): credential, holding account_key, shared_access_signature and service_data_auth_identity. Marked sensitive. Defaults to identity-based.

Optional (4): description (force-new), is_default, tags (force-new), timeouts.

Full input schemas
variable "name" {
  type = string
  # Force-new. Appears inside every azureml:// URI referencing this datastore.
}

variable "workspace_id" {
  type = string
  # Force-new. Anchored to .../Microsoft.MachineLearningServices/workspaces/<name>.
  # REJECTS A BARE GUID -- that is the workspace's OTHER id, emitted by the workspace
  # module as `workspace_id`. This argument wants `module.<workspace>.id`.
}

variable "storage_container_id" {
  type = string
  # Force-new. Anchored to .../blobServices/default/containers/<name>.
  # Rejects a file share ID (the sibling module's target) and a bare account ID.
}

variable "credential" {
  type = object({
    account_key                = optional(string)
    shared_access_signature    = optional(string)
    service_data_auth_identity = optional(string) # None | WorkspaceSystemAssignedIdentity | WorkspaceUserAssignedIdentity
  })

  default = {
    service_data_auth_identity = "WorkspaceSystemAssignedIdentity"
  }

  sensitive = true

  # TWO documented rules, both enforceable only because the fields share a variable:
  #   1. account_key CONFLICTS WITH shared_access_signature.
  #   2. identity "None" (or omitted) REQUIRES one of the two.
  # THE PROVIDER DEFAULTS THE IDENTITY TO "None" (credential-based). This module does not.
}

variable "description" {
  type    = string
  default = null
  # FORCE-NEW, unusually for a description.
}

variable "is_default" {
  type    = bool
  default = false
  # CANNOT BE `true` AT CREATION -- provider rule, and unenforceable in a validation.
  # Settable here; READ-ONLY on both sibling datastore resources.
}

variable "tags" {
  type    = map(string)
  default = {}
  # FORCE-NEW. A retagging sweep recreates the datastore.
}

variable "timeouts" {
  type = object({
    create = optional(string)
    read   = optional(string)
    update = optional(string)
    delete = optional(string)
  })
  default = null
  # Provider defaults: 30m / 5m / 30m / 30m.
}

⚠️ Object-type conversion silently discards extra keys. A misspelled key inside credential or timeouts is dropped with no error at all β€” so a typo'd service_data_identity (the sibling's spelling) would be silently ignored here, leaving the provider's None default in force. Check service_data_auth_identity_effective against what you intended.


🧾 Outputs

Output Description
id The datastore's Resource ID. Note dataStores, capital S.
name The datastore name, as it appears in every azureml:// URI.
datastore_uri_prefix πŸ”‘ The azureml:// prefix workloads use. Assembled from four values; in state nowhere.
workspace_id / workspace_name / workspace_resource_group / workspace_subscription_id The parent, echoed and decomposed.
storage_container_id / storage_container_name The container referenced.
storage_account_id Derived by truncating the container ID β€” a ready-made role-assignment scope.
storage_account_name / storage_account_resource_group Which account this actually points at.
authentication_mode Identity-based or Credential-based, in Microsoft's own words.
service_data_auth_identity_effective The identity in force, with the provider's None default applied.
uses_identity_based_access Derived β€” true on an empty call, because of the flip.
stores_no_credential_in_the_workspace_key_vault The goal, stated positively.
uses_account_key / uses_sas_token Which kind, if any. An account key controls the whole account.
supplied_credential_count πŸ”’ A count. Never a value, never a field name.
credential_supplied_but_identity_selected Legal, pointless, reported.
is_default Settable here; read-only on both siblings.
posture_summary The configuration as one reviewable line.
identity_based_access_needs_a_role_assignment Constant. Failure-timing table with two Never rows.
credential_based_access_is_readable_by_workspace_readers Constant. The RBAC asymmetry.
is_default_cannot_be_set_to_true_at_creation Constant. Published and uncheckable.
a_datastore_is_a_reference_and_deleting_it_deletes_no_data Constant. And what destroying it does break.
tags_and_description_are_force_new_on_this_resource Constant.
the_arm_type_is_a_child_of_the_workspace The ARM type string.
secure_by_default_applies_here_and_this_is_how Constant. The flip, and why it was safe to make.

No credential is emitted in any form, including a field name.


🧠 Architecture Notes

A datastore is a reference, and the only real decision is where the key lives. Microsoft offers two mechanisms. Credential-based access caches an account key or SAS in the workspace's Key Vault β€” and Microsoft names the consequence as a limitation: "users with Reader workspace access can access the credentials." Identity-based access stores nothing; "you don't save any authentication credentials. You only store the storage account information." The provider defaults to the first. This module defaults to the second, and uses_identity_based_access reads true on an empty call so the divergence is visible rather than surprising.

The flip was checked against the at-least-one-of rule before being made, and that check is the interesting part. A default that silently satisfies an at-least-one-of requirement removes a legal shape β€” which is exactly why a comparable flip was declined on another resource in this suite. Here it does not: a caller wanting credential-based access still writes service_data_auth_identity = "None" alongside a key or SAS, and nothing becomes unreachable. The flip only changes which shape you get for free.

Two documented rules span three fields, so the three fields are one variable. A validation {} block may reference only its own variable, so grouping account_key, shared_access_signature and service_data_auth_identity is what makes both rules parse-time errors: the conflict between the two credentials, and the requirement that one of them exist when the identity is None. Note the second is a conditional requirement rather than a plain exclusion β€” the credential-free shape is valid only when an identity is selected β€” which is a case grouping handles as neatly as a simple XOR.

Sensitivity is handled at the variable, because that is the only place it can be. sensitive applies to a variable and not to an attribute, so the whole credential object carries the mark even though service_data_auth_identity is not a secret. The consequence is contagion: try(var.credential.account_key, null) != null is itself a sensitive bool that Terraform refuses to emit, so every derived fact is unwrapped with nonsensitive() β€” which doubles as a guard, since it errors if its argument was not sensitive. What is deliberately not unwrapped is any field name: account_key versus shared_access_signature describe very different blast radii, so the kind of credential is itself disclosure. The count is what a review needs.

The most consequential thing this module cannot check is a role assignment. Identity-based access needs the workspace's managed identity to hold Storage Blob Data Reader on the storage account β€” except on the workspace's own default account, where Microsoft says it is unnecessary. Without it, the datastore is created cleanly, the plan is clean, and a job fails days later with an authorization error against storage. So identity_based_access_needs_a_role_assignment is a constant with a failure-timing table whose decisive rows read Never, and storage_account_id is derived from the container ID specifically so the composition can wire the grant without restating the account.

That derivation is a place to be careful, because getting it wrong is silent. The account name sits five segments back from the end of a container ID, not four β€” an off-by-one yields blobServices as the "storage account name", which validates, formats and reads plausibly in a summary string. The index arithmetic is commented with the segment count for that reason.

datastore_uri_prefix exists because the value that gets used is not in the resource's state. A notebook references azureml://subscriptions/…/resourcegroups/…/workspaces/…/datastores/<name>/paths/…, assembled from four separate values β€” and note resourcegroups is lower case and unhyphenated there, where an ARM Resource ID uses resourceGroups. That inconsistency is Azure's, and it produces URIs that look right and resolve to nothing, so the module builds the prefix rather than leaving it to be retyped.

Two force-new fields are surprising enough to be a constant. tags and description both force replacement on this resource, where almost every other resource in this library updates them in place. A subscription-wide retagging sweep therefore destroys and recreates every datastore it touches. What is updatable in place: the credentials, the identity and is_default β€” so rotating a credential is cheap and renaming is not.

One published rule is genuinely uncheckable, and that is stated rather than worked around. is_default can only be set to true on update. A variable validation runs against an input value with no knowledge of prior state, so it cannot see create-versus-update. The module reports the rule instead of pretending to enforce it.

The three sibling datastores are not interchangeable, and the differences were re-derived rather than copied. Two spellings of the identity argument (service_data_auth_identity here, service_data_identity on both siblings); two lifecycles for is_default (settable here, computed there); two authentication mechanisms (key or SAS here and on fileshare, a service principal on Data Lake Gen2); two storage-target arguments (storage_container_id here and on Data Lake Gen2, storage_fileshare_id on fileshare); and one rule the provider documents on this resource only. Reading one member of the trio and generalising would have produced three subtly wrong modules.

for_each, never count where several datastores share a workspace β€” and because they usually share a storage account too, one account-scoped role assignment covers all of them.


🧱 Design Principles

Concern This module's default (minimum call) Opt-out (you must type it)
Authentication identity-based β€” WorkspaceSystemAssignedIdentity, storing no credential service_data_auth_identity = "None" plus a key or SAS
Credential in state none accepted supply one; it lands in state and the workspace Key Vault
Credential disclosure presence and count only no opt-out, by design
Default datastore false β€” also the only value accepted at creation true, on a second apply
Storage lifecycle references only β€” creates and deletes nothing none; a datastore cannot own storage

πŸ”’ Where secure-by-default applies here, and how. Three arguments are required, so there is no empty call in the strict sense β€” but the one security-relevant default could be flipped and was: the provider's None caches a storage key behind a workspace Reader role; this module's WorkspaceSystemAssignedIdentity stores nothing. The flip was verified not to remove a legal shape first. The compensations for what cannot be defaulted β€” closed value sets, the restrictive fact named in each error message, and every unverifiable decision emitted β€” are enumerated in secure_by_default_applies_here_and_this_is_how.


πŸš€ Runbook

terraform init -backend=false
terraform validate
terraform fmt -check

Pin by tag β€” ?ref=v1.0.0 β€” never a branch. This library is plan-only: a human applies from CI.

terraform plan -out=tfplan
terraform show -json tfplan | jq '.planned_values.outputs'

⚠️ Two sequencing notes. To make a datastore the default, apply once with is_default = false and again with true. And confirm the workspace identity's Storage Blob Data Reader assignment exists before anyone runs a job β€” a clean apply here says nothing about read access.

⚠️ Never let a retagging sweep reach this resource. tags is force-new; the sweep will recreate every datastore and break every azureml:// reference mid-flight.


πŸ§ͺ Testing

The offline gate is init -backend=false β†’ validate β†’ fmt -check.

What only a root-module terraform console proves: that each validation {} block fires. terraform validate on a calling configuration does not run root-variable validations. Feed it deliberately bad .tfvars and read the line numbers, not the message text β€” Terraform hard-wraps messages, so grepping for a phrase silently matches nothing.

echo 'var.name' | terraform console -var-file=bad.tfvars

⚠️ Suppression is per-variable, so a short error list is not proof a check is missing. And proving the two credential rules needs two fixtures pulling in opposite directions: one setting both credentials, and one setting None with neither.

What only a real apply exercises: whether the workspace and container exist, whether the workspace identity holds Storage Blob Data Reader, whether a SAS has expired, whether soft delete blocks identity passthrough, and whether is_default = true is a create or an update. None of that is visible offline.


πŸ’¬ Example Output

id                                              = "/subscriptions/.../resourceGroups/rg-ml-eastus/providers/Microsoft.MachineLearningServices/workspaces/mlw-platform-eastus/dataStores/training_data"
name                                            = "training_data"
datastore_uri_prefix                            = "azureml://subscriptions/.../resourcegroups/rg-ml-eastus/workspaces/mlw-platform-eastus/datastores/training_data/paths/"
workspace_name                                  = "mlw-platform-eastus"
storage_account_name                            = "stmlplatformeastus"
storage_account_id                              = "/subscriptions/.../resourceGroups/rg-ml-eastus/providers/Microsoft.Storage/storageAccounts/stmlplatformeastus"
storage_container_name                          = "training-data"
authentication_mode                             = "Identity-based"
service_data_auth_identity_effective            = "WorkspaceSystemAssignedIdentity"
uses_identity_based_access                      = true
stores_no_credential_in_the_workspace_key_vault = true
uses_account_key                                = false
uses_sas_token                                  = false
supplied_credential_count                       = 0
credential_supplied_but_identity_selected        = false
is_default                                      = false
posture_summary                                 = "Identity-based (WorkspaceSystemAssignedIdentity) / container training-data on stmlplatformeastus"

ℹ️ supplied_credential_count = 0 alongside uses_identity_based_access = true is the shape to aim for: nothing in the workspace Key Vault, nothing in state, nothing to rotate.

ℹ️ Note the URI's lower-case resourcegroups against the Resource ID's resourceGroups. Both are correct for their own format, which is exactly why the prefix is emitted rather than hand-built.


πŸ” Troubleshooting

Symptom Cause Fix
workspace_id is a bare GUID… module.mlw.workspace_id was passed. The correctly-named output is the wrong one. Pass module.mlw.id.
Workspace ID rejected as not a workspace path A resource group, storage account or child resource ID. Pass the workspace's own ARM ID; the pattern is anchored at both ends.
storage_container_id is a FILE SHARE ID… A /fileServices/default/shares/ path. Use the fileshare datastore module; its argument is storage_fileshare_id.
Container ID rejected as having no container segment A bare storage account ID. Append /blobServices/default/containers/<name>.
credential may set account_key OR shared_access_signature… Both supplied. The provider documents them as conflicting. Choose one β€” or neither, and use an identity.
credential.service_data_auth_identity is "None" … and neither account_key nor shared_access_signature was supplied None is a commitment to a credential. Supply one, or set WorkspaceSystemAssignedIdentity.
shared_access_signature begins with "?" Copied from the portal's SAS token field. Strip the leading ?; start at sv=.
shared_access_signature is a full URL Copied from the portal's Blob SAS URL field. Pass only the part after the ?.
Identity value rejected as a different vocabulary SystemAssigned / UserAssigned β€” the ARM identity {} block's words. Use the Workspace-prefixed values.
Identity value rejected for case ARM is case-sensitive here. Exact camel case.
Apply fails: is_default cannot be set The provider allows true only on update. Uncheckable in a validation. Apply with false, then change to true and apply again.
Datastore created, but a job cannot read the data The workspace identity has no Storage Blob Data Reader on the account. No Terraform error, ever. Grant it; scope with this module's storage_account_id.
As above, and the account is behind a private endpoint The workspace identity also needs Reader on the private endpoint. Grant it on each endpoint in use (blob, file, dfs).
Identity passthrough fails on a credential-less blob datastore Soft delete is enabled on the storage account β€” a documented constraint. Disable it, or use a credential. Invisible from here.
A datastore that worked stops working, with no plan change A SAS expired. Nothing in Terraform tracks the expiry. Rotate the SAS β€” or move to identity-based access and stop rotating.
Someone with only workspace Reader obtained the storage key Expected, on a credential-based datastore. Microsoft documents it. Move to identity-based access; review who holds workspace Reader.
A retagging sweep destroyed every datastore tags is force-new on this resource. Keep changeable metadata on the workspace or storage account. No data was lost.
Notebooks broke after a rename name is force-new and embedded in every azureml:// URI. Recreate under the old name, or update the references.
A credential key was ignored Object-type conversion silently discards extra keys β€” likely service_data_identity, the sibling's spelling. Check service_data_auth_identity_effective.
terraform destroy β€” is the data gone? No. A datastore is a reference. Nothing to recover. But every azureml:// reference to it now resolves to nothing.

πŸ”— Related Docs


πŸ’™ "Infrastructure as Code should be standardized, consistent, and secure."