☁️ Google Cloud Vertex AI Dataset Terraform Module

Creates a single Vertex AI managed Dataset (google_vertex_ai_dataset) — a collection of DataItems and Annotations used as training/evaluation data for a custom ML model or AutoML pipeline. Targets hashicorp/google ~> 7.0, Terraform >= 1.12.0. First module in this catalog's data-ml (Vertex AI) domain.

• 🧠 Creates one google_vertex_ai_dataset — a managed container for DataItems and Annotations whose metadata_schema_uri determines the dataset's TYPE (image/text/tabular/video classification, object detection, and so on).
• 🔑 True standalone module — a Dataset's entity-level DataItems/Annotations are managed inside the Vertex AI API/console itself, never as Terraform resources, so there is no for_each-managed child collection here.
• 🔒 Optional customer-managed encryption (CMEK) via encryption_spec.kms_key_name, sourced from a terraform-google-kms-keyring crypto key id — never defaulted to a specific key.
• 🛡️ deletion_policy defaults to "PREVENT" — a documented extension to this module suite's secure-by-default table (see 🧱 Design Principles).
• 🌎 region is optional (Vertex AI region availability is narrower than general Compute Engine — see ✅ Provider/Versions).
• 🚫 No self_link — this resource exports only id and name, and the two are the same value (see 🧾 Outputs and 🧠 Architecture Notes).

💡 Why it matters: a Dataset is the first artifact in every Vertex AI training workflow — getting its type (metadata_schema_uri) and its encryption posture right at creation time matters disproportionately, because neither is a practical in-place change once real DataItems exist.

If these Terraform modules have been helpful to you or your organization, I'd appreciate your support in any of the following ways:

• ⭐ Star this repository to help others discover this Terraform module.
• 🤝 Connect with me on LinkedIn: linkedin.com/in/microsoftexpert
• ☕ Buy me a coffee: buymeacoffee.com/microsoftexpert

Whether it's a star, a professional connection, or a coffee, every gesture helps keep these modules actively maintained and continually improving. Thank you for being part of the community!

flowchart LR
 classDef thisModule fill:#174EA6,stroke:#174EA6,color:#ffffff,stroke-width:2px;
 classDef neutral fill:#F1F3F4,stroke:#9AA0A6,color:#202124,stroke-width:1px;

 subgraph dataml["data-ml domain (new)"]
 VDS["terraform-google-vertex-ai-dataset"]:::thisModule
 end

 subgraph security["Security domain"]
 KMS["terraform-google-kms-keyring"]:::neutral
 end

 subgraph storage["Storage domain"]
 BKT["terraform-google-storage-bucket"]:::neutral
 end

 subgraph networking["Networking domain"]
 VPC["terraform-google-vpc-network"]:::neutral
 end

 subgraph compute["Compute domain"]
 GKE["terraform-google-gke-cluster"]:::neutral
 end

 subgraph data["Data domain"]
 BQ["terraform-google-bigquery-dataset"]:::neutral
 end

 subgraph iamdom["IAM domain"]
 SA["terraform-google-service-account"]:::neutral
 end

 subgraph observability["Observability domain"]
 LS["terraform-google-log-sink"]:::neutral
 end

 KMS -. "optional CMEK: crypto_key_ids feeds encryption_spec.kms_key_name".-> VDS
 BKT -. "optional: custom metadata_schema_uri GCS object".-> VDS

This module founds the data-ml domain — there is no existing keystone/target sibling in this domain yet (terraform-google-vertex-ai-featurestore and terraform-google-vertex-ai-endpoint are authored alongside it in this same batch), so this module's own node is colored #174EA6 rather than the usual #4285F4 "this module" / #174EA6 "keystone sibling" pairing — there is no sibling to receive that second color yet. Both inbound edges are optional: terraform-google-kms-keyring only matters when a caller opts into CMEK; terraform-google-storage-bucket only matters when a caller publishes a genuinely custom metadata_schema_uri (most callers reference one of Google's own published schema catalog entries instead — see Example Library). No outbound consumer edge is shown from this module's outputs, since none exists in this library yet.

flowchart TD
 classDef thisModule fill:#174EA6,stroke:#174EA6,color:#ffffff,stroke-width:2px;
 classDef neutral fill:#F1F3F4,stroke:#9AA0A6,color:#202124,stroke-width:1px;

 IN["Inputs: display_name, metadata_schema_uri,<br/>region, encryption_spec,<br/>deletion_policy, labels, timeouts"]:::neutral
 DS["google_vertex_ai_dataset.this"]:::thisModule
 OUT["Outputs: id, name, display_name,<br/>create_time, update_time"]:::neutral

 IN -- "var.*" --> DS
 DS -- "id, name, display_name, create_time, update_time" --> OUT

Resource inventory: google_vertex_ai_dataset.this — exactly one resource, always created. No other resources, no for_each-managed children.

Schema notes that bite (verified against hashicorp/google v7.39.0 via the live provider schema, providerDocID 12684204, cross-checked against terraform providers schema -json):

• No self_link on this resource at all. The Attributes Reference lists only id, name, create_time, update_time, terraform_labels, effective_labels. id has the format {{name}} — id and name are the same value for this resource.
• metadata_schema_uri is effectively force-new, but neither available source can confirm a ForceNew flag. The live docs do not mark it "Immutable" (unusual for a force-new field in this provider), and terraform providers schema -json does not expose a ForceNew indicator for any attribute of any resource in this provider at all (confirmed by inspecting the full schema export — zero force_new occurrences anywhere in the file). Operationally, changing a Dataset's type post-creation is not a supported Vertex AI API operation — Terraform may show an in-place "update" at plan time that the API then rejects at apply time. See Troubleshooting.
• encryption_spec's doc text is a copy-paste artifact: "kms_key_name - (Optional) Required." The block itself is optional; kms_key_name is required once the block is used. Modeled here as a required string inside the encryption_spec object type.
• region has no companion location field on this resource (unlike google_vertex_ai_endpoint) — region is the only placement argument, and it is optional + computed (the provider/API resolves a value if omitted).
• Vertex AI region availability is narrower than general Compute Engine. Do not assume every Compute Engine region supports every Vertex AI feature — check https://cloud.google.com/vertex-ai/docs/general/locations before setting region.
• deletion_policy (DELETE/ABANDON/PREVENT) defaults to "DELETE" in the provider — this module overrides that default to "PREVENT" (see 🧱 Design Principles).
• All three create/update/delete timeouts are supported, each defaulting to 20 minutes in the provider.
• No import support — the provider docs state this resource does not support terraform import.

• roles/aiplatform.user on the target project — create/manage Datasets and most other Vertex AI resources. Default, least-privilege role for the applying principal.
• roles/aiplatform.admin only if the same applying principal must also set IAM policy directly on Vertex AI resources — do not default to admin.

• aiplatform.googleapis.com enabled on the target project (via terraform-google-project-services, applied before this module per the house authoring order).
• cloudkms.googleapis.com additionally required only when var.encryption_spec is set.
• Vertex AI region availability is narrower than general Compute Engine — verify the target region against https://cloud.google.com/vertex-ai/docs/general/locations before applying.
• If encryption_spec.kms_key_name is set, the Vertex AI service agent (service-<PROJECT_NUMBER>@gcp-sa-aiplatform.iam.gserviceaccount.com) must independently hold roles/cloudkms.cryptoKeyEncrypterDecrypter on the referenced key — this module does not, and cannot, grant that binding itself.

terraform-google-vertex-ai-dataset/
├── providers.tf # required_providers (hashicorp/google ~> 7.0) + required_version — no provider {} block
├── variables.tf # display_name, metadata_schema_uri, region, encryption_spec, deletion_policy, labels, timeouts
├── main.tf # google_vertex_ai_dataset.this — single keystone, no for_each children
├── outputs.tf # id, name, display_name, create_time, update_time — no self_link
├── README.md # this file
├── SCOPE.md # lightweight cross-module contract
└── examples/ # runnable example matching the Quick Start below

module "image_training_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "image-classification-training"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
  region              = "us-central1"
}

The caller's root module configures the google provider (project, region/zone, and authentication via ADC, Workload Identity Federation, or a service account key supplied out-of-band) — this module accepts none of those as variables (region here is this resource's own optional per-dataset argument, not a provider-level override).

Consumes

Emits

module "image_training_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "image-classification-training"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
  region              = "us-central1"
}

module "customer_churn_tabular_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "customer-churn-tabular"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml"
  region              = "us-central1"
}

module "support_ticket_text_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "support-ticket-classification"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/text_1.0.0.yaml"
  region              = "us-central1"
}

module "field_inspection_video_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "field-inspection-action-recognition"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/video_action_recognition_1.0.0.yaml"
  region              = "us-central1"
}

module "equipment_defect_detection_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "equipment-defect-object-detection"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_object_detection_1.0.0.yaml"
  region              = "us-central1"
}

module "cmek_training_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "prod-cmek-training-dataset"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
  region              = "us-central1"

  encryption_spec = {
    kms_key_name = module.kms_keyring.crypto_key_ids["vertex-ai-encryption-key"]
  }
}

module "custom_schema_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "custom-anomaly-schema-dataset"
  metadata_schema_uri = "gs://${module.custom_schema_bucket.name}/schemas/custom_anomaly_1.0.0.yaml"
  region              = "us-central1"
}

module "scratch_test_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "scratch-integration-test-dataset"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
  region              = "us-central1"
  deletion_policy     = "DELETE"
}

module "externally_managed_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "handed-off-to-ml-team-dataset"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml"
  region              = "us-central1"
  deletion_policy     = "ABANDON"
}

module "labeled_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "labeled-training-dataset"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
  region              = "us-central1"

  labels = {
    environment = "prod"
    domain      = "data-ml"
    owner       = "ml-platform-team"
  }
}

module "slow_region_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "slow-provisioning-region-dataset"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/video_classification_1.0.0.yaml"
  region              = "asia-southeast1"

  timeouts = {
    create = "30m"
    delete = "30m"
  }
}

module "default_region_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "default-region-training-dataset"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
}

locals {
  training_datasets = {
    "image-classification-v1" = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
    "tabular-churn-v1"        = "gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml"
  }
}

module "training_datasets" {
  source   = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"
  for_each = local.training_datasets

  display_name        = each.key
  metadata_schema_uri = each.value
  region              = "us-central1"
}

module "shared_reference_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "cross-team-reference-dataset"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
  region              = "us-central1"
}

resource "google_vertex_ai_dataset_iam_member" "ml_team_user" {
  dataset = module.shared_reference_dataset.id
  role    = "roles/aiplatform.user"
  member  = "group:ml-platform-team@financialpartners.com"
}

module "project_services" {
  source = "git::https://github.com/microsoftexpert/terraform-google-project-services.git?ref=v1.0.0"

  # Enables aiplatform.googleapis.com and cloudkms.googleapis.com for this composition.
}

module "kms_keyring" {
  source = "git::https://github.com/microsoftexpert/terraform-google-kms-keyring.git?ref=v1.0.0"

  key_ring_name = "casey-prod-vertex-ai-keyring"
  location      = "us-central1"

  crypto_keys = {
    "vertex-ai-encryption-key" = {
      purpose = "ENCRYPT_DECRYPT"
    }
  }

  depends_on = [module.project_services]
}

data "google_project" "current" {}

resource "google_kms_crypto_key_iam_member" "vertex_ai_encrypter" {
  crypto_key_id = module.kms_keyring.crypto_key_ids["vertex-ai-encryption-key"]
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
  member        = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-aiplatform.iam.gserviceaccount.com"
}

module "image_training_dataset" {
  source = "git::https://github.com/microsoftexpert/terraform-google-vertex-ai-dataset.git?ref=v1.0.0"

  display_name        = "prod-image-classification-training"
  metadata_schema_uri = "gs://google-cloud-aiplatform/schema/dataset/metadata/image_1.0.0.yaml"
  region              = "us-central1"

  encryption_spec = {
    kms_key_name = module.kms_keyring.crypto_key_ids["vertex-ai-encryption-key"]
  }

  labels = {
    environment = "prod"
    domain      = "data-ml"
  }

  depends_on = [google_kms_crypto_key_iam_member.vertex_ai_encrypter]
}

variable "display_name" {
  type = string
  # 1-128 chars enforced via validation {}
}

variable "metadata_schema_uri" {
  type = string
  # gs:// prefix enforced via validation {}
}

variable "region" {
  type    = string
  default = null
}

variable "encryption_spec" {
  type = object({
    kms_key_name = string
  })
  default = null
}

variable "deletion_policy" {
  type    = string
  default = "PREVENT"
  # One of DELETE/ABANDON/PREVENT enforced via validation {}
}

variable "labels" {
  type    = map(string)
  default = {}
  # GCP label key/value format + <= 64 entries enforced via validation {}
}

variable "timeouts" {
  type = object({
    create = optional(string)
    update = optional(string)
    delete = optional(string)
  })
  default = null
}

ℹ️ This resource has no self_link. id and name are numerically identical — both are still emitted, in that order, per this module suite's outputs-ordering convention, not because they carry different values.

• metadata_schema_uri determines the Dataset's TYPE and is effectively force-new, even though neither the live provider docs nor terraform providers schema -json mark it ForceNew (the schema-JSON format does not expose a ForceNew indicator for any attribute on any resource in this provider at all — confirmed by inspecting the full export). Changing a Dataset's type post-creation is not a supported Vertex AI API operation; Terraform may show an in-place "update" at plan time that the API then rejects at apply time. See Troubleshooting.
• id and name are the same value for this resource. id has the format {{name}}. Both are emitted per house convention (id first, then name), not because they differ — see Outputs.
• encryption_spec.kms_key_name's Cloud KMS key must be in the same region as this Dataset's region. The provider enforces this at apply time only; it cannot be checked at plan time since both values are plain strings to the type system. See Troubleshooting.
• region is a genuine per-resource argument, not an invented provider override. This resource has no location field (unlike google_vertex_ai_endpoint); region is the only placement argument the provider itself defines on google_vertex_ai_dataset. This module suite's convention against inventing project/region/zone module variables refers to modules manufacturing an override pattern the resource schema does not define — that is not the case here.
• No project variable. Per this module suite's authentication-model convention, project is a provider/caller concern; this module does not expose the resource's own optional project override as a variable, matching the house pattern used across this library's other standalone modules.
• labels is non-authoritative. This module only manages the labels present in its configuration; the resource's own effective_labels/terraform_labels computed attributes carry the full merged view, not exposed as an output in v1.0.0.
• A false lead avoided: an early lookup for this resource surfaced a dataset_id (Required) argument. That argument does not exist on google_vertex_ai_dataset — both the live provider schema and the schema-JSON ground truth confirm the real argument list is exactly display_name / metadata_schema_uri / labels / encryption_spec / region / project / deletion_policy. The dataset_id hit was noise, most likely from google_bigquery_dataset (which does take a required dataset_id), surfaced by the shared word "dataset."

cd terraform-google-vertex-ai-dataset
terraform init -backend=false
terraform validate
terraform fmt -check

Pin the module source to ?ref=v1.0.0 — never a branch. This library is plan-only from an authoring session; a human applies from CI with valid Workload Identity Federation or ADC credentials.

terraform validate confirms internal type/reference consistency — including the closed deletion_policy enum and the gs://-prefix format check on metadata_schema_uri. terraform fmt -check confirms canonical formatting. Neither can catch GCP API-level rejections: a metadata_schema_uri that is a syntactically valid gs:// string but not an actual, readable Vertex AI schema file will pass validate cleanly and fail only at apply time; likewise, a target region that Vertex AI does not support in a particular sub-feature, or an encryption_spec.kms_key_name in a different region than the Dataset, both pass plan and fail only at apply. Only a real terraform plan/apply against a live project, with valid credentials, exercises those paths — and that step belongs to the consuming CI pipeline, not this authoring session.

$ terraform output

id = "projects/casey-prod-ml/locations/us-central1/datasets/1234567890123456789"
name = "projects/casey-prod-ml/locations/us-central1/datasets/1234567890123456789"
display_name = "prod-image-classification-training"
create_time = "2026-07-12T14:03:22.123456Z"
update_time = "2026-07-12T14:03:22.123456Z"

• google_vertex_ai_dataset provider resource reference
• Vertex AI locations
• Vertex AI Dataset schema catalog (gs://google-cloud-aiplatform/schema/dataset/metadata/)
• terraform-google-kms-keyring (optional CMEK source, crypto key id output)
• terraform-google-storage-bucket (optional custom schema YAML source)
• terraform-google-project-services (must enable aiplatform.googleapis.com before this module applies)
• terraform-google-vertex-ai-featurestore, terraform-google-vertex-ai-endpoint (sibling modules founding this same data-ml domain in this authoring batch)
• This module's SCOPE.md