Skip to content

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

☁️ Azure Digital Twins Time Series Database Connection Terraform Module

The data-history connection (azurerm_digital_twins_time_series_database_connection): twin property updates flow through an Event Hub into an Azure Data Explorer table. The one member of this family that holds no secrets at all. Targets hashicorp/azurerm ~> 4.0.

Terraform Provider Module Type Resources

🧩 Overview

  • 🕰️ Declares the data-history connection for a Digital Twins instance, as a keystone resource named this. A twin holds only its current property values — without this, there is no record of what a property was an hour ago.
  • 🔐 The identity-based member of this family, and the contrast is the point. It accepts no keys and no connection strings: nothing secret passes through configuration, plan output, or state. Authentication is the instance's own system-assigned managed identity.
  • 🎫 So the security work moves from handling credentials to granting roles — including two grants that should be reduced once setup is done, which is the step most often skipped.
  • 🧊 Every field is force-new, including the two optional ones. There is no update operation, so the module's timeouts object carries no update.
  • 🚧 Documents the ordering trap up front: the Kusto cluster needs public network access enabled while the connection is created, because Digital Twins reaches in to build the table.
  • 📐 Validates the two URI inputs, which are both easy to get backwards and fail only at apply.

💡 Why it matters: This module is cheap to write and easy to get wrong in ways Terraform cannot see. Four of its eight inputs describe two resources twice — once as a Resource ID, once as a URI — and nothing checks that the pairs agree. One of those URIs must use sb:// rather than https://. And the connection will not create at all if the target cluster is locked down, which is the opposite of the order most teams would naturally do things in.

❤️ Support this project

If this module saves you time, please consider supporting its continued development:


🗺️ Where this fits in the family

flowchart LR
  rg["terraform-azurerm-resource-group"]
  uai["terraform-azurerm-user-assigned-identity: optional, via identity_ids"]
  dt["terraform-azurerm-digital-twins-instance: the family keystone, owns the system-assigned identity"]
  eg["terraform-azurerm-digital-twins-endpoint-eventgrid: KEY-BASED, no identity option EXISTS"]
  eh["terraform-azurerm-digital-twins-endpoint-eventhub: KEY-BASED, provider gap only"]
  sb["terraform-azurerm-digital-twins-endpoint-servicebus: KEY-BASED, provider gap only, TOPIC not queue"]
  ts["terraform-azurerm-digital-twins-time-series-database-connection: IDENTITY-BASED, no secrets at all"]
  egt["terraform-azurerm-eventgrid-topic: keys read via a data source, the module does not emit them"]
  ehar["terraform-azurerm-eventhub-authorization-rule: SEND only"]
  sbar["terraform-azurerm-servicebus-topic-authorization-rule: SEND only"]
  ehns["terraform-azurerm-eventhub-namespace: the hop for data history"]
  kc["terraform-azurerm-kusto-cluster"]
  kd["terraform-azurerm-kusto-database: must already exist, the table is created FOR you"]
  sa["terraform-azurerm-storage-account: dead-letter container plus a SAS token"]
  ra["terraform-azurerm-role-assignments: grants the DT identity its roles, reducible after setup"]
  routes["event routes in the twin graph: DATA PLANE, name the endpoint, not modelled by Terraform"]

  rg -->|"resource_group_name, location"| dt
  uai -->|"identity_ids"| dt
  dt -->|"id, BY ID"| eg
  dt -->|"id, BY ID"| eh
  dt -->|"id, BY ID"| sb
  dt -->|"id, BY ID"| ts
  egt -->|"endpoint URL plus TWO access keys"| eg
  ehar -->|"two connection strings"| eh
  sbar -->|"two connection strings"| sb
  ehns -->|"id, name, and an sb:// URI you COMPOSE"| ts
  kc -->|"id plus uri"| ts
  kd -->|"name"| ts
  sa -->|"dead_letter_storage_secret"| eg
  sa -->|"dead_letter_storage_secret"| eh
  sa -->|"dead_letter_storage_secret"| sb
  dt -->|"identity_principal_id"| ra
  ra -->|"Event Hubs Data Owner, Kusto Contributor, database Admin"| ts
  eg -->|"name"| routes
  eh -->|"name"| routes
  sb -->|"name"| routes

  classDef me fill:#0078D4,stroke:#004578,color:#fff;
  classDef keystone fill:#004578,stroke:#001f3f,color:#fff;
  classDef sib fill:#eef2f7,stroke:#b8c4d0,color:#1b1b1b;
  class dt keystone;
  class eg,eh,sb,ts me;
  class rg,uai,egt,ehar,sbar,ehns,kc,kd,sa,ra,routes sib;
Loading

🧬 What this module builds

flowchart TB
  what["WHAT IT BUYS: a twin holds only its CURRENT property values. This records the history."]
  ident["THE IDENTITY-BASED MEMBER OF THIS FAMILY: no keys, no connection strings, nothing secret in config, plan output, or state"]
  needs["so the instance needs a SYSTEM-ASSIGNED managed identity, enabled on the instance module, before this exists"]
  setup["setup grants to that identity: Event Hubs Data Owner, Kusto Contributor, and a database principal assignment with role Admin"]
  reduce["AFTER SETUP reduce to just Event Hubs Data Sender: the least-privilege end state, and easy to forget"]
  pna["ORDERING TRAP: the Kusto cluster must have PUBLIC NETWORK ACCESS ENABLED during creation, because DT reaches in to create the table. Restrict the network AFTERWARDS."]
  fail["with it disabled, setup fails with an internal-server-error that does not name the cause"]
  fn["EVERY field is FORCE-NEW, including the two optional ones, so there is no update timeout at all"]
  dbl["the namespace and the cluster are each supplied TWICE: a Resource ID and a URI. Nothing checks they agree."]
  sburi["eventhub_namespace_endpoint_uri must use sb://, and no module output exists so you COMPOSE it from the name"]
  kuri["kusto_cluster_uri is the cluster's uri output, not data_ingestion_uri and not the Resource ID"]
  db["the DATABASE must already exist. DT creates the TABLE inside it, plus its ingestion mapping."]
  tbl["changing kusto_table_name replaces the connection and LEAVES THE OLD TABLE behind holding the history so far"]
  subset["the provider exposes a SUBSET of the ARM API: no twin-lifecycle or relationship-lifecycle tables, no removal recording, no user-assigned identity"]
  this["terraform-azurerm-digital-twins-time-series-database-connection"]
  res["azurerm_digital_twins_time_series_database_connection.this"]
  out["outputs: id, name, plus the destination in the three pieces a Kusto query needs. No sensitive outputs, because there are no secrets."]

  what -->|"read this first"| this
  ident -->|"the contrast with the endpoints"| this
  needs -->|"prerequisite"| ident
  setup -->|"the real work"| needs
  reduce -->|"do this after"| setup
  pna -->|"before you start"| this
  fail -->|"the symptom"| pna
  fn -->|"set-once"| this
  dbl -->|"redundancy"| this
  sburi -->|"easy to get wrong"| dbl
  kuri -->|"easy to get wrong"| dbl
  db -->|"prerequisite"| this
  tbl -->|"replacement cost"| db
  subset -->|"what Terraform cannot express"| this
  this -->|"creates"| res
  res -->|"emits"| out

  classDef me fill:#0078D4,stroke:#004578,color:#fff;
  classDef keystone fill:#004578,stroke:#001f3f,color:#fff;
  classDef sib fill:#eef2f7,stroke:#b8c4d0,color:#1b1b1b;
  class this me;
  class res keystone;
  class what,ident,needs,setup,reduce,pna,fail,fn,dbl,sburi,kuri,db,tbl,subset,out sib;
Loading

Resource inventory

Resource Count Notes
azurerm_digital_twins_time_series_database_connection.this 1 The keystone. A child of the Digital Twins instance; supports no tags, and every field is force-new.

✅ Provider / Versions

Requirement Value
Terraform >= 1.12.0
hashicorp/azurerm ~> 4.0
Provider block None in this module. The caller configures provider "azurerm", including the mandatory features {} block, and supplies authentication.

Schema notes that bite — confirmed against the live provider schema:

  • Every field is force-new, including eventhub_consumer_group_name and kusto_table_name. The provider defines no update operation, which is why the module's timeouts object has no update key.
  • eventhub_namespace_endpoint_uri must use the sb:// protocol. https:// is the intuitive guess and is rejected by the service.
  • kusto_cluster_uri is the cluster's uri — not its data_ingestion_uri, and not its Resource ID.
  • The namespace and the cluster are each supplied twice, as a Resource ID and as a URI. Nothing checks that the two representations agree; a mismatch surfaces at apply.
  • No output exists for the Event Hubs namespace endpoint URI. Neither the resource nor its data source exposes one, so it has to be composed from the namespace name.
  • The Kusto database must already exist. Digital Twins creates the table inside it, along with its ingestion mapping.
  • Changing kusto_table_name replaces the connection and leaves the old table behind, still holding the history recorded up to that point. It is not renamed and not removed.
  • The provider exposes a subset of the ARM API. The resource type also supports separate tables for twin-lifecycle and relationship-lifecycle events, a switch for recording property removals, and a user-assigned identity option — none of which is available here. Terraform can express only twin and relationship property history, authenticated by the system-assigned identity.
  • No tags. The universal tail is timeouts only, minus update.

🔑 Required Azure RBAC Roles / Permissions

Two audiences here, and conflating them is the most common source of trouble. The caller needs to create the resource; the Digital Twins instance's own identity needs to do the work.

The caller:

Scope Role / permission Why
The Digital Twins instance Microsoft.DigitalTwins/digitalTwinsInstances/timeSeriesDatabaseConnections/write — held by Azure Digital Twins Data Owner, or Contributor on the instance The connection is a child of the instance.

The Digital Twins instance's system-assigned managed identity, during setup:

Scope Role Why
The event hub Azure Event Hubs Data Owner Configuring the hub for history delivery.
The Kusto cluster or database Contributor Creating the history table.
The Kusto database (principal assignment) Admin Creating the table and its ingestion mapping. This is a Kusto database principal assignment, not an Azure RBAC role.

💡 After setup these can be reduced. For ongoing operation the instance's identity needs only Azure Event Hubs Data Sender on the Event Hubs resource. Dropping Contributor and the database Admin assignment afterwards is the least-privilege end state — and it is the step that gets skipped, leaving a service identity with Contributor on an analytics cluster indefinitely.

Azure Prerequisites

  • The Microsoft.DigitalTwins, Microsoft.EventHub and Microsoft.Kusto resource providers registered on the subscription.
  • A Digital Twins instance with a system-assigned managed identity already enabled. This module cannot enable it.
  • An Event Hubs namespace with an event hub, preferably dedicated to data history, and preferably with a dedicated consumer group.
  • A Kusto cluster and an existing database in it. Digital Twins creates the table, not the database.
  • The three setup grants above, in place before this resource is created.
  • ⚠️ The Kusto cluster must have public network access enabled while the connection is created. With it disabled, setup fails. Apply network restrictions afterwards, and switch the database's data connection to a system-assigned identity at the same time.

📁 Module Structure

terraform-azurerm-digital-twins-time-series-database-connection/
├── providers.tf   # required_version + the pinned azurerm provider. No provider block.
├── variables.tf   # name, digital_twins_id, the Event Hub hop, the Kusto destination, timeouts
├── main.tf        # the keystone azurerm_digital_twins_time_series_database_connection.this
├── outputs.tf     # id, name, digital_twins_id, and the Kusto/Event Hub destination
├── README.md      # this document
├── SCOPE.md       # the cross-module contract
├── LICENSE        # MIT
└── .gitignore

⚙️ Quick Start

provider "azurerm" {
  features {}
}

module "dt_history" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-digital-twins-time-series-database-connection.git?ref=v1.0.0"

  name             = "adt-history"
  digital_twins_id = module.dt.id

  eventhub_namespace_id           = module.ehns.id
  eventhub_namespace_endpoint_uri = "sb://${module.ehns.name}.servicebus.windows.net"
  eventhub_name                   = module.ehns.eventhub_names["history"]

  kusto_cluster_id    = module.adx.id
  kusto_cluster_uri   = module.adx.uri
  kusto_database_name = module.adx_db.name
}

ℹ️ The caller configures the provider, its authentication, and the mandatory features {} block. This module declares none of them.

⚠️ Note what is not here: no keys, no connection strings. The instance's managed identity does the work — which is why the grants in the permissions section are the real prerequisite, not an afterthought.

🔌 Cross-Module Contract

Consumes

Input Type Source module
digital_twins_id string terraform-azurerm-digital-twins-instance → id
eventhub_namespace_id string terraform-azurerm-eventhub-namespace → id
eventhub_namespace_endpoint_uri string composed — sb://<namespace name>.servicebus.windows.net
eventhub_name string terraform-azurerm-eventhub-namespace → eventhub_names
eventhub_consumer_group_name string a consumer group on that namespace
kusto_cluster_id string terraform-azurerm-kusto-cluster → id
kusto_cluster_uri string terraform-azurerm-kusto-cluster → uri
kusto_database_name string terraform-azurerm-kusto-database → name

ℹ️ eventhub_namespace_endpoint_uri is the one input with no output to wire from — see example 3.

Emits

Output Description Consumed by
id The connection's Resource ID, nested beneath the instance. imports, state reading
name The connection name. Force-new. review
digital_twins_id The instance this connection belongs to. review
kusto_cluster_uri The cluster the history is written to. analytics tooling, dashboards
kusto_database_name The database holding the history table. analytics tooling, dashboards
kusto_table_name The history table, or null for the service default. analytics tooling, dashboards
eventhub_name The event hub history flows through. runbooks, troubleshooting
eventhub_consumer_group_name The consumer group ADX reads with, or null for the default. runbooks, troubleshooting

🔒 There are no sensitive outputs, because there are no secrets to withhold. That is worth stating explicitly given the other three modules in this family emit a dead-letter posture flag and carefully withhold credentials.

📚 Example Library

1 · The minimal call
module "dt_history" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-digital-twins-time-series-database-connection.git?ref=v1.0.0"

  name             = "adt-history"
  digital_twins_id = module.dt.id

  eventhub_namespace_id           = module.ehns.id
  eventhub_namespace_endpoint_uri = "sb://${module.ehns.name}.servicebus.windows.net"
  eventhub_name                   = module.ehns.eventhub_names["history"]

  kusto_cluster_id    = module.adx.id
  kusto_cluster_uri   = module.adx.uri
  kusto_database_name = module.adx_db.name
}

💡 Eight inputs, no credentials. The table name and consumer group are left to their service defaults here — example 5 argues for setting the consumer group anyway.

2 · The instance needs a system-assigned identity first
module "dt" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-digital-twins-instance.git?ref=v1.0.0"

  name                = "adt-twins-prod"
  resource_group_name = module.rg.name
  location            = module.rg.location

  identity = {
    type = "SystemAssigned" # required for data history
  }
}

⚠️ Not optional and not something this module can arrange. Without an identity on the instance there is nothing to authenticate the connection, and the apply fails.

ℹ️ The provider's resource does not accept a user-assigned identity for this connection, even though the underlying API supports one. System-assigned is the only shape Terraform can express here.

3 · The `sb://` URI you have to compose
# ✅ Correct.
eventhub_namespace_endpoint_uri = "sb://${module.ehns.name}.servicebus.windows.net"
# ❌ Refused at `terraform validate`.
eventhub_namespace_endpoint_uri = "https://${module.ehns.name}.servicebus.windows.net"
Error: Invalid value for variable

  eventhub_namespace_endpoint_uri must start with "sb://" - the service requires that
  protocol, for example "sb://my-namespace.servicebus.windows.net". An "https://" URI is
  the intuitive guess and is wrong. The provider itself checks only that the string is
  non-empty, so this is caught here or not at all before apply.

ℹ️ There is genuinely no output to wire this from: neither azurerm_eventhub_namespace nor its data source exposes an endpoint URI, so composition from the name is the supported approach rather than a workaround.

🔒 Every check on this argument is the module's. The provider carries StringIsNotEmpty on it and nothing else — no protocol check, no shape check, no comparison against eventhub_namespace_id. An https:// URI, a bare hostname, and the URI of an entirely different namespace all pass the provider's own validation untouched and fail at apply, after the create has begun doing real work.

💡 These fire at terraform validate — offline, without credentials — rather than at plan.

4 · Two representations of the same two resources
# The Event Hubs namespace, twice:
eventhub_namespace_id           = module.ehns.id                                    # Resource ID
eventhub_namespace_endpoint_uri = "sb://${module.ehns.name}.servicebus.windows.net"  # URI

# The Kusto cluster, twice:
kusto_cluster_id  = module.adx.id  # Resource ID
kusto_cluster_uri = module.adx.uri # URI

⚠️ Nothing validates that the ID and the URI in each pair refer to the same resource. Pass a URI from one cluster and an ID from another and the plan is clean; the failure arrives at apply, or worse, the connection is created pointing somewhere unintended.

💡 Deriving both from the same module reference — as above — makes the mismatch structurally impossible. Hard-coding either one is where this goes wrong.

5 · Give ADX its own consumer group
module "dt_history" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-digital-twins-time-series-database-connection.git?ref=v1.0.0"

  name             = "adt-history"
  digital_twins_id = module.dt.id

  eventhub_namespace_id           = module.ehns.id
  eventhub_namespace_endpoint_uri = "sb://${module.ehns.name}.servicebus.windows.net"
  eventhub_name                   = module.ehns.eventhub_names["history"]
  eventhub_consumer_group_name    = "adx-history"

  kusto_cluster_id    = module.adx.id
  kusto_cluster_uri   = module.adx.uri
  kusto_database_name = module.adx_db.name
}

💡 A consumer group tracks its own read position. Sharing $Default with any other reader of the same event hub makes the two compete, and you can lose history records without anything reporting an error.

⚠️ Force-new, like everything else here. Decide it at creation.

6 · Naming the history table
module "dt_history" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-digital-twins-time-series-database-connection.git?ref=v1.0.0"

  digital_twins_id             = var.digital_twins_id

  eventhub_name                = "eventhub-example"

  eventhub_namespace_endpoint_uri = "eventhub-namespace-endpoint-uri"

  eventhub_namespace_id        = var.eventhub_namespace_id

  kusto_cluster_id             = var.kusto_cluster_id

  kusto_cluster_uri            = "kusto-cluster-uri"

  kusto_database_name          = "kusto-database-example"

  name                         = "dt-history-example"
  kusto_table_name = "TwinPropertyHistory"
}

ℹ️ Left unset, the service names it AdtPropertyEvents. The module does not restate that default as its own, so the service keeps owning it — the output reads null in that case, and the table still exists under the service's name.

⚠️ Force-new, and the replacement is not clean: changing it leaves the old table behind holding the history recorded up to that point. It is not renamed and not removed, so you end up with two tables and a gap in whichever one you query.

7 · The ordering trap — public network access
# 1 · Create the cluster reachable.
module "adx" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster.git?ref=v1.0.0"

  name                = "adxtwinsprod"
  resource_group_name = module.rg.name
  location            = module.rg.location

  sku = {
    name     = "Standard_D13_v2"
    capacity = 2
  }

  public_network_access_enabled = true # required WHILE the connection is created
}

# 2 · Create the connection (this module).
# 3 · THEN restrict the network, and switch the database's data connection to a
#     system-assigned identity so the pieces can still reach each other.

⚠️ With public network access disabled, setup fails — Digital Twins cannot reach in to create the table and its ingestion mapping. The error is an internal-server-error that does not name the cause, which makes it an expensive thing to debug from first principles.

💡 This inverts the usual instinct of locking a resource down before wiring anything to it. Sequence it deliberately, and note that step 3 is a separate change that no single apply will do for you.

8 · No update operation at all
timeouts = {
  create = "60m"
  read   = "5m"
  delete = "30m"
  # no `update` — the object does not accept one
}

ℹ️ Every field on this resource is force-new, so the provider defines no update. The timeouts object omits the key rather than accepting and ignoring it, which turns a misunderstanding into a type error at plan time.

💡 Raise create if the cluster is busy. Creation is not a metadata write: Digital Twins builds the table and its ingestion mapping inside this timeout.

9 · Reducing the grants after setup
# DURING setup — all three needed by the INSTANCE's identity:
#   Azure Event Hubs Data Owner   on the event hub
#   Contributor                   on the Kusto cluster or database
#   database principal Admin      on the Kusto database

# AFTER setup — this is sufficient for ongoing operation:
module "dt_history_roles" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-role-assignments.git?ref=v1.0.0"

  scope = module.ehns.id

  role_assignments = {
    eventhub_sender = {
      scope                = module.ehns.id
      role_definition_name = "Azure Event Hubs Data Sender"
      principal_id         = module.dt.identity_principal_id
    }
  }
}

💡 The reduction is the part that gets skipped, and skipping it leaves a service identity holding Contributor on an analytics cluster forever. Data history itself needs only Data Sender once the table exists.

⚠️ Do not reduce before the connection is created — the setup grants are what let Digital Twins build the table. And if you later replace the connection, the setup grants are needed again.

10 · What Terraform cannot express here
Supported by the ARM API, NOT exposed by the pinned provider:
  adxRelationshipLifecycleEventsTableName   relationship lifecycle events -> its own table
  adxTwinLifecycleEventsTableName           twin lifecycle events -> its own table
  recordPropertyAndItemRemovals             record property/item DELETIONS too
  identity                                  user-assigned identity option

ℹ️ So a Terraform-managed connection historizes twin and relationship property updates, authenticated by the system-assigned identity, and nothing else. Lifecycle events — twins and relationships being created and deleted — are not recorded.

⚠️ Worth knowing before promising an audit trail. "We have data history" and "we can show when this twin was deleted" are different claims, and only the first one is true of what this module builds.

11 · Querying what it produces
output "history_query_target" {
  value = {
    cluster  = module.dt_history.kusto_cluster_uri
    database = module.dt_history.kusto_database_name
    table    = module.dt_history.kusto_table_name # null = AdtPropertyEvents
  }
}
AdtPropertyEvents
| where TimeStamp > ago(1h)
| where Id == "thermostat-42"
| project TimeStamp, Key, Value

💡 The destination is emitted in three pieces because that is exactly what a query needs. Reassembling it from a Resource ID is avoidable friction, and the table name in particular is not derivable from anything else.

ℹ️ Default ingestion latency is roughly ten minutes. If that is too slow, enable streaming ingestion on the cluster — a cluster-side setting, not something this connection controls.

12 · Destroying the connection keeps the data
# module.dt_history.azurerm_digital_twins_time_series_database_connection.this will be destroyed

ℹ️ The table and everything in it survive. Only the connection goes, so history stops accumulating and what has already been recorded stays queryable.

💡 That makes this a safe resource to replace — unlike most of the destructive plans in this family. The one cost is the gap: nothing is recorded between the destroy and the next create.

13 · The name constraint, and why it is not one regex
name = "adt-history" # 3-50 chars, letters/digits/hyphens, not all digits, no leading or trailing hyphen
# ❌ All refused at `terraform validate`:
name = "ab"             # too short - the floor is 3
name = "123"            # all digits
name = "-history"       # leading hyphen
name = "history-"       # trailing hyphen
name = "adt_history"    # underscore

⚠️ An earlier version of this module encoded the range as 2–49, and that was wrong in both directions. The provider checks len < 3 and len > 50 as two separate steps, so the legal range is 3 to 50. The old range refused a legal 50-character name — and because a failing validation {} blocks terraform destroy as well as apply, a connection already created with such a name could not have been destroyed through the module. Being stricter than the provider is the one direction a module must not take.

ℹ️ The provider expresses the rest as ^[A-Za-z0-9][A-Za-z0-9-]+[A-Za-z0-9]$ plus a dedicated all-digits check. The service documents it with lookaheads, which Terraform's Go RE2 engine does not support — that pattern fails to compile rather than matching — so the module carries the length, character-set and all-digits rules as three separate checks.

💡 Worth knowing generally: a service's own regex cannot always be pasted into a validation block, and a lookahead that fails to compile is a hard error at evaluation time rather than a permissive fallback.

14 · 🏗️ End-to-end composition
provider "azurerm" {
  features {}
}

module "rg" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-resource-group.git?ref=v1.0.0"

  name     = "rg-twins-prod"
  location = "eastus"
}

# 1 · The twin graph, WITH a system-assigned identity — the whole basis of this connection.
module "dt" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-digital-twins-instance.git?ref=v1.0.0"

  name                = "adt-twins-prod"
  resource_group_name = module.rg.name
  location            = module.rg.location

  identity = {
    type = "SystemAssigned"
  }
}

# 2 · The Event Hub hop, with a dedicated hub and consumer group for history.
module "ehns" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-eventhub-namespace.git?ref=v1.0.0"

  name                = "ehns-twins-prod"
  resource_group_name = module.rg.name
  location            = module.rg.location
  sku                 = "Standard"

  event_hubs = {
    history = {
      name              = "history"
      partition_count   = 4
      message_retention = 7
    }
  }

  consumer_groups = {
    adx = {
      name         = "adx-history"
      eventhub_key = "history"
    }
  }
}

# 3 · The analytics destination. Reachable during setup — see example 7.
module "adx" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-cluster.git?ref=v1.0.0"

  name                = "adxtwinsprod"
  resource_group_name = module.rg.name
  location            = module.rg.location

  sku = {
    name     = "Standard_D13_v2"
    capacity = 2
  }

  public_network_access_enabled = true
}

module "adx_db" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-kusto-database.git?ref=v1.0.0"

  name                = "twins-history"
  resource_group_name = module.rg.name
  location            = module.rg.location
  cluster_name        = module.adx.name
}

# 4 · The setup grants, to the INSTANCE's identity — not the caller's.
module "dt_history_setup_roles" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-role-assignments.git?ref=v1.0.0"

  scope = module.ehns.id

  role_assignments = {
    eventhub_owner = {
      scope                = module.ehns.id
      role_definition_name = "Azure Event Hubs Data Owner"
      principal_id         = module.dt.identity_principal_id
    }
    kusto_contributor = {
      scope                = module.adx.id
      role_definition_name = "Contributor"
      principal_id         = module.dt.identity_principal_id
    }
  }
}

# The Kusto database principal assignment with role Admin is a distinct resource type with no module
# in this suite yet, so declare it directly.
resource "azurerm_kusto_database_principal_assignment" "dt" {
  name                = "dt-history-admin"
  resource_group_name = module.rg.name
  cluster_name        = module.adx.name
  database_name       = module.adx_db.name

  tenant_id      = module.dt.identity_tenant_id
  principal_id   = module.dt.identity_principal_id
  principal_type = "App"
  role           = "Admin"
}

# 5 · The connection. This is the module.
module "dt_history" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-digital-twins-time-series-database-connection.git?ref=v1.0.0"

  name             = "adt-history"
  digital_twins_id = module.dt.id

  eventhub_namespace_id           = module.ehns.id
  eventhub_namespace_endpoint_uri = "sb://${module.ehns.name}.servicebus.windows.net"
  eventhub_name                   = module.ehns.eventhub_names["history"]
  eventhub_consumer_group_name    = "adx-history"

  kusto_cluster_id    = module.adx.id
  kusto_cluster_uri   = module.adx.uri
  kusto_database_name = module.adx_db.name

  timeouts = {
    create = "60m"
  }

  # Nothing in the arguments references the grants, so the ordering must be explicit.
  depends_on = [
    module.dt_history_setup_roles,
    azurerm_kusto_database_principal_assignment.dt,
  ]
}

output "twins_history" {
  value = {
    cluster  = module.dt_history.kusto_cluster_uri
    database = module.dt_history.kusto_database_name
    table    = module.dt_history.kusto_table_name # null = AdtPropertyEvents
  }
}

⚠️ Two steps deliberately remain after this applies: reduce the setup grants to Event Hubs Data Sender (example 9), and restrict the Kusto cluster's network (example 7). Neither can be part of the same apply — the connection needs the wider access while it is being created.

🔒 What the composition gets right: no credential appears anywhere, the identity is system-assigned rather than a secret, the event hub and consumer group are dedicated to history so nothing competes for the read position, and both URI/ID pairs derive from the same module reference so they cannot disagree.

💡 The depends_on is not decoration. Nothing in this module's arguments references the role assignments, so without it Terraform is free to create the connection before the grants exist — and role assignments take time to propagate even once created.

📥 Inputs

Input Type Default Notes
name string — Required. Force-new. Validated: 3–50 chars, letters/digits/hyphens, not all digits, beginning and ending with a letter or digit.
digital_twins_id string — Required. Force-new. The instance needs a system-assigned identity.
eventhub_namespace_id string — Required. Force-new.
eventhub_namespace_endpoint_uri string — Required. Force-new. Validated: must start sb://.
eventhub_name string — Required. Force-new.
eventhub_consumer_group_name string null Force-new. Null = $Default; prefer a dedicated group.
kusto_cluster_id string — Required. Force-new.
kusto_cluster_uri string — Required. Force-new. Validated: must start https://.
kusto_database_name string — Required. Force-new. Must already exist.
kusto_table_name string null Force-new. Null = AdtPropertyEvents.
timeouts object(...) null create / read / delete — no update.
Full schemas
variable "name" {
  type = string
  # Validated as separate checks rather than the service's own pattern: Terraform's regex engine
  # (Go RE2) has no lookahead, so `^(?![0-9]+$)(?!-)...` cannot be used — it fails to compile.
  #   3-50 chars, letters/digits/hyphens, not all digits, no leading or trailing hyphen.
}

variable "digital_twins_id" {
  type = string
  # ⚠️ The instance must already have a SYSTEM-ASSIGNED managed identity — this connection has no
  #    other way to authenticate. Enable it on the instance module, not here.
}

variable "eventhub_namespace_id" {
  type = string
  # ℹ️ The service wants the namespace as BOTH a Resource ID and an sb:// URI. They must agree;
  #    nothing checks that they do.
}

variable "eventhub_namespace_endpoint_uri" {
  type = string
  # Validated: must start with "sb://". No module output exists — compose it:
  #   "sb://${module.ehns.name}.servicebus.windows.net"
}

variable "eventhub_name" {
  type = string
  # 💡 Dedicate an event hub to history rather than sharing one with another consumer.
}

variable "eventhub_consumer_group_name" {
  type    = string
  default = null # service default: $Default
  # 💡 Supply a dedicated group. Sharing $Default with another reader makes the two compete for the
  #    read position and can cost you history records.
}

variable "kusto_cluster_id" {
  type = string
}

variable "kusto_cluster_uri" {
  type = string
  # Validated: must start with "https://". This is the cluster's `uri`, not `data_ingestion_uri`.
}

variable "kusto_database_name" {
  type = string
  # ⚠️ The database must already exist. Digital Twins creates the TABLE inside it.
}

variable "kusto_table_name" {
  type    = string
  default = null # service default: AdtPropertyEvents
  # ⚠️ Force-new, and the old table is LEFT BEHIND holding the history recorded up to that point.
}

variable "timeouts" {
  type = object({
    create = optional(string)
    read   = optional(string)
    delete = optional(string)
  })
  default = null
  # ⚠️ No `update` key, and that is not an omission: every field is force-new, so the provider
  #    defines no update operation.
}

🧾 Outputs

Output Description Sensitive
id The connection's Resource ID, nested beneath the instance. no
name The connection name. Force-new, like all ten arguments. no
digital_twins_id The instance this connection belongs to — and whose system-assigned identity is the only authentication path here. no
kusto_cluster_uri The cluster the history is written to. The QUERY endpoint, not the ingest- one. no
kusto_cluster_id The cluster's Resource ID, read back in canonical case. Use it to scope the cluster-side role assignments. no
kusto_database_name The database holding the history table. Must already exist. no
kusto_table_name The history table. Always populated, never null — the provider's schema defaults it to AdtPropertyEvents and reads that value back. no
kusto_query_reference Map of cluster URI, database and table — the three values a Kusto query needs, assembled. no
eventhub_name The event hub history flows through. Its retention is the history's blast radius. no
eventhub_namespace_id The namespace containing it — emitted to scope the namespace-side role assignment. no
eventhub_consumer_group_name The consumer group ADX reads with. Always populated, never null — the schema defaults it to $Default. no
table_is_the_service_default Whether the table name was inherited rather than chosen. Derived, so known at plan time — the only place the distinction survives. no
consumer_group_is_the_shared_default Whether ADX reads with the shared $Default. True is the case to review: a competing reader costs history records, with no error. no
destroy_leaves_the_kusto_table_behind Always true. The provider never sends cleanupConnectionArtifacts, so a destroy leaves the ADX table, its mapping and the data connection in place — stored, billed, unwritten. no
grants_no_access_by_itself Always true. The instance identity's roles on the Event Hub and the cluster are the real work, and the failure mode is silence, not an error. no
requires_public_kusto_access_during_setup Always true. Restrict the cluster's network after this applies, not before. no
every_argument_is_force_new Always true — all ten, which is why the timeouts type declares no update key. no
the_two_representations_are_never_compared_by_the_provider Always true. Four arguments describe two resources and the provider validates each in isolation; the name comparison is this module's. no

🔒 Nothing here is sensitive, because this resource accepts and produces no secrets. That is the substantive difference between it and the three endpoint modules in this family, and the reason to prefer data history over a hand-rolled export pipeline where either would do.

🧠 Architecture Notes

  • The identity-based design is why this module is the pleasant one. Its three siblings each hold two live credentials that reach Terraform state and cannot be avoided. This one holds none: the instance's system-assigned identity authenticates, so the security question becomes "which roles, for how long" rather than "where do these keys end up". Where a design choice exists between an endpoint and data history, that difference is worth weighing.

  • The grants are the real work, and they are reducible. Three grants to the instance's identity are needed during setup — Event Hubs Data Owner, Kusto Contributor, and a database principal assignment with role Admin — because Digital Twins builds the table on your behalf. Afterwards, Event Hubs Data Sender alone suffices. Leaving Contributor in place is the default outcome of not thinking about it, and the wrong end state.

  • The public-network-access ordering inverts the usual instinct. A locked-down cluster cannot be set up, because Digital Twins reaches in to create the table and mapping. So: open, connect, then restrict — and the restriction is a separate change, not part of the same apply. The failure mode if you get this backwards is an internal-server-error that names nothing.

  • Four inputs describe two resources, and THE PROVIDER NEVER COMPARES THEM. The Event Hubs namespace and the Kusto cluster are each supplied as both a Resource ID and a URI. The provider validates each in isolation — the two IDs through their parsers, the two URIs through a bare non-emptiness check — so a configuration naming one namespace by ID and a different one by URI passes its validation entirely and fails at apply. This module compares the NAME in each pair: the last segment of the ID against the first label of the URI host, lowercased. Only the first label is compared, so sovereign-cloud hostnames (servicebus.chinacloudapi.cn, kusto.usgovcloudapi.net) stay legal. Deriving both from the same module reference still makes a mismatch structurally impossible, and remains the better habit.

  • A destroy leaves the Azure Data Explorer table behind. The service's delete accepts a cleanupConnectionArtifacts option and the provider never sets it, so terraform destroy removes the connection record and leaves the table, its ingestion mapping and the cluster-side data connection in place — still stored, still billed, no longer written to. There is no argument to change that, and the same is true of any replacement, since every argument is force-new. Cleaning up is a manual step in Kusto.

  • Two arguments read back as their defaults rather than as null. kusto_table_name and eventhub_consumer_group_name carry schema defaults (AdtPropertyEvents and $Default), so leaving them null still sends those values and the attributes are never null. table_is_the_service_default and consumer_group_is_the_shared_default are derived from the inputs, which is the only place the distinction survives.

  • You cannot type $Default. The provider's consumer-group validator is ^[a-zA-Z0-9]([-._a-zA-Z0-9]{0,48}[a-zA-Z0-9])?$, which permits no dollar sign — so setting eventhub_consumer_group_name to the exact value the provider sends when you omit it is refused at terraform validate. The schema's own default is not a legal input to the schema. Omit the argument to get it. Note also that the consumer-group limit is 50 characters while the event hub's is 256, despite near-identical error messages.

  • One provider error names the wrong argument. When a refresh cannot parse the cluster ID returned by Azure, the message reads parsing `kusto_cluster_uri` — the value it could not parse is kusto_cluster_id. And the Kusto database-name validator prints a 4-to-22-character limit while enforcing 260.

  • The service's own name regex cannot be used directly. It relies on negative lookahead, which Go RE2 does not support — the pattern fails to compile rather than matching permissively. The same rules are enforced as separate checks. This generalizes: a documented service regex is not automatically a usable validation condition.

  • Everything is force-new, so there is nothing to update. The timeouts object omits update rather than accepting and ignoring it, which turns a misunderstanding into a plan-time type error. The practical consequence is that this is a set-once configuration — including the table name and consumer group, both of which look incidental and are not.

  • Replacement is unusually cheap here, with one asterisk. Destroying the connection leaves the table and its data intact, so history stops accumulating rather than disappearing. The asterisk is kusto_table_name: change it and the old table stays behind holding the history so far, leaving two tables and a gap in each.

  • The provider covers less than the API. Twin-lifecycle and relationship-lifecycle event tables, property-removal recording, and user-assigned identity all exist in the ARM API and none is exposed here. So a Terraform-managed connection records property updates only — relevant if anyone is treating data history as an audit trail for creations and deletions.

  • No tags. The resource type does not support them. Tag the instance, the namespace and the cluster instead.

🧱 Design Principles

Concern Secure default (empty call) Opt-out (caller must type it)
Credentials none accepted and none emitted — authentication is a managed identity —
Identity the instance's system-assigned identity, documented as a hard prerequisite —
Grant duration the setup grants are documented as reducible, with the end state named leave Contributor in place, knowingly
Network posture the required order is documented — open during setup, restrict afterwards —
URI correctness sb:// and https:// prefixes validated, because both fail only at apply —
Name correctness the service's constraint enforced without relying on an unsupported regex —
Service defaults left to the service (AdtPropertyEvents, $Default) rather than restated as module defaults set them explicitly, which is recommended for the consumer group
Force-new fields update omitted from timeouts so the set-once nature is a type error, not a surprise —
  • Before applying: confirm the instance has a system-assigned identity, and that all three setup grants are in place.
  • Before applying: confirm the Kusto cluster is reachable and the database already exists.
  • Before applying: confirm each ID/URI pair derives from the same resource.
  • After applying: reduce the grants to Event Hubs Data Sender, and restrict the cluster's network. Both are separate changes.

🚀 Runbook

terraform init -backend=false
terraform validate
terraform fmt -check
  • Pin the source to a tag — ?ref=v1.0.0 — never a branch.
  • Plan-only from here. A human applies from CI.
  • ⚠️ Order on first apply: identity, then grants, then this connection. Use depends_on — nothing in the arguments expresses it.
  • ⚠️ Raise create for a busy cluster; table creation happens inside that timeout.
  • ⚠️ Two follow-up changes after a successful apply: reduce the grants, restrict the network.
  • ℹ️ Destroying this is comparatively safe — the table and its data remain.

🧪 Testing

terraform validate and terraform fmt -check are the offline gate. They confirm:

  • all eight required inputs are present and typed string;
  • name satisfies the length, character, all-digits and hyphen rules;
  • eventhub_namespace_endpoint_uri starts with sb://;
  • kusto_cluster_uri starts with https://;
  • the timeouts value is an object of Go duration strings, with no update key because the provider defines no update operation - and note that passing one anyway is SILENTLY DISCARDED rather than refused, producing no error at validate, none at plan, and no trace in the resulting value;
  • the module declares no provider block.

What only plan and apply exercise:

  • whether the instance actually has a system-assigned identity;
  • whether that identity holds all three setup grants, and whether they have propagated;
  • whether the Kusto database exists;
  • whether the cluster permits public network access at creation time;
  • whether each ID/URI pair genuinely refers to the same resource;
  • whether table creation completes inside the create timeout.

Note the name validation was verified by evaluating the condition directly rather than trusting terraform validate, which does not evaluate module-input validations — a lookahead-based pattern would have shipped clean and failed in a caller's plan.

💬 Example Output

Outputs:

digital_twins_id             = "/subscriptions/00000000-.../providers/Microsoft.DigitalTwins/digitalTwinsInstances/adt-twins-prod"
eventhub_consumer_group_name = "adx-history"
eventhub_name                = "history"
id                           = "/subscriptions/00000000-.../digitalTwinsInstances/adt-twins-prod/timeSeriesDatabaseConnections/adt-history"
kusto_cluster_uri            = "https://adxtwinsprod.eastus.kusto.windows.net"
kusto_database_name          = "twins-history"
kusto_table_name             = null
name                         = "adt-history"

ℹ️ kusto_table_name = null means the service default is in place — the table exists as AdtPropertyEvents. The module does not restate that default as its own, so the null is accurate rather than missing.

🔍 Troubleshooting

Symptom Cause Fix
Plan rejects eventhub_namespace_endpoint_uri An https:// URI was passed. Use sb:// (example 3).
Plan rejects kusto_cluster_uri A Resource ID or the data_ingestion_uri was passed. Use the cluster's uri output.
Plan rejects name Too short, all digits, a leading/trailing hyphen, or an underscore. See example 13.
Plan rejects an update timeout Every field is force-new, so no update operation exists. Remove the key (example 8).
Apply fails: "the resource could not ACT due to an internal server error" The Kusto cluster has public network access disabled. Enable it for setup, restrict afterwards (example 7).
Apply fails with a permission error on the database The instance's identity lacks the database principal Admin assignment. Add it, wait for propagation, retry (example 14).
Apply fails reaching the event hub The identity lacks Azure Event Hubs Data Owner during setup. Grant it; Data Sender alone is not enough to create the connection.
Apply fails and the instance has no identity Data history needs a system-assigned identity. Enable it on the instance module (example 2).
Apply times out during creation Table and mapping creation exceeded create. Raise create; 60m is a reasonable starting point.
Apply succeeds but no rows appear Default ingestion latency is around ten minutes. Wait, then check. Enable streaming ingestion on the cluster if that is too slow.
Rows stopped appearing, no errors Another reader is sharing the $Default consumer group. Give ADX its own consumer group (example 5).
Two history tables exist, each with a gap kusto_table_name was changed; the old table was left behind. Expected. Query both, or backfill.
Twin deletions are not in the history The provider exposes no lifecycle-event tables. Not available through Terraform (example 10).
The identity still holds Contributor months later The post-setup reduction was skipped. Reduce to Event Hubs Data Sender (example 9).

🔗 Related Docs

💙 "Infrastructure as Code should be standardized, consistent, and secure."