Skip to content

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

☁️ Azure Data Factory Linked Service — Cosmos DB (MongoDB API) Terraform Module

A Data Factory's stored definition of how to reach a Cosmos DB account through the MongoDB API, targeting hashicorp/azurerm ~> 4.0.

Terraform azurerm Module Type Resources Caveat


🧩 Overview

  • 🔌 Creates one azurerm_data_factory_linked_service_cosmosdb_mongoapi — the definition a dataset or copy activity uses to reach a Cosmos DB account through the MongoDB API.
  • 🔴 The requires-import error names the WRONG resource. It says azurerm_data_factory_linked_service_cosmosdb — the SQL API sibling — so anyone who hits it is told to import a different resource type. A provider defect, reported here as a deliberately false output.
  • 🔴 The provider requires no credential at all. connection_string is the only one it accepts and it is optional, so the empty call applies cleanly and can never connect.
  • 🔴 The read never refreshes the connection string, so drift in it is invisible — which is also exactly why the credential is rotatable.
  • ⚠️ server_version_is_32_or_higher declares the account's version; it does not set it.

💡 Why it matters: two of this resource's three sharp edges are invisible until something fails, and the third tells you the wrong thing when it does.


❤️ Support this project

If this module saved you time:


🗺️ Where this fits in the family

flowchart TB
  RG["terraform-azurerm-resource-group"]
  ADF["terraform-azurerm-data-factory"]
  SQLAPI["terraform-azurerm-data-factory-linked-service-cosmosdb, SQL API"]
  MONGO["terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi"]
  ACCT["terraform-azurerm-cosmosdb-account"]
  SRC["a Cosmos DB account, never contacted by Terraform"]
  DS["a dataset, then a copy activity"]

  RG -->|"name"| ADF
  ADF -->|"id"| SQLAPI
  ADF -->|"id"| MONGO
  ACCT -->|"endpoint and key, or a connection string"| SQLAPI
  ACCT -->|"a connection string only"| MONGO
  SQLAPI -->|"connects at pipeline runtime, never at apply"| SRC
  MONGO -->|"connects at pipeline runtime, never at apply"| SRC
  SQLAPI -->|"name, referenced BY NAME and never by id"| DS
  MONGO -->|"name, referenced BY NAME and never by id"| DS

  classDef this fill:#0078D4,stroke:#004578,color:#ffffff,stroke-width:2px
  classDef keystone fill:#004578,stroke:#00243d,color:#ffffff,stroke-width:2px
  classDef sibling fill:#eef3f8,stroke:#b9c8d8,color:#1b2733
  class SQLAPI,MONGO this
  class ADF keystone
  class RG,ACCT,SRC,DS sibling
Loading

Both Cosmos DB linked services hang off the same factory and reach the same account. They differ in their credential model and in one provider defect, described below.


🧬 What this module builds

flowchart TB
  subgraph INPUTS["Inputs"]
    ID["name and data_factory_id, THE ONLY TWO FORCE-NEW FIELDS"]
    CS["connection_string, OPTIONAL, and the ONLY credential mode"]
    VER["server_version_is_32_or_higher, a declaration about the account"]
    META["database, integration_runtime_name, parameters, annotations, additional_properties"]
  end

  THIS["azurerm_data_factory_linked_service_cosmosdb_mongoapi.this"]

  subgraph OUTPUTS["Outputs"]
    MODE["credential_mode, one of two"]
    NONE["has_no_credential_at_all, and the empty call applies cleanly"]
    DRIFT["the read never refreshes the connection string, so drift is invisible"]
    ROT["...which is also why the credential IS rotatable"]
    VEROUT["the server version flag declares reality, it does not change it"]
    IMPORT["the import guard names the WRONG resource, a provider defect"]
  end

  ID --> THIS
  CS --> THIS
  VER --> THIS
  META --> THIS

  CS --> MODE
  CS --> NONE
  CS --> DRIFT
  CS --> ROT
  VER --> VEROUT
  META --> IMPORT

  classDef this fill:#0078D4,stroke:#004578,color:#ffffff,stroke-width:2px
  classDef sibling fill:#eef3f8,stroke:#b9c8d8,color:#1b2733
  class THIS this
  class ID,CS,VER,META,MODE,NONE,DRIFT,ROT,VEROUT,IMPORT sibling
Loading
Resource Count Notes
azurerm_data_factory_linked_service_cosmosdb_mongoapi.this 1 The keystone. A metadata record.
timeouts dynamic, 0..1 All four keys; none bounds a query.

✅ Provider / Versions

Requirement Value
Terraform >= 1.12.0
hashicorp/azurerm ~> 4.0
Provider block None. The caller configures the provider, including the mandatory features {} block.

Schema notes that bite — verified against the live provider source:

  • 🔴 The requires-import error names azurerm_data_factory_linked_service_cosmosdb, not this resource. The importer's own type check IS correct — it checks the MongoDB API linked-service type — so only the message is wrong, and it is wrong in the one place someone reads it under pressure.
  • 🔴 connection_string is Optional and is the only credential this resource accepts, with no ExactlyOneOf and no AtLeastOneOf. The empty call applies cleanly and can never connect.
  • ✅ The provider marks connection_string sensitive itself — unusual in this family, where most resources mark nothing.
  • 🔴 The read never sets connection_string. Drift in it is invisible and an import brings none.
  • ✅ The same fact makes the credential rotatable — the opposite of the Synapse and Snowflake linked services, where the read does refresh the connection string and a password inside it cannot be rotated.
  • ⚠️ server_version_is_32_or_higher is a declaration, not a setting. It tells Data Factory which wire protocol to speak; nothing checks it against the account, and a wrong value surfaces as a pipeline failure.
  • ⚠️ The connection string's only validator is StringIsNotEmpty. Nothing parses it.
  • ⚠️ The name validator refuses only an all-punctuation name. Its rule is one anchored regex over - . + ? / < > * % & : \ with a one-or-more quantifier, so it rejects a name composed entirely of those characters and nothing else — a leading digit passes, an empty string passes, and a name that merely contains them passes. Its error message names those characters as disallowed outright, so the message is far broader than the rule, which is the opposite of the usual failure. Azure itself is stricter, and its two published pages disagree with each other; both conditions are reported through outputs rather than refused.
  • ⚠️ Only name and data_factory_id are force-new. There is no location and no tags.
  • ℹ️ No CustomizeDiff, no ConflictsWith and no version gate — every check is reachable offline.

🔑 Required Azure RBAC Roles / Permissions

Permission Scope Why
Microsoft.DataFactory/factories/linkedservices/write the Data Factory Create and update.
Microsoft.DataFactory/factories/linkedservices/read the Data Factory Refresh and plan.
Microsoft.DataFactory/factories/linkedservices/delete the Data Factory Destroy — which breaks every dataset referencing it.
Data Factory Contributor the Data Factory The built-in role containing the above.
Microsoft.DocumentDB/databaseAccounts/listConnectionStrings/action the Cosmos DB account Only if the configuration reads the connection string from the account rather than a secret store.

🔴 A Cosmos DB connection string carries a full-access data-plane key, and no Azure RBAC role limits what it can do once issued.


Azure Prerequisites

  • An existing Data Factory.
  • An existing Cosmos DB account with the MongoDB API enabled, and a connection string for it.
  • The database, if one is named. Nothing here creates or verifies it.
  • A self-hosted integration runtime only where the account is reachable solely over a private endpoint.

📁 Module Structure

terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi/
├── providers.tf     # required_version + the pinned azurerm; no provider block
├── variables.tf     # 11 variables, 12 validations
├── main.tf          # the keystone, its timeouts block, and the derived locals
├── outputs.tf       # 48 outputs: the id first, then the facts no plan shows
├── README.md        # this file
├── SCOPE.md         # the cross-module contract
├── LICENSE          # MIT
└── .gitignore

⚙️ Quick Start

provider "azurerm" {
  features {}
}

module "cosmos_mongo_link" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"

  name              = "ls-cosmos-mongo"
  data_factory_id   = module.data_factory.id
  connection_string = var.cosmos_mongo_connection_string
  database          = "analytics"
}

🔴 The connection string is the only credential this resource accepts, and it is optional — so omitting it produces a linked service that applies cleanly and can never connect.

🔒 The caller configures the provider, its authentication, and the mandatory features {} block.


🔌 Cross-Module Contract

Consumes

Input Type Source
data_factory_id string terraform-azurerm-data-factory (id)
connection_string string (sensitive) a secret store, at the call site
database string caller
server_version_is_32_or_higher bool caller — a statement about the account

Emits

Output Consumed by
id imports, RBAC scoping
name datasets and copy activities
credential_mode assertions
credential_is_usable assertions

📚 Example Library

1 · The smallest correct call
module "cosmos_mongo_link" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"

  name              = "ls-cosmos-mongo"
  data_factory_id   = module.data_factory.id
  connection_string = var.cosmos_mongo_connection_string
  database          = "analytics"
}

ℹ️ The connection string is the only credential this resource accepts.

2 · The empty call, which the provider accepts
module "cosmos_mongo_link" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"

  name            = "ls-cosmos-mongo"
  data_factory_id = module.data_factory.id
}

output "state" {
  value = module.cosmos_mongo_link.credential_mode # => "NO_CREDENTIAL_AT_ALL"
}

🔴 This applies cleanly and can never connect. connection_string is Optional and nothing replaces it, so nothing fails until a pipeline runs.

3 · Asserting a credential exists
check "linked_service_can_connect" {
  assert {
    condition     = module.cosmos_mongo_link.credential_is_usable
    error_message = "This Cosmos DB MongoDB API linked service has no connection string and will fail at pipeline runtime."
  }
}

💡 Worth writing precisely because the provider requires nothing. Known at plan.

4 · The server version flag
module "cosmos_mongo_link" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"

  name                           = "ls-cosmos-mongo"
  data_factory_id                = module.data_factory.id
  connection_string              = var.cosmos_mongo_connection_string
  server_version_is_32_or_higher = true
}

⚠️ This declares the account's server version so Data Factory picks the right wire protocol. It does not upgrade or configure anything, and nothing checks it against reality — a wrong value applies cleanly and surfaces as a pipeline failure.

5 · The import error that names the wrong resource
output "import_guard_is_correct" {
  value = module.cosmos_mongo_link.the_import_guard_names_this_resource_correctly # => false
}

🔴 A provider defect, not a module one. If a linked service of this name already exists, the requires-import error tells you to import azurerm_data_factory_linked_service_cosmosdb — the SQL API sibling. Import this resource type instead. The importer's own type check is correct; only the message is wrong.

6 · Importing an existing linked service
terraform import 'module.cosmos_mongo_link.azurerm_data_factory_linked_service_cosmosdb_mongoapi.this' \
  "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.DataFactory/factories/<factory>/linkedservices/ls-cosmos-mongo"

⚠️ The import brings no connection string with it — the read never sets that field. The first plan afterwards proposes writing whatever the configuration says, without having read what is stored.

7 · Rotating the credential
module "cosmos_mongo_link" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"

  name              = "ls-cosmos-mongo"
  data_factory_id   = module.data_factory.id
  connection_string = var.cosmos_mongo_connection_string # regenerated in the account
}

✅ This works. Because the read never refreshes the field, state holds the last applied value and a change is a genuine in-place update — the opposite of the Synapse and Snowflake linked services.

8 · Drift you will never see
output "drift_blind_spot" {
  value = module.cosmos_mongo_link.the_read_never_refreshes_the_connection_string
}

⚠️ Constant true. Someone editing the connection string in the Data Factory portal produces no diff, ever. database IS refreshed, so drift in that is visible.

9 · Description, parameters and annotations
module "cosmos_mongo_link" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"

  name              = "ls-cosmos-mongo"
  data_factory_id   = module.data_factory.id
  connection_string = var.cosmos_mongo_connection_string

  description = "Analytics store, Mongo wire protocol"
  parameters  = { env = "prod" }
  annotations = ["owner:data-platform", "tier:gold"]
}

⚠️ annotations are not Azure resource tags — this resource supports none. They are an ordered list, so reordering them is a real diff.

10 · A private account through a self-hosted runtime
module "cosmos_mongo_link" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"

  name                     = "ls-cosmos-mongo"
  data_factory_id          = module.data_factory.id
  connection_string        = var.cosmos_mongo_connection_string
  integration_runtime_name = "ir-onprem-eastus"
}

ℹ️ Needed only where the account is reachable solely over a private endpoint. The runtime is referenced by name, so nothing proves it exists — a literal and a sibling module's name output are equally unverified here.

11 · Several linked services from one map
module "cosmos_mongo_links" {
  source   = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"
  for_each = var.cosmos_mongo_databases

  name              = each.key
  data_factory_id   = module.data_factory.id
  connection_string = each.value.connection_string
  database          = each.value.database
}

💡 Keys are the linked-service names, which is what datasets reference. Renaming a key destroys and recreates the record.

12 · Timeouts
timeouts = {
  create = "45m"
  read   = "10m"
  update = "45m"
  delete = "45m"
}

⚠️ These bound Terraform's wait on the Data Factory control-plane call, not any query against Cosmos DB. A fifth key would be silently discarded by object-type conversion, with no error at all.

13 · What this module refuses
# Refused offline, without credentials -- by THIS MODULE's own validations, which is
# the point: a provider schema rule fires only against a literal in a resource block,
# so none of the provider's own checks reaches you through a module call.
#   name              = "-.-"                 nothing but punctuation; mirrors the provider exactly
#   name              = ""                    Azure requires at least one character
#   data_factory_id   = "<a dataset id>"      anchored to the factory resource type
#   connection_string = ""                    omit the argument instead
#   connection_string = "/subscriptions/..."  that is a Resource ID, not a connection string
#
# NOT refused, and reported instead:
#   supplying no credential at all            reported by has_no_credential_at_all
#   name              = "1analytics"          a leading digit; the provider accepts it
#   name              = "mongo-ls"            a dash; the provider accepts it

💡 The provider accepts a credential-less linked service, so refusing it would reject legal input — and because a failed validation blocks terraform destroy too, it would strand anyone who already has one.

14 · Comparing against the SQL API sibling
output "sibling_difference" {
  value = module.cosmos_mongo_link.the_sql_api_sibling_has_an_endpoint_and_key_mode_with_pairing_traps
}

🔀 Constant true. azurerm_data_factory_linked_service_cosmosdb reaches the same account through the SQL API and adds an account_endpoint plus account_key mode — which the provider forbids combining with a connection string but does not require to appear together. This resource has one mode and none of those traps, and adds a server_version_is_32_or_higher toggle the sibling does not have.

15 · 🏗️ End-to-end composition
provider "azurerm" {
  features {}
}

module "resource_group" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-resource-group.git?ref=v1.0.0"

  name     = "rg-data-eastus"
  location = "eastus"
}

module "data_factory" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory.git?ref=v1.0.0"

  name                = "adf-corp-eastus"
  resource_group_name = module.resource_group.name
  location            = module.resource_group.location
}

module "cosmos_mongo_link" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-linked-service-cosmosdb-mongoapi.git?ref=v1.0.0"

  name                           = "ls-cosmos-mongo"
  data_factory_id                = module.data_factory.id
  connection_string              = var.cosmos_mongo_connection_string
  database                       = "analytics"
  server_version_is_32_or_higher = true

  description = "Analytics store, Mongo wire protocol"
  annotations = ["owner:data-platform"]
}

check "linked_service_can_connect" {
  assert {
    condition     = module.cosmos_mongo_link.credential_is_usable
    error_message = "This Cosmos DB MongoDB API linked service has no connection string and will fail at pipeline runtime."
  }
}

output "review" {
  value = {
    linked_service = module.cosmos_mongo_link.name
    mode           = module.cosmos_mongo_link.credential_mode
    drift_blind    = module.cosmos_mongo_link.the_read_never_refreshes_the_connection_string
    import_guard   = module.cosmos_mongo_link.the_import_guard_names_this_resource_correctly
  }
}

🏗️ import_guard is emitted on purpose: it reads false, and the one moment anyone needs that fact is the moment the provider tells them the wrong thing.


📥 Inputs

Required: name, data_factory_id. Optional: connection_string, database, server_version_is_32_or_higher, description, integration_runtime_name, parameters, annotations, additional_properties, timeouts.

Full schemas
variable "connection_string" {
  type      = string
  default   = null
  sensitive = true # marked by the PROVIDER here, unusually for this family
}

variable "server_version_is_32_or_higher" {
  type    = bool
  default = false # a DECLARATION about the account, not a setting applied to it
}

variable "timeouts" {
  type = object({
    create = optional(string)
    read   = optional(string)
    update = optional(string)
    delete = optional(string)
  })
  default = null
}

🧾 Outputs

Output Notes
id The Resource ID. Emitted first.
name What datasets reference.
data_factory_id, data_factory_name, resource_group_name, subscription_id Parent context.
credential_mode One of two.
credential_is_usable A connection string was supplied.
has_no_credential_at_all The empty call.
the_provider_requires_no_credential_at_all Constant.
the_import_guard_names_this_resource_correctly false — a provider defect.
server_version_is_32_or_higher The declared version.
the_server_version_flag_declares_reality_it_does_not_change_it Constant.
database The refreshed field.
the_read_never_refreshes_the_connection_string Constant.
the_credential_is_rotatable_because_the_read_ignores_it Constant.
force_new_fields, fields_that_can_change_after_creation Lifecycle.
the_sql_api_sibling_has_an_endpoint_and_key_mode_with_pairing_traps Constant.

No output emits a secret. The full list, with the reasoning behind each constant, is in outputs.tf.


🧠 Architecture Notes

The credential is optional and nothing replaces it. connection_string is the only credential this resource accepts, there is no ExactlyOneOf and no AtLeastOneOf, and the empty call applies cleanly. Nothing fails until a pipeline runs, so the module reports the state at plan time and refuses nothing — the provider accepts a credential-less linked service, and a failed validation {} would block terraform destroy for anyone who already has one.

The requires-import error names the wrong resource, and that is a provider defect. It says azurerm_data_factory_linked_service_cosmosdb, the SQL API sibling. The importer's own type check is correct, so the resource imports fine once you know what to type — but the message is wrong in exactly the moment someone is reading it under pressure. This module emits the_import_guard_names_this_resource_correctly as a deliberate false, which is unusual in this library and is the honest way to carry a fact that only surfaces at the worst time.

Drift in the credential is invisible; drift in the database name is not. The read sets database and the metadata but never connection_string. That asymmetry is also what makes the credential rotatable — state holds the last applied value, so a change is a real in-place update.

server_version_is_32_or_higher reads like a toggle and is a declaration. It tells Data Factory which wire protocol to speak. It does not upgrade the account, and nothing checks it against reality.

Renaming is destructive in a way the plan does not show. Datasets select a linked service by name and nothing points back from here to them.

Sensitivity is contagious, including to scalars. Every derived flag in main.tf is unwrapped individually with nonsensitive(); wrapping a constructed collection would not be enough, because element-level marks persist.


🧱 Design Principles

Concern This module's default Opt-out
Credential none assumed; the state is reported at plan supply connection_string
Connection string in plan output marked sensitive by the provider itself none — and it does not affect state
Emitting the credential never emitted; presence and mode only none
A provider-legal configuration never refused none needed
A provider defect emitted as a false output, not buried in prose none

The secure-by-default rule cannot fully apply: no credential is required, so the empty call is safe and useless at the same time. The module compensates by reporting the state at plan time.


🚀 Runbook

terraform init -backend=false
terraform validate
terraform fmt -check

Pin ?ref=v1.0.0, never a branch. Plan-only; a human applies from CI.


🧪 Testing

terraform validate exercises the module's twelve validations and the provider's schema validators — all offline, without credentials. This resource has no CustomizeDiff and no ConflictsWith at all, so nothing fires only at plan.

What validate cannot reach: whether the Cosmos DB account exists, whether the MongoDB API is enabled on it, whether the connection string works, whether the database exists, whether the named integration runtime exists, and whether the declared server version matches the account. None of those is checked by anything until a pipeline runs.


💬 Example Output

id                                            = "/subscriptions/.../factories/adf-corp-eastus/linkedservices/ls-cosmos-mongo"
name                                          = "ls-cosmos-mongo"
credential_mode                               = "connection_string"
credential_is_usable                          = true
has_no_credential_at_all                      = false
database                                      = "analytics"
server_version_is_32_or_higher                = true
the_import_guard_names_this_resource_correctly = false
force_new_fields                              = ["name", "data_factory_id"]

🔍 Troubleshooting

Symptom Cause Fix
A requires-import error naming azurerm_data_factory_linked_service_cosmosdb A provider defect — the message names the SQL API sibling instead of this resource. Import azurerm_data_factory_linked_service_cosmosdb_mongoapi, not what the message says.
connection_string must not be an empty or whitespace-only string. "" was passed to mean "none". Omit the argument — and note that omitting it leaves no credential at all.
connection_string looks like an Azure Resource ID. The Cosmos DB account's id was passed. Pass the connection string.
name must not consist ENTIRELY of the characters ... The name is nothing but punctuation. This mirrors the provider's real rule exactly. Use a name with at least one other character.
A name with a leading digit or a dash was expected to be refused and was not Neither the provider nor this module refuses those, by design. Read name_does_not_start_with_a_letter — Azure's own naming pages disagree on the leading character, and nothing in Terraform enforces either.
must be a Data Factory Resource ID A dataset, linked service or pipeline ID was passed. Pass the factory's id.
Pipeline fails to authenticate, but apply succeeded No credential is required; nothing here contacts Cosmos DB. Check credential_mode — it may be NO_CREDENTIAL_AT_ALL.
Pipeline connects but the wire protocol is wrong server_version_is_32_or_higher declares a version nothing verifies. Match it to the account.
A portal change to the connection string never appears in a plan The read never refreshes that field. Expected. Read the Data Factory directly.
Datasets break after a rename The record was replaced; datasets reference it by name. Update the datasets, or avoid renaming.

🔗 Related Docs


💙 "Infrastructure as Code should be standardized, consistent, and secure."