☁️ Azure Data Factory Flowlet Terraform Module

Manages one azurerm_data_factory_flowlet_data_flow — the reusable fragment of mapping-data-flow logic that other data flows embed. Targets hashicorp/azurerm ~> 4.0.

• 🎯 Creates one flowlet inside an existing Data Factory.
• 🧩 A flowlet is a fragment, not a flow: it runs as part of whichever mapping data flow embeds it.
• 📥 flow_source and sink are optional here — a flowlet may take both from the embedding flow, and usually does.
• 📝 Takes the Data Flow Script as either a single script string or an ordered script_lines list; at least one is required.
• 🔗 May embed other flowlets by name — and nothing prevents a cycle.
• 🚫 Carries no tags — the resource exposes none.

💡 Why it matters: a flowlet exists to be reused, so its blast radius is larger than a data flow's, not smaller. Several flows may embed it, each naming it as a plain string, and none of those references is a Terraform dependency. Renaming or destroying this resource applies cleanly and leaves every embedding flow failing at run time with no plan-time signal anywhere. That reuse is also why the provider makes source and sink optional here and required on the mapping data flow — the one schema difference between two otherwise near-identical resources.

If this module saves you time:

• ⭐ Star the repository
• 💼 Connect on LinkedIn
• ☕ Buy me a coffee

flowchart TB
  RG["azurerm_resource_group"]
  ADF["azurerm_data_factory"]
  DS["azurerm_data_factory_dataset_*"]
  LS["azurerm_data_factory_linked_service_*"]
  PL["azurerm_data_factory_pipeline"]
  DF["data_flow"]
  FL["flowlet_data_flow"]

  RG -->|"resource group"| ADF
  ADF -->|"data_factory_id"| DF
  ADF -->|"data_factory_id"| FL
  ADF -->|"data_factory_id"| DS
  ADF -->|"data_factory_id"| LS
  ADF -->|"data_factory_id"| PL
  DS -.->|"dataset by NAME"| DF
  LS -.->|"linked_service by NAME"| DF
  FL -.->|"flowlet by NAME, no dependency edge"| DF
  FL -.->|"a flowlet may embed another, CYCLE possible"| FL
  DF -.->|"invoked by NAME from Execute Data Flow"| PL

  classDef me fill:#0078D4,stroke:#004578,color:#ffffff
  classDef key fill:#004578,stroke:#002b47,color:#ffffff
  classDef ext fill:#F0F3F6,stroke:#9AA5B1,color:#1F2933
  class DF,FL me
  class ADF key
  class RG,DS,LS,PL ext

Every dashed edge is a name, not a reference — including the self-edge, because a flowlet may embed another flowlet and nothing prevents a cycle.

flowchart TB
  VN["name, data_factory_id -- the ONLY two force-new fields"]
  VS["script OR script_lines -- AtLeastOneOf, both legal together"]
  VSRC["flow_source -- OPTIONAL here, REQUIRED on the mapping data flow"]
  VSNK["sink -- OPTIONAL here"]
  VT["transformation -- OPTIONAL, and only THREE reference blocks"]
  R["azurerm_data_factory_flowlet_data_flow.this"]
  EMB["Embedded BY NAME by every consuming data flow"]
  OID["id, name, script_form"]
  OFLAG["defines_no_source_or_sink, referenced_flowlet_names"]
  OWARN["destroying_this_flowlet_breaks_every_data_flow_that_embeds_it"]

  VN --> R
  VS --> R
  VSRC --> R
  VSNK --> R
  VT --> R
  R -->|"no dependency edge in either direction"| EMB
  R --> OID
  R --> OFLAG
  R --> OWARN

  classDef me fill:#0078D4,stroke:#004578,color:#ffffff
  classDef key fill:#004578,stroke:#002b47,color:#ffffff
  classDef ext fill:#F0F3F6,stroke:#9AA5B1,color:#1F2933
  class R key
  class EMB me
  class VN,VS,VSRC,VSNK,VT,OID,OFLAG,OWARN ext

Schema notes that bite — each verified against the provider source at the pinned line:

• 🔴 source and sink are OPTIONAL here and REQUIRED on the mapping data flow. The provider uses SchemaForDataFlowletSourceAndSink here and SchemaForDataFlowSourceAndSink there — the same Elem, a different cardinality. That is the entire schema difference between the two resources.
• A flowlet with neither is legal and usual — it takes both from the embedding flow.
• Everything is referenced by name, in both directions, including this flowlet from the flows that embed it.
• Nothing prevents a cycle between two flowlets that embed each other.
• Nothing validates the script beyond StringIsNotEmpty.
• script and script_lines carry a mutual AtLeastOneOf; supplying both is legal and unresolved.
• script_lines is ordered and the order is the program.
• 🔴 source/sink and transformation are NOT the same shape — five reference blocks versus three.
• 🔴 An undeclared key on transformation is silently discarded — no error at validate, plan or apply.
• Only two force-new fields: name, data_factory_id.
• source is a RESERVED Terraform variable name, so this module's input is flow_source.
• No CustomizeDiff, no version gate, no tags.

⚠️ This module grants nothing and runs nothing. A flowlet is a definition; it executes with the Data Factory's managed identity as part of an embedding flow.

🔒 Write access to a shared flowlet is write access to every flow that embeds it. That is the difference from a data flow, whose blast radius is one flow. A flowlet exists to be reused, so a change here propagates to consumers that nothing in Terraform records.

• Microsoft.DataFactory registered in the subscription.
• An existing Data Factory.
• The datasets, linked services and other flowlets named anywhere in this flowlet must already exist. Nothing checks this at any Terraform stage.
• At least one mapping data flow that embeds this flowlet, if it is ever to run. A flowlet nothing embeds is inert and produces no error of any kind.

terraform-azurerm-data-factory-flowlet-data-flow/
├── providers.tf    # required_version + the pinned azurerm; no provider block
├── variables.tf    # 11 typed inputs, 14 validations
├── main.tf         # locals + the single keystone resource
├── outputs.tf      # 47 outputs; id first, then name, then the derived facts
├── README.md       # this file
├── SCOPE.md        # the cross-module contract
├── LICENSE         # MIT
└── .gitignore

provider "azurerm" {
  features {}
}

module "standardise_address" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-standardise-address"
  data_factory_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-data-eastus/providers/Microsoft.DataFactory/factories/adf-corp"

  script = file("${path.module}/flows/standardise_address.dfs")

  # No source and no sink: this fragment takes both from the flow that embeds it.
}

💡 That really is the whole call. A flowlet with neither a source nor a sink is the common case, and the mapping data flow sibling cannot be created that way at all.

⚠️ The input for sources is flow_source, not source — source is a meta-argument of the module block itself.

Consumes

Emits

module "standardise_address" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-standardise-address"
  data_factory_id = var.data_factory_id

  script = file("${path.module}/flows/standardise_address.dfs")
}

module "with_source" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-reference-lookup"
  data_factory_id = var.data_factory_id

  script = file("${path.module}/flows/reference_lookup.dfs")

  flow_source = {
    "countryCodes" = { dataset = { name = "ds-country-codes" } }
  }
}

module "dedupe" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-dedupe"
  data_factory_id = var.data_factory_id

  script = file("${path.module}/flows/dedupe.dfs")

  transformation = {
    "aggregateByKey" = {}
    "pickLatest"     = {}
  }
}

module "line_form" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-line-form"
  data_factory_id = var.data_factory_id

  script_lines = [
    "input1 derive(country = upper(country)) ~> upperCountry",
    "upperCountry output() ~> output1",
  ]
}

module "address_pipeline" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-address-pipeline"
  data_factory_id = var.data_factory_id

  script = file("${path.module}/flows/address_pipeline.dfs")

  transformation = {
    "standardise" = {
      flowlet = { name = "fl-standardise-address" }
    }
    "validate" = {
      flowlet = { name = "fl-validate-address" }
    }
  }
}

module "inline_source" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-inline-source"
  data_factory_id = var.data_factory_id

  script = file("${path.module}/flows/inline.dfs")

  flow_source = {
    "rawFiles" = { linked_service = { name = "ls-datalake" } }
  }
}

module "with_extra_refs" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-strict"
  data_factory_id = var.data_factory_id

  script = file("${path.module}/flows/strict.dfs")

  flow_source = {
    "incoming" = {
      dataset               = { name = "ds-incoming" }
      schema_linked_service = { name = "ls-schema-store" }
    }
  }

  sink = {
    "checked" = {
      dataset                 = { name = "ds-checked" }
      rejected_linked_service = { name = "ls-quarantine" }
    }
  }
}

module "both_forms" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-both-forms"
  data_factory_id = var.data_factory_id

  script       = file("${path.module}/flows/base.dfs")
  script_lines = ["input1 output() ~> output1"]
}

module "documented" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-standardise-address"
  data_factory_id = var.data_factory_id

  script = file("${path.module}/flows/standardise_address.dfs")

  description = "Uppercases country codes and trims postal codes."
  folder      = "shared/fragments"
  annotations = ["owner:data-platform", "shared:true"]
}

locals {
  fragments = ["standardise-address", "validate-email", "dedupe"]
}

module "shared" {
  source   = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"
  for_each = toset(local.fragments)

  name            = "fl-${each.value}"
  data_factory_id = var.data_factory_id
  folder          = "shared/fragments"

  script = file("${path.module}/flows/${each.value}.dfs")
}

output "flowlet_review" {
  value = {
    takes_from_parent = module.standardise_address.defines_no_source_or_sink
    sources           = module.standardise_address.source_names
    embeds_flowlets   = module.standardise_address.referenced_flowlet_names
    datasets          = module.standardise_address.referenced_dataset_names
    script_form       = module.standardise_address.script_form
    replace_on        = module.standardise_address.force_new_fields
  }
}

# There is NO output for "which data flows embed this flowlet", because this
# module cannot know. Embedding flows name the flowlet as a plain string, so
# nothing in Terraform records the relationship in that direction.
#
# The honest substitute is a naming and annotation convention the caller owns:

module "standardise_address" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-standardise-address"
  data_factory_id = var.data_factory_id
  folder          = "shared/fragments"

  script = file("${path.module}/flows/standardise_address.dfs")

  annotations = [
    "shared:true",
    "embedded-by:df-clean-customers",
    "embedded-by:df-clean-suppliers",
  ]
}

provider "azurerm" {
  features {}
}

module "resource_group" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-resource-group.git?ref=v1.0.0"

  name     = "rg-data-eastus"
  location = "eastus"
}

module "data_factory" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory.git?ref=v1.0.0"

  name                = "adf-corp"
  resource_group_name = module.resource_group.name
  location            = module.resource_group.location

  identity = {
    type = "SystemAssigned"
  }
}

module "standardise_address" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-flowlet-data-flow.git?ref=v1.0.0"

  name            = "fl-standardise-address"
  data_factory_id = module.data_factory.id
  folder          = "shared/fragments"

  script      = file("${path.module}/flows/standardise_address.dfs")
  description = "Uppercases country codes and trims postal codes."
  annotations = ["shared:true"]
}

module "clean_customers" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-data-factory-data-flow.git?ref=v1.0.0"

  name            = "df-clean-customers"
  data_factory_id = module.data_factory.id

  script = file("${path.module}/flows/clean_customers.dfs")

  flow_source = { "customersRaw" = { dataset = { name = "ds-customers-raw" } } }

  transformation = {
    "standardise" = {
      flowlet = { name = module.standardise_address.name }
    }
  }

  sink = { "customersClean" = { dataset = { name = "ds-customers-clean" } } }
}

output "flowlet_name" {
  value = module.standardise_address.name
}

Required: name, data_factory_id, and at least one of script / script_lines.

Optional: flow_source, sink, transformation, description, folder, annotations, timeouts.

Nothing is sensitive: this resource carries no credential. There is deliberately no "which flows embed this" output — see Example 12.

A flowlet's blast radius is larger than a data flow's. It exists to be reused, and every consumer names it as a plain string. Nothing in Terraform records that relationship, so destroying this resource — or renaming it, which is a destroy and a create — applies cleanly and leaves every embedding flow failing at run time. The module states this in destroying_this_flowlet_breaks_every_data_flow_that_embeds_it rather than leaving a reader to infer it from the data flow module's gentler equivalent.

The one schema difference is cardinality. The provider gives this resource SchemaForDataFlowletSourceAndSink (Optional) and the mapping data flow SchemaForDataFlowSourceAndSink (Required) — the same Elem, a different cardinality. The two provider resource files are otherwise near-identical, right down to sharing the transformation helper, which is exactly why this had to be read rather than assumed. A flowlet with neither source nor sink is the common shape; defines_no_source_or_sink reports it.

Composition is permitted and cycles are not prevented. A flowlet may embed another flowlet by name. Two that embed each other apply cleanly and fail only when a flow using either runs. This module emits referenced_flowlet_names so the outbound half is visible; it cannot see the inbound half, and does not pretend to — claiming cycle detection would be a protective claim that does not exist.

The three blocks look alike and are not the same shape. source and sink carry five reference blocks; transformation carries three. And because Terraform's object-type conversion silently discards undeclared keys, putting schema_linked_service on a transformation produces no error at validate, plan or apply — captured by running the case, not inferred.

Nothing validates the script, here or on the sibling. The failure surfaces when an embedding flow executes, which on a shared fragment may be in a pipeline nobody associated with this change.

source is reserved. It is a module block meta-argument, so the input is flow_source; the rendered resource still emits a source block exactly as the provider defines it.

features {} dependence. As with every module in this suite, the caller supplies provider "azurerm" { features {} }.

Two rules govern the validations. Never invent a constraint that could reject legal input — a validation {} failure blocks terraform destroy as well as apply. And enforce a configuration rule, report a service or runtime one — which is why this module refuses exactly where the provider refuses, and nowhere else. Notably it does not require a source or a sink, because the provider does not; the mapping data flow module does, because there it does.

terraform init -backend=false
terraform validate
terraform fmt -check

Pin the module with ?ref=v1.0.0 — never a branch. This module is plan-only; a human applies from CI.

Know which command reaches which check — the three stages are not interchangeable. terraform validate on a configuration that CALLS this module checks types and syntax only: it evaluates none of the module's variable values. The 14 validation {} blocks and the provider's AtLeastOneOf are reached at terraform plan — and neither needs credentials, because variable validation runs before the provider is configured. To exercise them without a plan at all, run the module as the root module and drive it through terraform console -var-file=....

Because this resource has no CustomizeDiff and no version gate, nothing is deferred to plan by the provider.

What no Terraform stage exercises at all: whether the script is valid, whether any referenced dataset, linked service or flowlet exists, whether a composition cycle exists, and whether anything embeds this flowlet at all. A flowlet nothing embeds applies cleanly, costs nothing and does nothing, and no error is produced anywhere.

$ terraform output
id                              = "/subscriptions/.../factories/adf-corp/dataflows/fl-standardise-address"
name                            = "fl-standardise-address"
data_factory_name               = "adf-corp"
defines_no_source_or_sink       = true
source_names                    = []
sink_names                      = []
transformation_names            = ["standardise"]
referenced_flowlet_names        = []
referenced_dataset_names        = []
script_form                     = "script"
folder                          = "shared/fragments"
force_new_fields                = ["name", "data_factory_id"]

• azurerm_data_factory_flowlet_data_flow
• azurerm_data_factory_data_flow
• Microsoft Learn — Flowlets in mapping data flows
• Microsoft Learn — Data flow script (DFS)
• Sibling modules: terraform-azurerm-data-factory-data-flow, terraform-azurerm-data-factory
• This module's SCOPE.md

💙 "Infrastructure as Code should be standardized, consistent, and secure."