🟧 AWS Kinesis Firehose (Amazon Data Firehose) Terraform Module

Secure-by-default Amazon Data Firehose delivery stream — one validated destination enum (S3 / extended S3 / Redshift / OpenSearch / HTTP endpoint), Direct PUT or Kinesis Data Stream sourcing, server-side encryption ON by default, and opt-in CloudWatch delivery-error logging, all from a single composite call. Built for the AWS provider v6.x.

• 🚚 Provisions a single aws_kinesis_firehose_delivery_stream keystone — "composite" by schema complexity, not resource count: one caller-selectable, multi-destination surface.
• 🎯 destination is a validated enum — s3 | extended_s3 | redshift | opensearch | http_endpoint — each backed by a deeply-typed object variable, rendered via a dynamic block so mutual exclusivity is structural, not just documented.
• 🌊 Source is direct_put (default) or kinesis_stream_source, a module-level modeling convenience over the presence/absence of kinesis_source_configuration — there is no literal "DirectPut" value on the AWS API itself.
• 🔒 Server-side encryption ON by default for non-Kinesis-stream sources — AWS-owned key unless a customer-managed CMK is supplied — and this module rejects the combination of SSE + a Kinesis Data Stream source at plan time (AWS guidance: don't combine them).
• 📝 CloudWatch delivery-error logging ON by default whenever a log group is supplied on the active destination's cloudwatch_logging_options.
• 🛡️ Backup / error-record capture defaults tuned per destination — Redshift defaults to s3_backup_mode = "Enabled" (overriding AWS's own "Disabled" default); OpenSearch and HTTP endpoint already default to capturing failed records.
• 🧷 iam_role_arn is a required input, never created here — this module never manages IAM; it always injects the one caller-supplied delivery role ARN as role_arn into whichever destination/source block is active.
• 🏷️ Universal tagging — var.tags flows to the one resource; tags_all surfaced as an output.

💡 Why it matters: Amazon Data Firehose is the load-bearing pipe between event producers (CloudWatch Logs, WAF, VPC Flow Logs, custom apps, Kinesis Data Streams) and durable storage/analytics destinations. Getting encryption, error-capture, and the least-privilege iam:PassRole wiring right here closes off a real data-loss and privilege-escalation surface for a regulated FI handling PII.

If these Terraform modules have been helpful to you or your organization, I'd appreciate your support in any of the following ways:

• ⭐ Star this repository to help others discover this Terraform module.
• 🤝 Connect with me on LinkedIn: linkedin.com/in/microsoftexpert
• ☕ Buy me a coffee: buymeacoffee.com/microsoftexpert

Whether it's a star, a professional connection, or a coffee, every gesture helps keep these modules actively maintained and continually improving. Thank you for being part of the community!

terraform-aws-kinesis-firehose is a Phase 2 (App Integration & Messaging) module. It is a pure consumer — it creates no IAM role, bucket, key, log group, or stream of its own; every one of those is wired in by ARN from a Phase 1 foundation module.

flowchart LR
 iam["terraform-aws-iam-role"]
 s3["terraform-aws-s3-bucket"]
 kms["terraform-aws-kms"]
 cwlg["terraform-aws-cloudwatch-log-group"]
 kstream["terraform-aws-kinesis-stream"]
 fh["terraform-aws-kinesis-firehose"]
 sqs["terraform-aws-sqs"]
 kfh2["terraform-aws-kinesis-firehose (sibling stream)"]
 msk["terraform-aws-msk"]
 mq["terraform-aws-mq"]

 iam -- "iam_role_arn (REQUIRED)" --> fh
 s3 -- "bucket_arn (primary / backup / error target)" --> fh
 kms -- "kms_key_arn (optional CMK, SSE)" --> fh
 cwlg -- "log_group_name (optional, delivery-error logging)" --> fh
 kstream -- "kinesis_source_stream_arn (optional source)" --> fh

 subgraph phase2["Phase 2 -- App Integration & Messaging"]
 fh
 sqs
 kfh2
 msk
 mq
 end

 style fh fill:#FF9900,color:#fff,stroke:#cc7a00,stroke-width:2px

ℹ️ terraform-aws-redshift (Phase 1 database family) and an OpenSearch module are not yet authored in this library — redshift_configuration.cluster_jdbcurl and opensearch_configuration.domain_arn are consumed as opaque caller-supplied strings until those modules exist.

flowchart TB
 subgraph FHMOD["terraform-aws-kinesis-firehose"]
 fh["aws_kinesis_firehose_delivery_stream.this<br/>keystone -- exactly one resource"]

 subgraph SRC["Source -- mutually exclusive, gated by var.source_type"]
 direct["Direct PUT<br/>no block rendered"]
 ksrc["kinesis_source_configuration<br/>kinesis_stream_source"]
 end

 subgraph SSE["Encryption"]
 sse["server_side_encryption<br/>ON by default, AWS-owned or CMK"]
 end

 subgraph DEST["Destination -- mutually exclusive, gated by var.destination"]
 exts3["extended_s3_configuration<br/>serves both s3 and extended_s3"]
 rs["redshift_configuration"]
 os["opensearch_configuration"]
 http["http_endpoint_configuration"]
 end

 subgraph SUB["Common nested sub-blocks per active destination"]
 s3cfg["s3_configuration<br/>required staging or target bucket"]
 backup["s3_backup_configuration<br/>extended_s3 and redshift only"]
 cwlo["cloudwatch_logging_options<br/>delivery-error logging"]
 proc["processing_configuration<br/>processors and parameters"]
 end

 fh --> SRC
 fh --> SSE
 fh --> DEST
 exts3 --> SUB
 rs --> SUB
 os --> SUB
 http --> SUB
 end

 style fh fill:#FF9900,color:#fff,stroke:#cc7a00,stroke-width:2px

No provider {} block is declared inside the module — the caller's configured provider (and its region/credentials) is inherited. No region variable — Firehose is a regional service with no us-east-1 global-resource coupling.

Least-privilege actions the Terraform identity needs. This is distinct from the runtime permissions the delivery role itself needs (granted on that role's own policy in terraform-aws-iam-role, not here).

⚠️ iam:PassRole is mandatory and must be scoped. Every CreateDeliveryStream / UpdateDestination call passes iam_role_arn to the firehose.amazonaws.com service principal. The Terraform identity's own policy must grant iam:PassRole on that specific role ARN (Resource = iam_role_arn), ideally further constrained with an iam:PassedToService = "firehose.amazonaws.com" condition key. Granting iam:PassRole on "*" lets the Terraform identity hand Firehose — or any future resource type it manages — any role in the account, including highly privileged ones. This is a textbook privilege-escalation path and is explicitly disallowed by our least-privilege posture.

ℹ️ No service-linked role is created or required for aws_kinesis_firehose_delivery_stream itself.

• No service-linked role for the delivery stream itself. (Some producers that feed Firehose — e.g. VPC/WAF/Network Firewall logging — may create their own service-linked roles; out of scope for this module.)
• IAM role trust policy (mandatory, caller-owned). The role passed as iam_role_arn MUST have a trust policy allowing the firehose.amazonaws.com service principal to sts:AssumeRole, and per AWS guidance SHOULD include an sts:ExternalId condition equal to the delivery-stream owner's AWS account ID to protect against the confused-deputy problem. This module does not create or validate that trust policy — author it in terraform-aws-iam-role and pass in the resulting arn.
• Kinesis Data Stream source. The delivery role's policy must additionally grant kinesis:DescribeStream, kinesis:GetShardIterator, kinesis:GetRecords, and kinesis:ListShards on the source stream (granted on the delivery role's own policy, not here).
• Destination-specific prerequisites:
• S3 / extended_s3: the bucket must exist; the delivery role's policy must grant s3:AbortMultipartUpload, s3:GetBucketLocation, s3:GetObject, s3:ListBucket, s3:ListBucketMultipartUploads, s3:PutObject, plus kms:GenerateDataKey / kms:Decrypt if the bucket uses SSE-KMS.
• Redshift: delivered via an S3 staging COPY, so the delivery role needs both S3 access AND Redshift JDBC/network reachability from Firehose's managed IPs (or VPC access for a private cluster). Prefer secrets_manager_configuration over inline username/password, and scope the Redshift-side grant to INSERT only.
• OpenSearch: a VPC-attached domain requires the delivery role to include ec2:CreateNetworkInterface / ec2:DescribeNetworkInterfaces / ec2:DeleteNetworkInterface permissions, and Firehose needs spare ENI capacity in the target subnets; a public domain only needs the domain's access policy to allow the delivery role.
• HTTP endpoint: the target must accept Firehose's request/response contract (including the endpoint's own access-key or Secrets-Manager-backed auth, if used).
• Quotas (per Region, soft unless noted):
• Firehose streams per account: 5,000 in us-east-1 / us-east-2 / us-west-2 / eu-west-1 / ap-northeast-1; 2,000 in several other Regions; 500 and 100 tiers for the remainder — see the AWS Firehose Quota page for the exact per-Region tier.
• Direct PUT combined PutRecord/PutRecordBatch throughput: 500,000 records/sec, 2,000 requests/sec, 5 MiB/sec in us-east-1 / us-west-2 / eu-west-1; 100,000 records/sec, 1,000 requests/sec, 1 MiB/sec elsewhere. This quota does not apply when the source is a Kinesis Data Stream — Firehose scales with the stream.
• Dynamic partitioning (extended_s3 only): 500 active partitions per stream by default, 1 GB/sec max throughput per active partition.
• Buffer hints are bounded per destination by AWS min/max (documented per field in variables.tf).

terraform-aws-kinesis-firehose/
├── providers.tf # terraform{} + required_providers (aws >= 6.0, < 7.0); no provider block
├── variables.tf # name, destination, iam_role_arn, source, SSE, per-destination objects, tags, timeouts
├── main.tf # locals (s3 -> extended_s3 normalization) + aws_kinesis_firehose_delivery_stream.this
├── outputs.tf # id + arn, name, destination, destination_id, version_id, tags_all
├── README.md # this file
└── SCOPE.md # in/out-of-scope, IAM, prerequisites, emits, gotchas, design decisions

module "firehose" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-app-events"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn # terraform-aws-iam-role

  extended_s3_configuration = {
    bucket_arn = module.events_bucket.arn # terraform-aws-s3-bucket
  }

  tags = {
    Environment = "prod"
    App         = "lending-portal"
  }
}

⚠️ Pin the source with ?ref=v1.0.0 — never a branch. iam_role_arn has no default; author the delivery role's trust policy and IAM permissions in terraform-aws-iam-role first.

module "firehose" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-clickstream"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn # terraform-aws-iam-role

  extended_s3_configuration = {
    bucket_arn = module.clickstream_bucket.arn # terraform-aws-s3-bucket
  }
}

# destination = "s3" is honored for interface completeness, but the live AWS
# provider schema has no distinct s3_configuration destination block anymore
# -- this module renders the same object onto extended_s3_configuration under
# the hood and stores destination = "extended_s3" on the real resource. See
# Architecture Notes. New streams should use destination = "extended_s3".
module "firehose_legacy" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-legacy-export"
  destination  = "s3"
  iam_role_arn = module.firehose_role.arn

  s3_configuration = {
    bucket_arn = module.export_bucket.arn
  }
}

module "firehose_from_stream" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-stream-to-s3"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  source_type               = "kinesis_stream_source"
  kinesis_source_stream_arn = module.orders_stream.arn # terraform-aws-kinesis-stream

  # AWS guidance: SSE should not be combined with a Kinesis Data Stream source
  # -- the module enforces this with a plan-time validation error.
  enable_server_side_encryption = false

  extended_s3_configuration = {
    bucket_arn = module.orders_bucket.arn
  }
}

module "firehose_redshift" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-redshift-loader"
  destination  = "redshift"
  iam_role_arn = module.firehose_role.arn

  redshift_configuration = {
    cluster_jdbcurl = "jdbc:redshift://casey-cluster.abc123.us-east-1.redshift.amazonaws.com:5439/analytics"
    data_table_name = "events"

    secrets_manager_configuration = {
      enabled    = true
      secret_arn = module.redshift_secret.arn # terraform-aws-secrets-manager
    }

    s3_configuration = {
      bucket_arn = module.redshift_staging_bucket.arn
    }

    # s3_backup_mode defaults to "Enabled" here (secure default) --
    # s3_backup_configuration is REQUIRED unless overridden to "Disabled".
    s3_backup_configuration = {
      bucket_arn = module.redshift_backup_bucket.arn
    }
  }
}

module "firehose_opensearch" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-logs-to-opensearch"
  destination  = "opensearch"
  iam_role_arn = module.firehose_role.arn

  opensearch_configuration = {
    domain_arn = "arn:aws:es:us-east-1:123456789012:domain/casey-logs"
    index_name = "app-logs"

    s3_configuration = {
      bucket_arn = module.opensearch_backup_bucket.arn
    }
  }
}

module "firehose_opensearch_vpc" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-logs-to-opensearch-vpc"
  destination  = "opensearch"
  iam_role_arn = module.firehose_role.arn # must also grant ec2:*NetworkInterface* -- see AWS Prerequisites

  opensearch_configuration = {
    domain_arn = "arn:aws:es:us-east-1:123456789012:domain/casey-logs-vpc" # opaque caller-supplied ARN -- no terraform-aws-opensearch module exists yet
    index_name = "app-logs"

    vpc_config = {
      subnet_ids         = values(module.vpc.private_subnet_ids) # terraform-aws-vpc
      security_group_ids = [module.opensearch_sg.id]             # terraform-aws-security-group
    }

    s3_configuration = {
      bucket_arn = module.opensearch_backup_bucket.arn
    }
  }
}

module "firehose_http" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-metrics-to-datadog"
  destination  = "http_endpoint"
  iam_role_arn = module.firehose_role.arn

  http_endpoint_configuration = {
    url  = "https://http-intake.logs.datadoghq.com/api/v2/logs"
    name = "datadog"

    secrets_manager_configuration = {
      enabled    = true
      secret_arn = module.datadog_api_key_secret.arn # terraform-aws-secrets-manager
    }

    s3_configuration = {
      bucket_arn = module.http_backup_bucket.arn
    }

    request_configuration = {
      content_encoding = "GZIP"
      common_attributes = [
        { name = "env", value = "prod" }
      ]
    }
  }
}

# Caller's provider block owns default_tags; resource tags win on key conflict.
provider "aws" {
  region = "us-east-1"
  default_tags {
    tags = { Owner = "platform", ManagedBy = "terraform" }
  }
}

module "firehose_tagged" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-tagged-stream"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  extended_s3_configuration = {
    bucket_arn = module.events_bucket.arn
  }

  tags = {
    Environment = "prod" # resource tag -- wins over default_tags on key conflict
    DataClass   = "internal"
  }
}
# module.firehose_tagged.tags_all == { Owner, ManagedBy, Environment, DataClass }

module "firehose_cmk" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-cmk-stream"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  kms_key_arn = module.firehose_kms.arn # terraform-aws-kms -- key policy must allow this account's Terraform identity + the delivery role

  extended_s3_configuration = {
    bucket_arn = module.events_bucket.arn
  }
}

# Opting out of the secure baseline. Only acceptable with a documented
# exception (e.g. a non-PII, ephemeral CI/sandbox stream where the cost of
# SSE key operations outweighs the risk). Never do this for PII-bearing data.
module "firehose_no_sse" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-sandbox-nosse"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  enable_server_side_encryption = false # DOCUMENTED EXCEPTION: non-PII sandbox data only

  extended_s3_configuration = {
    bucket_arn = module.sandbox_bucket.arn
  }
}

module "firehose_logged" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-logged-stream"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  extended_s3_configuration = {
    bucket_arn = module.events_bucket.arn

    # Supplying log_group_name turns delivery-error logging ON.
    cloudwatch_logging_options = {
      log_group_name = module.firehose_log_group.name # terraform-aws-cloudwatch-log-group
    }
  }
}

module "firehose_partitioned" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-partitioned-events"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  extended_s3_configuration = {
    bucket_arn = module.events_bucket.arn
    prefix     = "customer_id=!{partitionKeyFromQuery:customer_id}/"

    dynamic_partitioning_configuration = {
      enabled = true
    }

    processing_configuration = {
      enabled = true
      processors = [
        {
          type = "MetadataExtraction"
          parameters = [
            { parameter_name = "MetadataExtractionQuery", parameter_value = "{customer_id:.customer_id}" },
            { parameter_name = "JsonParsingEngine", parameter_value = "JQ-1.6" },
          ]
        }
      ]
    }
  }
}

locals {
  event_streams = {
    orders    = module.orders_bucket.arn
    shipments = module.shipments_bucket.arn
    returns   = module.returns_bucket.arn
  }
}

module "firehose_streams" {
  source   = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"
  for_each = local.event_streams

  name         = "casey-${each.key}-events"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  extended_s3_configuration = {
    bucket_arn = each.value
  }

  tags = { Domain = each.key }
}

import {
  to = module.firehose.aws_kinesis_firehose_delivery_stream.this
  id = "arn:aws:firehose:us-east-1:123456789012:deliverystream/casey-app-events"
}

module "firehose" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-app-events"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  extended_s3_configuration = {
    bucket_arn = module.events_bucket.arn
  }
}

# Customer-managed CMK for both the source stream and the delivery stream's SSE
module "firehose_kms" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kms?ref=v1.0.0"
  name   = "casey-firehose"
}

# Destination bucket
module "events_bucket" {
  source      = "git::https://github.com/microsoftexpert/terraform-aws-s3-bucket?ref=v1.0.0"
  name        = "casey-app-events-prod"
  kms_key_arn = module.firehose_kms.arn
}

# Delivery-error log group
module "firehose_log_group" {
  source      = "git::https://github.com/microsoftexpert/terraform-aws-cloudwatch-log-group?ref=v1.0.0"
  name        = "/casey/firehose/app-events"
  kms_key_arn = module.firehose_kms.arn
}

# Upstream Kinesis Data Stream source
module "orders_stream" {
  source      = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-stream?ref=v1.0.0"
  name        = "casey-orders"
  kms_key_arn = module.firehose_kms.arn
}

# Delivery role -- trust firehose.amazonaws.com, least-privilege S3 + Kinesis + KMS grants
data "aws_caller_identity" "current" {}

module "firehose_role" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-iam-role?ref=v1.0.0"
  name   = "casey-firehose-delivery"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "firehose.amazonaws.com" }
      Action    = "sts:AssumeRole"
      Condition = {
        StringEquals = { "sts:ExternalId" = data.aws_caller_identity.current.account_id }
      }
    }]
  })

  inline_policies = {
    firehose-delivery = {
      policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
          {
            Effect   = "Allow"
            Action   = ["s3:AbortMultipartUpload", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject"]
            Resource = [module.events_bucket.arn, "${module.events_bucket.arn}/*"]
          },
          {
            Effect   = "Allow"
            Action   = ["kinesis:DescribeStream", "kinesis:GetShardIterator", "kinesis:GetRecords", "kinesis:ListShards"]
            Resource = module.orders_stream.arn
          },
          {
            Effect   = "Allow"
            Action   = ["kms:GenerateDataKey", "kms:Decrypt"]
            Resource = module.firehose_kms.arn
          },
          {
            Effect   = "Allow"
            Action   = ["logs:PutLogEvents", "logs:CreateLogStream"]
            Resource = "${module.firehose_log_group.arn}:*"
          }
        ]
      })
    }
  }
}

# The Terraform identity applying this module needs iam:PassRole scoped to
# module.firehose_role.arn -- never "*" (see Required IAM Permissions above).

module "firehose" {
  source = "git::https://github.com/microsoftexpert/terraform-aws-kinesis-firehose?ref=v1.0.0"

  name         = "casey-orders-to-s3"
  destination  = "extended_s3"
  iam_role_arn = module.firehose_role.arn

  source_type               = "kinesis_stream_source"
  kinesis_source_stream_arn = module.orders_stream.arn

  # SSE must be disabled -- the source stream's own SSE-KMS already covers
  # data at rest; this module enforces the constraint at plan time.
  enable_server_side_encryption = false

  extended_s3_configuration = {
    bucket_arn         = module.events_bucket.arn
    compression_format = "GZIP"

    cloudwatch_logging_options = {
      log_group_name = module.firehose_log_group.name
    }
  }

  tags = { Environment = "prod", App = "orders-pipeline" }
}

ℹ️ High-level grouping:

• Core: name, destination (s3 | extended_s3 | redshift | opensearch | http_endpoint), iam_role_arn (required)
• Source: source_type (direct_put | kinesis_stream_source), kinesis_source_stream_arn
• Encryption: enable_server_side_encryption (default true), kms_key_arn
• Destination objects (exactly one required, matching destination): s3_configuration, extended_s3_configuration, redshift_configuration (sensitive), opensearch_configuration, http_endpoint_configuration (sensitive) — each a deeply-typed object mirroring the live provider schema field-for-field, with nested s3_configuration / s3_backup_configuration / cloudwatch_logging_options / processing_configuration / secrets_manager_configuration sub-objects as applicable
• Universal: tags, timeouts

• Primary: id, arn (equal to each other — see Architecture Notes)
• Attributes: name, destination (reflects the caller's selection), destination_id, version_id
• Tags: tags_all

ℹ️ No output is marked sensitive — the module emits no secrets. Note that redshift_configuration and http_endpoint_configuration inputs are marked sensitive = true (they may carry a password / access key); see Architecture Notes.

• ARN / ID format: arn:aws:firehose:<region>:<account-id>:deliverystream/<name>. id equals arn for this resource — confirmed via the live provider's identity schema, whose sole identity attribute is arn (the legacy terraform import form even uses the ARN string itself as the id).
• name and destination are treated as immutable (FORCE-NEW) by this module. The Firehose API exposes no RenameDeliveryStream operation, and UpdateDestination reconfigures an existing DestinationId's settings — it cannot switch a stream from one destination type to another (e.g. extended_s3 → redshift). Changing either value destroys and recreates the stream. Fields within the same destination block (buffering hints, prefixes, the delivery role, processing config) DO update in place.
• The live v6.53.0 provider schema has removed the standalone s3_configuration top-level destination block entirely — confirmed via terraform providers schema -json, not just registry prose (which merely calls "s3" "Deprecated, use extended_s3 instead"). This module still honors destination = "s3" as a caller-facing convenience: it renders var.s3_configuration onto the resource's real extended_s3_configuration block and always stores destination = "extended_s3" on the underlying resource, while the module's own destination output reflects the caller's original selection. Prefer "extended_s3" for all new streams.
• role_arn is the universal per-destination role argument name on this resource — every destination block, plus kinesis_source_configuration and the nested s3_configuration / s3_backup_configuration / secrets_manager_configuration / vpc_config blocks, all use role_arn for the delivery role. This module injects the single var.iam_role_arn everywhere rather than exposing one role variable per destination.
• tags ↔ tags_all ↔ default_tags: var.tags flows to the one resource's tags; tags_all is the computed merge of resource tags over provider default_tags (resource tags win on key conflict). default_tags is the caller's provider-block concern, never set inside this module.
• Server-side encryption must not be combined with a Kinesis Data Stream source. When the source is a Kinesis Data Stream, the stream's own SSE-KMS (configured in terraform-aws-kinesis-stream) already encrypts data at rest before Firehose reads and buffers it in memory. This module enforces the constraint with BOTH a variable validation block AND a resource lifecycle.precondition (defense in depth).
• destination_id and version_id are real computed attributes confirmed via the live provider JSON schema (not documented in the registry page's prose) — exposed as module outputs.
• redshift_configuration and http_endpoint_configuration are marked sensitive = true at the whole-variable level because they may carry a Redshift password or HTTP endpoint access key. Terraform cannot mark a single object attribute sensitive — only the whole variable — so non-secret fields in those objects (e.g. cluster_jdbcurl) are also redacted from plan output as an accepted trade-off.
• Destroy ordering / in-flight data. Deleting a delivery stream while it is actively buffering records can drop data that has not yet flushed to the destination — there is no "drain on destroy" semantic. Coordinate stream deletion with producers halting writes first.
• No us-east-1 constraint. Firehose is a regional service; the module declares no region variable and inherits the caller's provider region.

Secure-by-default posture and the explicit opt-out for each:

Why by-reference, not created here: the IAM delivery role, destination/backup S3 bucket, optional KMS CMK, and optional CloudWatch Log Group are all consumed by ARN from sibling Phase 1 foundation modules, never created by this module. This is the library's foundation-first design — one owner per resource type, wired together by ARN, keeps blast radius contained and lets each foundation module apply its own secure defaults independently.

v1.0.0 scope boundaries (documented, not gaps): only s3 / extended_s3 / redshift / opensearch / http_endpoint destinations and direct_put / kinesis_stream_source sources are implemented. elasticsearch (legacy), opensearchserverless, snowflake, splunk, and iceberg destinations, msk_source_configuration (Amazon MSK source), and data_format_conversion_configuration (Parquet/ORC conversion) all exist on the live resource but are explicitly out of scope for v1.0.0 — see SCOPE.md "Design decisions" for the rationale on each.

terraform init -backend=false
terraform validate
terraform fmt -check
terraform plan # requires valid AWS credentials (profile / SSO / OIDC) + a region
terraform apply
terraform output

⚠️ plan / apply require valid AWS credentials and a configured region (provider block / AWS_PROFILE / SSO / OIDC). Always pin the module source with ?ref=v1.0.0, never a branch.

• terraform init -backend=false && terraform validate — schema + reference integrity, including all cross-variable validation blocks (destination/source mutual exclusivity, SSE-vs-Kinesis-source rejection, backup-required-when-enabled).
• terraform fmt -check — formatting.
• terraform plan against a sandbox account — confirm exactly one destination block renders, and that enable_server_side_encryption is rejected in combination with source_type = "kinesis_stream_source".
• Verify delivery-error logging: after apply, put a malformed record and confirm it surfaces in the wired CloudWatch Log Group.
• Confirm iam:PassRole is scoped correctly on the Terraform identity's own policy before the first apply in a shared account.

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

arn = "arn:aws:firehose:us-east-1:123456789012:deliverystream/casey-app-events"
id = "arn:aws:firehose:us-east-1:123456789012:deliverystream/casey-app-events"
name = "casey-app-events"
destination = "extended_s3"
destination_id = "destinationId-000000000001"
version_id = "1"
tags_all = { "Environment" = "prod", "App" = "lending-portal" }

• Error: Invalid value for variable on s3_configuration / extended_s3_configuration / etc.: exactly one destination object must be set, matching var.destination. Supplying a destination object that doesn't match destination (or leaving the matching one unset) fails a validation block by design.
• enable_server_side_encryption must be false when source_type = "kinesis_stream_source": this is intentional — AWS documents that SSE should not be combined with a Kinesis Data Stream source. Set enable_server_side_encryption = false and rely on the source stream's own SSE-KMS.
• redshift_configuration.s3_backup_configuration is required when s3_backup_mode = "Enabled": the module defaults Redshift's backup mode to "Enabled" (a secure-by-default override of AWS's own "Disabled"). Either supply s3_backup_configuration or explicitly set s3_backup_mode = "Disabled" with a documented exception.
• AccessDenied on firehose:CreateDeliveryStream mentioning iam:PassRole: the Terraform identity's policy is missing (or too narrowly/incorrectly scoped) iam:PassRole on the exact iam_role_arn supplied. Never widen this to "*" — fix the Resource element to the role's ARN.
• Records not reaching the destination / silently dropped: confirm cloudwatch_logging_options.log_group_name is set on the active destination so delivery errors surface; check the destination-specific prerequisites (S3 bucket policy, Redshift network reachability, OpenSearch domain/VPC access policy, HTTP endpoint auth) in AWS Prerequisites above.
• Destination "switch" plans a destroy/recreate instead of an in-place update: expected — changing destination (or name) is modeled as FORCE-NEW because the Firehose API has no operation that migrates a stream between destination types.
• Tag drift / unexpected tags: caused by default_tags overlap. tags_all merges resource tags over provider default_tags with resource tags winning — if a value differs from what you set, a default_tags entry is colliding.
• Credential-chain failures (NoCredentialProviders / ExpiredToken): no valid credentials resolved. Set AWS_PROFILE, refresh SSO, or confirm OIDC role assumption in CI. The module never takes credentials as variables.
• Data loss on terraform destroy: there is no "drain on destroy" — halt producers before deleting a delivery stream that may still have buffered, undelivered records.

• Terraform Registry — hashicorp/aws provider: aws_kinesis_firehose_delivery_stream
• AWS — Amazon Data Firehose Developer Guide (destinations, buffering, dynamic partitioning)
• AWS — Data protection in Amazon Data Firehose (server-side encryption, Direct PUT vs. Kinesis Data Streams sources)
• AWS — Amazon Data Firehose Quota (streams per Region, Direct PUT throughput, dynamic-partitioning limits)
• AWS — Grant Amazon Data Firehose access to an Amazon Redshift / OpenSearch / HTTP endpoint destination (IAM role trust and permissions)
• — terraform-aws-iam-role, terraform-aws-s3-bucket, terraform-aws-kms, terraform-aws-cloudwatch-log-group, terraform-aws-kinesis-stream (sibling modules)

🧡 "Infrastructure as Code should be standardized, consistent, and secure."