❄️ Snowflake External S3-Compatible Stage Terraform Module

Manages a single Snowflake external stage targeting S3-API-compatible object storage that is not Amazon S3 itself — on-premises or MinIO-style endpoints (snowflake_stage_external_s3_compatible) — targeting snowflakedb/snowflake ~> 2.17.

• ❄️ Creates and owns exactly one snowflake_stage_external_s3_compatible.this — no children, no grants, no COPY INTO/loading-pipeline logic.
• 🧱 Targets S3-API-compatible object storage that is not Amazon S3 — on-premises or MinIO-style endpoints. Distinct from terraform-snowflake-stage-external-s3, which targets Amazon S3 itself.
• 🔌 Accepts the parent database and schema purely by reference (var.database / var.schema) — both owned by sibling terraform-snowflake-database / terraform-snowflake-schema instances.
• 🔒 credentials (an inline AWS-style access-key/secret-key pair) is the only authentication mechanism this resource exposes — there is no storage_integration reference path at all, unlike this module's S3/Azure/GCS siblings.
• 🚫 Has no encryption, use_privatelink_endpoint, or aws_access_point_arn fields — all three confirmed absent from the live schema, not accidentally dropped.

💡 Why it matters: an empty call to this module (no credentials, no directory, no file_format) produces the smallest possible stage definition — no directory table, no implied file format, no credentials sent to Snowflake — leaving every additional behavior an explicit, reviewable opt-in, consistent with this suite's "the empty call must produce the safe resource" philosophy.

If these Terraform modules have been helpful to you or your organization, I'd appreciate your support in any of the following ways:

• ⭐ Star this repository to help others discover this Terraform module.
• 🤝 Connect with me on LinkedIn: linkedin.com/in/microsoftexpert
• ☕ Buy me a coffee: buymeacoffee.com/microsoftexpert

Whether it's a star, a professional connection, or a coffee, every gesture helps keep these modules actively maintained and continually improving. Thank you for being part of the community!

flowchart LR
 classDef moduleNode fill:#29B5E8,color:#ffffff,stroke:#1b7ea6,stroke-width:1px;
 classDef keystoneNode fill:#11567F,color:#ffffff,stroke:#0b3e5c,stroke-width:1px;
 classDef neutralNode fill:#eef1f5,color:#1a1a1a,stroke:#c7ccd4,stroke-width:1px;

 DB["terraform-snowflake-database"]:::neutralNode
 SCHEMA["terraform-snowflake-schema"]:::neutralNode
 MOD["terraform-snowflake-stage-external-s3-compatible"]:::moduleNode
 RES["snowflake_stage_external_s3_compatible.this"]:::keystoneNode

 DB -->|"fully_qualified_name as database"| MOD
 SCHEMA -->|"fully_qualified_name as schema"| MOD
 MOD -->|"creates"| RES

This module consumes terraform-snowflake-database's and terraform-snowflake-schema's fully_qualified_name outputs as its required database/schema inputs — the same cross-reference convention every schema-scoped object module in this library uses. There is deliberately no terraform-snowflake-storage-integration-aws edge in this diagram — this resource has no storage_integration argument, so unlike its terraform-snowflake-stage-external-s3 sibling, there is nothing to wire from a storage-integration module here. No downstream consumer module exists yet in this catalog for this module's own fully_qualified_name output. Validated via the Mermaid Chart MCP before embedding.

flowchart LR
 classDef moduleNode fill:#29B5E8,color:#ffffff,stroke:#1b7ea6,stroke-width:1px;
 classDef keystoneNode fill:#11567F,color:#ffffff,stroke:#0b3e5c,stroke-width:1px;
 classDef neutralNode fill:#eef1f5,color:#1a1a1a,stroke:#c7ccd4,stroke-width:1px;

 subgraph INPUTS["Typed inputs"]
 direction TB
 I1["database / name / schema (required)"]:::neutralNode
 I2["url / endpoint (required)"]:::neutralNode
 I3["credentials (optional, sensitive)"]:::neutralNode
 I4["directory (optional)"]:::neutralNode
 I5["file_format (optional)"]:::neutralNode
 I6["comment (optional)"]:::neutralNode
 end

 RES["snowflake_stage_external_s3_compatible.this"]:::keystoneNode

 subgraph OUTPUTS["Outputs"]
 direction TB
 O1["fully_qualified_name"]:::neutralNode
 O2["name"]:::neutralNode
 O3["id"]:::neutralNode
 end

 INPUTS -->|"rendered 1:1, no for_each (standalone)"| RES
 RES -->|"computed attributes"| OUTPUTS

Resource inventory: exactly one resource, snowflake_stage_external_s3_compatible.this. No for_each, no count, no child resources of any kind — this is a standalone module. Validated via the Mermaid Chart MCP before embedding.

Schema notes that bite (verified against the live v2.18.0 schema):

• No storage_integration, encryption, use_privatelink_endpoint, or aws_access_point_arn fields exist on this resource. Confirmed absent, not an oversight — credentials is the only authentication mechanism available.
• credentials is not read back from DESCRIBE/SHOW STAGE output. The provider's own docs state this explicitly — rotating the S3-compatible endpoint's access key outside Terraform produces no diff on the next plan.
• directory.auto_refresh force-new / alter-on-unset asymmetry. Setting auto_refresh to a new explicit value forces stage replacement; unsetting it instead alters the whole directory field using the current enable value — not symmetric operations.
• refresh_on_create is create-only. Changes to it after the stage exists are silently ignored by the provider (not force-new, not applied).
• This resource is meant only for S3-compatible endpoints, not Amazon S3 itself. The provider's own docs explicitly say not to use s3:// URLs with this resource — use terraform-snowflake-stage-external-s3 instead. This module's url validation enforces the s3compat:// scheme at terraform validate time.
• Persistent diffs after terraform import may require enabling the IMPORT_BOOLEAN_DEFAULT experimental feature and reimporting — see the provider's MIGRATION_GUIDE.md.
• Several file_format sub-fields (e.g. csv.trim_space, json.multi_line) are STRING tri-state sentinels ("true" / "false" / the internal "default" value) on the provider resource, not native booleans — this module models them as real bool variables and converts internally.

ℹ️ The resource's own Terraform docs page describes argument/attribute shape only and carries no dedicated "required privileges" subsection — the privileges below are sourced from Snowflake's CREATE STAGE SQL reference, not the Terraform provider docs. Treat as a least-privilege starting point and verify against the executing role's actual grant set before relying on this list as exhaustive.

• CREATE STAGE on the target schema (to create the keystone snowflake_stage_external_s3_compatible.this) — the executing role must own the schema (OWNERSHIP) or hold this privilege explicitly granted on the schema.
• USAGE on the parent database and schema.
• No additional Snowflake privilege is required to populate credentials — it is stage-object metadata, not a Snowflake principal grant. Any access control on the S3-compatible endpoint itself (bucket policy, IAM-equivalent) is out of band on the storage provider's side.

• No preview feature flag required — snowflake_stage_external_s3_compatible is a GA ("Stable") resource as of provider v2.18.0.
• The parent database and schema referenced by var.database/var.schema must already exist.
• The S3-compatible endpoint itself must already exist and be network-reachable from Snowflake before the stage is usable for COPY INTO / directory-table refresh — this module cannot validate reachability at terraform validate/plan time.
• If var.file_format.format_name is set, the referenced snowflake_file_format object must already exist.
• Temporary stages are not applicable to this resource (per the provider's own docs).

terraform-snowflake-stage-external-s3-compatible/
├── providers.tf # required_providers + required_version — no provider {} block
├── variables.tf # deeply-typed variable schema, secure defaults, universal tail (comment)
├── main.tf # the total renderer — one keystone `this`, no children
├── outputs.tf # fully_qualified_name, then name, then id
├── README.md # this file
├── SCOPE.md # lightweight standalone spec (design intent, privileges, prerequisites, emits, gotchas)
└── examples/ # runnable example call sites

module "stage_minio_raw" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_MINIO_RAW"
  url      = "s3compat://raw-bucket/landing/"
  endpoint = "minio.internal.example.com"
}

The caller's root module configures the snowflake provider (account identifier, authentication, and any role alias) — this module never declares a provider {} block or a credential-shaped variable at the provider level. The call above produces the smallest possible stage: no credentials sent (an anonymous/public-read bucket), no directory table, no implied file format.

Consumes

Emits

module "stage_minimal" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_MINIMAL"
  url      = "s3compat://public-bucket/landing/"
  endpoint = "s3.my-provider.com"
}

module "stage_minio_onprem" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_MINIO_ONPREM"
  url      = "s3compat://onprem-bucket/exports/"
  endpoint = "minio.corp.internal:9000"
}

module "stage_with_credentials" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_MINIO_AUTHENTICATED"
  url      = "s3compat://private-bucket/landing/"
  endpoint = "minio.corp.internal:9000"

  credentials = {
    aws_key_id     = var.minio_access_key_id     # sourced from a secret store, never a literal
    aws_secret_key = var.minio_secret_access_key # sourced from a secret store, never a literal
  }
}

module "stage_invalid_credentials" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_INVALID"
  url      = "s3compat://private-bucket/landing/"
  endpoint = "minio.corp.internal:9000"

  credentials = {
    aws_key_id = "AKIAEXAMPLE" # aws_secret_key omitted — invalid
  }
}

module "stage_wrong_module" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_WRONG_SCHEME"
  url      = "s3://my-aws-bucket/path/" # invalid for this module
  endpoint = "s3.amazonaws.com"
}

module "stage_with_directory" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_DIRECTORY_ENABLED"
  url      = "s3compat://private-bucket/landing/"
  endpoint = "minio.corp.internal:9000"

  directory = {
    enable            = true
    refresh_on_create = true
    auto_refresh      = false
  }
}

module "stage_named_format" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_NAMED_FORMAT"
  url      = "s3compat://private-bucket/landing/"
  endpoint = "minio.corp.internal:9000"

  file_format = {
    format_name = "\"RAW\".\"LANDING\".\"FF_CSV_STANDARD\""
  }
}

module "stage_inline_csv" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_INLINE_CSV"
  url      = "s3compat://private-bucket/landing/"
  endpoint = "minio.corp.internal:9000"

  file_format = {
    csv = {
      compression         = "GZIP"
      field_delimiter     = "|"
      skip_header         = 1
      empty_field_as_null = true
      encoding            = "UTF8"
    }
  }
}

module "stage_inline_json" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_INLINE_JSON"
  url      = "s3compat://private-bucket/exports/"
  endpoint = "minio.corp.internal:9000"

  file_format = {
    json = {
      compression        = "AUTO"
      strip_outer_array  = true
      ignore_utf8_errors = false
    }
  }
}

module "stage_documented" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_VENDOR_EXPORTS"
  url      = "s3compat://vendor-bucket/exports/"
  endpoint = "s3.vendor-provider.com"
  comment  = "Nightly vendor export drop zone — MinIO-compatible endpoint"
}

module "stage_no_encryption_option" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_NO_ENCRYPTION_FIELD"
  url      = "s3compat://vendor-bucket/exports/"
  endpoint = "s3.vendor-provider.com"
}

module "stage_invalid_skip_header" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_INVALID_SKIP_HEADER"
  url      = "s3compat://private-bucket/landing/"
  endpoint = "minio.corp.internal:9000"

  file_format = {
    csv = {
      skip_header = -1 # invalid — the provider's own internal "unset" sentinel, not caller-settable
    }
  }
}

module "stage_production" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = "RAW"
  schema   = "LANDING"
  name     = "STAGE_PARTNER_FEED"
  url      = "s3compat://partner-feed-bucket/daily/"
  endpoint = "minio.partner-dc.example.com:9000"

  credentials = {
    aws_key_id     = var.partner_feed_access_key_id
    aws_secret_key = var.partner_feed_secret_access_key
  }

  directory = {
    enable       = true
    auto_refresh = true
  }

  file_format = {
    format_name = "\"RAW\".\"LANDING\".\"FF_CSV_STANDARD\""
  }

  comment = "Daily partner feed drop zone (MinIO-compatible endpoint)"
}

module "db_raw" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-database.git?ref=v1.0.0"

  name = "RAW"
}

module "schema_landing" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-schema.git?ref=v1.0.0"

  database = module.db_raw.fully_qualified_name
  name     = "LANDING"
}

module "stage_minio_feed" {
  source = "git::https://github.com/microsoftexpert/terraform-snowflake-stage-external-s3-compatible.git?ref=v1.0.0"

  database = module.db_raw.fully_qualified_name
  schema   = module.schema_landing.fully_qualified_name
  name     = "STAGE_MINIO_PARTNER_FEED"
  url      = "s3compat://partner-feed-bucket/daily/"
  endpoint = "minio.partner-dc.example.com:9000"

  credentials = {
    aws_key_id     = var.partner_feed_access_key_id
    aws_secret_key = var.partner_feed_secret_access_key
  }

  comment = "Daily partner feed drop zone"
}

Grouped summary: identity (database, name, schema), location (url, endpoint), authentication (credentials, sensitive), directory table (directory), file format (file_format), and the universal tail (comment).

file_format = object({
  format_name = optional(string, null)

  avro = optional(object({
    compression                = optional(string, null) # AUTO|GZIP|BROTLI|ZSTD|DEFLATE|RAW_DEFLATE|NONE
    null_if                    = optional(list(string), null)
    replace_invalid_characters = optional(bool, null)
    trim_space                 = optional(bool, null)
  }), null)

  csv = optional(object({
    binary_format                  = optional(string, null) # HEX|BASE64|UTF8
    compression                    = optional(string, null) # AUTO|GZIP|BZ2|BROTLI|ZSTD|DEFLATE|RAW_DEFLATE|NONE
    date_format                    = optional(string, null)
    empty_field_as_null            = optional(bool, null)
    encoding                       = optional(string, null) # closed enum — see variables.tf
    error_on_column_count_mismatch = optional(bool, null)
    escape                         = optional(string, null)
    escape_unenclosed_field        = optional(string, null)
    field_delimiter                = optional(string, null)
    field_optionally_enclosed_by   = optional(string, null)
    file_extension                 = optional(string, null)
    multi_line                     = optional(bool, null)
    null_if                        = optional(list(string), null)
    parse_header                   = optional(bool, null)
    record_delimiter               = optional(string, null)
    replace_invalid_characters     = optional(bool, null)
    skip_blank_lines               = optional(bool, null)
    skip_byte_order_mark           = optional(bool, null)
    skip_header                    = optional(number, null) # -1 sentinel forbidden — use null
    time_format                    = optional(string, null)
    timestamp_format               = optional(string, null)
    trim_space                     = optional(bool, null)
  }), null)

  json = optional(object({
    allow_duplicate            = optional(bool, null)
    binary_format              = optional(string, null) # HEX|BASE64|UTF8
    compression                = optional(string, null) # AUTO|GZIP|BZ2|BROTLI|ZSTD|DEFLATE|RAW_DEFLATE|NONE
    date_format                = optional(string, null)
    enable_octal               = optional(bool, null)
    file_extension             = optional(string, null)
    ignore_utf8_errors         = optional(bool, null)
    multi_line                 = optional(bool, null)
    null_if                    = optional(list(string), null)
    replace_invalid_characters = optional(bool, null)
    skip_byte_order_mark       = optional(bool, null)
    strip_null_values          = optional(bool, null)
    strip_outer_array          = optional(bool, null)
    time_format                = optional(string, null)
    timestamp_format           = optional(string, null)
    trim_space                 = optional(bool, null)
  }), null)

  orc = optional(object({
    null_if                    = optional(list(string), null)
    replace_invalid_characters = optional(bool, null)
    trim_space                 = optional(bool, null)
  }), null)

  parquet = optional(object({
    binary_as_text             = optional(bool, null)
    compression                = optional(string, null) # AUTO|LZO|SNAPPY|NONE
    null_if                    = optional(list(string), null)
    replace_invalid_characters = optional(bool, null)
    trim_space                 = optional(bool, null)
    use_logical_type           = optional(bool, null)
    use_vectorized_scanner     = optional(bool, null)
  }), null)

  xml = optional(object({
    compression                = optional(string, null) # AUTO|GZIP|BZ2|BROTLI|ZSTD|DEFLATE|RAW_DEFLATE|NONE
    disable_auto_convert       = optional(bool, null)
    ignore_utf8_errors         = optional(bool, null)
    preserve_space             = optional(bool, null)
    replace_invalid_characters = optional(bool, null)
    skip_byte_order_mark       = optional(bool, null)
    strip_outer_element        = optional(bool, null)
  }), null)
})

credentials is never re-exposed as an output — it is sensitive = true on the input side only, and the provider does not read it back from the live object either (see Troubleshooting).

• credentials's both-required-together constraint is enforced by the type system, not a validation {} block. The live schema documents aws_key_id/aws_secret_key as both Required within the (Optional) credentials block; variables.tf mirrors this exactly — object({ aws_key_id = string, aws_secret_key = string }) with no per-field optional wrapper — so a one-sided credentials object fails at terraform validate with a parse-time type error, before any API call. This is a deliberate correction against the authoring prompt's speculative fallback shape, made after Phase 1's live-schema grounding confirmed the type system alone already expresses the constraint.
• Tri-state bool-to-string-sentinel conversion, centralized. directory.auto_refresh / directory.refresh_on_create and every boolean-shaped file_format sub-field are modeled as real bool variables even though the underlying provider fields are STRING sentinels ("true"/"false"/internal "default", unsettable manually) — the same pattern already established by terraform-snowflake-schema (with_managed_access/is_transient) and terraform-snowflake-warehouse (auto_resume). main.tf performs every conversion in one locals.bool_str map rather than scattering ternaries across each dynamic block, keeping main.tf a "total renderer" per this suite's design convention.
• file_format's at-most-one-of-seven constraint (format_name, avro, csv, json, orc, parquet, xml) is enforced by a cross-attribute validation {} block using compact over a list of conditionally-populated markers — this cannot be expressed by the type system alone (Terraform's object type has no native mutual-exclusivity constraint between attributes).
• No for_each in this module. Standalone module, one keystone resource, no children — no for_each key-stability concern to document.
• No storage_integration relationship exists to model, deliberately — this is the one stage variant in this catalog with no reference-based auth path at all. Do not add one in a future revision without first confirming a live schema change; as of provider v2.18.0 the field does not exist on this resource.

🔒 Honest limitation, stated plainly per this suite's Secure-by-default table: the house-wide "prefer a reference over a raw secret" principle cannot be fully honored for this specific resource. Every other secret-bearing Snowflake module in this catalog that has one offers a storage_integration/similar reference-only path as its preferred pattern (terraform-snowflake-stage-external-s3, -external-azure, -external-gcs all do). This resource's live schema has no such field — credentials is the only mechanism the provider exposes for S3-compatible endpoints. This module therefore does the next-best thing: credentials is sensitive = true, is expected to be sourced from a secret store upstream and injected at apply time, and is never modeled as a plaintext literal — but it cannot be a bare reference the way its siblings' credentials can.

cd C:\GitHubCode\newsnowflakemodules\terraform-snowflake-stage-external-s3-compatible
terraform init -backend=false
terraform validate
terraform fmt -check

Pin ?ref=v1.0.0 in every call site — never a branch. This is a plan-only authoring pipeline; a human runs terraform apply from a governed CI pipeline.

• terraform init -backend=false / terraform validate / terraform fmt -check cover: type correctness (every object/scalar schema matches the live provider schema), HCL syntax, the url scheme rejection, the credentials both-required-together rejection (enforced by the type itself — no separate validation {} block exists for it), the file_format at-most-one-of-seven rejection, and every closed-enum validation {} block (compression, encoding, binary_format) catching a malformed input before any API call.
• What these three commands do not cover: privilege sufficiency (whether the executing role actually holds CREATE STAGE on the target schema), object-existence dependencies (whether the referenced database/schema/file-format actually exist, whether the S3-compatible endpoint is actually reachable), whether supplied credentials are valid against the endpoint, and whether credentials drift silently — since the provider does not read credentials back from DESCRIBE/SHOW STAGE, this module's state is authoritative only going forward from the last apply, and only a real terraform plan/apply against a live Snowflake account (and a reachable endpoint) exercises any of this. Per this library's plan-only posture, that step belongs to a human running it from governed CI, not to this authoring process.

$ terraform output

fully_qualified_name = "\"RAW\".\"LANDING\".\"STAGE_MINIO_PARTNER_FEED\""
name = "STAGE_MINIO_PARTNER_FEED"
id = "\"RAW\".\"LANDING\".\"STAGE_MINIO_PARTNER_FEED\""

credentials is never printed here, even redacted — it is a sensitive = true input, not an output, and this module does not re-expose it in any form.

• Provider resource docs: snowflake_stage_external_s3_compatible
• Sibling modules: terraform-snowflake-database, terraform-snowflake-schema, terraform-snowflake-stage-external-s3 (the Amazon S3 variant), terraform-snowflake-stage-internal, terraform-snowflake-stage-external-azure, terraform-snowflake-stage-external-gcs
• This module's SCOPE.md