👉 Subscribe to our newsletter to get:
- Real stories from real AWS projects
- No-nonsense DevOps tactics
- Cost, security & compliance patterns that actually work
- Expert guidance from engineers in the field
=========================================================================
Terraform module to compact small S3 objects into larger files with AWS Lambda and AWS Step Functions, cutting storage costs on tiers with minimum billable object sizes (e.g. 128 KB) and speeding up Amazon Athena queries.
This project is based on aws-samples/s3-small-object-compaction. We thank the original contributors for their work on the CDK-based solution that inspired this Terraform module.
The module deploys two variants of the compaction solution:
- A standalone Lambda function (via terraform-aws-modules/lambda/aws) that iterates over a list of Amazon S3 prefixes and compacts the objects in each into a single larger file
- An AWS Step Functions state machine (via terraform-aws-modules/step-functions/aws) using Distributed Map to invoke a compaction Lambda in parallel for each prefix, for faster compaction at scale
Both variants can be triggered on an EventBridge schedule (disabled by default, matching the upstream solution).
The handlers read the source and destination URIs from the invocation event, so the scheduled payload is not a permission boundary. The Lambda execution roles are therefore granted s3:ListBucket and s3:GetObject only on the key prefix of source_s3_uri, and an invocation that points at another prefix fails with AccessDenied. For that reason source_s3_uri must include a key prefix; a bare s3://bucket/ is rejected at plan time. The functions never delete source objects.
The state-machine role trusts states.amazonaws.com with an aws:SourceAccount check written as StringEqualsIfExists. Step Functions omits the confused-deputy context keys when it requests task credentials for Distributed Map child executions, so a strict condition denies every child; IfExists enforces the account check whenever the key is present and passes when the service omits it. A cross-account principal cannot hand the role to a state machine of their own, while a same-account principal already holding iam:PassRole on the role can, which is the posture of the upstream solution and of CDK-generated Distributed Map roles.
See examples/basic.
| Name | Version |
|---|---|
| terraform | >= 1.5.7 |
| aws | >= 6.28 |
| Name | Version |
|---|---|
| aws | 6.63.0 |
| Name | Source | Version |
|---|---|---|
| compact_lambda | terraform-aws-modules/lambda/aws | 8.8.0 |
| list_lambda | terraform-aws-modules/lambda/aws | 8.8.0 |
| standalone_compact_lambda | terraform-aws-modules/lambda/aws | 8.8.0 |
| step_function | terraform-aws-modules/step-functions/aws | 5.1.1 |
| Name | Type |
|---|---|
| aws_cloudwatch_event_rule.standalone | resource |
| aws_cloudwatch_event_rule.state_machine | resource |
| aws_cloudwatch_event_target.standalone | resource |
| aws_cloudwatch_event_target.state_machine | resource |
| aws_iam_role.events | resource |
| aws_iam_role.state_machine | resource |
| aws_iam_role_policy.events | resource |
| aws_iam_role_policy.state_machine | resource |
| aws_lambda_permission.standalone | resource |
| aws_caller_identity.current | data source |
| aws_iam_policy_document.events_assume | data source |
| aws_iam_policy_document.events_start_execution | data source |
| aws_iam_policy_document.state_machine | data source |
| aws_iam_policy_document.state_machine_assume | data source |
| aws_partition.current | data source |
| aws_region.current | data source |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| cloudwatch_logs_retention_in_days | Retention in days for the Lambda and Step Functions CloudWatch log groups | number |
14 |
no |
| compact_lambda_memory_size | Memory size in MB for the per-prefix compaction Lambda used by Step Functions | number |
128 |
no |
| compact_lambda_timeout | Timeout in seconds for the per-prefix compaction Lambda used by Step Functions | number |
300 |
no |
| create_standalone_lambda | Create the standalone compaction Lambda variant that processes all prefixes in a single invocation | bool |
true |
no |
| create_step_functions | Create the Step Functions variant that compacts prefixes in parallel with a Distributed Map | bool |
true |
no |
| date_format | Python strftime format of the date prefixes under the source URI, e.g. %Y/%m/%d for year/month/day | string |
"%Y/%m/%d" |
no |
| lambda_ephemeral_storage_size | Ephemeral storage (/tmp) in MB for the compaction Lambdas, 512 to 10240. The merged bytes of a single date prefix are staged in /tmp before upload, so this must exceed the largest day of source data under any one prefix, or that date fails mid-write with 'No space left on device' | number |
2048 |
no |
| lambda_runtime | Python runtime used by all Lambda functions | string |
"python3.12" |
no |
| list_lambda_memory_size | Memory size in MB for the prefix-listing Lambda used by Step Functions | number |
128 |
no |
| list_lambda_timeout | Timeout in seconds for the prefix-listing Lambda used by Step Functions | number |
60 |
no |
| name_prefix | Prefix used for naming all resources created by this module | string |
"s3-object-compaction" |
no |
| previous_days | How many days back to compact. Each daily date prefix in the range is compacted into one object | number |
1 |
no |
| schedule_enabled | Enable the EventBridge schedules. Disabled by default, matching the upstream solution | bool |
false |
no |
| schedule_expression | EventBridge schedule expression. Defaults to rate(previous_days days) when null | string |
null |
no |
| sfn_child_execution_type | Execution type of the Distributed Map child workflows. EXPRESS caps each date prefix at 5 minutes total regardless of compact_lambda_timeout, which a busy date on a large source cannot meet. STANDARD lifts that ceiling at Standard-workflow pricing | string |
"EXPRESS" |
no |
| sfn_max_concurrency | Maximum concurrent child executions of the Distributed Map | number |
100 |
no |
| sfn_tolerated_failure_count | Number of failed date prefixes the Distributed Map tolerates before the whole execution fails. Unset keeps the Step Functions default of zero, where one failed date fails the run after other dates have already written their output | number |
null |
no |
| sfn_tolerated_failure_percentage | Percentage (0-100) of failed date prefixes the Distributed Map tolerates before the whole execution fails. Unset keeps the Step Functions default of zero. If both count and percentage are set the run fails when either is exceeded | number |
null |
no |
| source_s3_uri | S3 URI holding the small objects to compact, e.g. s3://my-bucket/raw/. Date prefixes are appended to it. A key prefix is required: the Lambda IAM policies are scoped to it | string |
n/a | yes |
| standalone_lambda_memory_size | Memory size in MB for the standalone compaction Lambda | number |
1024 |
no |
| standalone_lambda_timeout | Timeout in seconds for the standalone compaction Lambda | number |
900 |
no |
| tags | Tags applied to all resources created by this module | map(string) |
{} |
no |
| target_s3_uri | S3 URI where compacted objects are written, e.g. s3://my-bucket/compacted/. Date prefixes are appended to it | string |
n/a | yes |
| Name | Description |
|---|---|
| compact_lambda_function_arn | ARN of the per-prefix compaction Lambda, empty string when create_step_functions is false |
| list_lambda_function_arn | ARN of the prefix-listing Lambda, empty string when create_step_functions is false |
| schedule_expression | EventBridge schedule expression used by both trigger rules |
| standalone_lambda_function_arn | ARN of the standalone compaction Lambda, empty string when create_standalone_lambda is false |
| standalone_lambda_function_name | Name of the standalone compaction Lambda, empty string when create_standalone_lambda is false |
| state_machine_arn | ARN of the compaction Step Functions state machine, null when create_step_functions is false |
| state_machine_name | Name of the compaction Step Functions state machine, null when create_step_functions is false |
| state_machine_role_arn | ARN of the IAM role assumed by the state machine, null when create_step_functions is false |
Apache 2.0 Licensed. See LICENSE for full details.