Skip to content

feat(glue-alpha): add extraPythonFiles support to PythonShellJob - #37130

Merged
mergify[bot] merged 6 commits into
aws:mainfrom
mazyu36:fix/issue-34448-python-shell-extra-py-files
Apr 30, 2026
Merged

feat(glue-alpha): add extraPythonFiles support to PythonShellJob#37130
mergify[bot] merged 6 commits into
aws:mainfrom
mazyu36:fix/issue-34448-python-shell-extra-py-files

Conversation

@mazyu36

@mazyu36 mazyu36 commented Mar 1, 2026

Copy link
Copy Markdown
Contributor

Issue

Closes #34448.

Reason for this change

PythonShellJob does not expose an extraPythonFiles property, preventing users from specifying additional Python dependencies (.py, .egg, .whl) in a type-safe way. The --extra-py-files argument is officially supported by AWS Glue for Python Shell jobs, and other job types like PySparkEtlJob already support this through the SparkJob base class.

Description of changes

  • Added extraPythonFiles?: Code[] property to PythonShellJobProps
  • Added logic in the PythonShellJob constructor to convert extraPythonFiles entries to S3 URLs and set the --extra-py-files default argument
  • The implementation is inlined in PythonShellJob rather than changing the class hierarchy, because SparkJob carries unrelated properties (extraJars, sparkUI, etc.) that are not applicable to Python Shell jobs

Describe any new or updated permissions being added

No new IAM permissions are required. The existing role already has access to S3 for the script location.

Description of how you validated changes

  • Added unit tests for single and multiple extraPythonFiles entries
  • Added integration test (ShellJobWithExtraPyFiles) using Code.fromAsset and deployed successfully
  • Ran yarn integ --update-on-failed to deploy and update snapshots

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@mazyu36
feat(glue-alpha): add extraPythonFiles support to PythonShellJob
5e97c98 
Add `extraPythonFiles` property to `PythonShellJobProps` to allow
specifying additional Python files (.py, .egg, .whl) for Python Shell
jobs via the `--extra-py-files` Glue argument.

Closes aws#34448
@aws-cdk-automation
aws-cdk-automation requested a review from a team March 1, 2026 01:46
@github-actions github-actions Bot added feature-request A feature should be added or improved. p2 distinguished-contributor [Pilot] contributed 50+ PRs to the CDK labels Mar 1, 2026
aws-cdk-automation
aws-cdk-automation previously requested changes Mar 1, 2026
View reviewed changes

@aws-cdk-automation aws-cdk-automation left a comment •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This review is outdated)

@github-actions

github-actions Bot commented Mar 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚠️ Experimental Feature: This security report is currently in experimental phase. Results may include false positives and the rules are being actively refined.
This security report is NOT a review blocker. Please try merge from main to avoid findings unrelated to the PR.
To suppress a specific rule, see Suppressing Rules.

TestsPassed ✅SkippedFailed
Security Guardian Results24 ran24 passed
TestResult
No test annotations available

@github-actions

github-actions Bot commented Mar 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

⚠️ Experimental Feature: This security report is currently in experimental phase. Results may include false positives and the rules are being actively refined.
This security report is NOT a review blocker. Please try merge from main to avoid findings unrelated to the PR.
To suppress a specific rule, see Suppressing Rules.

TestsPassed ✅SkippedFailed
Security Guardian Results with resolved templates24 ran24 passed
TestResult
No test annotations available

@aws-cdk-automation
aws-cdk-automation dismissed their stale review March 1, 2026 01:59

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

@aws-cdk-automation aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 1, 2026
@aws-cdk-automation aws-cdk-automation added pr/needs-maintainer-review This PR needs a review from a Core Team Member and removed pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. labels Mar 26, 2026
@leonmk-aws leonmk-aws self-assigned this Apr 22, 2026
@leonmk-aws leonmk-aws added the pr/needs-integration-tests-deployment Requires the PR to deploy the integration test snapshots. label Apr 28, 2026
@leonmk-aws
leonmk-aws temporarily deployed to deployment-integ-test April 28, 2026 11:08 — with GitHub Actions Inactive
leonmk-aws
leonmk-aws previously requested changes Apr 28, 2026
View reviewed changes

@leonmk-aws leonmk-aws left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution, LGTM, just a small comment regarding an additional unit test.

Comment thread packages/@aws-cdk/aws-glue-alpha/test/python-shell-job.test.ts
});
});
});

@leonmk-aws leonmk-aws Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a small unit test for the empty array edge case ?

@mazyu36 mazyu36 Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've added a test.

@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Apr 28, 2026
@mazyu36
test(glue-alpha): add empty array edge case test for extraPythonFiles
e3a7dc5 
Addresses review feedback on aws#37130 to verify that --extra-py-files is
not added to DefaultArguments when an empty array is provided.
@mazyu36
mazyu36 had a problem deploying to deployment-integ-test April 30, 2026 01:07 — with GitHub Actions Error
@mergify
mergify Bot dismissed leonmk-aws’s stale review April 30, 2026 01:07

Pull request has been modified.

@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Apr 30, 2026
@mazyu36
mazyu36 had a problem deploying to deployment-integ-test April 30, 2026 03:54 — with GitHub Actions Error
@mazyu36
mazyu36 had a problem deploying to deployment-integ-test April 30, 2026 05:26 — with GitHub Actions Failure
@mazyu36
mazyu36 requested a review from leonmk-aws April 30, 2026 06:16
@leonmk-aws leonmk-aws removed the pr/needs-integration-tests-deployment Requires the PR to deploy the integration test snapshots. label Apr 30, 2026
leonmk-aws
leonmk-aws approved these changes Apr 30, 2026
View reviewed changes

@leonmk-aws leonmk-aws left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution!

@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Apr 30, 2026
@mergify

mergify Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify

mergify Bot commented Apr 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Merge Queue Status

  • ✅ Entered queue — 2026-04-30 08:06 UTC · Rule: default-squash
  • ✅ Checks skipped · PR is already up-to-date
  • 🚫 Left the queue — 2026-04-30 08:07 UTC · at ce544b9d33d7a7fd658fb8d11ffeff7e3064676d

This pull request spent 13 seconds in the queue, including 2 seconds running CI.

Required conditions to merge

Reason

Pull request #37130 has been dequeued

The pull request could not be merged. This could be related to an activated branch protection or ruleset rule that prevents us from merging. Required status check "validate-pr" is in progress.

Hint

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.
If you do update this pull request, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

@leonmk-aws

leonmk-aws commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

@Mergifyio refresh

@mergify

mergify Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

refresh

✅ Pull request refreshed

@mergify

mergify Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify

mergify Bot commented Apr 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Merge Queue Status

  • ✅ Entered queue — 2026-04-30 08:10 UTC · Rule: default-squash
  • ✅ Checks skipped · PR is already up-to-date
  • ✅ Merged — 2026-04-30 08:10 UTC · at ce544b9d33d7a7fd658fb8d11ffeff7e3064676d · squash

This pull request spent 13 seconds in the queue, including 2 seconds running CI.

Required conditions to merge

@mergify
mergify Bot merged commit c9c6f9c into aws:main Apr 30, 2026
38 of 40 checks passed
@github-actions

github-actions Bot commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Apr 30, 2026

This branch had an error being deployed

1 failed and 1 inactive deployments
automation — ce544b9d Deployed Apr 30, 2026 by aws-cdk-automation via validate-pr #343039
deployment-integ-test — ce544b9d Deployed Apr 30, 2026 by mazyu36 via Deploy integration test snapshots (requires `pr/needs-integration-tests-deployment` label) #4935
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@leonmk-aws leonmk-aws leonmk-aws approved these changes
@aws-cdk-automation aws-cdk-automation aws-cdk-automation left review comments
+1 more reviewer
@badmintoncryer badmintoncryer badmintoncryer approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@leonmk-aws leonmk-aws

Labels

distinguished-contributor [Pilot] contributed 50+ PRs to the CDK feature-request A feature should be added or improved. p2

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

awws-glue-alpha: PytthonShellJob does not expose the ability to specify extraPythonFiles

4 participants

@mazyu36 @leonmk-aws @aws-cdk-automation @badmintoncryer