Skip to content

Commit 36d4976

Browse files
otaviomacedootaviomacedo
authored
feat(glue-alpha): default Flex jobs to Glue 5.0 (#38543)
The PySpark and Scala Spark Flex ETL jobs defaulted to Glue version 3.0 (Spark 3.1 / Python 3.7), which is end-of-life. Bump the default to Glue 5.0 (Spark 3.5 / Python 3.11). Users can still opt into older versions via the `glueVersion` prop. Python Shell jobs are intentionally left unchanged: the Glue version is ignored by the service for pyshell jobs, and they already default to the current Python 3.9 runtime. BREAKING CHANGE: PySparkFlexEtlJob and ScalaSparkFlexEtlJob now default to `GlueVersion.V5_0` instead of `V3_0`. Set `glueVersion` explicitly to keep the previous behavior. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent d81d2d6 commit 36d4976

13 files changed

Lines changed: 23 additions & 23 deletions

‎packages/@aws-cdk/aws-glue-alpha/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ new glue.PySparkStreamingJob(stack, 'PySparkStreamingJob', {
190190

191191
The flexible execution class is appropriate for non-urgent jobs such as
192192
pre-production jobs, testing, and one-time data loads. Flexible jobs default
193-
to Glue version 3.0 and worker type `G_2X`. The following best practice
193+
to Glue version 5.0 and worker type `G_2X`. The following best practice
194194
features are enabled by default:
195195
`—enable-metrics, —enable-spark-ui, —enable-continuous-cloudwatch-log`
196196

‎packages/@aws-cdk/aws-glue-alpha/lib/jobs/pyspark-flex-etl-job.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ export class PySparkFlexEtlJob extends SparkJob {
9595
scriptLocation: this.codeS3ObjectUrl(props.script),
9696
pythonVersion: PythonVersion.THREE,
9797
},
98-
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V3_0,
98+
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V5_0,
9999
workerType: props.workerType ? props.workerType : WorkerType.G_1X,
100100
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 10,
101101
maxRetries: props.maxRetries,

‎packages/@aws-cdk/aws-glue-alpha/lib/jobs/scala-spark-flex-etl-job.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ export class ScalaSparkFlexEtlJob extends SparkJob {
103103
name: JobType.ETL,
104104
scriptLocation: this.codeS3ObjectUrl(props.script),
105105
},
106-
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V3_0,
106+
glueVersion: props.glueVersion ? props.glueVersion : GlueVersion.V5_0,
107107
workerType: props.workerType ? props.workerType : WorkerType.G_1X,
108108
numberOfWorkers: props.numberOfWorkers ? props.numberOfWorkers : 10,
109109
maxRetries: props.maxRetries,

‎packages/@aws-cdk/aws-glue-alpha/test/integ.job-pyspark-flex-etl.js.snapshot/aws-glue-job-pysparkflex-etl.assets.json‎

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/@aws-cdk/aws-glue-alpha/test/integ.job-pyspark-flex-etl.js.snapshot/aws-glue-job-pysparkflex-etl.template.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@
114114
"--enable-observability-metrics": "true"
115115
},
116116
"ExecutionClass": "FLEX",
117-
"GlueVersion": "3.0",
117+
"GlueVersion": "5.0",
118118
"JobRunQueuingEnabled": false,
119119
"NumberOfWorkers": 10,
120120
"Role": {

‎packages/@aws-cdk/aws-glue-alpha/test/integ.job-pyspark-flex-etl.js.snapshot/manifest.json‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/@aws-cdk/aws-glue-alpha/test/integ.job-pyspark-flex-etl.js.snapshot/tree.json‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/@aws-cdk/aws-glue-alpha/test/integ.job-scalaspark-flex-etl.js.snapshot/aws-glue-job-scalasparkflex-etl.assets.json‎

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/@aws-cdk/aws-glue-alpha/test/integ.job-scalaspark-flex-etl.js.snapshot/aws-glue-job-scalasparkflex-etl.template.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@
114114
"--enable-observability-metrics": "true"
115115
},
116116
"ExecutionClass": "FLEX",
117-
"GlueVersion": "3.0",
117+
"GlueVersion": "5.0",
118118
"JobRunQueuingEnabled": false,
119119
"NumberOfWorkers": 10,
120120
"Role": {

‎packages/@aws-cdk/aws-glue-alpha/test/integ.job-scalaspark-flex-etl.js.snapshot/manifest.json‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)