🛡️ AWS Glue Spark Job uses a version past end of support🟢

• Contextual name: 🛡️ Spark Job uses a version past end of support🟢
• ID: /ce/ca/aws/glue/job-spark-version-support
• Tags:
  • 🟢 Policy with categories
  • 🟢 Policy with type
  • 🟢 Production policy
• Policy Type: BEST_PRACTICE
• Policy Categories: RELIABILITY

Stats

not available

Logic

• 🧠 prod.logic.yaml🟢
  • 📗 AWS Glue Job
  • 🔌 AWS Glue Job - object.extracts.yaml
  • 🧪 test-data.json

Description

Open File

Description

This policy identifies AWS Glue Spark batch and streaming job definitions configured with a Glue version that has passed AWS end of support (EOS).

Glue Version selects the managed runtime; it is not an Apache Spark version number. The check uses AWS Glue's lifecycle for that runtime, rather than upstream Apache Spark release age. It covers Spark jobs written in Python or Scala, including streaming jobs, without depending on worker type, execution class, or optional job arguments.

Rationale

AWS does not provide security patches or technical support for Glue versions past EOS and may not honor their SLAs. At EOL, AWS prevents new jobs and job runs, stops existing runs and interactive sessions, and removes the version from SDKs and APIs. Reviewing retained job definitions helps prevent failures when scheduled or infrequently used workloads are needed again.

Impact

Migrating to another Glue version can change Spark, Python, Scala, dependencies, connectors, permissions, and job behavior. Test workload compatibility, output correctness, performance, and bookmark handling before changing production jobs. A finding describes the collected configuration; it does not prove the job recently executed on the retired version.

... see more

Remediation

Open File

Remediation

Migrate the Spark job to a supported AWS Glue version

1. Confirm the job owner, triggers, schedules, data sources, downstream dependencies, current Glue version, and whether the definition is still needed. Retire an obsolete job through the organization's normal change process.
2. Review the current AWS Glue support policy, available versions, and the migration guidance linked from those release notes. Select a supported version compatible with the workload and Region.
3. Preserve the existing job configuration and script. Create a separate test job with the target Glue version, compatible worker settings, dependencies, connectors, and IAM/Lake Formation permissions. Avoid overlapping production writes while testing.
4. Update and test Spark, Python or Scala code and dependency versions as needed. Validate representative input data, output correctness, batch job bookmarks, retries, performance, logging, and downstream consumers. Streaming jobs use checkpoints rather than bookmarks: test checkpoint compatibility and restart behavior, and use separate test checkpoints and output destinations. Preserve production checkpoints; resetting them can reprocess data.

... see more

policy.yaml

Open File

Linked Framework Sections

Section	Sub Sections	Internal Rules	Policies	Flags	Compliance
💼 Cloudaware Framework → 💼 Infrastructure Modernization			39		no data