🛡️ AWS Glue Job is underutilized🟠🟢

• Contextual name: 🛡️ Job is underutilized🟠🟢
• ID: /ce/ca/aws/glue/job-underutilized
• Tags:
  • 🟢 Policy with categories
  • 🟠 Policy with internal.md
  • 🟢 Policy with type
  • 🟢 Production policy
• Policy Type: BEST_PRACTICE
• Policy Categories: COST, PERFORMANCE

Stats

not available

Logic

• 🧠 prod.logic.yaml🟢
  • 📗 AWS Glue Job
  • 🔌 AWS Glue Job - object.extracts.yaml
  • 🧪 test-data.json

Internal Notes 🟠

Open File

Implementation notes

Production decision

Collected CloudAware data is current and authoritative, as specified by the user. Freshness, collector timestamps, and sample-count metadata are not activation prerequisites. This policy is a production utilization-review heuristic; it does not calculate recoverable savings or prescribe an automatic capacity reduction.

The policy applies to established Spark batch ETL jobs. All four 14-day averages must be positive and below 20%, and completed tasks must be observed in both halves of the 14-day window. The 7-day total must be positive and less than the 14-day total. This excludes activity confined to either week without introducing an arbitrary minimum task count. Missing, zero-utilization, invalid, and insufficient-activity evidence remains UNDETERMINED.

The recurrence guard does not impose a minimum run duration. No run-duration field exists on the job table. Recurring short runs can be valid review candidates; remediation must establish a material benefit before changing capacity.

... see more

Description

Open File

Description

This policy identifies recurring Apache Spark batch ETL jobs with positive completed-task activity and 14-day average driver and executor CPU and JVM heap utilization all below 20%. Findings identify workloads for performance and cost-optimization review.

Rationale

Consistently low observed resource use during recurring processing can indicate oversized workers, insufficient parallelism, skew, or I/O bottlenecks. Reviewing these jobs can reveal opportunities to reduce DPU consumption or improve processing efficiency. Requiring task activity in both halves of the observation period excludes jobs observed only during one isolated part of that period.

Impact

Reducing capacity can increase runtime, cause memory or disk failures, or miss delivery deadlines. Averages can hide demanding stages, and JVM heap usage excludes other memory use. Confirm actual resource consumption and benchmark changes before reducing capacity. Findings do not prescribe a specific worker type or count or guarantee savings.

Audit

... see more

Remediation

Open File

Remediation

Review the workload before changing capacity. An INCOMPLIANT result identifies a tuning opportunity; it does not establish a specific safe reduction.

1. Review representative and peak-input runs over the last 14 days. Compare processing duration, input volume, CPU and memory peaks, disk pressure, failures, and completion deadlines. For UNDETERMINED results, first establish the required task activity and positive utilization measurements.
2. Identify the cause of low utilization. Address I/O bottlenecks, insufficient parallelism, or skew when those limit useful processing. Low JVM heap usage alone does not establish that a smaller worker has enough total memory or disk space.
3. Record the existing job configuration, including worker type/count, Glue version, arguments, and run-level overrides. Preserve a rollback copy.
4. In the AWS Glue console, open the job's Job details. For supported workloads, evaluate Automatically scale the number of workers. If already enabled, inspect actual DPU usage and executor demand before changing Maximum number of workers. Otherwise, test a smaller worker type or worker count where the workload supports it.

... see more

policy.yaml

Open File

Linked Framework Sections

Section	Sub Sections	Internal Rules	Policies	Flags	Compliance
💼 Cloudaware Framework → 💼 Resource Right-Sizing			22		no data
💼 Cloudaware Framework → 💼 Workload Efficiency			26		no data