🛡️ Azure Synapse Spark pool autoscale is disabled🟢

• Contextual name: 🛡️ Spark pool autoscale is disabled🟢
• ID: /ce/ca/azure/synapse/spark-pool-autoscale-disabled
• Tags:
  • 🟢 Policy with categories
  • 🟢 Policy with type
  • 🟢 Production policy
• Policy Type: BEST_PRACTICE
• Policy Categories: COST

Stats

not available

Logic

• 🧠 prod.logic.yaml🟢
  • 📗 Azure Synapse Big Data Pool
  • 🔌 Azure Synapse Big Data Pool - object.extracts.yaml
  • 🧪 test-data.json

Description

Open File

Description

Enable autoscale for Azure Synapse Analytics Apache Spark pools unless a pool has an approved fixed-capacity requirement. Autoscale adjusts the number of nodes between configured minimum and maximum limits as workload demand changes.

Rationale

Spark workloads can vary substantially in demand. Autoscale helps match pool capacity to that demand, reducing unnecessary compute consumption while retaining capacity for active workloads.

Impact

When autoscale is disabled, a Spark pool remains at a fixed node count and can continue to incur charges during periods when fewer nodes would suffice. Enabling autoscale can affect job startup time and throughput, so assess workload demand, capacity limits, quotas, and approved performance requirements before changing a pool.

Audit

This policy evaluates an Azure Synapse Spark pool based on its autoscale configuration.

The Spark pool is marked as INCOMPLIANT if Auto Scale: State is Disabled.

The Spark pool is marked as INAPPLICABLE if Provisioning State is not

... see more

Remediation

Open File

Remediation

Before you begin

Confirm that the pool's workloads can tolerate node scaling and establish approved minimum and maximum node counts. Review subscription quotas, cost limits, job concurrency, startup requirements, and any fixed-capacity exception.

Azure portal

1. Open the affected Azure Synapse workspace in Synapse Studio.
2. Select Manage, then Apache Spark pools.
3. Select the affected Spark pool and choose Edit.
4. Enable Autoscale and set minimum and maximum node counts that meet the approved workload and cost requirements and Azure service limits.
5. Save the configuration, then validate representative scheduled and interactive jobs after the change.
6. Confirm that Cloudaware reports Auto Scale: State as Enabled and populated node-count bounds after the next collection.

Automation

For infrastructure-as-code or API remediation, update the Microsoft.Synapse/workspaces/bigDataPools resource so that properties.autoScale.enabled is true. Set properties.autoScale.minNodeCount and properties.autoScale.maxNodeCount to

... see more

policy.yaml

Open File

Linked Framework Sections

Section	Sub Sections	Internal Rules	Policies	Flags	Compliance
💼 Cloudaware Framework → 💼 Resource Optimization			37		no data