Azure Daily 2022 by Gordon

Public preview: Azure Synapse Analytics Spark elastic pool storage

Azure Synapse Analytics Spark pools now support elastic pool storage. Apache Spark in Azure Synapse Ananlytics utilize temporary disk storage while the Spark pool is instantiated. For many Spark jobs, it's difficult to estimate the pools storage requirements, which can cause some Spark jobs to fail if the temporay storage is exhausted. Today we are enabling elastic pool storage which allows the Spark engine to monitor worker node temporary storage and attach additional disks if needed. No action is required by you, plus you should see less job failures as a result.

Source: Public preview: Azure Synapse Analytics Spark elastic pool storage

Azure Synapse Analytics