Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Dumps : Databricks Certified Associate Developer for Apache Spark 3.5 – Python

In Spark SQL and DataFrame operations, the configuration parameter spark.sql.shuffle.partitions defines the number of partitions created during shuffle operations such as join, groupBy, and distinct.

The default value (in Spark 3.5) is 200.

If this number is too low, Spark creates fewer tasks, leading to idle executors and poor cluster utilization.

Increasing this value allows Spark to create more tasks that can run in parallel across executors, effectively using more cluster resources.

Correct approach:

spark.conf.set("spark.sql.shuffle.partitions", 400)

This increases the parallelism level of shuffle stages and improves overall resource utilization.

Why the other options are incorrect:

B: Reducing partitions further would decrease parallelism and worsen the underutilization issue.

C: Dynamic resource allocation scales executors up or down based on workload, but it doesn’t fix low task parallelism caused by insufficient shuffle partitions.

D: Increasing dataset size is not a tuning solution and doesn’t address task-level under-parallelization.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

Spark SQL Configuration: spark.sql.shuffle.partitions — controls the number of shuffle partitions.

Databricks Exam Guide (June 2025): Section “Troubleshooting and Tuning Apache Spark DataFrame API Applications” — tuning strategies, partitioning, and optimizing cluster utilization.

===========

Broadcast joins work by sending the smaller DataFrame to all executors, eliminating the shuffle of the larger DataFrame.

From Spark documentation:

“Broadcast joins are efficient when one DataFrame is small enough to fit in memory. Spark avoids shuffling the larger table.”

DataFrame B (1 GB) fits within the default threshold and should be broadcasted.

It eliminates the need to shuffle the large DataFrame A.

Final Answer: B

The provided code defines a Pandas UDF of type Series-to-Series, where a new instance of the language model is created on each call, which happens per batch. This is inefficient and results in significant overhead due to repeated model initialization.

To reduce the frequency of model loading, the engineer should convert the UDF to an iterator-based Pandas UDF (Iterator[pd.Series] -> Iterator[pd.Series]). This allows the model to be loaded once per executor and reused across multiple batches, rather than once per call.

From the official Databricks documentation:

“Iterator of Series to Iterator of Series UDFs are useful when the UDF initialization is expensive… For example, loading a ML model once per executor rather than once per row/batch.”

— Databricks Official Docs: Pandas UDFs

Correct implementation looks like:

python

CopyEdit

@pandas_udf("string")

def translate_udf(batch_iter: Iterator[pd.Series]) -> Iterator[pd.Series]:

model = get_translation_model(target_lang='es')

for batch in batch_iter:

yield batch.apply(model)

This refactor ensures the get_translation_model() is invoked once per executor process, not per batch, significantly improving pipeline performance.