Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Dumps : Databricks Certified Associate Developer for Apache Spark 3.5 – Python

Broadcast joins work by sending the smaller DataFrame to all executors, eliminating the shuffle of the larger DataFrame.

From Spark documentation:

“Broadcast joins are efficient when one DataFrame is small enough to fit in memory. Spark avoids shuffling the larger table.”

DataFrame B (1 GB) fits within the default threshold and should be broadcasted.

It eliminates the need to shuffle the large DataFrame A.

Final Answer: B

In Spark SQL, the function date_sub(startDate, days) returns the date that is days before startDate.

If the days parameter is negative, Spark interprets it as subtracting a negative number, which effectively adds days to the date.

Example:

from pyspark.sql.functions import date_sub, lit

df = spark.createDataFrame([( "2024-10-01", )], ["dt"])

df.select(date_sub("dt", -5).alias("new_date")).show()

Output:

+----------+

| new_date |

+----------+

|2024-10-06|

+----------+

Why the other options are incorrect:

B: No error occurs; negative values are supported.

C: The start date changes if days ≠ 0.

D: Subtracting days would move the date backward, not forward.

To ensure that data written out to disk is sorted, it is important to consider how Spark writes data when saving to Parquet tables. The methods .sort() or .orderBy() apply a global sort but do not guarantee that the sorting will persist in the final output files unless certain conditions are met (e.g. a single partition via .coalesce(1) — which is not scalable).

Instead, the proper method in distributed Spark processing to ensure rows are sorted within their respective partitions when written out is:

sortWithinPartitions("column_name")

According to Apache Spark documentation:

"sortWithinPartitions() ensures each partition is sorted by the specified columns. This is useful for downstream systems that require sorted files."

This method works efficiently in distributed settings, avoids the performance bottleneck of global sorting (as in .orderBy() or .sort()), and guarantees each output partition has sorted records — which meets the requirement of consistently sorted data.

Thus:

Option A and B do not guarantee the persisted file contents are sorted.

Option C introduces a bottleneck via .coalesce(1) (single partition).

Option D correctly applies sorting within partitions and is scalable.