Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Online Access

f.to_date() converts a timestamp or string to a DateType.

Ideal for extracting the date component (year-month-day) from a full timestamp.

Example:

from pyspark.sql.functions import to_date

dates_df.withColumn("date", to_date("timestamp"))

A Spark job is triggered by an action (e.g., count, show).

The job is broken into stages, typically one per shuffle boundary.

Each stage is divided into multiple tasks, which are distributed across worker nodes.

Broadcast joins work by sending the smaller DataFrame to all executors, eliminating the shuffle of the larger DataFrame.

From Spark documentation:

“Broadcast joins are efficient when one DataFrame is small enough to fit in memory. Spark avoids shuffling the larger table.”

DataFrame B (1 GB) fits within the default threshold and should be broadcasted.

It eliminates the need to shuffle the large DataFrame A.

Final Answer: B

In Structured Streaming, a watermark defines the maximum delay for event-time data to be considered in stateful operations like deduplication or window aggregations.

Behavior:

df = df.withWatermark("event_timestamp", "30 minutes")

This sets a 30-minute watermark, meaning Spark will only keep track of events that arrive within 30 minutes of the latest event time seen so far. When used with:

df.dropDuplicates(["event_id", "event_timestamp"])

Spark removes duplicates that arrive within the watermark threshold (in this case, within 30 minutes).

Why other options are incorrect:

A: Watermarks do not remove all duplicates; they only manage those within the defined event-time window.

B: Watermark durations can be expressed as strings like "30 minutes", "10 seconds", etc., not only seconds.

D: Structured Streaming supports deduplication using withWatermark() and dropDuplicates().

References (Databricks Apache Spark 3.5 – Python / Study Guide):

PySpark Structured Streaming Guide — withWatermark() and dropDuplicates() methods for event-time deduplication.

Databricks Certified Associate Developer for Apache Spark Exam Guide (June 2025): Section “Structured Streaming” — Topic: Streaming Deduplication with and without watermark usage.