Databricks Certification Changed Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions

In Structured Streaming, checkpoints store state information (offsets, progress, and metadata) needed to resume a stream after a failure or restart.

Correct usage:

Set the checkpointLocation option when writing the streaming output:

streaming_df.writeStream \

format("delta") \

option("checkpointLocation", "/path/to/checkpoint/dir") \

start("/path/to/output")

Spark uses this checkpoint directory to recover progress automatically and maintain exactly-once semantics.

Why the other options are incorrect:

A/D: recoveryLocation is not a valid Spark configuration option.

B: Checkpointing must be configured in writeStream, not during readStream.

According to Spark’s watermarking rules:

“Records that are older than the watermark (event time < current watermark) are considered too late and are dropped.”

So, if a record’s event_time is earlier than (max event_time seen so far - 10 minutes), it is discarded.

Pandas API on Spark provides a distributed implementation of the Pandas DataFrame API on top of Apache Spark.

Advantages:

Executes transformations in parallel across all nodes and cores in the cluster.

Maintains Pandas-like syntax, making it easy for Python users to transition.

Enables scaling of existing Pandas code to handle large datasets without memory limits.

Therefore, it combines Pandas usability with Spark’s distributed power, offering both speed and scalability.

Why the other options are incorrect:

B: While it uses Python, that’s not its main advantage.

C: It runs distributed across the cluster, not on a single node.

D: Pandas API on Spark uses lazy evaluation, not eager computation.

Exactly-once guarantees in Spark Structured Streaming require micro-batch mode (default), not continuous mode.

Continuous mode (.trigger(continuous=...)) only supports at-least-once semantics and lacks full fault-tolerance.

trigger(availableNow=True) is a batch-style trigger, not suited for low-latency streaming.

So:

Option A uses micro-batching with a tight trigger interval → minimal latency + exactly-once guarantee.

Final Answer: A