Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam With Confidence Using Practice Dumps

In Structured Streaming, checkpoints store state information (offsets, progress, and metadata) needed to resume a stream after a failure or restart.

Correct usage:

Set the checkpointLocation option when writing the streaming output:

streaming_df.writeStream \

format("delta") \

option("checkpointLocation", "/path/to/checkpoint/dir") \

start("/path/to/output")

Spark uses this checkpoint directory to recover progress automatically and maintain exactly-once semantics.

Why the other options are incorrect:

A/D: recoveryLocation is not a valid Spark configuration option.

B: Checkpointing must be configured in writeStream, not during readStream.

To ensure that data written out to disk is sorted, it is important to consider how Spark writes data when saving to Parquet tables. The methods .sort() or .orderBy() apply a global sort but do not guarantee that the sorting will persist in the final output files unless certain conditions are met (e.g. a single partition via .coalesce(1) — which is not scalable).

Instead, the proper method in distributed Spark processing to ensure rows are sorted within their respective partitions when written out is:

sortWithinPartitions("column_name")

According to Apache Spark documentation:

"sortWithinPartitions() ensures each partition is sorted by the specified columns. This is useful for downstream systems that require sorted files."

This method works efficiently in distributed settings, avoids the performance bottleneck of global sorting (as in .orderBy() or .sort()), and guarantees each output partition has sorted records — which meets the requirement of consistently sorted data.

Thus:

Option A and B do not guarantee the persisted file contents are sorted.

Option C introduces a bottleneck via .coalesce(1) (single partition).

Option D correctly applies sorting within partitions and is scalable.

To create a new calculated column in Spark, use the .withColumn() method.

To remove an unwanted column, use the .drop() method.

Correct syntax:

from pyspark.sql.functions import col

employees_df = employees_df.withColumn("tax", col("salary") * 0.1).drop("age")

.withColumn("tax", col("salary") * 0.1) → adds a new column where tax = 10% of salary.

.drop("age") → removes the age column from the DataFrame.

Why the other options are incorrect:

B: lit(0.1) creates a constant value, not a calculated tax.

C: .dropField() is not a DataFrame API method (used only in struct field manipulations).

D: Adds 0.1 to salary instead of calculating 10%.