Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam With Confidence Using Practice Dumps

To exclude sensitive (PII) columns from a DataFrame, the easiest method is to use the .drop() function with the list of column names to remove.

Correct syntax:

df_user_non_pii = df_user.drop("name", "email", "birthdate")

This creates a new DataFrame containing all remaining columns.

Why the other options are incorrect:

B: .dropFields() is not valid for standard DataFrames — it’s used for struct fields only.

C: .select() would keep only PII columns, not remove them.

D: .remove() does not exist in Spark DataFrame API.

Underutilization with timeout warnings often indicates insufficient parallelism — meaning there aren’t enough executors to process all tasks concurrently.

Solution:

Increase the number of executors to allow more parallel task execution and better resource utilization.

Example configuration:

--conf spark.executor.instances=8

This distributes the workload more effectively across cluster nodes and reduces idle time for pending tasks.

Why the other options are incorrect:

A: Extending timeouts hides the symptom, not the root cause (lack of executors).

B: More memory per executor won’t fix scheduling bottlenecks.

C: Reducing partition size may increase overhead and does not fix resource imbalance.

In Spark SQL, the function date_sub(startDate, days) returns the date that is days before startDate.

If the days parameter is negative, Spark interprets it as subtracting a negative number, which effectively adds days to the date.

Example:

from pyspark.sql.functions import date_sub, lit

df = spark.createDataFrame([( "2024-10-01", )], ["dt"])

df.select(date_sub("dt", -5).alias("new_date")).show()

Output:

+----------+

| new_date |

+----------+

|2024-10-06|

+----------+

Why the other options are incorrect:

B: No error occurs; negative values are supported.

C: The start date changes if days ≠ 0.

D: Subtracting days would move the date backward, not forward.