Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Dumps : Databricks Certified Associate Developer for Apache Spark 3.5 – Python

To use a Python function as a UDF (User Defined Function) in Spark SQL, it must first be registered using spark.udf.register().

Correct usage:

spark.udf.register("cube_func", cube_func)

num_df.selectExpr("cube_func(num)").show()

This registers cube_func as a callable SQL function available in expressions or queries.

Why the other options are incorrect:

B: You must wrap with udf() or selectExpr; calling plain Python functions won’t work.

C: createDataFrame is for building DataFrames, not calling UDFs.

D: DataFrames cannot directly register UDFs.

Adaptive Query Execution (AQE) in Spark 3.x automatically optimizes query plans at runtime based on the actual data characteristics observed during job execution.

Key features of AQE include:

Dynamic switching of join strategies: Changes between sort-merge join and broadcast join based on actual shuffle sizes.

Coalescing shuffle partitions: Reduces small tasks and improves parallelism efficiency.

Handling skew joins: Dynamically splits large partitions to avoid data skew.

Thus, the most accurate answer describing AQE’s function is “dynamically switching join strategies.”

Why the other options are incorrect:

B: Table statistics are collected manually or by the metastore, not by AQE.

C: AQE benefits multi-stage jobs involving shuffles, not single-stage jobs.

D: Delta file optimization is handled by Databricks utilities, not AQE.

f.to_date() converts a timestamp or string to a DateType.

Ideal for extracting the date component (year-month-day) from a full timestamp.

Example:

from pyspark.sql.functions import to_date

dates_df.withColumn("date", to_date("timestamp"))