Free Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions Attempt

In Spark’s distributed architecture:

The Driver Node coordinates the execution of a Spark application. It converts the logical plan into a physical plan of stages and tasks.

The Executors, running on Worker Nodes, are responsible for executing tasks assigned by the driver and storing data (in memory or disk) during execution.

Key point:

Executors are the active agents that perform the actual computations on data partitions. Each executor runs multiple tasks in parallel using available CPU cores.

Why the other options are incorrect:

A (Driver Nodes): The driver schedules tasks; it doesn’t execute them.

C (CPU Cores): CPU cores execute within executors, but they are hardware, not Spark architectural components.

D (Worker Nodes): Worker nodes host executors but do not directly execute tasks; executors do.

References (Databricks Apache Spark 3.5 – Python / Study Guide):

Spark Architecture Components — Driver, Executors, Cluster Manager, Worker Nodes.

Databricks Exam Guide (June 2025): Section “Apache Spark Architecture and Components” — describes the roles of driver and executor nodes in distributed processing.

To view the schema of a Parquet file, you must use the DataFrameReader to load the Parquet data and call the .printSchema() method.

Correct syntax:

spark.read.parquet("events.parquet").printSchema()

This command loads the file metadata (without triggering a full read) and prints the column names, data types, and nullability information in a tree format.

Why the other options are incorrect:

A/D: SQL queries can’t directly introspect file schemas.

B: .show() displays data rows, not schema.

To use a Python function as a UDF (User Defined Function) in Spark SQL, it must first be registered using spark.udf.register().

Correct usage:

spark.udf.register("cube_func", cube_func)

num_df.selectExpr("cube_func(num)").show()

This registers cube_func as a callable SQL function available in expressions or queries.

Why the other options are incorrect:

B: You must wrap with udf() or selectExpr; calling plain Python functions won’t work.

C: createDataFrame is for building DataFrames, not calling UDFs.

D: DataFrames cannot directly register UDFs.

Adaptive Query Execution (AQE) dynamically selects join strategies based on actual data sizes at runtime. If the size of post-shuffle partitions is below the threshold set by:

spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold

then Spark prefers to use a shuffled hash join.

From the Spark documentation:

“AQE selects a shuffled hash join when the size of post-shuffle data is small enough to fit within the configured threshold, avoiding more expensive sort-merge joins.”

Therefore:

A is wrong — Cartesian joins are only used with no join condition.

B is correct — this is the optimized join for small partitioned shuffle data under AQE.

C and D are used under other scenarios but not for this case.

Final Answer: B