Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Exam Dumps : Databricks Certified Associate Developer for Apache Spark 3.0 Exam

Explanation

Correct code block:

transactionsDf.write.format("parquet").mode("overwrite").option("compression", "snappy").save(storeDir)

Solving this QUESTION NO: requires you to know how to access the DataFrameWriter (link below) from the DataFrame API - through DataFrame.write.

Another nuance here is about knowing the different modes available for writing parquet files that determine Spark's behavior when dealing with existing files. These, together with the compression

options are explained in the DataFrameWriter.parquet documentation linked below.

Finally, bracket __5__ poses a certain challenge. You need to know which command you can use to pass down the file path to the DataFrameWriter. Both save and parquet are valid options here.

More info:

- DataFrame.write: pyspark.sql.DataFrame.write — PySpark 3.1.1 documentation

- DataFrameWriter.parquet: pyspark.sql.DataFrameWriter.parquet — PySpark 3.1.1 documentation

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 58 (Databricks import instructions)

Explanation

Correct code block:

def count_to_target(target):

if target is None:

return

result = list(range(target))

return result

count_to_target_udf = udf(count_to_target, ArrayType(IntegerType()))

transactionsDf.select(count_to_target_udf('predError'))

Output of correct code block:

+--------------------------+

|count_to_target(predError)|

+--------------------------+

| [0, 1, 2]|

| [0, 1, 2, 3, 4, 5]|

| [0, 1, 2]|

| null|

| null|

| [0, 1, 2]|

+--------------------------+

This QUESTION NO: is not exactly easy. You need to be familiar with the syntax around UDFs (user-defined functions). Specifically, in this QUESTION NO: it is important to pass the correct types

to the udf

method - returning an array of a specific type rather than just a single type means you need to think harder about type implications than usual.

Remember that in Spark, you always pass types in an instantiated way like ArrayType(IntegerType()), not like ArrayType(IntegerType). The parentheses () are the key here - make sure you do not

forget those.

You should also pay attention that you actually pass the UDF count_to_target_udf, and not the Python method count_to_target to the select() operator.

Finally, null values are always a tricky case with UDFs. So, take care that the code can handle them correctly.

More info: How to Turn Python Functions into PySpark Functions (UDF) – Chang Hsin Lee – Committing my thoughts to words.

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 24 (Databricks import instructions)

Explanation

The output is the typical output of a DataFrame.printSchema() call. The DataFrame's RDD representation does not have a printSchema or formatSchema method (find available methods in the RDD

documentation linked below). The output of print(transactionsDf.schema) is this: StructType(List(StructField(transactionId,IntegerType,true),StructField(predError,IntegerType,true),StructField

(value,IntegerType,true),StructField(storeId,IntegerType,true),StructField(productId,IntegerType,true),StructField(f,IntegerType,true))). It includes the same information as the nicely formatted original

output, but is not nicely formatted itself. Lastly, the DataFrame's schema attribute does not have a print() method.

More info:

- pyspark.RDD: pyspark.RDD — PySpark 3.1.2 documentation

- DataFrame.printSchema(): pyspark.sql.DataFrame.printSchema — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 2, QUESTION NO: 52 (Databricks import instructions)