Databricks-Certified-Associate-Developer-for-Apache-Spark-3.0 Exam Dumps : Databricks Certified Associate Developer for Apache Spark 3.0 Exam

Explanation

Correct code block:

def count_to_target(target):

if target is None:

return

result = list(range(target))

return result

count_to_target_udf = udf(count_to_target, ArrayType(IntegerType()))

transactionsDf.select(count_to_target_udf('predError'))

Output of correct code block:

+--------------------------+

|count_to_target(predError)|

+--------------------------+

| [0, 1, 2]|

| [0, 1, 2, 3, 4, 5]|

| [0, 1, 2]|

| null|

| null|

| [0, 1, 2]|

+--------------------------+

This QUESTION NO: is not exactly easy. You need to be familiar with the syntax around UDFs (user-defined functions). Specifically, in this QUESTION NO: it is important to pass the correct types

to the udf

method - returning an array of a specific type rather than just a single type means you need to think harder about type implications than usual.

Remember that in Spark, you always pass types in an instantiated way like ArrayType(IntegerType()), not like ArrayType(IntegerType). The parentheses () are the key here - make sure you do not

forget those.

You should also pay attention that you actually pass the UDF count_to_target_udf, and not the Python method count_to_target to the select() operator.

Finally, null values are always a tricky case with UDFs. So, take care that the code can handle them correctly.

More info: How to Turn Python Functions into PySpark Functions (UDF) – Chang Hsin Lee – Committing my thoughts to words.

Static notebook | Dynamic notebook: See test 3, QUESTION NO: 24 (Databricks import instructions)

Explanation

Correct code block:

itemsDf.filter(col('supplier').contains('et'))

A mixup can easily happen here between isin and contains. Since we want to check whether a column "contains" the values et, this is the operator we should use here. Note that both methods are

methods of Spark's Column object. See below for documentation links.

A specific Column object can be accessed through the col() method and not the Column() method or through col[], which is an essential thing to know here. In PySpark, Column references a generic

column object. To use it for queries, you need to link the generic column object to a specific DataFrame. This can be achieved, for example, through the col() method.

More info:

- isin documentation: pyspark.sql.Column.isin — PySpark 3.1.1 documentation

- contains documentation: pyspark.sql.Column.contains — PySpark 3.1.1 documentation

Static notebook | Dynamic notebook: See test 1, QUESTION NO: 51 (Databricks import instructions)

Explanation

DAG stands for "Directing Acyclic Graph".

No, DAG stands for "Directed Acyclic Graph".

Spark strategically hides DAGs from developers, since the high degree of automation in Spark means that developers never need to consider DAG layouts.

No, quite the opposite. You can access DAGs through the Spark UI and they can be of great help when optimizing queries manually.

In contrast to transformations, DAGs are never lazily executed.

DAGs represent the execution plan in Spark and as such are lazily executed when the driver requests the data processed in the DAG.