You have an Azure Databricks workspace that is enabled for Unity Catalog
You have an Apache Spark Structured Streaming job that writes data to a Delta table.
After the cluster restarts, the streaming job reprocesses previously ingested data
You need to prevent the streaming job from reprocessing the data after the cluster restarts.
What should you do?
You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a Delta table named Orders.
You load the Orders table into an Apache Spark DataFrame named df.
You need to create a DataFrame that excludes rows where the order amount is null.
Solution: You run the following expression.
df.filter(df.order_amount.isNotNull())
Does this meet the goal?
You have an Azure Databricks workspace that contains a job in Lakeflow Jobs named Job1.
Job1 processes raw data files stored in Azure Storage.
New files arrive at unpredictable intervals.
You need to ensure that Job1 starts automatically when new files arrive and does NOT consume compute resources when no data is available.
Which type of job trigger should you use?
You have an Azure Databricks workspace that is enabled for Unity Catalog and contains a catalog named Catalog 1. Catalog 1 contains a table named Transactions. Transactions contains the following columns:
• transaction_id
• customet_name
• email address
• credit_card_number
• transaction_amount
You need to ensure that business analysts can query all the tows in the Transactions table. The solution must meet the following requirements:
• Prevent the analysts from seeing the full values in the email_address and credit_catd_number columns.
• Ensure that the analysts can see only the values after the @ character in each email address.
• Ensure that the analysts can see only the last four digits of each credit card number.
• Enable the analysts to query the table without errors.
• Follow the principle of least privilege.
What should you do?