Free Databricks-Certified-Data-Engineer-Associate Databricks Updates

The small country lookup DataFrame should be broadcast to the executors. A broadcast join copies the small side of the join to each worker, allowing every executor to join its partition of the billion-row events DataFrame locally. This avoids shuffling the very large DataFrame across the network and is effective when one relation is small enough to fit comfortably in executor memory. Broadcasting events_df would be impractical because it contains billions of rows. Caching or persisting both DataFrames can help when data is reused, but neither action by itself changes the join strategy or removes the expensive shuffle. Databricks exposes the PySpark broadcast function specifically to mark a DataFrame as suitable for a broadcast join. Therefore, option D is the correct optimization.

================

The data engineering team can use the spark.table method to access the Delta table sales in PySpark. This method returns a DataFrame representation of the Delta table, which can be used for further processing or testing. The spark.table method works for any table that is registered in the Hive metastore or the Spark catalog, regardless of the file format1. Alternatively, the data engineering team can also use the DeltaTable.forPath method to load the Delta table from its path2. References: 1: SparkSession | PySpark 3.2.0 documentation 2: Welcome to Delta Lake’s Python documentation page — delta-spark 2.4.0 documentation

Delta Sharing is an open protocol developed by Databricks that enables secure, real-time sharing of data across organizations without requiring recipients to have a Databricks account. It is specifically designed for scenarios where external partners need read-only access to data stored in Delta format. With Delta Sharing, data providers create a share and grant access to recipients via a secure mechanism (such as a profile file or URL-based access), ensuring that the data cannot be modified by the consumer. This approach eliminates the need for data duplication or manual file transfers, maintaining data freshness and governance. Option B is incorrect because Unity Catalog governs internal access within an organization, not external sharing without accounts. Option C is inefficient and breaks data lineage and freshness. Option D violates security best practices by granting excessive permissions. Delta Sharing ensures fine- grained access control, auditability, and secure external collaboration , making it the correct solution for this requirement.

=========