PDF Professional-Data-Engineer Study Guide

Set up Cloud Storage FUSE, and mount the Cloud Storage bucket on a Compute Engine Instance Remove the completed files from the TSV file Use a shell script to iterate through the TSV file and download the remaining URLs to the FUSE mount point.

Update the file checksums in the TSV file from using MD5 to SHA256. Remove the completed files from the TSV file and rerun the Storage Transfer Service job.

Renew the TLS certificate of the HTTPS endpoint Remove the completed files from the TSV file and rerun the Storage Transfer Service job.

Create a new TSV file for the remaining files by generating signed URLs with a longer validity period. Split the TSV file into multiple smaller files and submit them as separate Storage Transfer Service jobs in parallel.

Create a Cloud Dataproc Workflow Template

Create an initialization action to execute the jobs

Create a Directed Acyclic Graph in Cloud Composer

Create a Bash script that uses the Cloud SDK to create a cluster, execute jobs, and then tear down the cluster

Select random samples from the tables using the RAND() function and compare the samples.

Select random samples from the tables using the HASH() function and compare the samples.

Use a Dataproc cluster and the BigQuery Hadoop connector to read the data from each table and calculate a hash from non-timestamp columns of the table after sorting. Compare the hashes of each table.

Create stratified random samples using the OVER() function and compare equivalent samples from each table.

Create an application that provides an API. Write a tool to poll the API and write data to Cloud Storage as gzipped JSON files.

Create an application that writes to a Cloud SQL database to store the data. Set up periodic exports of the database to write to Cloud Storage and load into BigQuery.

Create an application that publishes events to Cloud Pub/Sub, and create Spark jobs on Cloud Dataproc to convert the JSON data to Avro format, stored on HDFS on Persistent Disk.

Create an application that publishes events to Cloud Pub/Sub, and create a Cloud Dataflow pipeline that transforms the JSON event payloads to Avro, writing the data to Cloud Storage and BigQuery.