Amazon Web Services Data-Engineer-Associate Exam With Confidence Using Practice Dumps

The Redshift UNLOAD command is specifically designed to export query results to Amazon S3, and AWS Glue can orchestrate this as part of a scheduled job. This is the cleanest and most appropriate approach for recurring weekly data transfers:

“Use the Redshift UNLOAD command with AWS Glue to export data to Amazon S3. This pattern enables routine exports of selected data to external locations.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

This avoids complexities of Redshift Spectrum or unsupported use of COPY commands in Lambda.

Amazon S3 natively supports checksum validation during uploads when objects are uploaded by using the AWS SDK. The SDK can automatically calculate a checksum, such as CRC32, on the client side and send it with the upload request. Amazon S3 validates the checksum upon receipt and stores it with the object, ensuring that the data uploaded is exactly the same as the source data.

Storing the checksum in the S3 object metadata provides a lightweight and cost-effective way to verify data integrity without requiring additional compute, storage, or post-processing steps. This approach validates the logs at upload time, which is the most efficient and reliable method.

Using AWS Lambda to calculate SHA-256 checksums would require additional compute, orchestration, and storage, increasing cost and operational complexity. Amazon S3 Object Lock enforces immutability and retention but does not validate data integrity between source and destination. Governance or compliance mode Object Lock cannot detect whether uploaded data matches the original logs.

Therefore, using the AWS SDK checksum validation during upload is the simplest, most efficient, and AWS-recommended approach for ensuring log integrity.

 This solution meets the requirement of logging all writes to the S3 bucket into another S3 bucket with the least operational effort. AWS CloudTrail is a service that records the API calls made to AWS services, including Amazon S3. By creating a trail of data events, you can capture the details of the requests that are made to the transactions S3 bucket, such as the requester, the time, the IP address, and the response elements. By specifying an empty prefix and write-only events, you can filter the data events to only include the ones that write to the bucket. By specifying the logs S3 bucket as the destination bucket, you can store the CloudTrail logs in another S3 bucket that is in the same AWS Region. This solution does not require any additional coding or configuration, and it is more scalable and reliable than using S3 Event Notifications and Lambda functions. References:

Logging Amazon S3 API calls using AWS CloudTrail

Creating a trail for data events

Enabling Amazon S3 server access logging