Amazon Web Services AWS Certified Associate Data-Engineer-Associate New Questions

S3 Lifecycle policies automate data retention and deletion. By specifying an expiration rule for 7 years, objects older than that period are permanently deleted without manual intervention.

“To automatically delete aged data, configure an S3 Lifecycle rule with an expiration policy for objects older than the retention period.”

– Ace the AWS Certified Data Engineer - Associate Certification - version 2 - apple.pdf

Problem Analysis:

The company uses DynamoDB for gaming data storage and needs to ingest data into Amazon OpenSearch Service in near real time.

Data updates must propagate quickly to OpenSearch for analytics or search purposes.

Key Considerations:

DynamoDB Streams provide near-real-time capture of table changes (inserts, updates, and deletes).

Integration with AWS Lambda allows seamless processing of these changes.

OpenSearch offers APIs for indexing and updating documents, which Lambda can invoke.

Solution Analysis:

Option A: Step Functions with Periodic Export

Not suitable for near-real-time updates; introduces significant latency.

Operationally complex to manage periodic exports and S3 data ingestion.

Option B: AWS Glue Job

AWS Glue is designed for ETL workloads but lacks real-time processing capabilities.

Option C: DynamoDB Streams + Lambda

DynamoDB Streams capture changes in near real time.

Lambda can process these streams and use the OpenSearch API to update the index.

This approach provides low latency and seamless integration with minimal operational overhead.

Option D: Custom OpenSearch Plugin

Writing a custom plugin adds complexity and is unnecessary with existing AWS integrations.

Implementation Steps:

Enable DynamoDB Streams for the relevant DynamoDB tables.

Create a Lambda function to process stream records:

Parse insert, update, and delete events.

Use OpenSearch APIs to index or update documents based on the event type.

Set up a trigger to invoke the Lambda function whenever there are changes in the DynamoDB Stream.

Monitor and log errors for debugging and operational health.

Amazon DynamoDB Streams Documentation

AWS Lambda and DynamoDB Integration

Amazon OpenSearch Service APIs

The RootDiskUsed metric for the MSK cluster indicates that the storage on the broker is reaching its capacity. The best solution is to expand the storage of the MSK broker and enable automatic storage expansion to prevent future alarms.

Expand MSK Broker Storage:

AWS Managed Streaming for Apache Kafka (MSK) allows you to expand the broker storage to accommodate growing data volumes. Additionally, auto-expansion of storage can be configured to ensure that storage grows automatically as the data increases.

Option A is the only choice that directly addresses the root cause: inconsistent postal-code formatting causing processing errors. In AWS Glue, a DynamicFrame can encounter issues when incoming data contains unexpected types or malformed values. Providing an explicit PySpark schema forces the postal code column to be interpreted consistently (for example, as a string with the expected structure), which prevents downstream steps from failing because of unexpected characters or mixed representations. After enforcing a consistent schema, the Glue job can standardize values (such as trimming extra digits, removing invalid characters, and normalizing case) and then write the corrected output back to Amazon S3 so the data lake is fixed at the source.

The document reinforces that AWS Glue is the managed ETL service used to transform data and move it between stores, which is exactly what is needed to correct bad values and persist clean outputs back into the lake. It also highlights that AWS Glue can perform validation as part of the ETL process, supporting the idea that data can be checked and corrected during processing rather than allowing bad values to break the workflow.

Options B, C, and D do not fix data quality or formatting issues; they relate to workflow state sharing, partition filtering, or internal implementation details, not cleansing invalid postal codes.