AWS Certified Data Engineer – AssociateData Ingestion and TransformationEasy
A data analytics team needs to process semi-structured log data (JSON format) generated by various applications. These logs are stored hourly in an Amazon S3 bucket. The team wants to perform ad-hoc queries directly on this data without loading it into a traditional database or setting up complex ETL pipelines. They also need to easily discover the schema of the JSON files for querying purposes. Which AWS service is the MOST cost-effective and flexible solution for this scenario?
- AAmazon Redshift
- BAWS Glue ETL
- CAmazon DynamoDB
- DAmazon Athena
Show answer & explanationAnswer & explanation
Correct answer: D. Amazon Athena
Amazon Athena is a serverless query service that allows you to analyze data directly in Amazon S3 using standard SQL. It's cost-effective as you only pay for the queries you run, and it works well with semi-structured data like JSON, especially when combined with AWS Glue Data Catalog for schema discovery, eliminating the need for ETL or a data warehouse.
Why the other options are wrong
- A. Amazon Redshift is a data warehouse, requiring data loading and management, which goes against the 'without loading into a traditional database or setting up complex ETL' requirement.
- B. AWS Glue ETL can transform and load data, but the requirement is to query directly without ETL or loading, making Athena a more direct fit for ad-hoc querying.
- C. Amazon DynamoDB is a NoSQL database, not designed for ad-hoc SQL querying of large volumes of semi-structured log files stored in S3.
Amazon Athena
An interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL.
- Serverless and pay-per-query
- Queries data directly in S3
- Supports various data formats (CSV, JSON, ORC, Parquet)
- Integrates with AWS Glue Data Catalog for schema management
Memory trick: Athena lets you ask S3 questions directly, like a wise oracle.