AWS Certified Data Engineer – AssociateData Storage and ManagementMedium
A data engineering team manages a large data lake on Amazon S3. They use AWS Glue Data Catalog to store metadata, but frequently add new partitions (folders) to their S3 data based on new data arriving. After adding new data and partitions, queries using Amazon Athena do not immediately reflect the new data. Which AWS Glue Data Catalog operation should the team perform to ensure Athena can query the newly added partitions?
- AExecute `MSCK REPAIR TABLE` in Amazon Athena.
- BUpdate the table schema manually in the Glue Data Catalog.
- CCreate a new Glue Crawler to re-scan the entire S3 path.
- DRun a Glue ETL job to transform the data.
Show answer & explanationAnswer & explanation
Correct answer: A. Execute `MSCK REPAIR TABLE` in Amazon Athena.
When new partitions are manually added to an S3 location that corresponds to an AWS Glue Data Catalog table, the `MSCK REPAIR TABLE` command in Athena (or a Glue Crawler) is necessary to update the Glue Data Catalog with the metadata for these new partitions. This ensures Athena can discover and query the new data.
Why the other options are wrong
- B. Manually updating the table schema is tedious and error-prone for new partitions and doesn't automatically discover them.
- C. While creating and running a Glue Crawler would also work, `MSCK REPAIR TABLE` is a more direct and often quicker way to update partitions for existing tables when the folder structure follows the partition scheme.
- D. Running a Glue ETL job is for data transformation, not for updating the Data Catalog with new partition metadata.
MSCK REPAIR TABLE
A DDL command in Amazon Athena (and Hive) used to update the AWS Glue Data Catalog with new partitions that have been manually added to the S3 location of a table.
- Discovers and adds new partitions to the Glue Data Catalog
- Ensures Athena can query newly added data
- Alternative to running a Glue Crawler for partition updates
- Scans the S3 path for new partition directories
Memory trick: MSCK REPAIR is the 'Master Key' to 'Repair' your catalog's missing partitions.