Microsoft Certified: Fabric Analytics Engineer AssociatePrepare and transform data (20-25%)Medium
A logistics company uses a legacy ERP system that generates daily CSV files containing shipment details. These files are dropped into a specific folder in Azure Data Lake Storage Gen2. The data engineering team needs to ingest these files into a Lakehouse table, but only after ensuring that all columns have the correct data types (e.g., 'ShipmentID' as integer, 'ShipmentDate' as date) and that any rows with missing critical values (e.g., 'ShipmentID' is null) are rejected. Which PySpark DataFrame operation is most suitable for enforcing data types and handling nulls during ingestion in a Spark notebook?
- A`.select()` and `.filter()`
- B`.orderBy()` and `.limit()`
- C`.withColumn()` and `.na.drop()`
- D`.join()` and `.groupBy()`
Show answer & explanationAnswer & explanation
Correct answer: C. `.withColumn()` and `.na.drop()`
`.withColumn()` is used to cast columns to specific data types, and `.na.drop()` is a DataFrame method specifically designed to remove rows containing null values, directly addressing the requirements.
Why the other options are wrong
- A. `.select()` is for column selection, and `.filter()` can remove rows, but `.withColumn()` is more direct for type casting, and `.na.drop()` is specifically for null handling.
- B. `.orderBy()` is for sorting, and `.limit()` is for restricting the number of rows; neither addresses data type enforcement or null handling.
- D. `.join()` is for combining DataFrames, and `.groupBy()` is for aggregation; neither directly addresses data type enforcement or null handling in this context.
PySpark Data Quality
Using PySpark DataFrame operations to enforce data types and handle missing values during data transformation.
- `.withColumn()` for type casting and new column creation.
- `.na.drop()` for removing rows with nulls.
- `.na.fill()` for imputing nulls.
Memory trick: With Spark, we polish columns and drop dirty rows.