Microsoft Certified: Fabric Analytics Engineer AssociateExplore and analyze data (15-20%)Medium
A data engineer is writing a PySpark script to process a large DataFrame `log_data_df` containing web server logs. The DataFrame has a `timestamp` column (string, in 'YYYY-MM-DD HH:MM:SS' format) and `request_path` (string). The engineer needs to extract the hour of the day from the `timestamp` column as an integer for further analysis. Which PySpark function should be used?
- AF.substring(F.col('timestamp'), 12, 2)
- BF.to_timestamp(F.col('timestamp')).hour
- CF.hour(F.col('timestamp'))
- DF.date_format(F.col('timestamp'), 'HH')
Show answer & explanationAnswer & explanation
Correct answer: C. F.hour(F.col('timestamp'))
The `F.hour()` function is specifically designed to extract the hour component from a timestamp or a string that can be cast to a timestamp. It returns an integer, which is exactly what the requirement specifies for further analysis.
Why the other options are wrong
- A. `F.substring()` is a string manipulation function. While it could extract the hour string, it's less robust (relies on fixed format) and still requires casting to integer, making it less ideal than a dedicated date/time function.
- B. `F.to_timestamp()` converts the string to a timestamp, but `.hour` is not a valid attribute or method to directly extract the hour from a Column object in PySpark. You would still need `F.hour()` after conversion.
- D. `F.date_format(..., 'HH')` would extract the hour as a *string*, not an integer, which then would require an additional cast to integer.
PySpark F.hour()
The `F.hour()` function in PySpark (from `pyspark.sql.functions`) extracts the hour component (0-23) from a timestamp, date, or a string that can be parsed as such, returning it as an integer.
- Part of `pyspark.sql.functions` module.
- Takes a Column object (timestamp, date, or string) as input.
- Returns an integer representing the hour (0-23).
- Automatically handles string to timestamp conversion if format is standard.
Memory trick: Functions for parts, format for strings, substring if you're desperate!