Professional Data EngineerManaging and securing dataMedium
A global media streaming platform uses Pub/Sub for ingesting real-time user activity data. Due to potential network disruptions or subscriber processing delays, they need to ensure that messages are not lost and can be reprocessed if an issue occurs. Specifically, messages should be retained for up to 7 days, and any messages that fail to be acknowledged after multiple delivery attempts should be automatically moved to a separate dead-letter topic for manual inspection and reprocessing. Which Pub/Sub features should they configure?
- AMessage ordering and exactly-once delivery
- BMessage retention and Dead-Letter Topics
- CSchema validation and message filtering
- DPush subscriptions and subscription expiration
Show answer & explanationAnswer & explanation
Correct answer: B. Message retention and Dead-Letter Topics
Pub/Sub's message retention allows messages to be stored for up to 7 days, ensuring they are available for reprocessing. Dead-Letter Topics automatically forward unacknowledged messages after a configured number of delivery attempts, preventing message loss and enabling manual recovery.
Why the other options are wrong
- A. Message ordering and exactly-once delivery address message sequence and duplication, not retention or handling of unacknowledged messages.
- C. Schema validation ensures message format, and message filtering selects specific messages, neither addresses retention or failed delivery.
- D. Push subscriptions deliver messages to an endpoint, and subscription expiration deletes the subscription, neither addresses the core problem of message retention and unacknowledged messages.
Pub/Sub Message Retention & Dead-Letter Topics
Pub/Sub message retention stores messages for a configurable duration for reprocessing, while Dead-Letter Topics automatically redirect messages that fail to be acknowledged after multiple delivery attempts, preventing data loss.
- Message retention allows replaying messages from a subscription.
- Dead-Letter Topics ensure reliable message delivery by isolating problematic messages.
- Both are critical for robust, fault-tolerant Pub/Sub architectures.
- Dead-lettering can be configured with a maximum delivery attempts threshold.
Memory trick: Retain the message, dead-letter the failed, so nothing's lost and the stream prevailed.