Professional Data Engineer flashcards

Using BigQuery table partitioning, especially by date, to leverage automatic cost reductions for older, less-accessed data that qualifies for long-term storage pricing.

• Divides table into smaller segments
• Data in partitions older than 90 days gets cheaper pricing
• Improves query performance by pruning partitions

Memory trick: Partition your BigQuery tables to pay less for old logs.

Cloud Spanner is a globally distributed, strongly consistent, and highly available relational database service that supports customer-managed encryption keys (CMEK) for data at rest.

• Enterprise-grade, highly scalable relational database.
• Strong global consistency.
• Supports CMEK for enhanced security and compliance.

Memory trick: Spanner Secures Critical Customer Keys Globally.

Pub/Sub subscriptions allow configuring message retention duration to automatically purge messages after a specified time. Dead-letter topics provide a mechanism to catch messages that cannot be successfully processed, preventing loss and enabling debugging or reprocessing.

• Message retention is configured at the subscription level (up to 7 days, or 31 days with topic retention).
• Dead-letter topics are configured on subscriptions.
• Messages sent to a dead-letter topic are not lost.
• Useful for compliance and robust message processing.

Memory trick: Retain the messages, but don't let them die, use a dead-letter, for failures high!

The process of managing BigQuery data from creation to archival or deletion, often using features like table expiration and data retention policies to optimize storage costs and meet compliance.

• Table expiration automatically deletes tables or partitions.
• Data retention policies define minimum data age before deletion or transition.
• Can be set at dataset or table level.

Memory trick: Expire the temporary, retain the old, manage with policies, and save on the gold.

Cloud Interconnect (Dedicated) provides a direct physical connection between an on-premises data center and Google's network, bypassing the public internet.

• Private, dedicated connection.
• High bandwidth and low latency.
• Enhanced security and compliance.
• Requires physical presence at a Google colocation facility.

Memory trick: Private paths for precious packets, bypassing the public's prying eyes.

In Apache Beam, watermarks are a measure of processing progress based on event time, and allowed lateness defines a grace period after a window's watermark passes during which late data can still be processed for that window.

• Watermarks indicate event-time completeness
• Allowed lateness handles data arriving after its window's end
• Prevents indefinite delays while ensuring data accuracy

Memory trick: Watermarks and Lateness: The clock and the grace period.

Core Google Cloud services for observing and managing operations, providing centralized log collection and analysis (Logging) and comprehensive metrics, dashboards, and alerting (Monitoring).

• Cloud Logging aggregates logs from all GCP services.
• Cloud Monitoring collects metrics, provides dashboards, and configures alerts.
• Essential for troubleshooting, performance analysis, and operational health.

Memory trick: Log everything, then Monitor the metrics.

A state in Dataflow streaming pipelines where the system's understanding of event time (watermark) falls significantly behind the actual current event time, leading to increased 'System Latency' and 'Data freshness' issues.

• Indicates a backlog of unprocessed event-time data
• Causes stale results in dashboards
• Often due to bottlenecks in processing, not necessarily resource exhaustion

Memory trick: Slow watermark means data's stuck in time's bottleneck.

Organization Policy Constraints are rules that apply across a Google Cloud organization, folders, or projects to enforce specific behaviors, such as restricting where resources can be created (Resource Location Restriction).

• Enforced at the organization, folder, or project level.
• Prevents creation of resources in non-compliant locations.
• Crucial for data residency and compliance (e.g., GDPR, HIPAA).
• Overrides individual user permissions for resource creation.

Memory trick: Organization Policy's Constraint, keeps your data's location without taint!

Cloud Audit Logs record administrative activities, system events, and data access on Google Cloud. For BigQuery, enabling Data Access logs captures detailed information about queries, including who ran them, the query text, and the resources (tables/columns) accessed, crucial for auditing and compliance.

• Records API calls that read or modify data.
• Captures user identity, operation, and affected resources.
• Essential for security, auditing, and compliance.
• Data Access logs are usually disabled by default for BigQuery and need explicit activation.

Memory trick: To know *who* touched *what* data, *audit* the *access*.

Cloud Key Management Service (KMS) provides various key types, including Hardware Security Module (HSM) keys, which are backed by FIPS 140-2 Level 3 validated HSMs for enhanced security and compliance.

• Manages cryptographic keys for various Google Cloud services.
• HSM keys offer strong, hardware-backed protection for encryption keys.
• Meets strict regulatory and compliance requirements.
• Integrates with other services like Cloud Storage for CMEK.

Memory trick: For keys so secure, KMS with HSM is the cure!

Cloud IAM defines who has access to which Google Cloud resources and what actions they can perform. Cloud Audit Logs, particularly Data Access logs, record data access attempts, providing a comprehensive audit trail of user activities on data, which together form a robust solution for data governance, security, and compliance.

• IAM specifies permissions (e.g., BigQuery Data Viewer, BigQuery Data Editor).
• Cloud Audit Logs record administrative, system, and data access events.
• Data Access logs capture detailed user activity on data, including queries.
• Essential for meeting compliance standards like GDPR and HIPAA.

Memory trick: IAM sets the rules, Audit Logs watch the game.

A Google Cloud service that allows administrators to enforce programmatic restrictions on how resources are configured and deployed across an organization, including restricting where resources can be created (resource location restriction).

• Enforces policies at the organization, folder, or project level.
• Uses constraints to define allowed or disallowed behaviors.
• The 'Resource location restriction' constraint specifically controls data residency.
• Provides centralized, automated governance.

Memory trick: Organization Policy sets the borders, Resource Location constraint draws the line, so data stays where it's meant to shine.

Cloud SQL is a fully managed relational database service on Google Cloud that supports various database engines. It integrates with Customer-Managed Encryption Keys (CMEK) for enhanced data security and compliance.

• Supports PostgreSQL, MySQL, and SQL Server.
• Offers high availability, replication, and automatic backups.
• CMEK provides customer control over encryption keys.

Memory trick: SQL on the Cloud with a Key, keeps your data safe, hip hip hooray!

A fully-managed service for executing Apache Beam pipelines for both batch and stream data processing, offering serverless auto-scaling.

• Serverless and auto-scaling
• Supports batch and stream processing
• Based on Apache Beam

Memory trick: Dataflow crafts data with serverless ease, a transforming river.

Cloud Storage lifecycle management allows you to automatically transition objects between storage classes (e.g., Standard to Nearline to Coldline to Archive) or delete them based on age or versioning rules, optimizing costs and meeting retention policies.

• Rules can be based on object age, creation date, number of versions.
• Transitions data to cheaper storage classes as it ages.
• Helps enforce data retention and deletion policies.
• Minimum storage durations apply to each class (e.g., Nearline 30 days, Coldline 90 days, Archive 365 days).

Memory trick: Standard, Nearline, Coldline, Archive: Your data's journey, from fresh to old archive!

A common Google Cloud pattern for building scalable, reliable, and low-latency real-time data pipelines by combining Cloud Pub/Sub for ingestion and Dataflow for processing.

• Cloud Pub/Sub handles high-volume, global message ingestion.
• Dataflow (streaming) provides flexible, autoscaling, low-latency processing.
• Ensures data reliability and dynamic scalability.

Memory trick: Pub/Sub gets the data in, Dataflow streams it through.

Customer-Managed Encryption Keys (CMEK) for Cloud Storage allow you to use encryption keys that you manage in Cloud Key Management Service (KMS) to encrypt your data at rest in Cloud Storage buckets.

• Provides enhanced control over encryption keys.
• Integrates with Cloud KMS for key management and rotation.
• Data is still encrypted by Google, but with your key.
• Meets many compliance requirements for key control.

Memory trick: CMEK is the Key for control, let IAM roll!

Cloud KMS EKM allows customers to use encryption keys that are managed outside Google Cloud, in their own external key management systems (like on-premises HSMs), while still leveraging Google Cloud services for data storage and processing. Cloud KMS acts as a proxy to these external keys.

• Integrates Google Cloud with customer's external key management systems.
• Keys remain outside Google's infrastructure, managed by the customer.
• Supports high-security requirements like FIPS 140-2 Level 3 HSMs.
• Data encrypted in Google Cloud is protected by keys never directly accessible by Google.

Memory trick: External Key Manager: Your keys, your house, Google accesses them (safely).

Google Cloud Storage is a highly scalable, durable, and available object storage service, commonly used as the foundation for data lakes on Google Cloud.

• Object storage for any data type
• High durability (11 nines) and availability
• Cost-effective for large volumes of data

Memory trick: Cloud Storage: The vast lake for all your data.

BigQuery table expiration automatically deletes tables after a specified duration. Scheduled queries allow you to run recurring SQL queries to transform or move data, enabling automated data lifecycle management.

• Table expiration is set at table creation or updated afterward.
• Scheduled queries can run at defined intervals (e.g., daily, weekly).
• Combine them for automated data retention and aggregation pipelines.
• Essential for compliance with data retention policies like GDPR.

Memory trick: Expire the old, aggregate the new, BigQuery's lifecycle makes compliance true!

Techniques used to reduce the amount of data moved between Dataflow workers during shuffle operations (e.g., GroupByKey), thus improving pipeline performance and reducing execution time.

• High shuffle time indicates a bottleneck.
• Key distribution optimization and combiner functions are key strategies.
• Reduces network I/O and worker load.

Memory trick: Combine keys smartly to reduce the shuffle dance.

A BigQuery feature that allows users to query data through a view without having direct access to the underlying tables, enabling data sharing while maintaining strict access control and data privacy.

• The view owner grants access to the view, not the underlying tables.
• Can be used to filter, project, or aggregate data.
• Queries against views are subject to the view's definition.
• All access is auditable via Cloud Audit Logs.

Memory trick: View the data, but don't touch the raw, audit every glance, to follow the law.

A fully managed, scalable NoSQL wide-column database service for large analytical and operational workloads, offering low-latency reads and writes at high throughput.

• Petabyte-scale, high throughput (millions of ops/sec)
• Low-latency reads and writes
• Flexible schema (wide-column store)
• Ideal for time-series, IoT, and operational analytics

Memory trick: Bigtable: Massive scale, blazing fast.

Google Cloud's unified platform for machine learning development, encompassing data preparation, model training, deployment, and monitoring.

• Offers managed services for ML lifecycle.
• Supports various ML frameworks.
• Scales for large datasets and complex models.

Memory trick: From vast data to live predictions, Vertex AI's got the full ML pipeline covered.

A record of the data's lifecycle, including its origin, where it moves, and what transformations it undergoes.

• Crucial for auditability, compliance, and debugging.
• Provides transparency into data quality and integrity.
• Often visualized as a graph of data sources, processes, and destinations.

Memory trick: Data Lineage is like a 'family tree' for your data, showing where it came from.

The process of continuously evaluating the performance of a deployed machine learning model by comparing its predictions against actual observed outcomes (ground truth).

• Detects concept drift by identifying degradation in model accuracy.
• Requires access to ground truth labels for comparison.
• Essential for maintaining model reliability in production.

Memory trick: Monitor the 'quality' of the 'predictions' like checking if a delivery arrived on time.

The process of tracking key performance indicators (KPIs) that measure the real-world impact and value generated by a deployed machine learning model.

• Connects model performance to organizational goals.
• Often involves A/B testing or observational studies.
• Helps identify misalignment between technical metrics and business value.

Memory trick: Monitor all angles: data, model, and business goals.

A centralized, managed service on Google Cloud for storing, serving, and sharing machine learning features.

• Supports both online (low-latency) and batch serving.
• Ensures consistency between training and serving features.
• Helps in feature reuse across different models.

Memory trick: The 'Feature Store' is like a fast-food counter for your ML model's ingredients.

A managed service within Vertex AI for deploying machine learning models to serve online predictions with high availability, low latency, and automatic scaling.

• Supports various model formats and custom containers.
• Provides managed autoscaling for variable traffic.
• Offers A/B testing and traffic splitting for model updates.

Memory trick: Vertex AI Endpoints: Your model's managed, auto-scaling launchpad for predictions.

A centralized repository for managing the lifecycle of machine learning models on Google Cloud, facilitating versioning, metadata tracking, and deployment.

• Stores model artifacts and metadata.
• Enables model versioning.
• Supports model deployment and discovery.

Memory trick: Register your models for an orderly lifecycle.

The practice of ensuring that data used in machine learning pipelines is accurate, complete, consistent, timely, and valid to produce reliable model performance.

• Critical for model performance and fairness.
• Involves validation, cleansing, and monitoring.
• Should be addressed at every stage of the data pipeline.

Memory trick: Validate and check data at every ML gate.

A fully managed service for discovering, classifying, and protecting sensitive data at scale, including PII, PHI, and financial data.

• Detects over 150 types of sensitive data.
• Supports de-identification, redaction, tokenization.
• Works across various Google Cloud data sources.

Memory trick: DLP prevents data from leaking out.

A Vertex AI service for tracking and managing machine learning experiments, including parameters, metrics, and artifacts.

• Ensures reproducibility of ML research.
• Allows systematic comparison of different model runs.
• Integrates with other Vertex AI services.

Memory trick: For every 'experiment', Vertex AI keeps a 'record' like a lab notebook.

A phenomenon in machine learning where the statistical properties of the target variable, which the model is trying to predict, change over time.

• The relationship between input features and the target variable shifts.
• Can occur even if input feature distributions remain stable.
• Often requires model retraining or adaptation.

Memory trick: The 'concept' of what makes a good recommendation has 'drifted' away.

The geographical location where data is stored and processed, often mandated by legal or regulatory requirements.

• Crucial for compliance in regulated industries.
• Requires careful selection of cloud resource locations.
• Different from data locality or data sovereignty, though related.

Memory trick: Keep the data 'resident' in its own 'region' like a local.

A managed service on Google Cloud for deploying machine learning models for online (real-time) or batch predictions.

• Supports various ML frameworks.
• Offers automatic scaling and high availability.
• Provides endpoints for model serving.

Memory trick: Vertex AI is the peak for ML predictions.

A phenomenon where the statistical properties of the input features to a machine learning model change over time, leading to degraded model performance.

• Also known as covariate shift.
• Can be caused by real-world changes (e.g., market trends, new data sources).
• Requires model retraining or adaptation.

Memory trick: Drifting features make models lose their way.

A Google Cloud service within Vertex AI that provides automated monitoring of deployed machine learning models to detect data drift, concept drift, and performance degradation.

• Detects data and concept drift.
• Monitors model performance metrics.
• Integrates with Vertex AI Endpoints.

Memory trick: Vertex AI watches your model's health, alerting you if it drifts or declines.

A Vertex AI capability that helps users understand the predictions of their machine learning models by identifying feature attributions.

• Provides insights into model behavior and transparency.
• Supports various explanation methods (e.g., SHAP, integrated gradients).
• Crucial for responsible AI and debugging.

Memory trick: Explainable AI 'explains' why the model did what it did, like a detective.

A managed service on Google Cloud for training machine learning models at scale, supporting custom code and various hardware configurations.

• Handles infrastructure provisioning and scaling.
• Supports custom training jobs and hyperparameter tuning.
• Allows use of GPUs and other accelerators.

Memory trick: Vertex AI 'Training' is like a gym for your models, with all the heavy equipment managed.

A component of Vertex AI that helps users understand the output of their machine learning models by providing feature attributions for predictions.

• Enhances model transparency and interpretability.
• Supports various explanation methods (e.g., SHAP, LIME).
• Crucial for regulatory compliance and debugging.

Memory trick: Explainable AI sheds light on 'why' your model predicted what it did.

A Google Cloud service that helps discover, classify, and protect sensitive data across various Google Cloud products and on-premises environments, offering de-identification techniques.

• Detects over 150 types of sensitive data.
• Offers redaction, tokenization, format-preserving encryption.
• Crucial for regulatory compliance (e.g., HIPAA, GDPR).

Memory trick: DLP scrubs your data, making PII invisible before ML sees it.

A serverless MLOps platform on Google Cloud for orchestrating and automating end-to-end machine learning workflows, based on Kubeflow Pipelines.

• Enables repeatable and scalable ML pipelines.
• Manages dependencies and resource allocation.
• Supports custom components and various ML frameworks.

Memory trick: Pipeline your ML, automate the whole flow.

A Vertex AI feature that allows distributing incoming prediction requests across multiple deployed model versions on a single endpoint.

• Enables canary deployments and A/B testing.
• Ensures zero-downtime model updates.
• Managed at the Vertex AI Endpoint level.

Memory trick: Traffic lights guide the new model smoothly onto the road.

Google Cloud Dataflow is a fully managed, serverless service for executing Apache Beam pipelines, enabling unified programming for both batch and streaming data processing with auto-scaling and high performance.

• Unified programming model for batch and streaming.
• Serverless and fully managed service.
• Auto-scaling for dynamic workloads.
• Supports complex data transformations and aggregations.

Memory trick: Data Flow, One Code, Both Streams.

Google Cloud's fully managed, serverless, and highly scalable enterprise data warehouse for analytics, designed for petabyte-scale data and complex SQL queries.

• Serverless architecture, no infrastructure to manage.
• Columnar storage optimized for analytical queries.
• Scales automatically to petabytes of data and thousands of queries.
• Cost-effective with pricing based on data scanned per query.

Memory trick: Big Query for Big Questions.

Dataflow is a fully managed, serverless service for executing Apache Beam pipelines, providing robust fault tolerance, exactly-once processing, and automatic scaling for continuous stream transformations.

• Exactly-once processing guarantees.
• Automatic scaling and resource management.
• Built-in fault tolerance and recovery mechanisms.
• Supports stateful processing for windowing and aggregations.

Memory trick: Dataflow: Your data 'flows' reliably, even when things 'blow'.

A Google Cloud service that allows customers to protect their data at rest with encryption keys managed in an external key management system (KMS) or hardware security module (HSM) that they control.

• Provides cryptographic operations using externally-managed keys.
• Keys never leave the external KMS/HSM.
• Enhances compliance for highly regulated industries.

Memory trick: Keys Everywhere, Managed Securely.

BigQuery's long-term storage is a cost-optimization feature where data that has not been edited for 90 consecutive days automatically receives a 50% discount on storage costs, while remaining immediately available for queries.

• Automatic transition for data not edited for 90 days.
• Provides a 50% discount on storage costs.
• Data remains immediately queryable.
• Ideal for historical, infrequently accessed data.

Memory trick: BigQuery: Save on Storage, Save on Scans.

A fully managed, globally scalable, and durable messaging service that enables asynchronous communication between independent applications, acting as a real-time event bus.

• Decouples publishers and subscribers.
• Scales automatically to millions of events/second.
• Offers durable message storage and guaranteed delivery.
• Supports message retention (default 7 days, configurable).

Memory trick: To send a 'Pub/Sub' message, think of a global post office that never loses a letter, no matter how many are sent.

A serverless, highly scalable, and cost-effective enterprise data warehouse designed for petabyte-scale analytics, offering robust security and compliance features for complex analytical queries.

• Fully managed and serverless.
• Scales automatically to petabytes.
• Optimized for analytical queries (OLAP).
• Cost-effective for massive datasets (storage and query pricing models).

Memory trick: For a mountain of sales data, you need a 'Big Query' to find the gold, not a small shovel.

Google Cloud Dataplex is an intelligent data fabric that enables organizations to unify distributed data, manage, monitor, and govern data across data lakes, data warehouses, and data marts, providing capabilities for data discovery, quality, and lineage.

• Unified data management across diverse data stores.
• Automated data discovery and metadata management.
• Built-in data quality monitoring and remediation.
• Provides data lineage tracking for compliance and understanding.

Memory trick: Dataplex: Data's Perfect Ledger.

This pipeline leverages Cloud Pub/Sub for ingestion, Dataflow for stream processing, Cloud Bigtable for real-time operational access, and BigQuery for historical analytical storage, providing a comprehensive solution for diverse data needs.

• Pub/Sub: Scalable message ingestion.
• Dataflow: Managed stream processing (ETL).
• Cloud Bigtable: Low-latency operational store.
• BigQuery: Petabyte-scale historical analytics.

Memory trick: Pub/Sub gets it IN, Dataflow makes it SHINE, Bigtable shows it NOW, BigQuery saves it FOREVER.

Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow, enabling users to author, schedule, and monitor pipelines programmatically.

• Uses Python DAGs for workflow definition.
• Provides robust scheduling and monitoring.
• Integrates with various Google Cloud services.
• Ideal for complex ETL, ML pipelines, and batch jobs.

Memory trick: Composer 'conducts' your data orchestra.

Cloud Bigtable is a NoSQL wide-column database optimized for large analytical and operational workloads, particularly well-suited for high-throughput, low-latency time-series data like IoT sensor readings due to its efficient storage and retrieval mechanisms.

• Handles millions of writes/reads per second.
• Designed for petabyte-scale data.
• Low-latency access, ideal for time-series and operational data.
• Supports multi-region replication for high availability.

Memory trick: BigTable stores Big Time-Series.

Cloud Bigtable is a sparsely populated table that can scale to billions of rows and thousands of columns, enabling petabyte-scale data storage with very high throughput and low-latency access for operational workloads.

• NoSQL wide-column store.
• Designed for high throughput and low latency.
• Ideal for time-series data, IoT, and operational analytics.

Memory trick: Bigtable is the 'Big, Fast Table' for your operational needs.

Processing and analyzing data as it arrives to derive immediate insights and enable rapid decision-making, crucial for scenarios like live event monitoring or fraud detection.

• Requires low-latency data ingestion and processing.
• Must handle high throughput and sudden data spikes.
• Provides immediate feedback for operational adjustments.
• Often involves message queues, stream processors, and analytical databases.

Memory trick: To monitor a live stream, you need a lightning-fast data river that never stops flowing, feeding directly to your control panel.

A comprehensive security and compliance strategy involving geographical data placement, cryptographic protection of data, precise control over who can access data, and detailed logs of all data interactions.

• Data residency: data remains within specified geographic boundaries.
• Encryption: data is protected at rest and in transit.
• Access control: granular permissions dictate who can interact with data.
• Auditability: all data access and administrative actions are logged.

Memory trick: To protect sensitive data, think of a secure vault: it has a specific location, strong locks, restricted access, and a complete logbook of every entry.

Dataflow is a fully managed, serverless service for executing Apache Beam pipelines, enabling scalable and cost-effective batch and stream data processing without infrastructure management.

• Serverless: No infrastructure to manage.
• Autoscaling: Dynamically adjusts resources.
• Supports Python, Java, Go Beam SDKs.
• Ideal for complex ETL, batch analytics, and stream processing.

Memory trick: Dataflow lets your data 'flow' without server 'woes'.