Certification Exam
A test validating skills in a specific technology.
Getting Started: Exam Overview
Free knowledge base
Professional Data Engineer — key terms, tricks & tips
Everything from the course in one searchable place: 253 entries. Use it to review before a practice test or look up a word you forgot.
253 results
Exam Domains
Major subject areas covered by the certification exam.
Getting Started: Exam Overview
Multiple-Choice
Questions with one correct answer from several options.
Getting Started: Exam Overview
Multiple-Select
Questions requiring selection of all correct answers.
Getting Started: Exam Overview
Proctoring
Supervision of an exam to ensure fairness and integrity.
Getting Started: Exam Overview
Time Management
Strategically allocating time to complete all exam questions.
Getting Started: Exam Overview
Pass/Fail Notification
Immediate result after completing the certification exam.
Getting Started: Exam Overview
Understanding the PDE Exam Structure & Format
DREAM: Design, Realize, Evaluate, Analyze, Manage. Remember these steps for any data engineering project, and you'll cover all exam domains!
Getting Started: Exam Overview
Understanding the PDE Exam Structure & Format
The exam assesses your ability to design, build, operationalize, secure, and monitor data processing systems on Google Cloud, focusing on scalability, availability, and fault tolerance.
Getting Started: Exam Overview
Understanding the PDE Exam Structure & Format
Not reading the entire question or all answer choices before selecting an answer.
Getting Started: Exam Overview
Understanding the PDE Exam Structure & Format
Spending too much time on a single difficult question, leading to insufficient time for easier ones.
Getting Started: Exam Overview
Understanding the PDE Exam Structure & Format
Failing to consider all constraints (cost, latency, security) mentioned in scenario-based questions.
Getting Started: Exam Overview
Official Documentation
Google's authoritative guides for all Cloud products.
Getting Started: Exam Overview
Solution Architectures
Pre-designed patterns for integrating multiple Cloud services.
Getting Started: Exam Overview
Best Practices
Recommended methods for optimal performance, cost, and security.
Getting Started: Exam Overview
Google Cloud Blog
Source for product announcements, updates, and use cases.
Getting Started: Exam Overview
Whitepapers
In-depth technical documents on specific Cloud topics.
Getting Started: Exam Overview
Case Studies
Examples of how companies use Google Cloud solutions.
Getting Started: Exam Overview
Community Forums
Online platforms for user discussion and troubleshooting.
Getting Started: Exam Overview
Navigating Google Cloud Resources for PDE
Docs, Solutions, Blogs, Papers, Cases, Community! (DSBPCC) – The six pillars of Google Cloud knowledge!
Getting Started: Exam Overview
Navigating Google Cloud Resources for PDE
The exam expects you to know *where* to find information and *how* to apply it, not just memorize every detail. Look for keywords like 'best practices,' 'recommended architecture,' or 'cost optimization' in questions, as these often point to concepts covered in solution guides.
Getting Started: Exam Overview
Navigating Google Cloud Resources for PDE
Relying solely on community forums without verifying information against official documentation.
Getting Started: Exam Overview
Navigating Google Cloud Resources for PDE
Ignoring solution architectures and best practice guides, leading to suboptimal designs.
Getting Started: Exam Overview
Navigating Google Cloud Resources for PDE
Not checking for the latest documentation version, especially for rapidly evolving services.
Getting Started: Exam Overview
Scalability
Ability of a system to handle increased workload.
Designing Robust Data Systems
Reliability
Probability a system performs its function without failure.
Designing Robust Data Systems
Fault Tolerance
System's ability to continue operating despite component failures.
Designing Robust Data Systems
High Availability
System design to operate continuously with minimal downtime.
Designing Robust Data Systems
Vertical Scaling
Adding more resources to a single machine.
Designing Robust Data Systems
Horizontal Scaling
Adding more machines to distribute workload.
Designing Robust Data Systems
Encryption at Rest
Data is encrypted when stored on disk.
Designing Robust Data Systems
IAM
Identity and Access Management for granular permissions.
Designing Robust Data Systems
Scalability, Reliability, Fault Tolerance & Security
Imagine a 'SRFTS' party: Scalability brings more guests (workload), Reliability ensures the music never stops, Fault Tolerance means if a speaker breaks, another kicks in, and Security keeps out party crashers!
Designing Robust Data Systems
Scalability, Reliability, Fault Tolerance & Security
The exam often tests the distinction between high availability and fault tolerance. High availability aims to minimize downtime, while fault tolerance ensures continued operation even with component failures. A fault-tolerant system is inherently highly available, but a highly available system might not be fully fault-tolerant if it relies on manual intervention for recovery.
Designing Robust Data Systems
Scalability, Reliability, Fault Tolerance & Security
Confusing High Availability with Disaster Recovery; HA is for local failures, DR is for widespread outages.
Designing Robust Data Systems
Scalability, Reliability, Fault Tolerance & Security
Over-engineering for maximum 'nines' of availability when not truly needed, leading to unnecessary cost and complexity.
Designing Robust Data Systems
Scalability, Reliability, Fault Tolerance & Security
Neglecting security in favor of performance, leaving critical data vulnerable.
Designing Robust Data Systems
Cloud SQL
Managed relational database service for MySQL, PostgreSQL, SQL Server.
Designing Robust Data Systems
Cloud Spanner
Globally distributed, strongly consistent, petabyte-scale relational database.
Designing Robust Data Systems
BigQuery
Serverless, highly scalable enterprise data warehouse for analytics.
Designing Robust Data Systems
Dataflow
Fully managed service for executing Apache Beam batch and streaming pipelines.
Designing Robust Data Systems
Pub/Sub
Real-time messaging service for ingesting and delivering events.
Designing Robust Data Systems
Cloud Storage
Durable, highly available object storage for all data types.
Designing Robust Data Systems
Firestore
NoSQL document database for mobile, web, and serverless applications.
Designing Robust Data Systems
Bigtable
Wide-column NoSQL database for large analytical and operational workloads.
Designing Robust Data Systems
Choosing GCP Products for Data Solutions
SQL, Spanner, Store, Stream, BigQuery, Bigtable: 'S-S-S-S-B-B' for your data needs!
Designing Robust Data Systems
Choosing GCP Products for Data Solutions
The exam frequently presents scenarios and asks you to choose the 'most appropriate' or 'most cost-effective' GCP data product. Pay close attention to keywords like 'transactional,' 'real-time,' 'petabyte-scale,' 'unstructured,' 'low-latency,' and 'fully managed' to guide your selection.
Designing Robust Data Systems
Choosing GCP Products for Data Solutions
Using Cloud SQL for petabyte-scale analytical queries instead of BigQuery.
Designing Robust Data Systems
Choosing GCP Products for Data Solutions
Attempting real-time streaming analytics with Dataproc batch jobs instead of Dataflow.
Designing Robust Data Systems
Choosing GCP Products for Data Solutions
Storing unstructured data in Cloud Spanner when Cloud Storage is more appropriate and cost-effective.
Designing Robust Data Systems
Data Pipeline
Series of steps to move, transform, and deliver data.
Designing Robust Data Systems
Data Warehouse
Structured repository for reporting and analysis (schema-on-write).
Designing Robust Data Systems
Data Lake
Repository for raw, unstructured data (schema-on-read).
Designing Robust Data Systems
Schema-on-Write
Data conforms to schema upon ingestion (data warehouse).
Designing Robust Data Systems
Schema-on-Read
Schema applied when data is accessed (data lake).
Designing Robust Data Systems
ETL
Extract, Transform, Load; common pipeline pattern.
Designing Robust Data Systems
ELT
Extract, Load, Transform; common pipeline pattern.
Designing Robust Data Systems
Designing Data Pipelines, Warehouses & Lakes
Think 'LAKE' for 'Lots of Any Kind of Everything' (raw data) and 'WAREHOUSE' for 'Well-Organized, Ready for Heavy-Duty Reporting' (structured data).
Designing Robust Data Systems
Designing Data Pipelines, Warehouses & Lakes
The exam often tests your ability to choose the right data storage and processing solution for a given scenario. Pay close attention to keywords like 'raw data,' 'unstructured,' 'machine learning' (suggests data lake) versus 'structured,' 'reporting,' 'BI dashboards' (suggests data warehouse). Also, differentiate between batch and streaming pipeline requirements.
Designing Robust Data Systems
Designing Data Pipelines, Warehouses & Lakes
Using a data warehouse for raw, unstructured data that will primarily be used for machine learning, leading to unnecessary schema definition and transformation overhead.
Designing Robust Data Systems
Designing Data Pipelines, Warehouses & Lakes
Attempting to perform complex, real-time analytics directly on a traditional data warehouse not optimized for such workloads, resulting in slow query performance.
Designing Robust Data Systems
Designing Data Pipelines, Warehouses & Lakes
Not establishing clear data governance and quality processes for data ingested into a data lake, leading to a 'data swamp' that is difficult to use.
Designing Robust Data Systems
Batch Processing
Processing data in large, discrete chunks at scheduled intervals.
Designing Robust Data Systems
Streaming Processing
Processing data continuously as it arrives in real-time.
Designing Robust Data Systems
Data Governance
Policies and procedures for managing data quality, security, and compliance.
Designing Robust Data Systems
Cost Optimization
Strategies to reduce cloud spending while maintaining performance.
Designing Robust Data Systems
Data Catalog
A metadata management service for data discovery and governance.
Designing Robust Data Systems
Cloud IAM
Google Cloud's Identity and Access Management service for permissions.
Designing Robust Data Systems
BigQuery Reservations
A pricing model for BigQuery providing fixed-rate query capacity.
Designing Robust Data Systems
Batch, Streaming, Governance & Cost Optimization
B.A.T.C.H. for Batch: Big Analytics, Timely, Chunks, Historical. S.T.R.E.A.M. for Streaming: Swift, Timely, Real-time, Events, Always Moving.
Designing Robust Data Systems
Batch, Streaming, Governance & Cost Optimization
The exam often tests your ability to choose between batch and streaming solutions based on latency requirements and data volume. Pay close attention to scenario questions that describe 'real-time' vs. 'daily reports' needs. For governance, remember Cloud IAM for access, Data Catalog for discovery, and Audit Logs for compliance.
Designing Robust Data Systems
Batch, Streaming, Governance & Cost Optimization
Confusing batch and streaming use cases; applying batch solutions where real-time is needed, or vice-versa.
Designing Robust Data Systems
Batch, Streaming, Governance & Cost Optimization
Ignoring data governance, leading to security vulnerabilities, compliance issues, and data quality problems.
Designing Robust Data Systems
Batch, Streaming, Governance & Cost Optimization
Failing to monitor and optimize cloud spending, resulting in unexpectedly high bills.
Designing Robust Data Systems
Batch, Streaming, Governance & Cost Optimization
Over-provisioning resources for data processing jobs, leading to unnecessary costs.
Designing Robust Data Systems
Batch Ingestion
Processing data in large chunks at scheduled intervals.
Building & Operating Data Pipelines
Streaming Ingestion
Continuous processing of data as it arrives in real-time.
Building & Operating Data Pipelines
Cloud Pub/Sub
Scalable, asynchronous messaging service for real-time data.
Building & Operating Data Pipelines
Cloud Dataflow
Fully managed service for batch and streaming data processing.
Building & Operating Data Pipelines
Storage Transfer Service
Moves large amounts of data into Cloud Storage efficiently.
Building & Operating Data Pipelines
Data Transformation
Cleaning, enriching, aggregating, or restructuring data.
Building & Operating Data Pipelines
Data Ingestion & Transformation Techniques
To remember the core ingestion services: 'P-S-D-S' for Pub/Sub (Streaming), Storage Transfer (Batch), Dataflow (Both), Dataproc (Batch/Open Source).
Building & Operating Data Pipelines
Data Ingestion & Transformation Techniques
The exam frequently asks about choosing the MOST appropriate GCP service for a given ingestion or transformation scenario. Pay close attention to keywords like 'real-time,' 'batch,' 'large volume,' 'on-premises,' 'database migration,' or 'streaming analytics' to guide your service selection.
Building & Operating Data Pipelines
Data Ingestion & Transformation Techniques
Confusing batch and streaming use cases and services. Remember, real-time needs Pub/Sub and streaming Dataflow.
Building & Operating Data Pipelines
Data Ingestion & Transformation Techniques
Underestimating the complexity of data cleaning and transformation; it's often the most time-consuming part.
Building & Operating Data Pipelines
Data Ingestion & Transformation Techniques
Not considering the cost implications of different ingestion and transformation services for varying data volumes.
Building & Operating Data Pipelines
Dataproc
Managed service for Apache Spark/Hadoop clusters for big data.
Building & Operating Data Pipelines
Stream Processing
Processing data continuously as it arrives, in real-time or near real-time.
Building & Operating Data Pipelines
Data Storage & Processing on GCP
STAMP your data: Storage, Transformation, Analytics, Messaging, Processing. This helps remember the categories of services.
Building & Operating Data Pipelines
Data Storage & Processing on GCP
The exam frequently presents scenarios and asks you to choose the 'best' or 'most cost-effective' GCP service. Look for keywords like 'real-time,' 'low-latency,' 'unstructured,' 'petabyte-scale SQL,' 'managed relational,' or 'Apache Spark/Hadoop' to guide your service selection.
Building & Operating Data Pipelines
Data Storage & Processing on GCP
Confusing Cloud SQL (relational) with Cloud Spanner (globally distributed relational) or Firestore (NoSQL document).
Building & Operating Data Pipelines
Data Storage & Processing on GCP
Choosing Dataproc for simple ETL when Dataflow or Cloud Data Fusion would be more serverless and cost-effective.
Building & Operating Data Pipelines
Data Storage & Processing on GCP
Underestimating the importance of Pub/Sub for real-time data ingestion in streaming architectures.
Building & Operating Data Pipelines
DAG
Directed Acyclic Graph; a collection of tasks with dependencies.
Building & Operating Data Pipelines
Operator
A template for a task, defining a specific piece of work.
Building & Operating Data Pipelines
Sensor
A special operator that waits for a condition to be met.
Building & Operating Data Pipelines
Scheduler
Airflow component that parses DAGs and schedules tasks.
Building & Operating Data Pipelines
Webserver
Airflow component providing the UI for monitoring.
Building & Operating Data Pipelines
Worker
Executes tasks as instructed by the scheduler.
Building & Operating Data Pipelines
Hook
Interface to external platforms and databases.
Building & Operating Data Pipelines
Orchestrating Data Pipelines with Cloud Composer
Composer's DAGs Orchestrate Tasks: DAGs are the core, Orchestration is the goal, Tasks are the steps.
Building & Operating Data Pipelines
Orchestrating Data Pipelines with Cloud Composer
On the exam, expect questions about Cloud Composer's role in orchestrating complex data pipelines, its integration with other GCP services, and the core components of Apache Airflow (DAGs, Operators, Sensors, Scheduler, Webserver, Workers). Pay attention to scenarios involving task dependencies and failure handling.
Building & Operating Data Pipelines
Orchestrating Data Pipelines with Cloud Composer
Overlooking the importance of idempotency in tasks, leading to incorrect results on retries.
Building & Operating Data Pipelines
Orchestrating Data Pipelines with Cloud Composer
Not properly configuring Airflow connections and variables, causing tasks to fail due to authentication or missing parameters.
Building & Operating Data Pipelines
Orchestrating Data Pipelines with Cloud Composer
Ignoring Cloud Composer environment sizing, leading to performance bottlenecks or excessive costs for your workloads.
Building & Operating Data Pipelines
Cloud Monitoring
Google Cloud service for collecting metrics, events, and metadata.
Building & Operating Data Pipelines
Cloud Logging
Google Cloud service for centralized log collection and analysis.
Building & Operating Data Pipelines
Structured Logging
Logging where entries are formatted as parseable JSON objects.
Building & Operating Data Pipelines
Log-based Metrics
Metrics derived from log entries, integrated with Cloud Monitoring.
Building & Operating Data Pipelines
Autoscaling
Automatically adjusts compute resources based on workload.
Building & Operating Data Pipelines
Preemptible VMs
Low-cost, short-lived virtual machines suitable for fault-tolerant jobs.
Building & Operating Data Pipelines
Data Lifecycle
Managing data through its stages, from creation to deletion.
Building & Operating Data Pipelines
Hot Spot
An uneven distribution of data or work, causing bottlenecks.
Building & Operating Data Pipelines
Monitoring, Logging, Troubleshooting & Optimization
To remember the troubleshooting flow: 'Mighty Little Trouble-Shooters Find Interesting Solutions Quickly!' (Monitor, Log, Troubleshoot, Fix, Solve, Query)
Building & Operating Data Pipelines
Monitoring, Logging, Troubleshooting & Optimization
The exam frequently tests your knowledge of which Google Cloud services to use for specific monitoring and logging tasks. Memorize that Cloud Monitoring is for metrics and alerts, and Cloud Logging is for detailed log analysis and troubleshooting. Keywords like 'real-time performance,' 'resource utilization,' or 'alerting' point to Cloud Monitoring. Keywords like 'error messages,' 'stack traces,' or 'debugging' point to Cloud Logging.
Building & Operating Data Pipelines
Monitoring, Logging, Troubleshooting & Optimization
Ignoring alerts: Not responding to alerts can lead to prolonged outages or data quality issues.
Building & Operating Data Pipelines
Monitoring, Logging, Troubleshooting & Optimization
Lack of structured logging: Unstructured logs are difficult to parse and make troubleshooting much harder.
Building & Operating Data Pipelines
Monitoring, Logging, Troubleshooting & Optimization
Not optimizing queries: Running inefficient queries, especially in BigQuery, can lead to unexpectedly high costs.
Building & Operating Data Pipelines
Latency
Time taken for a data operation to complete.
Building & Operating Data Pipelines
Throughput
Amount of data processed per unit of time.
Building & Operating Data Pipelines
Materialized Views
Pre-computed query results for faster BigQuery access.
Building & Operating Data Pipelines
Dataflow Prime
Intelligent Dataflow service for cost and performance optimization.
Building & Operating Data Pipelines
Storage Class
Tier of Cloud Storage based on access frequency and cost.
Building & Operating Data Pipelines
Egress Costs
Cost associated with data leaving a network or region.
Building & Operating Data Pipelines
Advanced Performance & Cost Optimization
To 'PACE' your pipeline: Partition, Autoscale, Cluster, Egress-minimize.
Building & Operating Data Pipelines
Advanced Performance & Cost Optimization
The exam frequently tests BigQuery pricing models (on-demand vs. flat-rate) and optimization techniques like partitioning, clustering, and materialized views. For Dataflow, know autoscaling, machine types, and Dataflow Prime benefits. Understand that moving data between regions incurs egress costs.
Building & Operating Data Pipelines
Advanced Performance & Cost Optimization
Using `SELECT *` in BigQuery production queries without a `LIMIT` or strong `WHERE` clause, leading to high scan costs.
Building & Operating Data Pipelines
Advanced Performance & Cost Optimization
Not enabling autoscaling or choosing inappropriate machine types for Dataflow jobs, resulting in over-provisioning or under-provisioning.
Building & Operating Data Pipelines
Advanced Performance & Cost Optimization
Ignoring Cloud Storage lifecycle policies, leading to expensive storage of infrequently accessed data in Standard class.
Building & Operating Data Pipelines
ML Pipeline
Automated workflow for machine learning from data to deployment.
Operationalizing Machine Learning
Vertex AI Pipelines
Managed service for orchestrating and automating ML workflows on GCP.
Operationalizing Machine Learning
Vertex AI Training
Managed service for training custom machine learning models.
Operationalizing Machine Learning
Vertex AI Endpoints
Managed service for deploying and serving ML models for predictions.
Operationalizing Machine Learning
Kubeflow Pipelines
Open-source platform for building and deploying portable ML workflows.
Operationalizing Machine Learning
Feature Engineering
Process of transforming raw data into features for ML models.
Operationalizing Machine Learning
Orchestration
Automating and managing the execution of complex workflows.
Operationalizing Machine Learning
Designing & Building ML Pipelines on GCP
To remember the ML pipeline stages, think: 'I Prepare To Evaluate Deploy Predict!' (Ingest, Prepare, Train, Evaluate, Deploy, Predict)
Operationalizing Machine Learning
Designing & Building ML Pipelines on GCP
The exam often tests your ability to choose the right GCP service for each stage of an ML pipeline. Look for keywords like 'large-scale data transformation' (Dataflow), 'managed model training' (Vertex AI Training), 'real-time predictions' (Vertex AI Endpoints), and 'workflow automation' (Vertex AI Pipelines).
Operationalizing Machine Learning
Designing & Building ML Pipelines on GCP
Underestimating the complexity of data preprocessing and feature engineering, which often consumes the most time in an ML project.
Operationalizing Machine Learning
Designing & Building ML Pipelines on GCP
Failing to implement proper monitoring and alerting for deployed models, leading to undetected performance degradation.
Operationalizing Machine Learning
Designing & Building ML Pipelines on GCP
Not using a dedicated orchestration tool like Vertex AI Pipelines, resulting in manual, brittle, and non-reproducible workflows.
Operationalizing Machine Learning
Online Prediction
Real-time inference serving with low latency for immediate results.
Operationalizing Machine Learning
Batch Prediction
Asynchronous inference for large datasets, not requiring immediate results.
Operationalizing Machine Learning
Canary Deployment
Gradually rolling out a new model version to a small subset of users.
Operationalizing Machine Learning
A/B Testing
Comparing two model versions simultaneously with different user groups.
Operationalizing Machine Learning
Model Drift
Degradation of model performance due to changes in data or relationships.
Operationalizing Machine Learning
Data Drift
Changes in the statistical properties of the input features over time.
Operationalizing Machine Learning
Concept Drift
Changes in the relationship between input features and the target variable.
Operationalizing Machine Learning
Vertex AI Model Monitoring
GCP service for detecting data and concept drift in deployed models.
Operationalizing Machine Learning
Deploying & Monitoring ML Models
To remember types of drift: 'Data' is about the 'D'ata you 'D'eliver; 'Concept' is about the 'C'hanges in the 'C'ore idea.
Operationalizing Machine Learning
Deploying & Monitoring ML Models
The exam expects you to differentiate between online and batch prediction, and understand the purpose of canary deployments and A/B testing. Be familiar with Vertex AI Endpoints for deployment and Vertex AI Model Monitoring for drift detection.
Operationalizing Machine Learning
Deploying & Monitoring ML Models
Deploying a model directly to 100% of traffic without any staged rollout or testing, risking widespread negative impact if issues arise.
Operationalizing Machine Learning
Deploying & Monitoring ML Models
Neglecting to monitor ML-specific metrics (like accuracy or F1-score) and only focusing on technical metrics (like latency), missing critical performance degradation.
Operationalizing Machine Learning
Deploying & Monitoring ML Models
Not having an automated process for model retraining or an incident response plan for when model performance inevitably degrades due to drift.
Operationalizing Machine Learning
Data Quality
Accuracy, completeness, consistency, and reliability of data.
Operationalizing Machine Learning
Data Validation
Process of checking data for correctness, compliance, and consistency.
Operationalizing Machine Learning
Model Versioning
Tracking and managing different iterations of an ML model.
Operationalizing Machine Learning
Model Metadata
Descriptive information about an ML model, its training, and performance.
Operationalizing Machine Learning
Model Registry
Centralized repository for managing the lifecycle of ML models.
Operationalizing Machine Learning
Schema Validation
Verifying that data conforms to expected structure and types.
Operationalizing Machine Learning
Managing ML Models & Data Quality for ML
DQ-VTM: Data Quality, Validation, Transformation, Model Registry. Remember, Data Quality is Very Truly Marvelous!
Operationalizing Machine Learning
Managing ML Models & Data Quality for ML
On the exam, look for questions that test your understanding of how to maintain model performance over time. Keywords like 'data drift,' 'model decay,' 'reproducibility,' and 'rollback' often point to solutions involving robust data validation, model versioning, and model registries. Be familiar with Vertex AI Model Registry's capabilities.
Operationalizing Machine Learning
Managing ML Models & Data Quality for ML
Neglecting continuous data validation in production, leading to silent model degradation.
Operationalizing Machine Learning
Managing ML Models & Data Quality for ML
Failing to version models and their associated training data, making reproducibility and rollbacks difficult or impossible.
Operationalizing Machine Learning
Managing ML Models & Data Quality for ML
Treating model deployment as a 'fire and forget' operation without proper metadata tracking and registry usage.
Operationalizing Machine Learning
Reproducibility
Ability to re-create an ML experiment or result exactly.
Operationalizing Machine Learning
Data Versioning
Tracking changes and retrieving specific versions of datasets.
Operationalizing Machine Learning
Code Versioning
Managing and tracking changes to source code, typically with Git.
Operationalizing Machine Learning
Environment Management
Ensuring consistent software dependencies and configurations.
Operationalizing Machine Learning
Experiment Tracking
Logging metadata like hyperparameters, metrics, and artifacts.
Operationalizing Machine Learning
ML Lineage
The historical record of how an ML model was created and evolved.
Operationalizing Machine Learning
Containerization
Packaging an application and its dependencies into a portable unit.
Operationalizing Machine Learning
Ensuring Reproducibility in ML Workflows
Remember R.E.C.O.R.D. for Reproducibility: **R**aw data, **E**nvironment, **C**ode, **O**rchestration, **R**esults (metrics), **D**ocumentation.
Operationalizing Machine Learning
Ensuring Reproducibility in ML Workflows
The exam often tests your understanding of the components that ensure reproducibility. Look for keywords like 'versioning data', 'versioning code', 'environment consistency', 'experiment tracking', and 'pipeline orchestration' as solutions to reproducibility challenges.
Operationalizing Machine Learning
Ensuring Reproducibility in ML Workflows
Failing to version datasets, leading to ambiguity about which data was used for training.
Operationalizing Machine Learning
Ensuring Reproducibility in ML Workflows
Not containerizing environments, causing 'works on my machine' issues when moving between stages or collaborators.
Operationalizing Machine Learning
Ensuring Reproducibility in ML Workflows
Manually tracking experiment parameters, which is prone to errors and incompleteness.
Operationalizing Machine Learning
Encryption in Transit
Data is encrypted while moving between systems, protecting it during network transfer.
Ensuring Solution Quality & Best Practices
GDPR
General Data Protection Regulation; EU law on data protection and privacy.
Ensuring Solution Quality & Best Practices
HIPAA
Health Insurance Portability and Accountability Act; US law for protected health information.
Ensuring Solution Quality & Best Practices
Data Residency
The physical location where data is stored, often mandated by compliance rules.
Ensuring Solution Quality & Best Practices
Idempotency
An operation that produces the same result whether executed once or multiple times.
Ensuring Solution Quality & Best Practices
Security, Compliance & Reliability Design
Remember 'SCR' for Security, Compliance, Reliability. Think of a 'SCR'eaming data engineer if these aren't handled!
Ensuring Solution Quality & Best Practices
Security, Compliance & Reliability Design
The exam often asks about specific Google Cloud services that address security, compliance, or reliability. Keywords like 'data residency,' 'least privilege,' 'encryption,' 'disaster recovery,' and specific regulations (GDPR, HIPAA) are common. Know which services (e.g., Cloud KMS, VPC Service Controls, BigQuery's multi-region options) fit each category.
Ensuring Solution Quality & Best Practices
Security, Compliance & Reliability Design
Confusing security with compliance: Security is about protection; compliance is about following rules, which often uses security measures.
Ensuring Solution Quality & Best Practices
Security, Compliance & Reliability Design
Neglecting to test disaster recovery plans: A plan is useless if it hasn't been validated to work under pressure.
Ensuring Solution Quality & Best Practices
Security, Compliance & Reliability Design
Over-provisioning permissions: Granting too much access (violating least privilege) is a common security vulnerability.
Ensuring Solution Quality & Best Practices
Failover
Automatic switch to a standby system upon primary system failure.
Ensuring Solution Quality & Best Practices
Data Lineage
Understanding the origin, movement, and transformations of data.
Ensuring Solution Quality & Best Practices
Scalability, High Availability & Data Governance
S.H.A.G. - Scalability, High Availability, Governance. Remember to SHAG your data solutions for robust design!
Ensuring Solution Quality & Best Practices
Scalability, High Availability & Data Governance
The exam often presents scenarios where you need to choose the most appropriate Google Cloud service to meet requirements for scalability, high availability, or data governance. Look for keywords like 'handle growing data,' 'minimize downtime,' 'ensure data quality,' or 'comply with regulations.'
Ensuring Solution Quality & Best Practices
Scalability, High Availability & Data Governance
Underestimating future growth, leading to non-scalable designs.
Ensuring Solution Quality & Best Practices
Scalability, High Availability & Data Governance
Failing to identify single points of failure, compromising high availability.
Ensuring Solution Quality & Best Practices
Scalability, High Availability & Data Governance
Neglecting data governance, resulting in data quality issues or compliance breaches.
Ensuring Solution Quality & Best Practices
Lifecycle Policy
Automated rules to manage object transitions between storage classes.
Ensuring Solution Quality & Best Practices
Data Egress
Data transferred out of a cloud region or to the internet.
Ensuring Solution Quality & Best Practices
Right-sizing
Matching resource allocation to actual workload requirements.
Ensuring Solution Quality & Best Practices
Budget Alerts
Notifications when cloud spending approaches a set threshold.
Ensuring Solution Quality & Best Practices
Partitioning
Dividing a table into smaller segments for improved query performance and cost.
Ensuring Solution Quality & Best Practices
Clustering
Ordering data within partitions by specified columns for faster filtering.
Ensuring Solution Quality & Best Practices
Cost Optimization Strategies for Data Solutions
For storage classes, remember 'S-N-C-A': Standard is New, Nearline is Not-so-new, Coldline is Cold, Archive is Ancient. Each gets cheaper but slower.
Ensuring Solution Quality & Best Practices
Cost Optimization Strategies for Data Solutions
The exam often tests your knowledge of Google Cloud Storage classes and their typical use cases, as well as BigQuery cost drivers. Memorize the cost implications of `SELECT *` in BigQuery and the benefits of partitioning/clustering.
Ensuring Solution Quality & Best Practices
Cost Optimization Strategies for Data Solutions
Over-provisioning resources without considering actual usage patterns, leading to unnecessary compute costs.
Ensuring Solution Quality & Best Practices
Cost Optimization Strategies for Data Solutions
Ignoring Cloud Storage lifecycle policies, keeping infrequently accessed data in expensive Standard storage.
Ensuring Solution Quality & Best Practices
Cost Optimization Strategies for Data Solutions
Using `SELECT *` in BigQuery queries, causing full table scans and higher processing costs.
Ensuring Solution Quality & Best Practices
Cost Optimization Strategies for Data Solutions
Neglecting to monitor billing reports and set budget alerts, leading to unexpected high bills.
Ensuring Solution Quality & Best Practices
Unit Testing
Testing individual components or functions of a data pipeline.
Ensuring Solution Quality & Best Practices
Alerting Policy
A rule that triggers notifications when monitored conditions are met.
Ensuring Solution Quality & Best Practices
Data Freshness
How recently data was updated or ingested into a system.
Ensuring Solution Quality & Best Practices
End-to-End Testing
Testing the entire data pipeline from source to destination.
Ensuring Solution Quality & Best Practices
Testing, Validation, Monitoring & Alerting
To ensure your data is 'TV-M.A.D.' (Tested, Validated, Monitored, Alerted, and Delivered), remember these steps for a healthy pipeline!
Ensuring Solution Quality & Best Practices
Testing, Validation, Monitoring & Alerting
The exam often tests your ability to choose the right Google Cloud service for monitoring, logging, and alerting. Know that Cloud Monitoring handles metrics and alerts, while Cloud Logging centralizes logs. Be prepared to differentiate between data validation and data testing.
Ensuring Solution Quality & Best Practices
Testing, Validation, Monitoring & Alerting
Not implementing automated tests and validation, leading to manual, error-prone checks.
Ensuring Solution Quality & Best Practices
Testing, Validation, Monitoring & Alerting
Setting up too many alerts for non-critical issues (alert fatigue) or too few for critical ones.
Ensuring Solution Quality & Best Practices
Testing, Validation, Monitoring & Alerting
Failing to monitor data quality metrics, focusing only on infrastructure performance.
Ensuring Solution Quality & Best Practices
Lifecycle Management
Automated rules for transitioning or deleting objects based on age/conditions.
Managing & Securing Data Assets
Object
A piece of data stored in Cloud Storage, along with its metadata.
Managing & Securing Data Assets
Bucket
A container for objects in Cloud Storage.
Managing & Securing Data Assets
Managing Data Storage & Lifecycle
SQL for Structured Queries, Spanner for Spreading Globally, Bigtable for Big Data Tables. Remember the 'S' for SQL and 'S' for Spanner's global 'S'pread.
Managing & Securing Data Assets
Managing Data Storage & Lifecycle
The exam often tests your ability to choose the most cost-effective storage solution. Pay attention to keywords like 'infrequently accessed,' 'archival,' 'high-throughput,' or 'transactional' to guide your service and storage class selection.
Managing & Securing Data Assets
Managing Data Storage & Lifecycle
Storing all data in Cloud Storage Standard, leading to unnecessarily high costs.
Managing & Securing Data Assets
Managing Data Storage & Lifecycle
Not implementing lifecycle policies, resulting in manual and inefficient data management.
Managing & Securing Data Assets
Managing Data Storage & Lifecycle
Using a relational database (e.g., Cloud SQL) for petabyte-scale, low-latency NoSQL workloads, leading to performance issues.
Managing & Securing Data Assets
Least Privilege
Granting minimum necessary permissions to perform a task.
Managing & Securing Data Assets
Role
A collection of permissions that can be granted to a member.
Managing & Securing Data Assets
Service Account
Special account used by applications or VMs, not human users.
Managing & Securing Data Assets
CMEK
Customer-Managed Encryption Keys; customer controls encryption keys.
Managing & Securing Data Assets
VPC Service Controls
Creates security perimeters to prevent data exfiltration.
Managing & Securing Data Assets
Cloud Audit Logs
Records administrative activities and data access events.
Managing & Securing Data Assets
Data Access Control & Security Best Practices
Remember 'PAL' for Permissions, Accounts, Logs: Permissions (least privilege), Accounts (service accounts), Logs (audit logs).
Managing & Securing Data Assets
Data Access Control & Security Best Practices
The exam frequently tests your understanding of IAM roles and their scope. Know the difference between primitive, predefined, and custom roles, and when to use each. Keywords like 'least privilege', 'data exfiltration', and 'service account' are strong indicators for IAM-related questions.
Managing & Securing Data Assets
Data Access Control & Security Best Practices
Granting primitive roles (Owner, Editor) instead of specific predefined roles.
Managing & Securing Data Assets
Data Access Control & Security Best Practices
Using a single service account for multiple applications with varying needs.
Managing & Securing Data Assets
Data Access Control & Security Best Practices
Forgetting to audit IAM policies regularly, leading to stale or over-provisioned access.
Managing & Securing Data Assets
CCPA
California Consumer Privacy Act, US state law on consumer data rights.
Managing & Securing Data Assets
PHI
Protected Health Information, covered by HIPAA.
Managing & Securing Data Assets
PCI DSS
Payment Card Industry Data Security Standard for card data.
Managing & Securing Data Assets
Data Minimization
Collecting only essential data for a specific purpose.
Managing & Securing Data Assets
Right to Erasure
Individual's right to have personal data deleted (GDPR).
Managing & Securing Data Assets
Shared Responsibility Model
Cloud provider secures the cloud, customer secures in the cloud.
Managing & Securing Data Assets
Data Privacy & Compliance Regulations
G-D-P-R: 'Good Data Practices Rule!' reminds you of its comprehensive nature. C-C-P-A: 'Consumers Control Personal Access' highlights individual rights.
Managing & Securing Data Assets
Data Privacy & Compliance Regulations
The exam often asks about the core principles and individual rights granted by GDPR and CCPA. Keywords to spot include 'right to be forgotten' (GDPR), 'opt-out of sale' (CCPA), and 'data residency.' Remember that GDPR applies to EU residents, and CCPA/CPRA to California residents, regardless of company location.
Managing & Securing Data Assets
Data Privacy & Compliance Regulations
Assuming Google Cloud's compliance certifications mean your application is automatically compliant; you still have a shared responsibility.
Managing & Securing Data Assets
Data Privacy & Compliance Regulations
Confusing the scope of GDPR (EU residents globally) with CCPA (California residents).
Managing & Securing Data Assets
Data Privacy & Compliance Regulations
Underestimating the technical effort required to implement data subject rights (e.g., deletion, access requests) across complex data landscapes.
Managing & Securing Data Assets
Data Owner
Accountable for specific datasets, defines usage and access.
Managing & Securing Data Assets
Data Steward
Manages data quality, definitions, and access requests.
Managing & Securing Data Assets
Data Loss Prevention (DLP)
Identifies, classifies, and protects sensitive data.
Managing & Securing Data Assets
Implementing Data Governance Policies
GOVERN: **G**uidelines, **O**wnership, **V**alidation, **E**nforcement, **R**eporting, **N**etwork (of stakeholders).
Managing & Securing Data Assets
Implementing Data Governance Policies
The exam often tests your understanding of which Google Cloud services map to specific data governance functions. Memorize that Data Catalog is for metadata and discovery, DLP for sensitive data, and IAM for access control.
Managing & Securing Data Assets
Implementing Data Governance Policies
Confusing data governance (strategic framework) with data management (operational tasks).
Managing & Securing Data Assets
Implementing Data Governance Policies
Underestimating the importance of non-technical aspects like roles, policies, and culture.
Managing & Securing Data Assets
Implementing Data Governance Policies
Failing to integrate data governance tools with existing data pipelines and services.
Managing & Securing Data Assets