Microsoft Azure Data FundamentalsDescribe an analytics workload on AzureMedium
A data science team is building a fraud detection model. They need to process large volumes of historical transaction data (petabytes) and apply complex machine learning algorithms, including feature engineering and model training. The team prefers to use open-source frameworks like Apache Spark and Python libraries. Which Azure service would be most appropriate for their needs?
- AAzure SQL Database
- BAzure Databricks
- CAzure Synapse Analytics dedicated SQL pool
- DAzure Stream Analytics
Show answer & explanationAnswer & explanation
Correct answer: B. Azure Databricks
Azure Databricks is an Apache Spark-based analytics platform optimized for the Azure cloud. It provides a collaborative environment for data scientists and engineers to process large datasets, run complex Spark jobs, and perform machine learning tasks using Python, Scala, R, and SQL.
Why the other options are wrong
- A. Azure SQL Database is a relational database and not designed for petabyte-scale big data processing or complex machine learning with Spark.
- C. Azure Synapse Analytics dedicated SQL pool is a data warehousing solution optimized for structured data and T-SQL queries, not for Spark-based ML.
- D. Azure Stream Analytics is for real-time stream processing, not for batch processing of historical data or complex ML model training.
Azure Databricks
A unified, Apache Spark-based analytics platform optimized for Azure, providing collaborative workspaces for data engineering, data science, and machine learning.
- Managed Apache Spark clusters for big data processing.
- Supports Python, Scala, R, SQL for data science and ML.
- Ideal for large-scale ETL, stream processing, and advanced analytics.
Memory trick: Databricks is where data scientists get to 'spark' their models.