Course Overview
Data pipelines typically fall under one of the Extract and Load (EL), Extract, Load and Transform (ELT) or Extract, Transform and Load (ETL) paradigms. This course describes which paradigm should be used and when for batch data. Furthermore, this course covers several technologies on Google Cloud for data transformation including BigQuery, executing Spark on Dataproc, pipeline graphs in Cloud Data Fusion and serverless data processing with Dataflow. Learners get hands-on experience building data pipeline components on Google Cloud using Qwiklabs.
Who should attend
This course is intended for developers who are responsible for designing pipelines and architectures for data processing.
Prerequisites
- Experience with data modeling and ETL (extract, transform, load) activities.
- Experience with developing applications by using a common programming language such as Python or Java.
Course Objectives
- Review different methods of data loading: EL, ELT and ETL and when to use what.
- Run Hadoop on Dataproc, use Cloud Storage, and optimize Dataproc jobs.
- Build your data processing pipelines by using Dataflow.
- Manage data pipelines with Data Fusion and Cloud Composer
Outline: Building Batch Data Pipelines on Google Cloud (BBDP)
Module 1 - Introduction to Building Batch Data Pipelines
Topics:
- EL, ELT, ETL
- Quality considerations
- How to conduct operations in BigQuery
- Shortcomings
- ETL to solve data quality issues
Objectives:
- Review different methods of loading data into your data lakes and warehouses: EL, ELT and ETL
Module 2 - Executing Spark on Dataproc
Topics:
- The Hadoop ecosystem
- Run Hadoop on Dataproc
- Cloud Storage instead of HDFS
- Optimizing Dataproc
Objectives:
- Review the Hadoop ecosystem.
- Discuss how to lift and shift your existing Hadoop workloads to the cloud using Dataproc.
- Explain when to use Cloud Storage instead of HDFS storage.
- Explain how to optimize your Dataproc jobs.
Module 3 - Serverless Data Processing with Dataflow
Topics:
- Introduction to Dataflow
- Why customers value Dataflow
- Dataflow pipelines
- Aggregate with GroupByKey and Combine
- Side inputs and windows
- Dataflow templates
Objectives:
- Identify the features that customers value in Dataflow.
- Discuss core concepts in Dataflow.
- Review the use of Dataflow templates and SQL.
- Write a simple Dataflow pipeline and run it both locally and on the cloud.
- Identify map and reduce operations, execute the pipeline, and use command line parameters.
- Read data from BigQuery into Dataflow and use the output of a pipeline as a sideinput to another pipeline
Module 4 - Manage Data Pipelines with Cloud Data Fusion and Cloud Composer
Topics:
- Building batch data pipelines visually with Cloud Data Fusion
- Components
- UI overview
- Building a pipeline
- Exploring data using Wrangler
- Orchestrating work between Google Cloud services with Cloud Composer
- Apache Airflow environment
- DAGs and operators
- Workflow scheduling
- Monitoring and logging
Objectives:
- Discuss how to manage your data pipelines with Data Fusion and Cloud Composer.
- Summarize how Cloud Data Fusion allows data analysts and ETL developers to wrangle data and build pipelines in a visual way.
- Describe how Cloud Composer can help to orchestrate the work across multiple Google Cloud services.