We can host this training at your preferred location. Contact us!

The two key components of any data pipeline are data lakes and warehouses. This course highlights use-cases for each type of storage and dives into the available data lake and warehouse solutions on Google Cloud in technical detail. Also, this course describes the role of a data engineer, the benefits of a successful data pipeline to business operations, and examines why data engineering should be done in a cloud environment.

This is the first course of the Data Engineering on Google Cloud series. After completing this course, enroll in the Building Batch Data Pipelines on Google Cloud course.

Introduction to Data Engineering

This module discusses the role of data engineering and motivates the claim why data engineering should be done in the Cloud

Module introduction
The role of a data engineer
Data engineering challenges
Introduction to BigQuery
Data lakes and data warehouses
Transactional databases versus data warehouses
Partner effectively with other data teams
Manage data access and governance
Demo: Finding PII in your dataset with the DLP API
Build production-ready pipelines
Google Cloud customer case study
Lab Intro: Using BigQuery to do Analysis
LAB: Using BigQuery to do Analysis: In this lab, you analyze 2 different public datasets, run queries on them, separately and then combined, to derive interesting insights.
QUIZ

Building a Data Lake

In this module, we describe what data lake is and how to use Cloud Storage as your data lake on Google Cloud.

Module Introduction
Introduction to data lakes
Data storage and ETL options on Google Cloud
Build a data lake using Cloud Storage
Secure Cloud Storage
Store all sorts of data types
Cloud SQL as a relational data lake
Lab Intro: Loading Taxi Data into Google Cloud SQL
LAB: Loading Taxi Data into Google Cloud SQL 2.5:In this lab you will import data from CSV text files into Cloud SQL and then carry out some basic data analysis using simple queries.
QUIZ

Building a Data Warehouse

In this module, we talk about BigQuery as a data warehousing option on Google Cloud

Module Introduction
The modern data warehouse
Introduction to BigQuery
Demo: Querying TB of data in seconds
Get started with BigQuery
Load data into BigQuery
Lab Intro: Loading Data into BigQuery
LAB: Loading data into BigQuery: This lab focuses on how to ingest data into tables inside of BigQuery.
Explore schemas
Demo: Exploring Schemas
Schema design
Nested and repeated fields
Demo: Nested and repeated fields
Design the optimal schema for BigQuery
Lab Intro: Working with JSON and Array data in BigQuery
LAB: Working with JSON and Array data in BigQuery 2.5: In this lab you will work with semi-structured data (ingesting JSON, Array data types) inside of BigQuery. You will practice loading, querying, troubleshooting, and unnesting various semi-structured datasets.
Optimize with partitioning and clustering
Lab Intro: Partitioned Tables in BigQuery
LAB: Partitioned Tables in Google BigQuery:This lab focuses on how to query partitioned datasets and how to create your own dataset partitions to improve query performance, which reduces cost.
Review
QUIZ

Upcoming Trainings

Join our public courses in our Singapore facilities. Private class trainings will be organized at the location of your preference, according to your schedule.

Classroom / Virtual Classroom

09 July 2024