Overview
Data Engineers design solutions that ensure maximum flexibility and scalability, while meeting all required security controls.
Get hands-on experience with designing and building data processing systems on Google Cloud. This course uses lectures, demos, and hand-on labs to show you how to design data processing systems, build end-to-end data pipelines, analyze data, and implement machine learning.
This Google Cloud course covers structured, unstructured, and streaming data.
Why Choose KORNERSTONE
- Accredited, practitioner-led, expert certificate trainers to provide high quality training
- Official training material included
- Practical, scenario-based learning – focus on application, not just theory
- Excellent Passing Rate
Skills Covered
- Design and build data processing systems on Google Cloud.
- Process batch and streaming data by implementing autoscaling data pipelines on Dataflow.
- Derive business insights from extremely large datasets using BigQuery.
- Leverage unstructured data using Spark and ML APIs on Dataproc.
- Enable instant insights from streaming data.
Prerequisites
To get the most of out of this course, participants should have:
- Prior Google Cloud experience using Cloud Shell and accessing products from the Google Cloud console.
- Basic proficiency with a common query language such as SQL.
- Experience with data modeling and ETL (extract, transform, load) activities.
- Experience developing applications using a common programming language such as Python.
Target Audience
This class is intended for experienced developers who are responsible for managing big data transformations including:
- Extracting, loading, transforming, cleaning, and validating data.
- Designing pipelines and architectures for data processing.
- Creating and maintaining machine learning and statistical models.
- Querying datasets, visualizing query results and creating reports
Course Curriculum
Module 1: Data engineering tasks and components
Topics
- The role of a data engineer
- Data sources versus data syncs
- Data formats
- Storage solution options on Google Cloud
- Metadata management options on Google Cloud
- Share datasets using Analytics Hub
Module 2: Data replication and migration
Topics
- Replication and migration architecture
- The gcloud command line tool
- Moving datasets
- Datastream
Module 3: The extract and load data pipeline pattern
Topics
- Extract and load architecture
- The bq command line tool
- BigQuery Data Transfer Service
- BigLake
Module 4: The extract, load, and transform data pipeline pattern
Topics
- Extract, load, and transform (ELT) architecture
- SQL scripting and scheduling with BigQuery
- Dataform
Module 5: The extract, transform, and load data pipeline pattern
Topics
- Extract, transform, and load (ETL) architecture
- Google Cloud GUI tools for ETL data pipelines
- Batch data processing using Dataproc
- Streaming data processing options
- Bigtable and data pipelines
Module 6: Automation techniques
Topics
- Automation patterns and options for pipelines
- Cloud Scheduler and Workflows
- Cloud Composer
- Cloud Run functions
- Eventarc
Module 7: Introduction to data engineering
Topics
- Data engineer’s role
- Data engineering challenges
- Introduction to BigQuery
- Data lakes and data warehouses
- Transactional databases versus data warehouses
- Effective partnership with other data teams
- Management of data access and governance
- Building of production-ready pipelines
- Google Cloud customer case study
Module 8: Build a Data Lake
Topics
- Introduction to data lakes
- Data storage and ETL options on Google Cloud
- Building of a data lake using Cloud Storage
- Secure Cloud Storage
- Store all sorts of data types
- Cloud SQL as your OLTP system
Module 9: Build a data warehouse
Topics
- The modern data warehouse
- Introduction to BigQuery
- Get started with BigQuery
- Loading of data into BigQuery
- Exploration of schemas
- Schema design
- Nested and repeated fields
- Optimization with partitioning and clustering
Module 10: Introduction to building batch data pipelines
Topics
- EL, ELT, ETL
- Quality considerations
- Ways of executing operations in BigQuery
- Shortcomings
- ETL to solve data quality issues
Module 11: Execute Spark on Dataproc
Topics
- The Hadoop ecosystem
- Run Hadoop on Dataproc
- Cloud Storage instead of HDFS
- Optimize Dataproc
Module 12: Serverless data processing with Dataflow
Topics
- Introduction to Dataflow
- Reasons why customers value Dataflow
- Dataflow pipelines
- Aggregating with GroupByKey and Combine
- Side inputs and windows
- Dataflow templates
Module 13: Manage data pipelines with Cloud Data Fusion and Cloud Composer
Topics
- Build batch data pipelines visually with Cloud Data Fusion
- Components
- UI overview
- Building a pipeline
- Exploring data using Wrangler
- Orchestrate work between Google Cloud services with Cloud Composer
- Apache Airflow environment
- DAGs and operators
- Workflow scheduling
- Monitoring and logging
Module 14: Introduction to processing streaming data
Topics
- Process streaming data
Module 15: Serverless messaging with Pub/Sub
Topics
- Introduction to Pub/Sub
- Pub/Sub push versus pull
- Publishing with Pub/Sub code
Module 16: Dataflow streaming features
Topics
- Steaming data challenges
- Dataflow windowing
Module 17: High-throughput BigQuery and Bigtable streaming features
Topics
- Streaming into BigQuery and visualizing results
- High-throughput streaming with Bigtable
- Optimizing Bigtable performance
Module 18: Advanced BigQuery functionality and performance
Topics
- Analytic window functions
- GIS functions
- Performance considerations
Dates & Locations
Exam & Certification
Google Cloud Professional Data Engineer Certification
A Google Professional Data Engineer enables data-driven decision making by collecting, transforming, and publishing data. A data engineer should be able to design, build, operationalize, secure, and monitor data processing systems with a particular emphasis on security and compliance; scalability and efficiency; reliability and fidelity; and flexibility and portability. A data engineer should also be able to leverage, deploy, and continuously train pre-existing machine learning models.
