Description

Build a realistic GCP data engineering project that ingests retail data from Cloud SQL for MySQL into BigQuery using Google Cloud Dataflow.You will work through the project as a data engineer—from understanding the business context and source systems to defining the pipeline contract, designing incremental loads, setting up Google Cloud resources, implementing the solution, validating the output, and orchestrating repeatable runs.In this project, you will:Prepare the source: Set up retail source tables in Cloud SQL for MySQLConfigure networking: Connect Cloud SQL and Dataflow using private networkingBuild the ingestion pipeline: Move data from MySQL into BigQuery using DataflowLoad data incrementally: Use watermarks to process new and updated recordsCreate BigQuery tables: Build staging and raw tables with partitioning and clusteringMerge incremental data: Use BigQuery MERGE procedures to update target tablesOrchestrate the pipeline: Deploy and schedule workflows using Cloud Composer and Apache AirflowValidate the results: Query the loaded data and verify initial and incremental runs in BigQueryMore than isolated service demonstrationsYou will build the project in your own Google Cloud account using the provided source code, sample retail data, configuration files, SQL scripts, and infrastructure setup scripts.The focus is not only on getting the pipeline to run. You will understand why each Google Cloud service is used, how Cloud SQL, Dataflow, BigQuery, and Cloud Composer work together, and how the pipeline handles new and updated source records during subsequent runs.By the end of the course, you will have a complete GCP data engineering project that you can practice, adapt for your portfolio, and explain clearly in interviews.This course contains a promotion.