Streaming Data Pipeline From Confluent Kafka To Aws Mysql Db

Hands on course to build Real Time Data Streaming Pipeline from Confluent Kafka to Lambda to AWS MySQL RDS from scratch

What you'll learn

Hands On course to learn to use Confluent Kafka & AWS Cloud Platforms

Learn basics of navigating & using AWS Cloud Platform services like Lambda, RDS-MYSQL, IAM, Cloudwatch, Secrets Manager & more

Learn basics of Confluent Platform and overview of services like: Topics, Connectors, API Keys, Clusters & more

Watch & Learn to build on your AWS + mConfluent Accounts- Real Time Data Streaming pipeline from Confluent to AWS RDS using Serverless Lambda Compute

Requirements

Be able to create AWS Cloud Platform Account (Free Tier)

Be able to create Confluent Kafka Platform Account

Basic understanding of Python

Description

In this hands-on course, participants will follow along step-by-step to build a real-time streaming data pipeline that sends data from Confluent Kafka to AWS Lambda and finally into an AWS RDS MySQL Database. This course is designed to provide practical, real-world skills by walking through each component of the architecture, ensuring that learners not only understand the concepts but also apply them directly in a cloud environment.On the AWS side, participants will gain experience working with several important services, including AWS IAM for securing resources, AWS Secrets Manager for managing sensitive credentials, and AWS CloudWatch for monitoring and logging the data pipeline in action. These services are essential for building secure, scalable, and reliable applications in the cloud.On the Confluent Kafka side, learners will set up a fully managed Kafka cluster, create topics for message streaming, and configure a fully managed Source Connector to simulate real-world data ingestion. This gives participants valuable exposure to enterprise-grade Kafka infrastructure without the overhead of managing the platform themselves.By the end of this course, participants will have built a working, scalable pipeline, gained insights into cloud-native architectures, and acquired hands-on experience that can be directly applied to real-world projects or professional roles.This course is suitable for budding Cloud Engineers, mid level Data Engineers, Product Owners, Product Mangers, Scrum Masters and Technology Leaders looking to get a hands-on experience of building a Real time Streaming Data Pipeline.

Overview

Section 1: Introduction

Lecture 1 Introduction & High Level Architecture of end product that we will build

Section 2: Confluent related Activities

Lecture 2 Create Free Confluent.io account to use for data streaming using Kafka

Lecture 3 Confluent: Create Topic, API Keys & Explore Confluent Kafka Topic capabilities

Lecture 4 Confluent: Setup Datagen Source Connector to produce data Kafka Topic

Section 3: AWS Account creation & Exploration

Lecture 5 AWS Cloud Platform: Create Account & Explore Services

Section 4: AWS + Confluent Integration

Lecture 6 AWS: Create & Explore Lambda Serverless Compute

Lecture 7 Configure Lambda to consume from Confluent Kafka Topic & verify in CloudWatch

Lecture 8 Create & Explore AWS RDS MySQL Database

Lecture 9 Connect to AWS MySql DB via MySQL Workbench & create Schema/Table

Section 5: Parse the event & extract data fields

Lecture 10 Parse event and extract fields and write to Cloudwatch

Lecture 11 Update Lambda to connect to RDS + Create Layer and add pymysql module to it

Lecture 12 Update Lambda Python code to insert extracted fields into MySQL RDS

Section 6: End to End Data Pipeline execution & Validation

Lecture 13 Execute End to End flow and observe at all Intermediate steps

Lecture 14 Summary, Wrap Up and Next Steps!

Beginner Cloud Developers looking to build Real Time Data pipeline development skills and add a real life project to resume,MId Level to Sr. Cloud Developers to quickly gain understanding of building Real Time Streaming Data Pipelines using AWS + Confluent Platforms,Product Owners looking to build understanding of capabilities, risks & issues in building data pipeline using AWS + Confluent Kafka platform,Product Managers looking to build understanding of capabilities, risks & issues in building data pipeline using AWS + Confluent Kafka platform,Technology Leaders looking to deepen understanding of capabilities, efforts, risks & issues in building data pipeline using AWS + Confluent Kafka platform,Solution Architects looking for building hands-on Data Pipeline using AWS MySQL DB + Confluent Kafka Platforms