Confluent Kafka Administration Basics

The message streaming platform

What you'll learn

Kafka on confluent platform

Kafka in kraft mode

Real time integration solutions with kafka

Full fledged hands on with confluent platform kafka

Requirements

Operating a computer and linux basic commands

Description

     Confluent is a company founded by the creators of Apache Kafka that offers a data streaming platform, including Confluent Cloud and Confluent Platform, designed to build and manage Kafka-based streaming solutions. Confluent's platform provides a complete end-to-end solution for data streaming, offering features like elastic scalability, resiliency, low latency, pre-built connectors, and enterprise-grade security. They also offer various tools and services, including Confluent Hub for pre-built connectors and extensions, and Confluent's documentation for Apache Kafka client libraries.     Kafka is an open-source distributed streaming system used for stream processing, real-time data pipelines, and data integration at scale. Originally created to handle real-time data feeds at LinkedIn in 2011, Kafka quickly evolved from a messaging queue to a full-fledged event streaming platform, capable of handling over one million messages per second, or trillions of messages per day     In modern days more than 4000+ fortune 500 companies are using kafka for their messaging and integration platform needs. Kafka is used in Retail industries, Banking industries, Insurance industries, Biofore industries, E-Commerce industries, Aviation industries and many more. Technically kafka can be used to in ETL pipelines inorder to maintain proper communication between two systems.      The magic of kafka is "The producer no need to aware of the consumer and consumer no need to aware of the producer, but still communication can be established between them"

Overview

Section 1: Introduction

Lecture 1 Kafka Introduction

Lecture 2 Advantages and use cases of kafka

Lecture 3 Kafka flavors and versions

Lecture 4 Roles and Responsibilities of a kafka admin

Section 2: Kafka components

Lecture 5 Topics, Partitions, offsets and Replication factor

Lecture 6 Broker

Lecture 7 Controller (Kraft mode)

Lecture 8 Producer

Lecture 9 Consumer

Section 3: Kafka single node installation

Lecture 10 Lab setup with linux environment

Lecture 11 SSH to the Linux server

Lecture 12 Confluent kafka installation (Single node)

Lecture 13 Kafka service setup and start in kraft mode

Section 4: Kafka Administration (Using Command Line Interface (CLI))

Lecture 14 Confluent kafka CLI - Topics management

Lecture 15 Confluent kafka CLI - Producer operations

Lecture 16 Confluent kafka CLI - Consumer operations

Lecture 17 Consumer groups and its significance in kafka

Lecture 18 Resetting the offsets for a consumer group in CLI

Section 5: Kafka Administration (using GUI Tools)

Lecture 19 Offset explorer installation

Lecture 20 Connecting to a kafka cluster and topics management with offset explorer

Lecture 21 Consumer groups management with offset explorer

Section 6: Kafka partition reassignment

Lecture 22 Scale out kafka cluster from 1 node broker to 2 node brokers

Lecture 23 Increasing the replication factor of a topic to 2 with partition reassignment

Section 7: Kafka Broker level Configurations

Lecture 24 Broker level configurations

Section 8: Kafka REST Proxy Setup

Lecture 25 Kafka REST

IT engineers who want to learn kafka