Automating Ml Pipelines For Song Recommendation System

Automate Song Recommendations with Docker, MLFlow, and CI/CD Practices for Machine Learning Algorithms.

What you'll learn

Understand the Math Behind ML Algorithms: You will learn the mathematical concepts that underlie popular machine learning algorithms.

Implement Machine Learning Algorithms: You will gain hands-on experience in coding and applying various machine learning algorithms.

Design and Build MLFlow Tracking: You will learn how to use MLFlow for tracking and managing machine learning experiments effectively.

Implement Microservices with Docker: You will learn how to create and manage microservices for automating machine learning pipelines using Docker.

Automate Model Training and Evaluation: You will learn to use Airflow triggers to automate the process of training and evaluating machine learning models.

Set Up Git CI/CD for a Song Recommender App: You will learn how to implement CI/CD for a song recommendation web app.

Requirements

Basic Knowledge of Python programming, as it will be used for implementing machine learning algorithms and building ML pipeline microservices.

A desire to learn and experiment with machine learning and microservices is encouraged.

Description

Math Behind Machine Learning Algorithms:K-Nearest Neighbors (KNN): A method for finding similar songs based on user preferences.Random Forest (RF): An algorithm that combines many decision trees for better predictions.Principal Component Analysis (PCA): A technique for reducing the number of features while retaining important information.K-Means Clustering: A way to group similar songs together based on features.Collaborative Filtering: Making recommendations based on user interactions and preferences.Data Processing Techniques:Feature Engineering (Feature Importance using Random Forest): Feature importance analysis and creating new features from existing data to improve model accuracy.Data Pre-processing (Missing Data Imputation): Cleaning and preparing data for analysis.Evaluation and Tuning:Hyperparameter Tuning (Collaborative Filtering, KNN, Naive Bayes Classifier): Adjusting the settings of algorithms to improve performance.Evaluation Metrics (Precision, Recall, ROC, Accuracy, MSE): Methods to measure how well the model performs.Data Science Fundamentals:TF-IDF (Term Frequency and Inverse Document Frequency): A technique for analyzing the importance of words in song lyrics.Correlation Analysis: Understanding how different features relate to each other.T-Test: A statistical method for comparing groups of data.Automation Tools:Building Microservices using Docker: Use containers to run applications consistently across different environments.Airflow: Automate workflows and schedule tasks for running ML models.MLFlow: Manage and track machine learning experiments and models effectively.By the end of the course, you will know how to build and automate the training, evaluation, and deployment of an ML model for a song recommendation system using these tools, libraries and techniques.

Overview

Section 1: Introduction

Lecture 1 Course Introduction

Section 2: Machine Learning - Math Intuition

Lecture 2 Math Behind Collaborative Filtering

Lecture 3 Math Behind KNN (Euclidean Distance)

Lecture 4 Math Behind Naive Bayes (Bayes Theorem)

Lecture 5 Math Behind TF and IDF

Lecture 6 Math Behind Cosine Similarity

Lecture 7 Evaluation Metric - MSE

Lecture 8 Math Behind - K-Means Clustering (Unsupervised Learning)

Lecture 9 Math Behind Principal Component Analysis

Lecture 10 Math Behind Pearson Correlation

Lecture 11 Math Behind - T-Statistic Test

Lecture 12 Evaluation Metrics - Classification Models

Section 3: ML Experimentation - Supervised & Unsupervised Learning

Lecture 13 Module Artifacts

Lecture 14 Project Env Setup (Conda)

Lecture 15 Import required libraries

Lecture 16 Understanding the features in data

Lecture 17 Data Preprocessing

Lecture 18 Feature Engineering

Lecture 19 Pearson Correlation Analysis

Lecture 20 T-Test Statistics

Lecture 21 Collaborative Filtering - User Genre Matrix

Lecture 22 Creation of user similarity network visualization (Cosine Similarity)

Lecture 23 Songs Recommender Engine Model - Collaborative Filtering

Lecture 24 Fetch Songs Recommendation - Collaborative Filtering Model

Lecture 25 KNN and Naive Bayes Model Pipeline

Lecture 26 Model Hyperparameter Tuning

Lecture 27 Best Estimator Recommendation

Lecture 28 K-Means Clustering and PCA

Section 4: Airflow - Automate Collaborative Filtering model training and deployment

Lecture 29 Module Artifacts

Lecture 30 Code Environment Setup

Lecture 31 MLFlow Lifecycle and Commands

Lecture 32 Airflow Lifecycle and Commands

Lecture 33 DAG Setup - Data Splitting, User Genre Matrix Generation, Training & Evaluation

Lecture 34 train_and_deploy.py W/O Airflow

Lecture 35 Optional - DAG Assets Validation

Section 5: Building Microservices for MLFlow and Airflow using Docker

Lecture 36 docker-compose.yml Lifecycle (Theory)

Lecture 37 Dockerfile (Python and Airflow)

Lecture 38 Microservices - docker-compose.yml

Lecture 39 Building Docker Image for Python

Lecture 40 Building Docker Image for Airflow

Section 6: ML Pipeline Orchestration - Airflow Triggers and MLFlow Experiments

Lecture 41 Build and Compose up the Microservices

Lecture 42 Orchestrating the ML Job Triggers and Logs

Section 7: Song Recommender System Web App

Lecture 43 Import required modules

Lecture 44 Load Pkl Model

Lecture 45 Fallback condition for recommender system

Lecture 46 Load and Fetch cache Data

Lecture 47 Building UI for song recommender system

Lecture 48 Filter and Join recommendations

Lecture 49 Testing the recommender app in localhost environment

Lecture 50 Push the codebase to Github repository

Lecture 51 Deploy recommender app to Streamlit cloud with Github CI/CD

Section 8: Challenges / Takeaways / Homework

Lecture 52 Automating ML Pipeline Song Recommendation: Challenges / Takeaways / Homework

Lecture 53 Thank you!

Lecture 54 Codebase Artifacts

Students pursuing studies in data science, computer science, or related disciplines who want to enhance their practical skills in machine learning and automation.,Individuals looking to deepen their understanding of machine learning and its applications in real-world scenarios, particularly in recommendation systems.,Programmers interested in expanding their skill set to include machine learning concepts and automation practices using tools like Docker, MLFlow, and Airflow.,Professionals wanting to learn how to build and automate machine learning pipelines and improve their workflow efficiency.,Anyone with a foundational knowledge of machine learning who wants to gain practical experience in implementing algorithms and automating processes.,Individuals looking to enhance their qualifications and job prospects by adding machine learning and automation expertise to their portfolio.