Azure Data Engineering End-To-End Course (English)

Learn multiple tools in Azure data engineering stack through this course, all in one bundle!

What you'll learn

Understand end-to-end flow of Azure data engineering stack

Be ready to appear for interviews and crack them easily

Become independent to maximum level in handling tasks

End-to-end project covering entire lifecycle

Requirements

No pre-requistes for this course as it created from scratch for everyone for every single tool/technology making it easier to learn

Description

This course covers multiple tools and technologies needed to become an azure data engineer.The best part is there are no pre-requisites!Anyone can enroll and learn through this course.Our videos are simple to understand, to the point, short and yet covering everything you need.The content we are offering in this course is immense and requires full dedication, self -discipline and daily learning to ensure a completion and to make you independent skilled professional.We have trained 1000's of students and shaped their career and you could be the next one, join us by enrolling in the course and benefit immensely through our course offering.You will get best of both quality and quantity!In this course you will learn below tools and technologies from scratch:SQL : Learn structured query language in Microsoft SQL Server.Data warehousing : Learn fundamental concepts of data warehousing.Azure cloud : Learn about cloud computing, benefits and different services.Azure Data factory: Learn ETL on azure cloud, code free.Python programming: Learn programming in python with simple way.Big data fundamentals: Master the big data concepts to build strong foundation.Databricks: Learn databricks, the leading data platform.PySpark: Learn big data processing in pyspark on databricks.Delta lake: Learn the delta lake features.Spark structured streamingAzure devopsEnd-to-end project (Release in progress)This course is for everyone from beginner to architect level!

Overview

Section 1: Introduction to Azure Data Engineering

Lecture 1 Introduction

Section 2: SQL

Lecture 2 001. Introduction to SQL

Lecture 3 002. SQL Server Installation

Lecture 4 003. Create And Drop Database

Lecture 5 004. System Databases

Lecture 6 Create and Drop Tables

Lecture 7 006. Insert Data In Table

Lecture 8 007. Select

Lecture 9 008. InstallAdventureWorksSampleDB

Lecture 10 009. GetUniqueValues

Lecture 11 010. Sort Data

Lecture 12 011. Comments

Lecture 13 012. Filter Data

Lecture 14 013.Filter With Wild Characters

Lecture 15 014. Aggregate Function

Lecture 16 015. Grouping Rows

Lecture 17 016. Select..Into

Lecture 18 017. Create Table With Primary Key

Lecture 19 018. Create Table With NOT NULL Constraint

Lecture 20 019. Create Table With Unique Constraint

Lecture 21 020. Create Table With Check Constraint

Lecture 22 021. Create Table With Default Constraint

Lecture 23 022. Create Table With AutoIncrement

Lecture 24 023.Update Rows In Table

Lecture 25 024. Delete Rows From Table

Lecture 26 025. String Functions - Part 1

Lecture 27 026. String Functions - Part 2

Lecture 28 027. Scenario - Combine Names

Lecture 29 028. Scenario - Extract First And Last Names

Lecture 30 029. Date Time Functions - Part 1

Lecture 31 030. Date Time Functions - Part 2

Lecture 32 031. Date Time Functions - Part 3

Lecture 33 032. Data Type - Part 1 - Integers

Lecture 34 033. Data Types - Part 2 - Approximate Numeric

Lecture 35 034. Data Types - Part 3 - Date Time

Lecture 36 035. Data Types - Part 4 - Strings

Lecture 37 036. Data Types - Part 5 - Unique identifier

Lecture 38 037. Data Types - Part 6 - bit

Lecture 39 038. Data Type Conversion

Lecture 40 039. Joins Part 1 – Introduction to joins

Lecture 41 040. Joins Part 2- Using Joins

Lecture 42 041. Joins Part 3- Join 3 tables

Lecture 43 042. Joins Part 4- Using Expressions

Lecture 44 043. Joins Part 4- Outer Joins

Lecture 45 044. Scenario - Joins Part 6 – Self Join

Lecture 46 045. Fact And Dimensions

Lecture 47 046. IIF Function

Lecture 48 047. CASE

Lecture 49 048. IIF & CASE multiple conditions

Lecture 50 049. Scenario - Custom sorting

Lecture 51 50. UNION & UNION ALL

Lecture 52 051. INTERSECT

Lecture 53 052. EXCEPT

Lecture 54 053. Foreign key constraint

Lecture 55 054. Subquery - PART 1

Lecture 56 055. Subquery - PART2 -IN & NOT IN

Lecture 57 056. Subquery - PART3 - Update

Lecture 58 057.Subquery - PART 4-Derived Table

Lecture 59 058. Subquery - PART 5 - EXISTS, Correlated subquery

Lecture 60 059. Scenario - Subquery - PART 6 - Get contribution yearwise

Lecture 61 060. HAVING Clause

Lecture 62 061. TOP Clause

Lecture 63 062. Scenario - Get TOP & BOTTOM Products

Lecture 64 063. Window Functions-PART1 - ROW_NUMBER

Lecture 65 064. Window Functions - PART 2 - RANK & DENSE_RANK

Lecture 66 065. Window Functions - PART 3-LAG & LEAD

Lecture 67 066. Window Functions - PART4 - FIRST_VALUE & LAST_VALUE

Lecture 68 067. SQL Classification

Lecture 69 068. ALTER TABLE - PART 1 - Columns

Lecture 70 069. ALTER TABLE - PART 2- Constraints

Lecture 71 070. OFFSET

Lecture 72 071. COALESCE Function

Lecture 73 072. MERGE

Lecture 74 073. SCHEMA

Lecture 75 074. GROUP BY WITH ROLLUP

Lecture 76 075. GROUP BY WITH CUBE

Lecture 77 076. PIVOT

Lecture 78 077. UNPIVOT

Lecture 79 078. VIEWS

Lecture 80 079. Common Table Expression - Part1 - Introduction

Lecture 81 080. Common TableExpression - PART 2 - MultiPart

Lecture 82 081. Common Table Expression - PART 3 - Recursion

Lecture 83 082. Variables

Lecture 84 083. IF..ELSE

Lecture 85 084. WHILE LOOP

Lecture 86 085. Temp Tables

Lecture 87 086. Stored Procedures

Lecture 88 087. Converting a table to JSON

Lecture 89 088. Creating Nested JSON Output

Lecture 90 089. Output Clause

Lecture 91 090. Slowly Changing Dimensions

Section 3: SQL Scenarios

Lecture 92 1. Find Average

Lecture 93 2. Joins Challenge

Lecture 94 3. Identify and delete duolicate rows

Lecture 95 4. Custom Merging

Lecture 96 005.MissingData-FindMissingDepartments

Lecture 97 006.Find2ndLargestSalary

Lecture 98 7. 2nd Largest Salary In Each Department

Lecture 99 8. RowNumbering

Lecture 100 9. Find Alternate Rows

Lecture 101 10. Palindrome

Lecture 102 11. FindDuplicateEmails

Lecture 103 12. UpdateCorrectGender

Lecture 104 13. Transform Student Info

Lecture 105 14. Defect Classification

Lecture 106 015. Daily Running Total

Lecture 107 16. MTD Total

Lecture 108 17. QTD Total

Lecture 109 18. YTD Total

Lecture 110 19. Previous Year Sales And YOY Growth

Lecture 111 20. 3 month moving sum

Section 4: 4. Data warehousing

Lecture 112 1.Introduction to DataWarehousing

Lecture 113 2. Data Loads

Lecture 114 3. Storage Layout Models

Lecture 115 4. Robin Round Distribution

Lecture 116 5. Hash Distribution

Lecture 117 6.ACID Properties

Lecture 118 7. Normalization

Section 5: 5. Azure

Lecture 119 1. Cloud Computing

Lecture 120 2. Cloud Providers

Lecture 121 3. Azure Introduction

Lecture 122 4. Create Azure Trial Account

Lecture 123 5. Getting Started With Azure

Lecture 124 6. Create Data Lake

Lecture 125 7. Create Azure SQL Server And Database

Lecture 126 8. CreateDataFactory

Lecture 127 9. Upgrade To Pay-as-you-go

Section 6: 6.Azure Data Factory

Lecture 128 1. Introduction

Lecture 129 2. Copy Data Within Datalake

Lecture 130 3. Copy Entire Folder Specific Files

Lecture 131 4. Copy data from ADLS to SQLDB & vice versa

Lecture 132 5. Copy Data Additional Columns

Lecture 133 6. Set Variable Activity

Lecture 134 7. Copy files in timeframe

Lecture 135 8. Get MetaData Activity

Lecture 136 9. For Each Activity

Lecture 137 10. Copy Each File to New table

Lecture 138 11. Truncate & load

Lecture 139 12. Copy data with upsert

Lecture 140 13. Append Variable Activity

Lecture 141 14. Using SQL Queries In Copy Data Activity

Lecture 142 15. Column Mapping In Copy Data Activity

Lecture 143 16. Delete Activity

Lecture 144 17. Copy Data With Stored Procedure

Lecture 145 18. Stored Procedure Activity

Lecture 146 19. Lookup Activity

Lecture 147 20. Filter Activity

Lecture 148 21. IF Activity

Lecture 149 22. Switch Activity

Lecture 150 23. Script Activity

Lecture 151 24. Validation Activity

Lecture 152 25. Convert CSV to JSON

Lecture 153 26. Copy JSON to SQL DB

Lecture 154 27. Execute Pipeline Activity

Lecture 155 28. Copy Data If File Exists

Lecture 156 29. Parameters Vs Variables

Lecture 157 30. Delete Blank Files

Lecture 158 31. Copy Header Less CSV

Lecture 159 32. Retry Logic

Lecture 160 33. Copy Behavior

Lecture 161 34. Max Rows Per File

Lecture 162 35. Split Data Single Criteria

Lecture 163 36. Split Data Multiple Criteria

Lecture 164 37. Consolidate Data From Multiple Files

Lecture 165 38. Consolidate Data From Mutiple Folders

Lecture 166 39. Copy Data Custom Mappings SingleTable

Lecture 167 40. Copy Data Custom Mappings - MultipleTables

Lecture 168 41. Copy Data Pipe Character

Lecture 169 42. Copy Data Quote Character

Lecture 170 43. Introduction To DataFlows

Lecture 171 44. Select Transformation

Lecture 172 45. SortTransformation

Lecture 173 46. Filter Transformation

Lecture 174 47. Derived Column Transformation

Lecture 175 48. Conditional Split Transformation

Lecture 176 49. Cast Transformation

Lecture 177 50. Surrogate Key Transformation

Lecture 178 51.Aggregate Transformation

Lecture 179 52. Pivot Transformation

Lecture 180 53. Unpivot Transformation

Lecture 181 54.Rank Transformation

Lecture 182 55. Window Transformation

Lecture 183 56. Union Transformation

Lecture 184 57. Lookup Transformation

Lecture 185 58. Join Transformation

Lecture 186 59. Exists Transformation

Lecture 187 60. Flatten Transformation

Lecture 188 61. Parse Transformation

Lecture 189 62. Stringify Transformation

Lecture 190 65. Integration Runtime

Lecture 191 66. Install Self Hosted Integration Runtime

Lecture 192 67.Copy Data From Local To ADLS

Lecture 193 68. Copy Data From Local to Azure SQL DB

Lecture 194 69. Copy Data Local SQL db to Azure SQL db

Lecture 195 070. Copy Multiple Tables From Local sql DB to Azure sql DB

Lecture 196 071. Using Key Vault

Lecture 197 072. Schedule Trigger

Section 7: 7.Introduction To Databricks

Lecture 198 1. Introduction To Databricks

Lecture 199 2. Community Edition Sign Up

Lecture 200 3. Databricks Platform Overview

Section 8: 8.Python Programming

Lecture 201 1. Introduction And Installation

Lecture 202 2. print

Lecture 203 3. Variables

Lecture 204 4. Concatenate

Lecture 205 5. Interpolation

Lecture 206 6. If..else

Lecture 207 7. Input

Lecture 208 8. IF..ELIF..ELSE

Lecture 209 9. For..Loop

Lecture 210 10. While..Loop

Lecture 211 11. Break&Continue

Lecture 212 12. Lists-Part1-IntroductionAndIndexing

Lecture 213 13. Lists-Part2-Len,Existence&Loop

Lecture 214 14. Lists-Part 3 – Add&RemoveItems

Lecture 215 15. Lists-Part4-Count,Copy,Reverse&Sort

Lecture 216 16. Lists-Part5-UpdatingLists

Lecture 217 17. Set

Lecture 218 18. Tuple

Lecture 219 19. Dictionary

Lecture 220 20. Comments

Lecture 221 21. StringOperations-Part1

Lecture 222 22. StringOperations-Part2

Lecture 223 23.StringOperations-Part3

Lecture 224 24. ListComprehensions

Lecture 225 25. dir

Lecture 226 26. DateTime-Part1-datetimeModule

Lecture 227 27. DateTime-Part2-timedelta&timestamp

Lecture 228 28. DateTime-Part3-monthdeltaModule

Lecture 229 29. DateTime-Part4-relativedeltaModule

Lecture 230 30.DateTime-Part5-Formatting

Lecture 231 31.ExceptionHandling

Lecture 232 32. None

Lecture 233 33. Random

Lecture 234 34. Functions-Part1-Introduction

Lecture 235 35. Functions-Part2-ScopeOfVariable

Lecture 236 36. Functions-Part3-Arguments

Lecture 237 37.Functions-Part4-DocStrings

Lecture 238 38. Lambda

Lecture 239 39. Map

Lecture 240 40. Reduce

Lecture 241 41. Recursion

Lecture 242 42. Generators

Lecture 243 43. Decorators

Section 9: 9. Big Data Fundamentals

Lecture 244 1. IntroductionToBigData

Lecture 245 2. Evolution Of Big Data

Lecture 246 3. Distributed Computing

Lecture 247 4. Features

Lecture 248 5. Hadoop Ecosystem

Lecture 249 6. TypesOfProcessing

Section 10: 10.Databricks & PySpark

Lecture 250 1. IntroductionToDatabricksAndPyspark

Lecture 251 2. Introduction To Apache Spark

Lecture 252 3. SparkComponents And API

Lecture 253 4. Spark Architecture

Lecture 254 5. RDD

Lecture 255 6. Create RDD from List

Lecture 256 7. Control Partitions In RDD

Lecture 257 8. CreateRDDfromTextfile

Lecture 258 9. Transformations On RDD

Lecture 259 10. Lineage Graph

Lecture 260 11. Understanding DAG fundamentals

Lecture 261 12. Mapreduce Working

Lecture 262 13. ReduceByKey Vs ReduceByKey Locally

Lecture 263 14. GroupByKey

Lecture 264 15. Filter Transformation On RDD

Lecture 265 16. SortBy&SortByKeyTransformationsOnRDD

Lecture 266 017. Extract Top Bottom From RDD

Lecture 267 18. Save RDD as Textfile

Lecture 268 19. Coalesce And Repartition On RDD

Lecture 269 20. IntroductionToAccumulators

Lecture 270 21. Implementing Accumulators

Lecture 271 22. Broadcast Variables

Lecture 272 23. Introduction To Dataframes

Lecture 273 24. ReadCSVInDataframe

Lecture 274 25. DataFrameInsights

Lecture 275 26. SelectColumnsFromDataFrame

Lecture 276 27. Add Modify Columns In DataFrame

Lecture 277 28. Drop Columns In DataFrame

Lecture 278 29. RenameColumnsInDataFrame

Lecture 279 30. SortColumnsInDataFrame

Lecture 280 31. FilterDataInDataFrame

Lecture 281 32. RemoveDuplicatesFromDataFrame

Lecture 282 33. CombineDataFrames

Lecture 283 34. PatternBasedFiltersInDataframe

Lecture 284 35. AddColumnOnBasisOfConditionInPyspark

Lecture 285 36. CaseConversionInDataFrame

Lecture 286 37. AggregationsOnDataFrame

Lecture 287 38. AggregationsWithGroupBy

Lecture 288 39.Pivot&UnpivotOnDataFrame

Lecture 289 40. WindowFunctions

Lecture 290 41. FillNullValuesInDataFrame

Lecture 291 42. ReadCSVoptions

Lecture 292 43. ImposeSchemaOnDataFrame

Lecture 293 44. WriteModesInDataFrame

Lecture 294 45. DateFunctions

Lecture 295 46. ConvertRddtoDFandViceVersa

Lecture 296 47. Explode

Lecture 297 48. WorkingWithArraytypeColumn

Lecture 298 49. dbutils

Lecture 299 50. AccessADLSgen2UsingAccessKey

Lecture 300 51. SignUpForAzureDatabricks

Lecture 301 52. Widgets

Lecture 302 53. CallingOtherNotebooks

Lecture 303 54. ReadJsonFiles

Lecture 304 55. ReadMultipleFilesAndGetFileNames

Lecture 305 56. ParquetFileFormat

Lecture 306 57. AccessADLSusingSAStoken

Lecture 307 58. AccessADLSusingOAUTH

Lecture 308 59. Read&WriteParquetFiles

Lecture 309 60. DeltaFormat

Lecture 310 61. CreateTempViews

Lecture 311 62. CreateManagedAndUnManagedTables

Lecture 312 63. Partitioning

Lecture 313 64. Bucketing

Lecture 314 65. ControlNumberOfRecordsWhileWriting

Lecture 315 66. MaxPartitionBytes

Lecture 316 67. SparkQueryExecutionPlans

Lecture 317 68. Joins

Lecture 318 69. SparkExecutionPlansForTransformations

Lecture 319 70. BroadCastHashJoinAlgorithm

Lecture 320 71. SortMergeJoinAlgorithm

Lecture 321 72. ShuffleHashJoinAlgorithm

Lecture 322 73. SortMergeBucketJoin

Lecture 323 74. JoinHints

Lecture 324 75. SparkSQL

Lecture 325 76. SparkMemoryManagement

Lecture 326 77. GarbageCollection

Lecture 327 78. CPUTerminologies

Lecture 328 79. DatabricksCompute&Clusters

Lecture 329 80. ClusterManager

Lecture 330 81. ResourceAllocation

Lecture 331 82. DynamicAllocation

Lecture 332 83. SerializationAndDeserialization

Lecture 333 84. CacheAndPersist

Lecture 334 85. HashFunctions

Section 11: 11.DeltaLake & Lakehouse

Lecture 335 1.HistoryOfDataArchitectures

Lecture 336 2. IntroductionToDeltaLake

Lecture 337 3. ReadAndWriteDeltaFormat

Lecture 338 4. UnderstandingDeltaLog

Lecture 339 5. VersionHistoryAndTimeTravel

Lecture 340 6. CheckPointing

Lecture 341 7. CreateDeltaTable

Lecture 342 8. GeneratedColumns

Lecture 343 9. CreatePartitionedTable

Lecture 344 10. SchemaEvolution

Lecture 345 11. CopyInto

Lecture 346 12. Merge

Lecture 347 13. ColumnStatistics

Lecture 348 14.Optimize-Vaccum-Zorder

Lecture 349 15. LiquidClustering

Lecture 350 16. ChangeDataFeed

Lecture 351 17. ReorgTable

Lecture 352 18. DeletionVectors

Lecture 353 19.SCD-Type 1

Lecture 354 20.SCD-Type 2

Section 12: 12.Databricks

Lecture 355 1.HiveMetaStore

Lecture 356 2. IntroductionToUnityCatalog

Lecture 357 3. SettingUpUnityCatalog

Lecture 358 4.CreateCatalog, Schema & Table

Lecture 359 5. Autoloader

Section 13: 13.Spark Structured Streaming

Lecture 360 1.SparkStructuredSteaming-Introduction

Lecture 361 2. ReadStream

Lecture 362 3. WriteStream

Lecture 363 4. OutputModes

Lecture 364 5.Sources&Sinks

Lecture 365 6. Triggers

Lecture 366 7. Joins

Lecture 367 8. Stateful Vs Stateless

Lecture 368 9. WindowOperations

Section 14: 14. Azure synapse analytics

Lecture 369 1.IntroductionToAzureSynapseAnalytics

Lecture 370 2.SQL Pools

Lecture 371 3. DedicatedSQLPool

Lecture 372 4. ServerlessSQLPool

Lecture 373 5. DistributionTypes

Lecture 374 6.ProvisionAzureSynapseAnalytics

Lecture 375 7. WorkingWithServerlessSQLPool

Section 15: 15. End-to-end project

Lecture 376 1. Introduction

Lecture 377 2. UnderstandingRequirements

Lecture 378 3. PurchaseDataModel

Lecture 379 4. SalesModel

Lecture 380 5. HRModel

Lecture 381 6. ResourceSetUp

Lecture 382 7. GettingStartedWithAzureDevops

Lecture 383 8. UnderstandingDataSource

Lecture 384 9. GettingDatabricksReady

Lecture 385 10. DevelopingPurchaseNotebooks

Lecture 386 11. DevelopingPurchaseModel-Part2

Lecture 387 12. TaskPlanning

Lecture 388 13.DevelopingPurchaseBronzeNotebooks

Lecture 389 14.FirstDemo

Lecture 390 15.DevelopingSilverNotebooks-Part1

Lecture 391 16.DevelopingSilverAndOtherNotebooks

Lecture 392 17.DevelopingOtherAndSilverNotebooks-Part2

Lecture 393 18. CreatingRawAndBronzeSalesNotebooks

Lecture 394 19.CreatingSalesSilverNotebooks-Part1

Lecture 395 020.CreatingSalesSilverNotebookPart2

Lecture 396 021.CreatingHrModelNotebooks

Lecture 397 22. DevelopingDatabricksWorkflows

Anyone can enroll this course from beginner, intermediate to architect level!,Anyone looking to transition into azure data engineering or existing data engineers looking to not just enhance their skillset but also learn in depth.