High-Performance PySpark: Advanced Strategies for Optimal Data Processing
High-Performance PySpark: Advanced Strategies for Optimal Data Processing
.MP4, AVC, 1280x720, 30 fps | English, AAC, 2 Ch | 1h 22m | 158 MB
Instructor: Ameena Ansari
.MP4, AVC, 1280x720, 30 fps | English, AAC, 2 Ch | 1h 22m | 158 MB
Instructor: Ameena Ansari
Master the art of efficient data processing with this advanced PySpark course designed for data engineers. Instructor Ameena Ansari shows you the essentials of optimizing the data cleaning process and defining schemas to streamline ingestion at scale. Explore various data formats and compression techniques to ensure seamless performance, even with massive datasets. By the end of this course, you'll have the tools and skills you need to transform and ingest high-quality data using PySpark pipelines that are both scalable and efficient.
This course is integrated with GitHub Codespaces, an instant cloud developer environment that offers all the functionality of your favorite IDE without the need for any local machine setup. With GitHub Codespaces, you can get hands-on practice from any machine, at any time—all while using a tool that you’ll likely encounter in the workplace. Check out “Using GitHub Codespaces" with this course to learn how to get started.
Learning objectives
- Master data cleaning techniques for handling missing values, outlier detection, data normalization, and data transformation.
- Structure data schemas for improved performance and scalability.
- Work with a variety of data formats such as Parquet, ORC, Avro, JSON, and CSV.
- Leverage compression techniques using Gzip, Snappy, and LZO.
- Minimize data shuffling and skew, optimizing joins, aggregations, and repartitioning.
High-Performance PySpark: Advanced Strategies for Optimal Data Processing
