- 🎥 6.1.1 Introduction to Batch Processing
- 🎥 6.1.2 Introduction to Spark
Follow these instructions to install Spark:
🎥 6.2.1 (Optional) Installing Spark (Linux)
Alternatively, if the setups above don't work, you can run Spark in Google Colab.
Note
It's advisable to invest some time in setting things up locally rather than immediately jumping into this solution
- 🎥 6.3.1 First Look at Spark/PySpark
- 🎥 6.3.2 Spark Dataframes
- 🎥 6.3.3 (Optional) Preparing Yellow and Green Taxi Data
Script to prepare the Dataset download_data.sh
Note
The other way to infer the schema (apart from pandas) for the csv files, is to set the inferSchema option to true while reading the files in Spark.
- 🎥 6.3.4 SQL with Spark
- 🎥 6.4.1 Anatomy of a Spark Cluster
- 🎥 6.4.2 GroupBy in Spark
- 🎥 6.4.3 Joins in Spark
- 🎥 6.5.1 Operations on Spark RDDs
- 🎥 6.5.2 Spark RDD mapPartition
- 🎥 6.6.1 Connecting to Google Cloud Storage
- 🎥 6.6.2 Creating a Local Spark Cluster
- 🎥 6.6.3 Setting up a Dataproc Cluster
- 🎥 6.6.4 Connecting Spark to Big Query
Did you take notes? You can share them here
- Notes by Alvaro Navas
- Sandy's DE Learning Blog
- Notes by Alain Boisvert
- Alternative : Using docker-compose to launch spark by rafik
- Marcos Torregrosa's blog (spanish)
- Notes by Victor Padilha
- Notes by Oscar Garcia
- Notes by HongWei
- 2024 videos transcript by Maria Fisher
- 2025 Notes by Manuel Guerra
- 2025 Notes by Gabi Fonseca
- 2025 Notes on Installing Spark on MacOS (with Anaconda + brew) by Gabi Fonseca
- 2025 Notes by Daniel Lachner
- 2026 Notes by Ajay Katte
- 2026 Video PySpark installation on Windows (no pip install) by khanh
- Add your notes here (above this line)