10 PySpark Projects for Practice (Beginner to Advanced, With Code)

Reading PySpark tutorials only gets you so far. The fastest way to actually learn PySpark (and to have something real to talk about in a data engineering interview) is to build small, end-to-end projects with messy, realistic data.

Below are 10 PySpark projects for practice, ordered from beginner to advanced. Each one comes with a public dataset (or a tiny inline dataset), the exact tasks to complete, tested starter code, and stretch goals that turn it into a portfolio project. You can run every starter snippet in the free PySpark Online Compiler without installing Spark.

How to Use These PySpark Practice Projects

  • Pick by level. Projects 1–4 are beginner (read, clean, aggregate), 5–7 are intermediate (window functions, data quality, segmentation), and 8–10 are advanced (SCD Type 2, data skew, medallion pipelines).
  • Try the tasks first. Read the task list, write your own solution, and only then compare with the starter code.
  • Start small, then scale. Run the logic on a sample in the online compiler, then run it on the full dataset locally or on Databricks Free Edition.
  • Write it up. Push each project to GitHub with a short README: problem, dataset, approach, results, and what you would do differently at 10x scale. That README is what interviewers read.

PySpark Projects at a Glance

  1. Retail Sales Analysis — beginner — filters, groupBy, aggregations
  2. NYC Taxi Trip Analysis — beginner — Parquet, timestamps, broadcast joins
  3. MovieLens Ratings Analysis — beginner — joins, explode, split
  4. Web Server Log Analysis — beginner/intermediate — regex parsing, text files
  5. Data Cleaning Pipeline — intermediate — deduplication, nulls, data quality
  6. Customer RFM Segmentation — intermediate — window functions, ntile
  7. Clickstream Sessionization — intermediate — lag, running sums
  8. SCD Type 2 Customer Dimension — advanced — anti/semi joins, unions
  9. Fixing a Skewed Join with Salting — advanced — performance tuning
  10. Medallion (Bronze/Silver/Gold) Pipeline — advanced — idempotent incremental loads

1. Retail Sales Analysis (Beginner)

The classic first PySpark project: take raw e-commerce transactions and answer business questions. It teaches the 80% of PySpark you will use every day.

  • Dataset: Online Retail II from the UCI Machine Learning Repository (~1M invoice lines from a UK online retailer, 2009–2011).
  • Skills you practice: reading CSV with a schema, filter, withColumn, groupBy().agg(), date formatting, sorting.

What to build

  1. Load the data and cast InvoiceDate to a timestamp.
  2. Remove cancelled invoices (invoice numbers starting with C) and rows with negative quantity.
  3. Add a revenue column (quantity * price).
  4. Compute monthly revenue and number of orders.
  5. Find the top 10 products and top 10 countries by revenue.
  6. Find the share of revenue that comes from the top 1% of customers.

Starter code

1from pyspark.sql import SparkSession, functions as F
2
3spark = SparkSession.builder.appName("retail-sales").getOrCreate()
4
5# A few sample rows in the same shape as the Online Retail II dataset
6data = [
7    ("536365", "85123A", "WHITE HANGING HEART T-LIGHT HOLDER", 6, "2010-12-01 08:26:00", 2.55, 17850, "United Kingdom"),
8    ("536366", "22633", "HAND WARMER UNION JACK", 6, "2010-12-01 08:28:00", 1.85, 17850, "United Kingdom"),
9    ("C536379", "D", "Discount", -1, "2010-12-01 09:41:00", 27.50, 14527, "United Kingdom"),
10    ("536370", "22728", "ALARM CLOCK BAKELIKE PINK", 24, "2010-12-01 08:45:00", 3.75, 12583, "France"),
11    ("539993", "22727", "ALARM CLOCK BAKELIKE RED", 24, "2011-01-04 10:00:00", 3.75, 12583, "France"),
12]
13cols = ["invoice", "stock_code", "description", "quantity",
14        "invoice_date", "price", "customer_id", "country"]
15
16df = (spark.createDataFrame(data, cols)
17      .withColumn("invoice_date", F.to_timestamp("invoice_date")))
18
19# 1. Clean: drop cancelled invoices (start with "C") and returns
20sales = (df
21    .filter(~F.col("invoice").startswith("C"))
22    .filter(F.col("quantity") > 0)
23    .withColumn("revenue", F.round(F.col("quantity") * F.col("price"), 2)))
24
25# 2. Monthly revenue and order count
26monthly = (sales
27    .groupBy(F.date_format("invoice_date", "yyyy-MM").alias("month"))
28    .agg(F.round(F.sum("revenue"), 2).alias("revenue"),
29         F.countDistinct("invoice").alias("orders"))
30    .orderBy("month"))
31monthly.show()
32
33# 3. Revenue by country
34(sales.groupBy("country")
35      .agg(F.round(F.sum("revenue"), 2).alias("revenue"))
36      .orderBy(F.desc("revenue"))
37      .show())

New to these functions? See the tutorials on filtering data, grouping data and date functions.

Stretch goals (make it portfolio-ready)
  • Define an explicit StructType schema instead of relying on inferSchema (and explain why in your README).
  • Write the cleaned data to Parquet partitioned by year and month.
  • Add a month-over-month growth column using lag().

2. NYC Taxi Trip Analysis (Beginner)

NYC taxi data is big, public, and full of bad records — a great way to practise working with timestamps and real-world data quality problems.

  • Dataset: NYC TLC Trip Record Data (monthly yellow-taxi Parquet files) plus the Taxi Zone Lookup CSV, both from the NYC Taxi & Limousine Commission website.
  • Skills you practice: reading Parquet, timestamp maths, hour(), filtering outliers, broadcast joins.

What to build

  1. Load one month of yellow taxi trips and the zone lookup table.
  2. Compute trip duration in minutes and remove impossible trips (negative or zero duration, zero distance, negative fares, trips over 3 hours).
  3. Find trips and average fare by hour of day.
  4. Join with the zone lookup and find the 10 busiest pickup zones.
  5. Calculate average tip percentage by payment type.

Starter code

1from pyspark.sql import SparkSession, functions as F
2
3spark = SparkSession.builder.appName("nyc-taxi").getOrCreate()
4
5trips = spark.read.parquet("yellow_tripdata_2024-01.parquet")
6zones = spark.read.csv("taxi_zone_lookup.csv", header=True, inferSchema=True)
7
8# Trip duration in minutes + remove obviously bad records
9clean = (trips
10    .withColumn("duration_min",
11        (F.unix_timestamp("tpep_dropoff_datetime")
12         - F.unix_timestamp("tpep_pickup_datetime")) / 60)
13    .filter((F.col("duration_min") > 1) & (F.col("duration_min") < 180))
14    .filter((F.col("trip_distance") > 0) & (F.col("fare_amount") > 0)))
15
16# Demand and average fare by hour of day
17by_hour = (clean
18    .groupBy(F.hour("tpep_pickup_datetime").alias("pickup_hour"))
19    .agg(F.count("*").alias("trips"),
20         F.round(F.avg("fare_amount"), 2).alias("avg_fare"),
21         F.round(F.avg("duration_min"), 1).alias("avg_minutes"))
22    .orderBy("pickup_hour"))
23by_hour.show(24)
24
25# Busiest pickup zones (zones table is tiny -> broadcast it)
26top_zones = (clean
27    .join(F.broadcast(zones), clean.PULocationID == zones.LocationID)
28    .groupBy("Borough", "Zone")
29    .count()
30    .orderBy(F.desc("count"))
31    .limit(10))
32top_zones.show(truncate=False)

The zone table only has ~265 rows, so broadcasting it avoids a shuffle. Learn more in join strategies in Spark.

Stretch goals (make it portfolio-ready)
  • Load a full year (12 files) and compare how long the job takes with and without caching the cleaned DataFrame.
  • Find the most common pickup → drop-off zone pairs.
  • Detect and report the percentage of bad records per month as a data quality metric.

3. MovieLens Ratings Analysis (Beginner)

A small, clean dataset with multiple related tables — perfect for practising joins and array functions.

  • Dataset: MovieLens from GroupLens (start with ml-latest-small: ~100k ratings, ~9k movies).
  • Skills you practice: joins, explode, split, aggregations with filters, sorting on multiple columns.

What to build

  1. Load movies.csv and ratings.csv.
  2. Find the top 10 movies by average rating with at least 50 ratings.
  3. Split the pipe-separated genres column and compute the average rating per genre.
  4. Find the 10 most active users.
  5. Convert the Unix timestamp column to a date and show the number of ratings per year.

Starter code

1from pyspark.sql import SparkSession, functions as F
2
3spark = SparkSession.builder.appName("movielens").getOrCreate()
4
5movies = spark.read.csv("movies.csv", header=True, inferSchema=True)
6ratings = spark.read.csv("ratings.csv", header=True, inferSchema=True)
7
8# Top-rated movies with at least 50 ratings
9movie_stats = (ratings
10    .groupBy("movieId")
11    .agg(F.count("*").alias("num_ratings"),
12         F.round(F.avg("rating"), 2).alias("avg_rating"))
13    .filter("num_ratings >= 50")
14    .join(movies, "movieId")
15    .orderBy(F.desc("avg_rating"), F.desc("num_ratings")))
16movie_stats.select("title", "avg_rating", "num_ratings").show(10, truncate=False)
17
18# Average rating per genre ("Action|Comedy" -> one row per genre)
19genre_stats = (movies
20    .withColumn("genre", F.explode(F.split("genres", r"\|")))
21    .join(ratings, "movieId")
22    .groupBy("genre")
23    .agg(F.round(F.avg("rating"), 2).alias("avg_rating"),
24         F.count("*").alias("num_ratings"))
25    .orderBy(F.desc("num_ratings")))
26genre_stats.show()

Need a refresher on joins? Read joining data in PySpark.

Stretch goals (make it portfolio-ready)
  • Build a simple movie recommender with pyspark.ml.recommendation.ALS and evaluate it with RMSE.
  • Use a window function to find the top 3 movies within each genre.
  • Repeat the analysis on the 25M-rating version and note what changes in the Spark UI.

4. Web Server Log Analysis (Beginner/Intermediate)

Not all data comes as neat CSVs. In this project you parse raw text logs with regular expressions — a very common task in real data engineering jobs.

  • Dataset: NASA HTTP access logs (July and August 1995), a well-known public web server log dataset.
  • Skills you practice: spark.read.text, regexp_extract, parsing custom timestamp formats, handling malformed lines.

What to build

  1. Read the raw log file as text (one row per line).
  2. Parse host, timestamp, HTTP method, endpoint, status code and response size with a regex.
  3. Count lines that fail to parse — never silently drop data.
  4. Find the top 10 endpoints, the top 10 endpoints returning 404, and requests per hour.
  5. Find the number of unique hosts per day.

Starter code

1from pyspark.sql import SparkSession, functions as F
2
3spark = SparkSession.builder.appName("log-analysis").getOrCreate()
4
5# Each line looks like:
6# 199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245
7logs = spark.read.text("NASA_access_log_Jul95")
8
9pattern = r'^(\S+) \S+ \S+ \[([^\]]+)\] "(\S+) (\S+)\s*\S*" (\d{3}) (\S+)'
10
11parsed = (logs.select(
12        F.regexp_extract("value", pattern, 1).alias("host"),
13        F.regexp_extract("value", pattern, 2).alias("ts_raw"),
14        F.regexp_extract("value", pattern, 3).alias("method"),
15        F.regexp_extract("value", pattern, 4).alias("endpoint"),
16        F.regexp_extract("value", pattern, 5).alias("status"),
17        F.regexp_extract("value", pattern, 6).alias("bytes_raw"))
18    .filter(F.col("host") != "")   # lines that did not match the pattern
19    .withColumn("ts", F.to_timestamp("ts_raw", "dd/MMM/yyyy:HH:mm:ss Z"))
20    .withColumn("status", F.col("status").cast("int"))
21    .withColumn("bytes",
22        F.when(F.col("bytes_raw") == "-", 0).otherwise(F.col("bytes_raw").cast("long"))))
23
24# Top endpoints, top 404s, traffic per hour
25parsed.groupBy("endpoint").count().orderBy(F.desc("count")).show(10, truncate=False)
26
27(parsed.filter(F.col("status") == 404)
28       .groupBy("endpoint").count()
29       .orderBy(F.desc("count"))
30       .show(10, truncate=False))
31
32parsed.groupBy(F.hour("ts").alias("hour")).count().orderBy("hour").show(24)

Text-processing functions like regexp_extract are covered in the string functions tutorial.

Stretch goals (make it portfolio-ready)
  • Write malformed lines to a separate "bad records" folder.
  • Find the hosts with the most 4xx/5xx errors — a basic anomaly detector.
  • Package the parser as a reusable function with unit tests (pytest + a local SparkSession).

5. Data Cleaning Pipeline (Intermediate)

Real pipelines spend most of their time on data quality. This project simulates a messy customer feed: duplicates, inconsistent casing, multiple date formats, invalid emails and nulls.

  • Dataset: the inline sample below — then generate a larger messy file yourself (or use any public customer/CRM sample).
  • Skills you practice: trim/lower/initcap, rlike validation, row_number() for deduplication, fillna, quarantining bad rows.

What to build

  1. Standardise names (trim + title case) and emails (trim + lower case).
  2. Parse signup_date, which arrives in two different formats.
  3. Keep only the latest record per customer_id using a window function (not dropDuplicates).
  4. Validate emails and send invalid rows to a quarantine DataFrame.
  5. Fill missing names with a default value.

Starter code

1from pyspark.sql import SparkSession, Window, functions as F
2
3spark = SparkSession.builder.appName("data-cleaning").getOrCreate()
4
5raw = spark.createDataFrame([
6    (1, "  asha ", "ASHA@MAIL.COM ", "2024-01-05", "2024-03-01 10:00:00"),
7    (1, "Asha",    "asha@mail.com",  "05/01/2024", "2024-04-01 09:00:00"),
8    (2, "Ravi",    "ravi[at]mail.com", "2024-02-11", "2024-03-02 11:00:00"),
9    (3, None,      "meera@mail.com", None,         "2024-03-03 12:00:00"),
10], ["customer_id", "name", "email", "signup_date", "updated_at"])
11
12# Parse dates that arrive in two different formats
13signup = (F.when(F.col("signup_date").rlike(r"^\d{4}-\d{2}-\d{2}$"),
14                 F.to_date("signup_date", "yyyy-MM-dd"))
15           .when(F.col("signup_date").rlike(r"^\d{2}/\d{2}/\d{4}$"),
16                 F.to_date("signup_date", "dd/MM/yyyy")))
17
18# Keep only the latest record per customer
19latest = Window.partitionBy("customer_id").orderBy(F.col("updated_at").desc())
20
21clean = (raw
22    .withColumn("name", F.initcap(F.trim("name")))
23    .withColumn("email", F.lower(F.trim("email")))
24    .withColumn("signup_date", signup)
25    .withColumn("updated_at", F.to_timestamp("updated_at"))
26    .withColumn("is_valid_email", F.col("email").rlike(r"^[\w.+-]+@[\w-]+\.[\w.]+$"))
27    .withColumn("rn", F.row_number().over(latest))
28    .filter("rn = 1")
29    .drop("rn"))
30
31good = clean.filter("is_valid_email").fillna({"name": "Unknown"})
32quarantine = clean.filter(~F.col("is_valid_email"))
33
34good.show()
35quarantine.show()

Why row_number() instead of dropDuplicates()? dropDuplicates keeps an arbitrary row, while the window lets you decide *which* row wins. More on handling nulls and window functions.

Stretch goals (make it portfolio-ready)
  • Produce a data quality report: row counts in, rows out, rows quarantined, nulls per column.
  • Make the date parsing configurable (list of accepted formats).
  • Add schema enforcement so an unexpected new column fails the job loudly.

6. Customer RFM Segmentation (Intermediate)

RFM (Recency, Frequency, Monetary) is a standard marketing analytics technique. It reuses the cleaned data from Project 1 and adds window functions on top.

  • Dataset: the cleaned sales DataFrame from Project 1 (Online Retail II).
  • Skills you practice: datediff, countDistinct, ntile() window functions, when/otherwise business rules.

What to build

  1. For each customer compute recency (days since last purchase), frequency (number of invoices) and monetary value (total revenue).
  2. Score each metric from 1 to 5 using ntile(5).
  3. Assign segments such as Champions, Loyal, At Risk and Needs Attention.
  4. Report the number of customers and average spend per segment.

Starter code

1from pyspark.sql import Window, functions as F
2
3# `sales` = cleaned DataFrame from Project 1 (one row per invoice line, with revenue)
4snapshot_date = (sales
5    .agg(F.date_add(F.max(F.to_date("invoice_date")), 1).alias("d"))
6    .first()["d"])
7
8rfm = (sales
9    .filter(F.col("customer_id").isNotNull())
10    .groupBy("customer_id")
11    .agg(F.datediff(F.lit(snapshot_date), F.max(F.to_date("invoice_date"))).alias("recency"),
12         F.countDistinct("invoice").alias("frequency"),
13         F.round(F.sum("revenue"), 2).alias("monetary")))
14
15# Score each metric 1-5 (5 = best). Lower recency is better, so sort it descending.
16rfm = (rfm
17    .withColumn("r_score", F.ntile(5).over(Window.orderBy(F.desc("recency"))))
18    .withColumn("f_score", F.ntile(5).over(Window.orderBy("frequency")))
19    .withColumn("m_score", F.ntile(5).over(Window.orderBy("monetary")))
20    .withColumn("segment",
21        F.when((F.col("r_score") >= 4) & (F.col("f_score") >= 4), "Champions")
22         .when((F.col("r_score") >= 3) & (F.col("f_score") >= 3), "Loyal")
23         .when(F.col("r_score") <= 2, "At Risk")
24         .otherwise("Needs Attention")))
25
26rfm.groupBy("segment").agg(
27    F.count("*").alias("customers"),
28    F.round(F.avg("monetary"), 2).alias("avg_spend")
29).show()

Interview tip: a window with orderBy but no partitionBy moves all rows to a single partition. It is fine here (one row per customer), but be ready to explain why it would not scale to billions of rows.

Stretch goals (make it portfolio-ready)
  • Compare segments across two time periods to see which customers moved from Champions to At Risk.
  • Replace ntile with percent_rank and compare the results.
  • Export the segments to a CSV that a marketing team could use.

7. Clickstream Sessionization (Intermediate)

Sessionization — grouping a user's events into sessions with a 30-minute inactivity timeout — is one of the most popular PySpark coding interview questions. Build it once properly and you will never forget it.

  • Dataset: the inline sample below; for scale, generate a few million synthetic events with spark.range() and rand().
  • Skills you practice: lag(), running sum() over a window, min_by, timestamp arithmetic.

What to build

  1. Order each user's events by time.
  2. Compute the gap since the previous event with lag().
  3. Flag a new session when the gap is over 30 minutes (or it is the user's first event).
  4. Build a session id with a running sum of the flag.
  5. Summarise each session: start, end, duration, number of events and landing page.

Starter code

1from pyspark.sql import SparkSession, Window, functions as F
2
3spark = SparkSession.builder.appName("sessionization").getOrCreate()
4
5events = (spark.createDataFrame([
6        ("u1", "2024-05-01 10:00:00", "home"),
7        ("u1", "2024-05-01 10:05:00", "product"),
8        ("u1", "2024-05-01 10:50:00", "home"),     # > 30 min gap -> new session
9        ("u1", "2024-05-01 10:52:00", "cart"),
10        ("u2", "2024-05-01 09:00:00", "home"),
11        ("u2", "2024-05-01 09:10:00", "checkout"),
12    ], ["user_id", "event_time", "page"])
13    .withColumn("event_time", F.to_timestamp("event_time")))
14
15w = Window.partitionBy("user_id").orderBy("event_time")
16
17sessions = (events
18    .withColumn("prev_time", F.lag("event_time").over(w))
19    .withColumn("gap_min",
20        (F.col("event_time").cast("long") - F.col("prev_time").cast("long")) / 60)
21    .withColumn("is_new_session",
22        F.when(F.col("prev_time").isNull() | (F.col("gap_min") > 30), 1).otherwise(0))
23    .withColumn("session_num", F.sum("is_new_session").over(w))
24    .withColumn("session_id",
25        F.concat_ws("-", "user_id", F.col("session_num").cast("string"))))
26
27summary = (sessions
28    .groupBy("user_id", "session_id")
29    .agg(F.min("event_time").alias("session_start"),
30         F.max("event_time").alias("session_end"),
31         F.count("*").alias("events"),
32         F.expr("min_by(page, event_time)").alias("landing_page"))
33    .withColumn("duration_min",
34        (F.col("session_end").cast("long") - F.col("session_start").cast("long")) / 60))
35summary.orderBy("session_id").show(truncate=False)

See lead and lag window functions and rowsBetween / rangeBetween for the window concepts used here.

Stretch goals (make it portfolio-ready)
  • Compute the conversion rate: sessions that reach a checkout page.
  • Find the most common 3-page paths with collect_list over a window.
  • Rewrite the same logic in Spark SQL and compare the physical plans.

8. SCD Type 2 Customer Dimension (Advanced)

Slowly Changing Dimensions (Type 2) keep full history when an attribute changes — for example, a customer moving cities. It is a staple of data warehouse interviews.

  • Dataset: the inline dimension and daily updates below.
  • Skills you practice: left_anti and left_semi joins, unionByName, effective dates and current flags.

What to build

  1. Start with a dimension that has effective_from, effective_to and is_current columns.
  2. Detect new and changed customers in the daily updates.
  3. Expire the old version of changed rows (set effective_to and is_current = false).
  4. Insert new versions and brand-new customers as current rows.
  5. Leave unchanged customers untouched.

Starter code

1from pyspark.sql import SparkSession, functions as F
2
3spark = SparkSession.builder.appName("scd2").getOrCreate()
4
5schema = ("customer_id int, name string, city string, "
6          "effective_from string, effective_to string, is_current boolean")
7dim = spark.createDataFrame([
8    (1, "Asha", "Pune",  "2024-01-01", None, True),
9    (2, "Ravi", "Delhi", "2024-01-01", None, True),
10], schema)
11
12updates = spark.createDataFrame([
13    (1, "Asha",  "Mumbai"),    # city changed  -> expire old row, add new version
14    (2, "Ravi",  "Delhi"),     # no change     -> keep as is
15    (3, "Meera", "Chennai"),   # new customer  -> insert
16], "customer_id int, name string, city string")
17
18load_date = "2024-06-01"
19current = dim.filter("is_current")
20history = dim.filter(~F.col("is_current"))
21
22# Rows that are new OR different from the current version
23new_versions = (updates
24    .join(current, ["customer_id", "name", "city"], "left_anti")
25    .withColumn("effective_from", F.lit(load_date))
26    .withColumn("effective_to", F.lit(None).cast("string"))
27    .withColumn("is_current", F.lit(True)))
28
29changed_ids = new_versions.select("customer_id")
30
31# Close out the old version of changed customers
32expired = (current
33    .join(changed_ids, "customer_id", "left_semi")
34    .withColumn("effective_to", F.lit(load_date))
35    .withColumn("is_current", F.lit(False)))
36
37unchanged = current.join(changed_ids, "customer_id", "left_anti")
38
39dim_v2 = (history
40    .unionByName(expired)
41    .unionByName(unchanged)
42    .unionByName(new_versions))
43
44dim_v2.orderBy("customer_id", "effective_from").show()

This version uses plain DataFrames so it runs anywhere. In production you would usually implement the same logic with a Delta Lake MERGE INTO. Note: if tracked columns can be null, compare them with null-safe equality (<=> / eqNullSafe).

Stretch goals (make it portfolio-ready)
  • Re-run the job with the same updates and prove it is idempotent (no duplicate versions).
  • Track changes with a hash of all tracked columns (sha2(concat_ws(...))) instead of comparing column by column.
  • Re-implement it with Delta Lake MERGE and compare the code.

9. Fixing a Skewed Join with Salting (Advanced)

Data skew is the #1 reason Spark jobs get stuck at 199/200 tasks. In this project you create skew on purpose, observe it in the Spark UI, and fix it with salting.

  • Dataset: synthetic — generated with spark.range() so you can control exactly how skewed it is.
  • Skills you practice: diagnosing skew, salting keys, crossJoin, reading the Spark UI, AQE.

What to build

  1. Generate 1M orders where 80% belong to a single customer.
  2. Disable auto-broadcast and AQE so you can see the skew.
  3. Run a normal join and look at task durations in the Spark UI.
  4. Add a random salt to the big table and replicate the small table once per salt value.
  5. Re-run the join and compare the slowest task time.

Starter code

1from pyspark.sql import SparkSession, functions as F
2
3spark = SparkSession.builder.appName("skew-salting").getOrCreate()
4
5# Turn off auto-broadcast and AQE so the skew is visible in the Spark UI
6spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
7spark.conf.set("spark.sql.adaptive.enabled", "false")
8
9# 80% of orders belong to customer 1 -> one huge partition after the shuffle
10orders = (spark.range(0, 1_000_000)
11    .withColumn("customer_id",
12        F.when(F.rand(seed=42) < 0.8, F.lit(1))
13         .otherwise((F.rand(seed=7) * 1000).cast("int")))
14    .withColumn("amount", F.round(F.rand(seed=1) * 100, 2)))
15
16customers = (spark.range(0, 1000).withColumnRenamed("id", "customer_id")
17    .withColumn("tier", F.when(F.col("customer_id") % 10 == 0, "gold").otherwise("silver")))
18
19orders.groupBy("customer_id").count().orderBy(F.desc("count")).show(5)
20
21# Salting: spread the hot key over N buckets
22SALT = 16
23orders_salted = orders.withColumn("salt", (F.rand(seed=3) * SALT).cast("int"))
24customers_salted = customers.crossJoin(
25    spark.range(SALT).withColumnRenamed("id", "salt"))
26
27joined = orders_salted.join(customers_salted, ["customer_id", "salt"])
28
29joined.groupBy("tier").agg(F.round(F.sum("amount"), 2).alias("revenue")).show()

In real life, if the smaller table fits in memory, a broadcast join is simpler than salting — salting is for when both sides are large. Read the theory in data skew and salting and the step-by-step guide on how to fix data skew in Spark.

Stretch goals (make it portfolio-ready)
  • Re-enable AQE (spark.sql.adaptive.skewJoin.enabled) and compare it with manual salting.
  • Salt only the hot keys instead of every key.
  • Write up the before/after task times with Spark UI screenshots — this makes an excellent portfolio post.

10. Medallion (Bronze/Silver/Gold) Pipeline (Advanced)

Put everything together into a small lakehouse-style pipeline: raw files land in bronze, get cleaned and de-duplicated in silver, and are aggregated for reporting in gold. Design it so re-running it never creates duplicates.

  • Dataset: any of the datasets above split into daily CSV drops (for example, the Online Retail data split by invoice date) in a landing/orders/ folder.
  • Skills you practice: layered architecture, ingestion metadata, deduplication, dynamic partition overwrite, idempotency.

What to build

  1. Bronze: read new files as-is and add _source_file and _ingested_at metadata columns.
  2. Silver: cast types, de-duplicate by business key (latest updated_at wins), partition by date.
  3. Gold: build a daily revenue table for dashboards.
  4. Re-run the whole pipeline twice and check that silver and gold row counts do not change.

Starter code

1from pyspark.sql import SparkSession, Window, functions as F
2
3spark = SparkSession.builder.appName("medallion").getOrCreate()
4spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
5
6# BRONZE: land raw files as-is + ingestion metadata
7bronze = (spark.read.csv("landing/orders/", header=True)
8    .withColumn("_source_file", F.input_file_name())
9    .withColumn("_ingested_at", F.current_timestamp()))
10bronze.write.mode("append").parquet("lake/bronze/orders")
11
12# SILVER: typed, de-duplicated (latest version of each order wins)
13latest = Window.partitionBy("order_id").orderBy(F.col("updated_at").desc())
14silver = (spark.read.parquet("lake/bronze/orders")
15    .withColumn("order_date", F.to_date("order_ts"))
16    .withColumn("amount", F.col("amount").cast("decimal(10,2)"))
17    .withColumn("updated_at", F.to_timestamp("updated_at"))
18    .withColumn("rn", F.row_number().over(latest))
19    .filter("rn = 1")
20    .drop("rn"))
21silver.write.mode("overwrite").partitionBy("order_date").parquet("lake/silver/orders")
22
23# GOLD: business-level aggregate
24gold = (spark.read.parquet("lake/silver/orders")
25    .groupBy("order_date")
26    .agg(F.count("*").alias("orders"),
27         F.sum("amount").alias("revenue")))
28gold.write.mode("overwrite").parquet("lake/gold/daily_revenue")

Dynamic partition overwrite only replaces the partitions present in the new data, which keeps reruns safe. Read more about idempotency in data ingestion pipelines and partitioning and bucketing.

Stretch goals (make it portfolio-ready)
  • Process only new files by keeping a list of already-ingested files (or use Structured Streaming with trigger(availableNow=True)).
  • Swap Parquet for Delta Lake and use MERGE in the silver layer.
  • Add data quality checks between layers and fail the run when they break.

How to Present PySpark Projects on Your Resume

  • Lead with the outcome, not the tool: "Built a sessionization pipeline over 5M click events that computes conversion per session" beats "Used PySpark window functions".
  • Mention scale and performance: data size, runtime before/after, and what you tuned (partitions, broadcast joins, salting, AQE).
  • Show data quality thinking: how many records were bad, where they went, and how you detected them.
  • Keep the code clean: small functions, a config file for paths, and a few unit tests with a local SparkSession.

Practice More with Spark Playground

Projects build depth; coding questions build speed. Once you are comfortable with a few of these projects, practise interview-style problems in the PySpark coding interview questions section, keep the PySpark cheat sheet open while you code, and test ideas instantly in the PySpark online compiler.

FAQ

Which PySpark project is best for beginners?

Start with the Retail Sales Analysis or the MovieLens project. They use small, clean-ish datasets and cover the core DataFrame operations: reading files, filtering, joining and aggregating.

Can I practice PySpark online without installing Spark?

Yes. You can run all of the inline-data snippets in this article in the free Spark Playground PySpark online compiler. For the full datasets, use a local install (pip install pyspark) or Databricks Free Edition.

How many PySpark projects do I need for a data engineering resume?

Two or three well-documented projects are better than ten half-finished ones. A good combination is one analytics project (1–3), one window-function project (6–7) and one pipeline or performance project (8–10).

Do I need a cluster to practice PySpark?

No. Spark runs in local mode on a laptop, which is enough for datasets up to a few GB. A cluster only matters when you want to practise tuning executors and partitions at real scale.