PySpark Coding Interview Questions - Practice Online

Solve the most common PySpark coding interview questions asked in Data Engineering, Data Analyst and Data Science roles.
Use PySpark APIs or Spark SQL with temporary view to solve these coding questions.
You can also check the most popular conceptual questions here.

1. Load and Transform Data

Practice loading a CSV file and apply basic transformations on columns.

PopularEasySelectDropFiltering

2. Handling Null Values

Clean the dataset by filtering out or replacing null values in various columns.

BasicEasySelectCleaningFiltering

3. Total Purchases by Customer

Group data by customer and compute the total purchase amount per user.

BasicEasySelectGroupingFilteringCasting

4. Running Payroll

Help Human Resources calculate pay for each employee.

BasicMediumJoinsArithmeticConditional

5. Food and Beverage Sales

Calculate total sales quantity, revenue, and stock for each product.

BasicMediumAggregate FunctionsJoinsData Cleaning

6. Discounts on Products

Add a new column calculating discounted prices for products using arithmetic operations.

BasicEasySelectArithmeticCasting

7. Load & Transform JSON file

Read a nested JSON file and flatten it using explode and array-handling techniques.

PopularMediumSelectJsonExplodeArrays

8. Employee Earnings

Use window functions to find employees whose salary is higher than the department average.

AccentureHardSelectWindowsGroupingJoiningFunctions

9. Remove Duplicates From Dataset

Identify and remove duplicate records based on custom logic using window functions.

PopularMediumFilteringWindow FunctionsDatesGrouping

10. Word Count Program in PySpark

Implement a word count logic using PySpark RDD transformations on a text file.

PopularMediumRDDTextfileGrouping

11. Group By and Aggregate List

Group records and aggregate values into lists using advanced group and array functions.

Tiger AnalyticsHardGroupingAggregationFunctionsArrays

12. Monthly Transaction Summary

Summarize transactions month-wise by grouping and using date functions to extract months.

AmazonMediumGroupingAggregationDate FunctionsTransactions

13. Top Players Summary

Generate a summary of top players using joins, aggregations, and string operations.

ITC InfotechHardGroupingAggregationString FunctionsJoins

14. Daily Total Sales

Calculate total sales for each store on a daily basis using grouping and aggregation.

WalmartEasyAggregationGroupingDate

15. Top 5 Products by Sales

Find the top 5 products with the highest total sales across all stores for a given day.

WalmartMediumAggregationGroupingSortingLimit

16. Products with Increasing Sales

Identify products whose total sales revenue has increased every year.

DeloitteHardWindowJoinFilteringPivot

17. Remove Outliers from Trip Data

Given a dataset of trip costs and customer ratings, remove rows that contain outliers.

VISAMediumQuantilesFilteringIQRData Cleaning

18. Driver Details for Rides

Given datasets of rides and drivers, join them to produce ride-level data.

VISAEasyJoinsSelectAlias

19. Customer Loyalty Score

Calculate customer loyalty scores based on number of trips and ratings.

VISAHardJoinsAggregationFilteringGroupingArithmetic

20. Track Employee History

Implement SCD Type 2 logic to track historical changes in employee records.

DeloitteHardSCD2JoinsFilteringUnion

21. Employee Attendance

Transform employee attendance records to show count of each attendance status.

TCS, InfosysMediumPivotGroupingAggregation

22. Daily Stock Price Change

Calculate day-over-day change in closing stock prices.

NielsenMediumWindow FunctionsLagTime Series

23. Analyze Review Sentiment

Use a PySpark UDF to classify reviews based on keywords.

TargetHardUDFText ProcessingString Manipulation

24. Sessionize Clickstream Data

Assign session IDs and visit numbers to user clickstream data.

NielsenHardWindow FunctionsTime DifferenceSessionization

25. Combine Transactions Data

Unify records from multiple sources with different columns

BasicEasyUnionSchema EvolutionData Integration

26. Categorize Products by Price

Classify products into categories based on price

BasicEasyConditional ColumnData TransformationCategorization

27. Aggregate Item Weight per Person

Compute total weight of each item per person

AmazonEasyAggregationGroupbyData Summarization

28. Maximize Items Within Budget

Determine the maximum number of items that can be purchased within a fixed budget using PySpark.

EPAMHardKnapsackWindow FunctionsAggregationBudget Optimization

29. Running Balance per User

Compute the running balance for each user over time.

AmazonMediumWindow FunctionsCumulative SumEvent StreamTime Series

30. Top N Sales per Store

Find the Top N highest sales transactions for each store.

AmazonMediumWindow FunctionsRow NumberRankingTop N

31. User Sessionization with Time

Define user sessions and calculate event counts, start and end times

NielsenHardWindow FunctionsLagCumulative SumSessionizationTime Series

32. Employee Attendance Streaks

Calculate each employee's longest consecutive attendance streak.

PayPalHardWindow FunctionsLagGroupingDate Functions

33. Items Purchased by User

Perform various operations on the array column.

BasicMediumArray Functions

34. Sales Segmentation

Categorize sales records into buckets, and compute average sales per category.

UberMediumFilteringConditional LogicAggregations

35. Add Metadata to Customer Data

Load multiple CSV files and include metadata columns.

BasicMediumMetadataDataframeFile Handling

36. Tallying Election Results

Determine the seats won by each party in the election.

DocusignHardWindowsJoinsAggregationRanking

37. Call Center

Count unique callers per date and sum total call durations.

WorkdayEasyAggregate FunctionsDistinct Function

38. Social Media PII

Anonymize social media data and extract email domains.

BasicMediumRegular Expressions

39. Total Rental Income

Calculate total rental income for each landlord

BasicHardString FunctionsComplex JoinsAggregate Functions

40. E-Commerce Stats

Given product and order data from an e-commerce platform, calculate required KPIs

BasicMediumAggregate FunctionsComplex Joins

41. Private Equity Data Merge

Combine firm, fund, and investment data using joins and remove rows with all null values.

BasicMediumComplex JoinsNull Handling

42. Compute Annual GDP Growth Rate

As an economist, calculate the year-over-year GDP growth rate for each country.

WayfairHardWindow FunctionsCase When StatementsMathematical Functions

43. Word Count Program (Part 2)

Clean the text file and count the occurrences of each word in a text file.

GoogleMediumString ProcessingDataframeRegexExplodeAggregation

44. Most Frequent Error Codes

Given application log events, determine the most frequent error codes.

Juniper NetworksMediumAggregationGroupbySortingTop-kLogs

45. Palindrome Words

Check for palindromic words in the data.

HiLabsMediumStringExplodeTransformationBoolean

46. Exam Attendance Report

Produce a report for examination attendance.

HiLabsMediumJoinGroupbyAggregationNull-handling

47. Top Users by Event Types

Compute the top 3 users with the highest number of distinct event types on that day.

CargillMediumWindowGroupbyAggregation

48. Student Marks to Grade

Assign a letter grade based on the marks range.

AmazonEasyCase-whenColumn-derivationData-cleaning

49. Positive Balance Customer Details

For customers with strictly positive balance, format balance to exactly 2 decimal places and add details.

AmazonMediumJoinFilterFormatting

50. Top Customers by Spend

Find top 2 customers by total spend in each region.

EPAMHardJoinWindowRankAggregateFilter

51. Active Subscription Watch Time

Calculate each user’s watch minutes within active subscription dates.

BasicHardJoinDate-filterBetweenAggregateSubscription

52. Ledger Reconciliation Variance Report

Reconcile source vs target totals and compute variance.

BasicHardJoinReconciliationAggregateVarianceNull-handling

53. ETL Job Statistics

Calculate job execution statistics including latest run date, failure counts, and status.

AmazonHardJoinWindowAggregateNull-handling

54. Unsold Products

Find products that were never sold.

AmazonMediumJoinFilterAggregate

55. Connected Airports

Find all distinct airport pairs that have flights between them.

AmazonMediumDistinctFilterScalar Functions

56. Dynamic Schema Transformation

Rename legacy columns, cast them to the correct data types, and derive new USD columns.

BasicMediumRenameCastLoopsDynamic Transformations

57. Flight Delay Analysis

Parse flight departure timestamps, categorize delays, and summarize counts per airline.

BasicMediumTimestampDatetimeCase-whenGroupbyAggregation

58. Price History Tracker

Calculate how many days each product price was active before the next change.

BasicMediumWindow FunctionsLeadDate FunctionsNull Handling

59. Forward Fill Sensor Readings

Fill null sensor readings with the last known non-null value per device.

BasicMediumWindow FunctionsNull HandlingForward FillTime Series

60. Struct Column Flattening

Flatten nested struct fields to the top level dynamically.

BasicMediumStructSchemaDynamic TransformationsFlattenStructtype

61. Order Date Breakdown

Add year, month, and day columns extracted from an order date.

BasicEasyDate FunctionsColumn Derivation

62. Customer Purchase Gaps

Compute the average number of days between consecutive purchases per customer.

AmazonHardWindow FunctionsLagDate FunctionsAggregationGroupby

63. Event Type Pivot

Count user events per type and pivot them into separate columns.

BasicMediumPivotGroupbyAggregationNull Handling

64. Engaged Shoppers Analysis

Find users with 2+ sessions who also ordered that day; aggregate non-returned orders and top platform.

WalmartHardJoinDate FunctionsConditional AggregationWindow FunctionsGroupby

65. Department Budget Utilization

Audit salary costs against department budgets and identify the top earner in each department.

TwitterHardJoinAggregationWindow FunctionsArithmeticConditional Flags

66. Parse Product Attributes JSON

Parse a JSON string column to extract product attributes and filter by rating.

BasicMediumFrom_jsonStructtypeSchemaNested DataFiltering

67. Clean Support Ticket Tags

Remove a deprecated tag and deduplicate tag arrays on support tickets.

BasicEasyArray FunctionsArray_removeArray_distinctFiltering

68. Quarterly Sales Growth Unpivot

Unpivot wide quarterly sales into long format and compute growth rate.

WayfairHardUnpivotStackWindow FunctionsLagGrowth Rate

69. Rolling Sales Average

Compute a trailing 3-day moving average per store and flag days above trend.

TwitterMediumWindow FunctionsRowsbetweenMoving AverageSliding Window

70. Daily New vs Returning Users

Split each day's active users into new and returning, with a running total.

IBMMediumWindow FunctionsCountdistinctCumulative SumDate FunctionsUser Analytics

71. Expand Subscription Months

Expand subscription date ranges into monthly rows and compute monthly MRR.

BasicMediumDate FunctionsSequenceExplodeAggregation

72. Merge Overlapping Bookings

Merge overlapping room bookings into continuous busy blocks per room.

BasicHardWindow FunctionsGaps And IslandsCumulative SumTimestamp

73. Sales Subtotals Report

Build region and category subtotals plus a grand total as labelled rows.

BasicMediumRollupGroupingAggregationNull Handling

74. Product Catalog Upsert

Merge a daily supplier feed into a product catalog with updates and inserts.

MicrosoftMediumSCD1UpsertFull Outer JoinWindow FunctionsNull Handling

75. Signup Funnel Conversion

Track users through an ordered signup funnel and compute conversion rates.

TikTokHardFunnel AnalysisJoinsTimestampWindow FunctionsAggregation

76. Salary Quartile Bands

Assign salary quartiles and percentile positions within each department.

GoogleMediumWindow FunctionsNtilePercent_rankRanking

77. Ordered Purchase Path

Build each user's chronological page path, collapsing consecutive repeats.

BasicMediumArray FunctionsCollect_listWindow FunctionsLag

78. Median Delivery Time

Compute the median delivery time and on-time rate for each courier.

GoogleEasyMedianAggregationTimestampGroupby

79. Feature Flag Map

Build a per-user map of current feature flags and read values from it.

MicrosoftMediumMaptypeMap FunctionsWindow FunctionsHigher-order Functions

80. Duplicate Ride Fares

Flag retried fare charges within 10 minutes and compute refunds per rider.

StripeMediumWindow FunctionsLagTimestampConditional Aggregation

81. EV Charger Utilization

Pair start/stop events into charging sessions and total the hours per charger.

AmazonHardWindow FunctionsLeadSessionizationTimestamp

82. Full-Track Learners

Find students who completed a course in every active learning track.

MicrosoftMediumJoinsRelational DivisionCountdistinctFiltering

83. Gym Member Retention

Compute month-over-month member retention, handling gaps between months.

MetaMediumSelf JoinDate FunctionsRetentionAggregation

84. Game Launch Month Sales

Find each game's launch month sales across platforms and its share of total.

JPMorganMediumWindow FunctionsDate FunctionsAggregationString Functions

85. Grocery Basket Pairs

Find item pairs bought together in at least two orders.

WalmartHardSelf JoinDistinctAggregationMarket Basket

86. Smart Meter Readings

Sum odd- and even-positioned meter readings per meter per day.

GoogleMediumWindow FunctionsRow_numberConditional AggregationDate Functions

87. Most Common Cart Size

Find the most common cart size from pre-aggregated regional counts.

AlibabaEasyAggregationModeWindow FunctionsTies

88. Podcasts With No Listens

Find podcasts with no qualifying listens in October 2024.

MetaEasyAnti JoinFilteringDate Filter

89. Duplicate Rental Listings

Find agents who listed the same property more than once.

LinkedInEasyGroupbyHavingString FunctionsData Cleaning

90. Streaming Device Split

Count viewing sessions by device bucket and compute the mobile minutes share.

New York TimesEasyConditional AggregationString FunctionsData Cleaning

91. Last Visit Checkouts

Count the books each member borrowed on their most recent library visit.

WalmartEasyWindow FunctionsMaxFilteringAggregation

92. Workout Span

Find days between each member's first and last workout in 2024.

MetaEasyDate FunctionsDatediffCountdistinctFiltering

93. Top Support Agents

Find agents with the top two resolved-ticket counts in August, ties included.

MicrosoftEasyDense_rankWindow FunctionsFilteringTies

94. Top Delivery Cities

Find the top 3 cities by delivered orders, excluding unknown cities.

RobinhoodEasyJoinsNull HandlingAggregationTop N

95. Monthly Hotel Ratings

Compute monthly average guest ratings per hotel, ignoring invalid stars.

AmazonEasyDate FunctionsAggregationData Cleaning

Stay tuned - we're adding more interview questions!

Interested in contributing? We'd love your help - click "Help Us Improve" on the bottom right.