PySpark Coding Interview Questions - Practice Online
Solve the most common PySpark coding interview questions asked in Data Engineering, Data Analyst and Data Science roles.
Use PySpark APIs or Spark SQL with temporary view to solve these coding questions.
You can also check the most popular conceptual questions here.
| # | Title & Description | Difficulty | Tags | Status | |
|---|---|---|---|---|---|
1 | Load and Transform DataPractice loading a CSV file and apply basic transformations on columns. | Easy | PopularSelectDropFiltering | ||
2 | Handling Null ValuesClean the dataset by filtering out or replacing null values in various columns. | Easy | SelectCleaningFiltering | ||
3 | Total Purchases by CustomerGroup data by customer and compute the total purchase amount per user. | Easy | SelectGroupingFilteringCasting | ||
4 | Running PayrollHelp Human Resources calculate pay for each employee. | Medium | JoinsArithmeticConditional | ||
5 | Food and Beverage SalesCalculate total sales quantity, revenue, and stock for each product. | Medium | Aggregate FunctionsJoinsData Cleaning | ||
6 | Discounts on ProductsPremiumAdd a new column calculating discounted prices for products using arithmetic operations. | Easy | SelectArithmeticCasting | ||
7 | Load & Transform JSON filePremiumRead a nested JSON file and flatten it using explode and array-handling techniques. | Medium | PopularSelectJsonExplodeArrays | ||
8 | Employee EarningsPremiumUse window functions to find employees whose salary is higher than the department average. | Hard | AccentureSelectWindowsGroupingJoiningFunctions | ||
9 | Remove Duplicates From DatasetPremiumIdentify and remove duplicate records based on custom logic using window functions. | Medium | PopularFilteringWindow FunctionsDatesGrouping | ||
10 | Word Count Program in PySparkImplement a word count logic using PySpark RDD transformations on a text file. | Medium | PopularRDDTextfileGrouping | ||
11 | Group By and Aggregate ListPremiumGroup records and aggregate values into lists using advanced group and array functions. | Hard | Tiger AnalyticsGroupingAggregationFunctionsArrays | ||
12 | Monthly Transaction SummaryPremiumSummarize transactions month-wise by grouping and using date functions to extract months. | Medium | AmazonGroupingAggregationDate FunctionsTransactions | ||
13 | Top Players SummaryPremiumGenerate a summary of top players using joins, aggregations, and string operations. | Hard | ITC InfotechGroupingAggregationString FunctionsJoins | ||
14 | Daily Total SalesPremiumCalculate total sales for each store on a daily basis using grouping and aggregation. | Easy | WalmartAggregationGroupingDate | ||
15 | Top 5 Products by SalesPremiumFind the top 5 products with the highest total sales across all stores for a given day. | Medium | WalmartAggregationGroupingSortingLimit | ||
16 | Products with Increasing SalesPremiumIdentify products whose total sales revenue has increased every year. | Hard | DeloitteWindowJoinFilteringPivot | ||
17 | Remove Outliers from Trip DataPremiumGiven a dataset of trip costs and customer ratings, remove rows that contain outliers. | Medium | VISAQuantilesFilteringIQRData Cleaning | ||
18 | Driver Details for RidesPremiumGiven datasets of rides and drivers, join them to produce ride-level data. | Easy | VISAJoinsSelectAlias | ||
19 | Customer Loyalty ScorePremiumCalculate customer loyalty scores based on number of trips and ratings. | Hard | VISAJoinsAggregationFilteringGroupingArithmetic | ||
20 | Track Employee HistoryPremiumImplement SCD Type 2 logic to track historical changes in employee records. | Hard | DeloitteSCD2JoinsFilteringUnion | ||
21 | Employee AttendancePremiumTransform employee attendance records to show count of each attendance status. | Medium | TCS, InfosysPivotGroupingAggregation | ||
22 | Daily Stock Price ChangePremiumCalculate day-over-day change in closing stock prices. | Medium | NielsenWindow FunctionsLagTime Series | ||
23 | Analyze Review SentimentPremiumUse a PySpark UDF to classify reviews based on keywords. | Hard | TargetUDFText ProcessingString Manipulation | ||
24 | Sessionize Clickstream DataPremiumAssign session IDs and visit numbers to user clickstream data. | Hard | NielsenWindow FunctionsTime DifferenceSessionization | ||
25 | Combine Transactions DataPremiumUnify records from multiple sources with different columns | Easy | UnionSchema EvolutionData Integration | ||
26 | Categorize Products by PricePremiumClassify products into categories based on price | Easy | Conditional ColumnData TransformationCategorization | ||
27 | Aggregate Item Weight per PersonPremiumCompute total weight of each item per person | Easy | AmazonAggregationGroupbyData Summarization | ||
28 | Maximize Items Within BudgetPremiumDetermine the maximum number of items that can be purchased within a fixed budget using PySpark. | Hard | EPAMKnapsackWindow FunctionsAggregationBudget Optimization | ||
29 | Running Balance per UserPremiumCompute the running balance for each user over time. | Medium | AmazonWindow FunctionsCumulative SumEvent StreamTime Series | ||
30 | Top N Sales per StorePremiumFind the Top N highest sales transactions for each store. | Medium | AmazonWindow FunctionsRow NumberRankingTop N | ||
31 | User Sessionization with TimePremiumDefine user sessions and calculate event counts, start and end times | Hard | NielsenWindow FunctionsLagCumulative SumSessionizationTime Series | ||
32 | Employee Attendance StreaksPremiumCalculate each employee's longest consecutive attendance streak. | Hard | PayPalWindow FunctionsLagGroupingDate Functions | ||
33 | Items Purchased by UserPremiumPerform various operations on the array column. | Medium | Array Functions | ||
34 | Sales SegmentationPremiumCategorize sales records into buckets, and compute average sales per category. | Medium | UberFilteringConditional LogicAggregations | ||
35 | Add Metadata to Customer DataPremiumLoad multiple CSV files and include metadata columns. | Medium | MetadataDataframeFile Handling | ||
36 | Tallying Election ResultsPremiumDetermine the seats won by each party in the election. | Hard | DocusignWindowsJoinsAggregationRanking | ||
37 | Call CenterPremiumCount unique callers per date and sum total call durations. | Easy | WorkdayAggregate FunctionsDistinct Function | ||
38 | Social Media PIIPremiumAnonymize social media data and extract email domains. | Medium | Regular Expressions | ||
39 | Total Rental IncomePremiumCalculate total rental income for each landlord | Hard | String FunctionsComplex JoinsAggregate Functions | ||
40 | E-Commerce StatsPremiumGiven product and order data from an e-commerce platform, calculate required KPIs | Medium | Aggregate FunctionsComplex Joins | ||
41 | Private Equity Data MergePremiumCombine firm, fund, and investment data using joins and remove rows with all null values. | Medium | Complex JoinsNull Handling | ||
42 | Compute Annual GDP Growth RatePremiumAs an economist, calculate the year-over-year GDP growth rate for each country. | Hard | WayfairWindow FunctionsCase When StatementsMathematical Functions | ||
43 | Word Count Program (Part 2)PremiumClean the text file and count the occurrences of each word in a text file. | Medium | GoogleString ProcessingDataframeRegexExplodeAggregation | ||
44 | Most Frequent Error CodesPremiumGiven application log events, determine the most frequent error codes. | Medium | Juniper NetworksAggregationGroupbySortingTop-kLogs | ||
45 | Palindrome WordsPremiumCheck for palindromic words in the data. | Medium | HiLabsStringExplodeTransformationBoolean | ||
46 | Exam Attendance ReportPremiumProduce a report for examination attendance. | Medium | HiLabsJoinGroupbyAggregationNull-handling | ||
47 | Top Users by Event TypesPremiumCompute the top 3 users with the highest number of distinct event types on that day. | Medium | CargillWindowGroupbyAggregation | ||
48 | Student Marks to GradePremiumAssign a letter grade based on the marks range. | Easy | AmazonCase-whenColumn-derivationData-cleaning | ||
49 | Positive Balance Customer DetailsPremiumFor customers with strictly positive balance, format balance to exactly 2 decimal places and add details. | Medium | AmazonJoinFilterFormatting | ||
50 | Top Customers by SpendPremiumFind top 2 customers by total spend in each region. | Hard | EPAMJoinWindowRankAggregateFilter | ||
51 | Active Subscription Watch TimePremiumCalculate each user’s watch minutes within active subscription dates. | Hard | JoinDate-filterBetweenAggregateSubscription | ||
52 | Ledger Reconciliation Variance ReportPremiumReconcile source vs target totals and compute variance. | Hard | JoinReconciliationAggregateVarianceNull-handling | ||
53 | ETL Job StatisticsPremiumCalculate job execution statistics including latest run date, failure counts, and status. | Hard | AmazonJoinWindowAggregateNull-handling | ||
54 | Unsold ProductsPremiumFind products that were never sold. | Medium | AmazonJoinFilterAggregate | ||
55 | Connected AirportsPremiumFind all distinct airport pairs that have flights between them. | Medium | AmazonDistinctFilterScalar Functions | ||
56 | Dynamic Schema TransformationPremiumRename legacy columns, cast them to the correct data types, and derive new USD columns. | Medium | RenameCastLoopsDynamic Transformations | ||
57 | Flight Delay AnalysisPremiumParse flight departure timestamps, categorize delays, and summarize counts per airline. | Medium | TimestampDatetimeCase-whenGroupbyAggregation | ||
58 | Price History TrackerPremiumCalculate how many days each product price was active before the next change. | Medium | Window FunctionsLeadDate FunctionsNull Handling | ||
59 | Forward Fill Sensor ReadingsPremiumFill null sensor readings with the last known non-null value per device. | Medium | Window FunctionsNull HandlingForward FillTime Series | ||
60 | Struct Column FlatteningPremiumFlatten nested struct fields to the top level dynamically. | Medium | StructSchemaDynamic TransformationsFlattenStructtype | ||
61 | Order Date BreakdownPremiumAdd year, month, and day columns extracted from an order date. | Easy | Date FunctionsColumn Derivation | ||
62 | Customer Purchase GapsPremiumCompute the average number of days between consecutive purchases per customer. | Hard | AmazonWindow FunctionsLagDate FunctionsAggregationGroupby | ||
63 | Event Type PivotPremiumCount user events per type and pivot them into separate columns. | Medium | PivotGroupbyAggregationNull Handling | ||
64 | Engaged Shoppers AnalysisPremiumFind users with 2+ sessions who also ordered that day; aggregate non-returned orders and top platform. | Hard | WalmartJoinDate FunctionsConditional AggregationWindow FunctionsGroupby | ||
65 | Department Budget UtilizationAudit salary costs against department budgets and identify the top earner in each department. | Hard | TwitterJoinAggregationWindow FunctionsArithmeticConditional Flags | ||
66 | Parse Product Attributes JSONParse a JSON string column to extract product attributes and filter by rating. | Medium | From_jsonStructtypeSchemaNested DataFiltering | ||
67 | Clean Support Ticket TagsRemove a deprecated tag and deduplicate tag arrays on support tickets. | Easy | Array FunctionsArray_removeArray_distinctFiltering | ||
68 | Quarterly Sales Growth UnpivotPremiumUnpivot wide quarterly sales into long format and compute growth rate. | Hard | WayfairUnpivotStackWindow FunctionsLagGrowth Rate | ||
69 | Rolling Sales AveragePremiumCompute a trailing 3-day moving average per store and flag days above trend. | Medium | TwitterWindow FunctionsRowsbetweenMoving AverageSliding Window | ||
70 | Daily New vs Returning UsersPremiumSplit each day's active users into new and returning, with a running total. | Medium | IBMWindow FunctionsCountdistinctCumulative SumDate FunctionsUser Analytics | ||
71 | Expand Subscription MonthsPremiumExpand subscription date ranges into monthly rows and compute monthly MRR. | Medium | Date FunctionsSequenceExplodeAggregation | ||
72 | Merge Overlapping BookingsPremiumMerge overlapping room bookings into continuous busy blocks per room. | Hard | Window FunctionsGaps And IslandsCumulative SumTimestamp | ||
73 | Sales Subtotals ReportPremiumBuild region and category subtotals plus a grand total as labelled rows. | Medium | RollupGroupingAggregationNull Handling | ||
74 | Product Catalog UpsertPremiumMerge a daily supplier feed into a product catalog with updates and inserts. | Medium | MicrosoftSCD1UpsertFull Outer JoinWindow FunctionsNull Handling | ||
75 | Signup Funnel ConversionPremiumTrack users through an ordered signup funnel and compute conversion rates. | Hard | TikTokFunnel AnalysisJoinsTimestampWindow FunctionsAggregation | ||
76 | Salary Quartile BandsPremiumAssign salary quartiles and percentile positions within each department. | Medium | GoogleWindow FunctionsNtilePercent_rankRanking | ||
77 | Ordered Purchase PathPremiumBuild each user's chronological page path, collapsing consecutive repeats. | Medium | Array FunctionsCollect_listWindow FunctionsLag | ||
78 | Median Delivery TimePremiumCompute the median delivery time and on-time rate for each courier. | Easy | GoogleMedianAggregationTimestampGroupby | ||
79 | Feature Flag MapPremiumBuild a per-user map of current feature flags and read values from it. | Medium | MicrosoftMaptypeMap FunctionsWindow FunctionsHigher-order Functions | ||
80 | Duplicate Ride FaresPremiumFlag retried fare charges within 10 minutes and compute refunds per rider. | Medium | StripeWindow FunctionsLagTimestampConditional Aggregation | ||
81 | EV Charger UtilizationPremiumPair start/stop events into charging sessions and total the hours per charger. | Hard | AmazonWindow FunctionsLeadSessionizationTimestamp | ||
82 | Full-Track LearnersPremiumFind students who completed a course in every active learning track. | Medium | MicrosoftJoinsRelational DivisionCountdistinctFiltering | ||
83 | Gym Member RetentionPremiumCompute month-over-month member retention, handling gaps between months. | Medium | MetaSelf JoinDate FunctionsRetentionAggregation | ||
84 | Game Launch Month SalesPremiumFind each game's launch month sales across platforms and its share of total. | Medium | JPMorganWindow FunctionsDate FunctionsAggregationString Functions | ||
85 | Grocery Basket PairsPremiumFind item pairs bought together in at least two orders. | Hard | WalmartSelf JoinDistinctAggregationMarket Basket | ||
86 | Smart Meter ReadingsPremiumSum odd- and even-positioned meter readings per meter per day. | Medium | GoogleWindow FunctionsRow_numberConditional AggregationDate Functions | ||
87 | Most Common Cart SizePremiumFind the most common cart size from pre-aggregated regional counts. | Easy | AlibabaAggregationModeWindow FunctionsTies | ||
88 | Podcasts With No ListensPremiumFind podcasts with no qualifying listens in October 2024. | Easy | MetaAnti JoinFilteringDate Filter | ||
89 | Duplicate Rental ListingsPremiumFind agents who listed the same property more than once. | Easy | LinkedInGroupbyHavingString FunctionsData Cleaning | ||
90 | Streaming Device SplitCount viewing sessions by device bucket and compute the mobile minutes share. | Easy | New York TimesConditional AggregationString FunctionsData Cleaning | ||
91 | Last Visit CheckoutsCount the books each member borrowed on their most recent library visit. | Easy | WalmartWindow FunctionsMaxFilteringAggregation | ||
92 | Workout SpanFind days between each member's first and last workout in 2024. | Easy | MetaDate FunctionsDatediffCountdistinctFiltering | ||
93 | Top Support AgentsFind agents with the top two resolved-ticket counts in August, ties included. | Easy | MicrosoftDense_rankWindow FunctionsFilteringTies | ||
94 | Top Delivery CitiesFind the top 3 cities by delivered orders, excluding unknown cities. | Easy | RobinhoodJoinsNull HandlingAggregationTop N | ||
95 | Monthly Hotel RatingsCompute monthly average guest ratings per hotel, ignoring invalid stars. | Easy | AmazonDate FunctionsAggregationData Cleaning |
Your Progress
Leaderboard
1. Load and Transform Data
Practice loading a CSV file and apply basic transformations on columns.
2. Handling Null Values
Clean the dataset by filtering out or replacing null values in various columns.
3. Total Purchases by Customer
Group data by customer and compute the total purchase amount per user.
4. Running Payroll
Help Human Resources calculate pay for each employee.
5. Food and Beverage Sales
Calculate total sales quantity, revenue, and stock for each product.
6. Discounts on Products
Add a new column calculating discounted prices for products using arithmetic operations.
7. Load & Transform JSON file
Read a nested JSON file and flatten it using explode and array-handling techniques.
8. Employee Earnings
Use window functions to find employees whose salary is higher than the department average.
9. Remove Duplicates From Dataset
Identify and remove duplicate records based on custom logic using window functions.
10. Word Count Program in PySpark
Implement a word count logic using PySpark RDD transformations on a text file.
11. Group By and Aggregate List
Group records and aggregate values into lists using advanced group and array functions.
12. Monthly Transaction Summary
Summarize transactions month-wise by grouping and using date functions to extract months.
13. Top Players Summary
Generate a summary of top players using joins, aggregations, and string operations.
14. Daily Total Sales
Calculate total sales for each store on a daily basis using grouping and aggregation.
15. Top 5 Products by Sales
Find the top 5 products with the highest total sales across all stores for a given day.
16. Products with Increasing Sales
Identify products whose total sales revenue has increased every year.
17. Remove Outliers from Trip Data
Given a dataset of trip costs and customer ratings, remove rows that contain outliers.
18. Driver Details for Rides
Given datasets of rides and drivers, join them to produce ride-level data.
19. Customer Loyalty Score
Calculate customer loyalty scores based on number of trips and ratings.
20. Track Employee History
Implement SCD Type 2 logic to track historical changes in employee records.
21. Employee Attendance
Transform employee attendance records to show count of each attendance status.
22. Daily Stock Price Change
Calculate day-over-day change in closing stock prices.
23. Analyze Review Sentiment
Use a PySpark UDF to classify reviews based on keywords.
24. Sessionize Clickstream Data
Assign session IDs and visit numbers to user clickstream data.
25. Combine Transactions Data
Unify records from multiple sources with different columns
26. Categorize Products by Price
Classify products into categories based on price
27. Aggregate Item Weight per Person
Compute total weight of each item per person
28. Maximize Items Within Budget
Determine the maximum number of items that can be purchased within a fixed budget using PySpark.
29. Running Balance per User
Compute the running balance for each user over time.
30. Top N Sales per Store
Find the Top N highest sales transactions for each store.
31. User Sessionization with Time
Define user sessions and calculate event counts, start and end times
32. Employee Attendance Streaks
Calculate each employee's longest consecutive attendance streak.
33. Items Purchased by User
Perform various operations on the array column.
34. Sales Segmentation
Categorize sales records into buckets, and compute average sales per category.
35. Add Metadata to Customer Data
Load multiple CSV files and include metadata columns.
36. Tallying Election Results
Determine the seats won by each party in the election.
37. Call Center
Count unique callers per date and sum total call durations.
38. Social Media PII
Anonymize social media data and extract email domains.
39. Total Rental Income
Calculate total rental income for each landlord
40. E-Commerce Stats
Given product and order data from an e-commerce platform, calculate required KPIs
41. Private Equity Data Merge
Combine firm, fund, and investment data using joins and remove rows with all null values.
42. Compute Annual GDP Growth Rate
As an economist, calculate the year-over-year GDP growth rate for each country.
43. Word Count Program (Part 2)
Clean the text file and count the occurrences of each word in a text file.
44. Most Frequent Error Codes
Given application log events, determine the most frequent error codes.
45. Palindrome Words
Check for palindromic words in the data.
46. Exam Attendance Report
Produce a report for examination attendance.
47. Top Users by Event Types
Compute the top 3 users with the highest number of distinct event types on that day.
48. Student Marks to Grade
Assign a letter grade based on the marks range.
49. Positive Balance Customer Details
For customers with strictly positive balance, format balance to exactly 2 decimal places and add details.
50. Top Customers by Spend
Find top 2 customers by total spend in each region.
51. Active Subscription Watch Time
Calculate each user’s watch minutes within active subscription dates.
52. Ledger Reconciliation Variance Report
Reconcile source vs target totals and compute variance.
53. ETL Job Statistics
Calculate job execution statistics including latest run date, failure counts, and status.
54. Unsold Products
Find products that were never sold.
55. Connected Airports
Find all distinct airport pairs that have flights between them.
56. Dynamic Schema Transformation
Rename legacy columns, cast them to the correct data types, and derive new USD columns.
57. Flight Delay Analysis
Parse flight departure timestamps, categorize delays, and summarize counts per airline.
58. Price History Tracker
Calculate how many days each product price was active before the next change.
59. Forward Fill Sensor Readings
Fill null sensor readings with the last known non-null value per device.
60. Struct Column Flattening
Flatten nested struct fields to the top level dynamically.
61. Order Date Breakdown
Add year, month, and day columns extracted from an order date.
62. Customer Purchase Gaps
Compute the average number of days between consecutive purchases per customer.
63. Event Type Pivot
Count user events per type and pivot them into separate columns.
64. Engaged Shoppers Analysis
Find users with 2+ sessions who also ordered that day; aggregate non-returned orders and top platform.
65. Department Budget Utilization
Audit salary costs against department budgets and identify the top earner in each department.
66. Parse Product Attributes JSON
Parse a JSON string column to extract product attributes and filter by rating.
67. Clean Support Ticket Tags
Remove a deprecated tag and deduplicate tag arrays on support tickets.
68. Quarterly Sales Growth Unpivot
Unpivot wide quarterly sales into long format and compute growth rate.
69. Rolling Sales Average
Compute a trailing 3-day moving average per store and flag days above trend.
70. Daily New vs Returning Users
Split each day's active users into new and returning, with a running total.
71. Expand Subscription Months
Expand subscription date ranges into monthly rows and compute monthly MRR.
72. Merge Overlapping Bookings
Merge overlapping room bookings into continuous busy blocks per room.
73. Sales Subtotals Report
Build region and category subtotals plus a grand total as labelled rows.
74. Product Catalog Upsert
Merge a daily supplier feed into a product catalog with updates and inserts.
75. Signup Funnel Conversion
Track users through an ordered signup funnel and compute conversion rates.
76. Salary Quartile Bands
Assign salary quartiles and percentile positions within each department.
77. Ordered Purchase Path
Build each user's chronological page path, collapsing consecutive repeats.
78. Median Delivery Time
Compute the median delivery time and on-time rate for each courier.
79. Feature Flag Map
Build a per-user map of current feature flags and read values from it.
80. Duplicate Ride Fares
Flag retried fare charges within 10 minutes and compute refunds per rider.
81. EV Charger Utilization
Pair start/stop events into charging sessions and total the hours per charger.
82. Full-Track Learners
Find students who completed a course in every active learning track.
83. Gym Member Retention
Compute month-over-month member retention, handling gaps between months.
84. Game Launch Month Sales
Find each game's launch month sales across platforms and its share of total.
85. Grocery Basket Pairs
Find item pairs bought together in at least two orders.
86. Smart Meter Readings
Sum odd- and even-positioned meter readings per meter per day.
87. Most Common Cart Size
Find the most common cart size from pre-aggregated regional counts.
88. Podcasts With No Listens
Find podcasts with no qualifying listens in October 2024.
89. Duplicate Rental Listings
Find agents who listed the same property more than once.
90. Streaming Device Split
Count viewing sessions by device bucket and compute the mobile minutes share.
91. Last Visit Checkouts
Count the books each member borrowed on their most recent library visit.
92. Workout Span
Find days between each member's first and last workout in 2024.
93. Top Support Agents
Find agents with the top two resolved-ticket counts in August, ties included.
94. Top Delivery Cities
Find the top 3 cities by delivered orders, excluding unknown cities.
95. Monthly Hotel Ratings
Compute monthly average guest ratings per hotel, ignoring invalid stars.