PySpark Coding Interview Questions - Practice Online
Solve the most common PySpark coding interview questions asked in Data Engineering, Data Analyst and Data Science roles.
Use PySpark APIs or Spark SQL with temporary view to solve these coding questions.
You can also check the most popular conceptual questions here.
| # | Title & Description | Difficulty | Tags | Status | |
|---|---|---|---|---|---|
1 | Load and Transform DataPractice loading a CSV file and apply basic transformations on columns. | Easy | PopularSelectDropFiltering | ||
2 | Handling Null ValuesClean the dataset by filtering out or replacing null values in various columns. | Easy | SelectCleaningFiltering | ||
3 | Total Purchases by CustomerGroup data by customer and compute the total purchase amount per user. | Easy | SelectGroupingFilteringCasting | ||
4 | Running PayrollHelp Human Resources calculate pay for each employee. | Medium | JoinsArithmeticConditional | ||
5 | Food and Beverage SalesCalculate total sales quantity, revenue, and stock for each product. | Medium | Aggregate FunctionsJoinsData Cleaning | ||
6 | Discounts on ProductsPremiumAdd a new column calculating discounted prices for products using arithmetic operations. | Easy | SelectArithmeticCasting | ||
7 | Load & Transform JSON filePremiumRead a nested JSON file and flatten it using explode and array-handling techniques. | Medium | PopularSelectJsonExplodeArrays | ||
8 | Employee EarningsPremiumUse window functions to find employees whose salary is higher than the department average. | Hard | AccentureSelectWindowsGroupingJoiningFunctions | ||
9 | Remove Duplicates From DatasetPremiumIdentify and remove duplicate records based on custom logic using window functions. | Medium | PopularFilteringWindow FunctionsDatesGrouping | ||
10 | Word Count Program in PySparkImplement a word count logic using PySpark RDD transformations on a text file. | Medium | PopularRDDTextfileGrouping | ||
11 | Group By and Aggregate ListPremiumGroup records and aggregate values into lists using advanced group and array functions. | Hard | Tiger AnalyticsGroupingAggregationFunctionsArrays | ||
12 | Monthly Transaction SummaryPremiumSummarize transactions month-wise by grouping and using date functions to extract months. | Medium | GroupingAggregationDate FunctionsTransactions | ||
13 | Top Players SummaryPremiumGenerate a summary of top players using joins, aggregations, and string operations. | Hard | ITC InfotechGroupingAggregationString FunctionsJoins | ||
14 | Daily Total SalesPremiumCalculate total sales for each store on a daily basis using grouping and aggregation. | Easy | WalmartAggregationGroupingDate | ||
15 | Top 5 Products by SalesPremiumFind the top 5 products with the highest total sales across all stores for a given day. | Medium | WalmartAggregationGroupingSortingLimit | ||
16 | Products with Increasing SalesPremiumIdentify products whose total sales revenue has increased every year. | Hard | DeloitteWindowJoinFilteringPivot | ||
17 | Remove Outliers from Trip DataPremiumGiven a dataset of trip costs and customer ratings, remove rows that contain outliers. | Medium | VISAQuantilesFilteringIQRData Cleaning | ||
18 | Driver Details for RidesPremiumGiven datasets of rides and drivers, join them to produce ride-level data. | Easy | VISAJoinsSelectAlias | ||
19 | Customer Loyalty ScorePremiumCalculate customer loyalty scores based on number of trips and ratings. | Hard | VISAJoinsAggregationFilteringGroupingArithmetic | ||
20 | Track Employee HistoryPremiumImplement SCD Type 2 logic to track historical changes in employee records. | Hard | DeloitteSCD2JoinsFilteringUnion | ||
21 | Employee AttendancePremiumTransform employee attendance records to show count of each attendance status. | Medium | TCS, InfosysPivotGroupingAggregation | ||
22 | Daily Stock Price ChangePremiumCalculate day-over-day change in closing stock prices. | Medium | NielsenWindow FunctionsLagTime Series | ||
23 | Analyze Review SentimentPremiumUse a PySpark UDF to classify reviews based on keywords. | Hard | TargetUDFText ProcessingString Manipulation | ||
24 | Sessionize Clickstream DataPremiumAssign session IDs and visit numbers to user clickstream data. | Hard | NielsenWindow FunctionsTime DifferenceSessionization | ||
25 | Combine Transactions DataPremiumUnify records from multiple sources with different columns | Easy | UnionSchema EvolutionData Integration | ||
26 | Categorize Products by PricePremiumClassify products into categories based on price | Easy | Conditional ColumnData TransformationCategorization | ||
27 | Aggregate Item Weight per PersonPremiumCompute total weight of each item per person | Easy | AmazonAggregationGroupbyData Summarization | ||
28 | Maximize Items Within BudgetPremiumDetermine the maximum number of items that can be purchased within a fixed budget using PySpark. | Hard | EPAMKnapsackWindow FunctionsAggregationBudget Optimization | ||
29 | Running Balance per UserPremiumCompute the running balance for each user over time. | Medium | Window FunctionsCumulative SumEvent StreamTime Series | ||
30 | Top N Sales per StorePremiumFind the Top N highest sales transactions for each store. | Medium | Window FunctionsRow NumberRankingTop N | ||
31 | User Sessionization with TimePremiumDefine user sessions and calculate event counts, start and end times | Hard | NielsenWindow FunctionsLagCumulative SumSessionizationTime Series | ||
32 | Employee Attendance StreaksPremiumCalculate each employee's longest consecutive attendance streak. | Hard | PayPalWindow FunctionsLagGroupingDate Functions | ||
33 | Items Purchased by UserPremiumPerform various operations on the array column. | Medium | Array Functions | ||
34 | Sales SegmentationPremiumCategorize sales records into buckets, and compute average sales per category. | Medium | UberFilteringConditional LogicAggregations | ||
35 | Add Metadata to Customer DataPremiumLoad multiple CSV files and include metadata columns. | Medium | MetadataDataframeFile Handling | ||
36 | Tallying Election ResultsPremiumDetermine the seats won by each party in the election. | Hard | DocusignWindowsJoinsAggregationRanking | ||
37 | Call CenterPremiumCount unique callers per date and sum total call durations. | Easy | Aggregate FunctionsDistinct Function | ||
38 | Social Media PIIPremiumAnonymize social media data and extract email domains. | Medium | Regular Expressions | ||
39 | Total Rental IncomePremiumCalculate total rental income for each landlord | Hard | String FunctionsComplex JoinsAggregate Functions | ||
40 | E-Commerce StatsPremiumGiven product and order data from an e-commerce platform, calculate required KPIs | Medium | Aggregate FunctionsComplex Joins |
🏆 Leaderboard
1. Load and Transform Data
Practice loading a CSV file and apply basic transformations on columns.
2. Handling Null Values
Clean the dataset by filtering out or replacing null values in various columns.
3. Total Purchases by Customer
Group data by customer and compute the total purchase amount per user.
4. Running Payroll
Help Human Resources calculate pay for each employee.
5. Food and Beverage Sales
Calculate total sales quantity, revenue, and stock for each product.
6. Discounts on Products
Add a new column calculating discounted prices for products using arithmetic operations.
7. Load & Transform JSON file
Read a nested JSON file and flatten it using explode and array-handling techniques.
8. Employee Earnings
Use window functions to find employees whose salary is higher than the department average.
9. Remove Duplicates From Dataset
Identify and remove duplicate records based on custom logic using window functions.
10. Word Count Program in PySpark
Implement a word count logic using PySpark RDD transformations on a text file.
11. Group By and Aggregate List
Group records and aggregate values into lists using advanced group and array functions.
12. Monthly Transaction Summary
Summarize transactions month-wise by grouping and using date functions to extract months.
13. Top Players Summary
Generate a summary of top players using joins, aggregations, and string operations.
14. Daily Total Sales
Calculate total sales for each store on a daily basis using grouping and aggregation.
15. Top 5 Products by Sales
Find the top 5 products with the highest total sales across all stores for a given day.
16. Products with Increasing Sales
Identify products whose total sales revenue has increased every year.
17. Remove Outliers from Trip Data
Given a dataset of trip costs and customer ratings, remove rows that contain outliers.
18. Driver Details for Rides
Given datasets of rides and drivers, join them to produce ride-level data.
19. Customer Loyalty Score
Calculate customer loyalty scores based on number of trips and ratings.
20. Track Employee History
Implement SCD Type 2 logic to track historical changes in employee records.
21. Employee Attendance
Transform employee attendance records to show count of each attendance status.
22. Daily Stock Price Change
Calculate day-over-day change in closing stock prices.
23. Analyze Review Sentiment
Use a PySpark UDF to classify reviews based on keywords.
24. Sessionize Clickstream Data
Assign session IDs and visit numbers to user clickstream data.
25. Combine Transactions Data
Unify records from multiple sources with different columns
26. Categorize Products by Price
Classify products into categories based on price
27. Aggregate Item Weight per Person
Compute total weight of each item per person
28. Maximize Items Within Budget
Determine the maximum number of items that can be purchased within a fixed budget using PySpark.
29. Running Balance per User
Compute the running balance for each user over time.
30. Top N Sales per Store
Find the Top N highest sales transactions for each store.
31. User Sessionization with Time
Define user sessions and calculate event counts, start and end times
32. Employee Attendance Streaks
Calculate each employee's longest consecutive attendance streak.
33. Items Purchased by User
Perform various operations on the array column.
34. Sales Segmentation
Categorize sales records into buckets, and compute average sales per category.
35. Add Metadata to Customer Data
Load multiple CSV files and include metadata columns.
36. Tallying Election Results
Determine the seats won by each party in the election.
37. Call Center
Count unique callers per date and sum total call durations.
38. Social Media PII
Anonymize social media data and extract email domains.
39. Total Rental Income
Calculate total rental income for each landlord
40. E-Commerce Stats
Given product and order data from an e-commerce platform, calculate required KPIs