PySpark Online Test: 30 Spark Quiz Questions with Answers (MCQ)

Answer: A. count()

count() is an action: it triggers a Spark job and returns a result to the driver. filter, select and withColumn are lazy transformations that only build the plan. Read more in transformations and actions.

Answer: D. Transformations are only executed when an action is called

Spark records transformations in a plan (a DAG) and only runs them when an action such as count, show or write is called. This lets the optimizer see the whole query before running it. See lazy evaluation in Spark.

Answer: C. collect()

collect() returns every row to the driver as a Python list, which can crash the driver with an OutOfMemoryError on large data. show, take and limit only return a small number of rows.

Answer: C. 200

The default is 200 partitions after every shuffle (joins, groupBy, distinct). With Adaptive Query Execution enabled, Spark can automatically merge small post-shuffle partitions.

Answer: A. left_anti

left_anti keeps left rows without a match (like NOT EXISTS in SQL). left_semi does the opposite and keeps left rows that do have a match. See joining data in PySpark.

Answer: D. fillna()

fillna() (also available as df.na.fill()) replaces nulls, for example df.fillna({"city": "Unknown"}). dropna() removes rows with nulls instead. More in handling nulls.

Answer: C. Parquet

Parquet stores data column by column, so Spark reads only the columns a query needs and compresses data much better. CSV, JSON and Avro are row-based. See common file formats.

Answer: A. Scans the data to detect column types, which costs an extra pass over the file

With inferSchema=True Spark reads the data to guess types such as int, double or timestamp. This is convenient but slower on big files, so production jobs usually pass an explicit schema. See inferSchema in PySpark.

Answer: D. SparkSession

SparkSession combines the older SQLContext and HiveContext and gives access to the SparkContext. You create it with SparkSession.builder.getOrCreate(). See setting up a Spark session.

Answer: C. persist() lets you choose the storage level, cache() uses the default level

Both keep a computed DataFrame for reuse. cache() uses the default storage level, while persist() accepts a level such as MEMORY_ONLY, MEMORY_AND_DISK or DISK_ONLY. Both are lazy and take effect on the next action.

Answer: D. Driver

The driver builds the logical and physical plans, splits work into jobs, stages and tasks, and sends tasks to executors. The cluster manager only allocates resources. See Spark architecture.

Answer: A. At every shuffle boundary (wide transformation)

Stages are split at shuffle boundaries. All narrow transformations between two shuffles run together in one stage. See jobs, stages and tasks.

Answer: C. One per partition

Each task processes exactly one partition, so a stage reading 10 partitions runs 10 tasks. Executor cores decide how many of those tasks run at the same time.

Answer: A. groupBy().agg()

groupBy needs all rows with the same key in the same partition, so data is shuffled across the network. filter, select and withColumn are narrow: each output partition depends on one input partition.

Answer: D. Lineage

Spark keeps the lineage (the chain of transformations) of every RDD and DataFrame. If a partition is lost, Spark recomputes only that partition from its lineage instead of relying on replicated copies. See RDDs in Spark.

Answer: C. Analysis, Logical optimization, Physical planning, Code generation

Catalyst first resolves names (analysis), applies rule-based rewrites such as predicate pushdown (logical optimization), picks a physical plan using a cost model (physical planning), and finally generates JVM bytecode (code generation). See Catalyst optimizer.

Answer: D. All of the above

AQE uses real statistics collected after each shuffle stage to merge small partitions, change join strategies and handle skewed joins. It is enabled by default since Spark 3.2. See Adaptive Query Execution.

Answer: B. 10 MB

Tables smaller than 10 MB (by Spark's size estimate) are broadcast automatically. You can force it with F.broadcast(df) or disable automatic broadcasting by setting the threshold to -1. See join strategies.

Answer: C. 0.6

spark.memory.fraction defaults to 0.6. Execution memory (shuffles, joins, sorts) and storage memory (cache, broadcast) share this region, and the remaining 40% is user memory. See executor memory.

Answer: A. Apache Mesos

Mesos support was deprecated in Spark 3.2 and removed in Spark 4.0. YARN, Kubernetes and Spark's built-in standalone manager are supported.

Answer: B. 2

Only rows 1 (IT, 100) and 3 (IT, 120) match both conditions. Row 5 is in IT but its salary is 70. Remember to wrap each condition in brackets when combining them with & or |.

Answer: C. 4

rank() gives ties the same rank and then leaves a gap: 100 gets 1, both 90s get 2, and 80 gets 4. dense_rank() would give 3, and row_number() would also give 4 but never repeats a value.

Answer: A. 4

coalesce() can only reduce the number of partitions because it merges existing ones without a full shuffle. Asking for more partitions than exist leaves the count unchanged. Use repartition(10) to increase it. See repartition and coalesce.

Answer: C. 4

A left join keeps every left row, but a left row is repeated once for every match. id 1 matches two rows on the right, so it appears twice; ids 2 and 3 appear once with nulls. That is 4 rows. Duplicate keys on the right are a common cause of row explosions after joins.

Answer: B. 1

Comparing null with anything returns null, not true, so the null row is silently dropped by the filter. Only the row with 2 is kept. To keep nulls, use (F.col("x") != 1) | F.col("x").isNull(), or a null-safe comparison.

Answer: B. 3

groupBy treats null as its own group, so the result has three groups: "a", "b" and null. This often surprises people who expect nulls to be ignored.

Answer: B. 5 and 3

union() behaves like SQL UNION ALL and keeps duplicates, so 2 + 3 = 5 rows. Adding distinct() removes the duplicates, leaving 1, 2 and 3. Also note that union matches columns by position; use unionByName to match by name.

Answer: D. coalesce(10)

coalesce() merges existing partitions on the same executors, so it avoids a full shuffle. It is the usual way to reduce the number of output files before a write. repartition() always shuffles.

Answer: C. Data skew

One task processing far more data than the others is the classic sign of data skew: one key has many more rows than the rest. Fixes include AQE skew join handling, broadcasting the smaller table, or salting the hot key. See data skew and salting.

Answer: A. A regular Python UDF

A regular Python UDF is a black box to the Catalyst optimizer and sends every row between the JVM and a Python worker. Built-in functions and SQL expressions run inside the JVM, and pandas UDFs process data in Arrow batches, so both are much faster.