PySpark Online Compiler: Write, Run, and Test Code Effectively

🎯 Ready for a challenge?

Try solving real PySpark interview questions asked by Walmart, Amazon, VISA and other top companies.

In this PySpark Playground, you have access to several sample datasets stored in the /samples/ folder. These datasets can be used to test your PySpark code and understand how to work with real-world data.

Example: `customers.csv`

The customers.csv file is a sample dataset that contains customer information. You can load this file into a DataFrame using PySpark and apply various transformations and actions on it. Below is a code snippet to help you get started:

# Initialize Spark session
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Spark Playground').getOrCreate()

# Load the customers.csv dataset
df = spark.read.format('csv').option('header', 'true').load('/samples/customers.csv')

# Show the first few rows of the DataFrame
df.show(5)

# Display the DataFrame using the display() function.
display(df)

Available Sample Datasets:

All datasets are present in the /samples/ folder.

File Name	Format	Notes
`customers.csv`	CSV	Customer IDs, names, purchase history
`sales.csv`	CSV	Transactions with date, ID, and amount
`products.csv`	CSV	Product ID, name, and price info
`products.json`	JSON	Pretty-printed; use `multiLine=true` when reading in Spark
`organizations.orc`	ORC	Columnar format, efficient for large-scale analytics
`titanic.parquet`	Parquet	Default Spark format; columnar and highly optimized.

Try it out!

Use the provided datasets to explore various PySpark functions such as select(), filter(), groupBy(), and more. This will help you get hands-on practice with data processing in PySpark.

Environment and Library versions:

Python : 3.8.20
PySpark : 3.5.5
Pandas : 2.0.3

Follow for Spark tips & updates

TABLE STDOUT STDERR

Use display(df) function to show the final DataFrame in this section.