Skip to content
BENIBIX Learn

PySpark

Data⚙️ ProcessingDistributed Processing

Also known as Spark Python API

The way to use Apache Spark from Python.

Think of it like

It is like a remote for a big machine. You press keys in Python, and Spark does the hard work.

Example

A data scientist moves a pandas script to PySpark. It now runs on a year of data instead of one week.

Why it matters

Python is the language most data people know. PySpark lets them use a cluster without learning Scala.

Definition

PySpark is the Python API for Apache Spark. You write DataFrame steps in Python, and Spark runs them across the cluster. Most built-in steps run inside Spark at full speed. Your own Python functions are slower, because data has to be passed between Spark and Python. It is the most common way people use Spark today.

Example code

python
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.getOrCreate()
clicks = spark.read.parquet("s3://lake/clicks/")
clicks.groupBy("country").agg(F.count("*")).show()
Count clicks by country. Spark splits the work over the cluster.

Common mistake

Writing your own Python code for jobs Spark can already do. The built-in tools are many times faster.

Key takeaways

  1. 1
    Python on top. Spark does the work.
  2. 2
    Built-ins are fast. They run inside Spark.
  3. 3
    Own functions cost. Data moves to Python and back.