# PySpark

> The way to use Apache Spark from Python.

**Field:** Data Engineering  
**Category:** Processing  
**Also known as:** Spark Python API

## Think of it like

It is like a remote for a big machine. You press keys in Python, and Spark does the hard work.

## Definition

PySpark is the Python API for Apache Spark. You write DataFrame steps in Python, and Spark runs them across the cluster. Most built-in steps run inside Spark at full speed. Your own Python functions are slower, because data has to be passed between Spark and Python. It is the most common way people use Spark today.

## Example

A data scientist moves a pandas script to PySpark. It now runs on a year of data instead of one week.

## Example code

```python
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.getOrCreate()
clicks = spark.read.parquet("s3://lake/clicks/")
clicks.groupBy("country").agg(F.count("*")).show()
```

Count clicks by country. Spark splits the work over the cluster.

## Why it matters

Python is the language most data people know. PySpark lets them use a cluster without learning Scala.

## Common mistake

Writing your own Python code for jobs Spark can already do. The built-in tools are many times faster.

## Three principles

- **Python on top.** Spark does the work.
- **Built-ins are fast.** They run inside Spark.
- **Own functions cost.** Data moves to Python and back.

## Related terms

- [Apache Spark](https://learn.benibix.com/term/data/apache-spark)
- [DataFrame](https://learn.benibix.com/term/data/dataframe)
- [pandas](https://learn.benibix.com/term/data/pandas)
- [Python](https://learn.benibix.com/term/development/python)

---

Source: https://learn.benibix.com/term/data/pyspark
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
