⚙️Apache Spark
Also known as Spark, Spark SQL, Spark Cluster
A free tool that works on big data with lots of computers.
Think of it like
It is like a chef who splits a huge order among many cooks. Each cook keeps their food on the bench.
Example
Each night, a Spark job runs on 40 computers. It joins 3 billion clicks with orders and writes the result to the lake.
Why it matters
It is the most used engine for big data jobs. Databricks and most cloud data tools are built on it.
Definition
Common mistake
Key takeaways
- 1Driver and executors. One plans, many do the work.
- 2In memory. Faster than MapReduce.
- 3Many languages. Python, SQL, Scala, Java, R.