# Web Scraping

> A program that reads web pages and copies data out of them.

**Field:** Software Development  
**Category:** Integration & Automation  
**Also known as:** Scraping, Web Scraper, Screen Scraping, Data Scraping

## Think of it like

Think of it like copying prices from shop windows. A robot walks down the street and writes them down.

## Definition

Web scraping is a program that reads web pages and pulls data out of them. It loads a page, finds the parts it wants and saves them, often as a table. People scrape when a site has no API to ask. Scrapers break easily, because a small change to the page can move the data. Many sites set rules about scraping in their terms and their robots.txt file.

## Example

Jack scrapes the prices of 50 bikes from a shop's pages each night. His sheet now tracks every price change.

## Example code

```python
import requests
from bs4 import BeautifulSoup

page = requests.get("https://example.com/bikes")
soup = BeautifulSoup(page.text, "html.parser")
prices = [p.text for p in soup.select(".price")]
```

Fetch the page, then pick out every element marked price.

## Why it matters

A lot of useful data lives only on web pages. Scraping is often the only way to collect it.

## Common mistake

Scraping fast and often from one site. It can slow the site down and get you blocked.

## Three principles

- **Read the page.** Load it like a browser would.
- **Pick the parts.** Find the data and save it.
- **Be polite.** Go slow and follow the site's rules.

## Related terms

- [Browser Automation](https://learn.benibix.com/term/development/browser-automation)
- [Headless Browser](https://learn.benibix.com/term/development/headless-browser)
- [Application Programming Interface (API)](https://learn.benibix.com/term/development/api)
- [HyperText Markup Language (HTML)](https://learn.benibix.com/term/development/html)
- [Python](https://learn.benibix.com/term/development/python)

## Told apart from

- [Web Scraping vs Browser Automation](https://learn.benibix.com/compare/web-scraping-vs-browser-automation): People think web scraping and browser automation are one job, and they are not. Scraping reads a page and saves the data it finds, like prices. Browser automation acts on a page. It clicks, types and sends forms. The same tools can do both, but only automation changes anything on the site.

![A script on the left drives a browser in the middle that has no window, labelled headless. The browser loads a web page on the right. Two outputs come back: web scraping pulls prices out of the page into a table, and browser automation clicks buttons and fills in a form. Below, a separate row shows a software robot doing the same clicks across desktop apps, labelled RPA.](https://learn.benibix.com/diagrams/dev3-web-automation.svg)

---

Source: https://learn.benibix.com/term/development/web-scraping
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
