Web Scraping with BeautifulSoup

Web Scraping with BeautifulSoup

After this lesson you will: read the structure of an HTML page, use BeautifulSoup to extract data from it, and know when you should (or should not) scrape.

1. What scraping is and when to use it

Web scraping means automatically pulling data out of web pages that are written for human readers. Prefer an API whenever the service offers one, because an API is more stable and follows the rules more closely. Scrape only when there is no API and collecting the data is allowed.

2. HTML in one minute

A web page is made of nested tags. Each tag can have attributes such as class, id, href:

<ul id="items">
  <li class="item"><a href="/p/1">Pen</a> <span class="price">3.5</span></li>
</ul>

3. Parsing HTML with BeautifulSoup

The example below uses a ready-made piece of HTML, so you can run it immediately without a network connection:

from bs4 import BeautifulSoup

html = """<html><body><h1>Shop</h1><ul id="items"><li class="item"><a href="/p/1">Pen</a> <span class="price">3.5</span></li><li class="item"><a href="/p/2">Book</a> <span class="price">12</span></li><li class="item sale"><a href="/p/3">Bag</a> <span class="price">20</span></li></ul></body></html>"""

soup = BeautifulSoup(html, "html.parser")
print(soup.h1.text)                       # Shop
print(len(soup.find_all("li")))           # 3   (every li tag)
print(soup.find("a")["href"])             # /p/1  (first a tag, read its href attribute)
print(soup.find_all("a")[1].get("href"))  # /p/2
print(soup.find("table"))                 # None  (find returns None when nothing matches)

• find(...) returns the first matching element (or None); find_all(...) returns a list of all matching elements.
• .text gets the text inside; ["href"] or .get("href") gets an attribute value. .get returns None instead of raising an error when the attribute is missing.

4. CSS selectors: select and select_one

The syntax is the same as CSS: tag_name.class, #id, and a space to select a child tag.

print(len(soup.select("li.item")))          # 3
print(len(soup.select("li.sale")))          # 1   (only the item that also has class sale)
print(soup.select_one("li.sale a").text)    # Bag
prices = [float(p.text) for p in soup.select("span.price")]
print(prices)        # [3.5, 12.0, 20.0]
print(sum(prices))   # 35.5

Note: <li class="item sale"> has two classes, so it matches both li.item and li.sale. Extracted text is always a string, so you need float()/int() to do calculations.

5. With a real page

import requests
resp = requests.get("https://books.toscrape.com/", timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")
titles = [a["title"] for a in soup.select("article.product_pod h3 a")]

The site books.toscrape.com was built specifically for practicing scraping. Use it, and do not practice on other people's sites.

6. Scraping responsibly

• Check robots.txt (for example https://domain-name/robots.txt) and the site's terms of use before collecting data.
• Do not fire too many requests in quick succession: wait between calls (time.sleep) so you do not overload the server.
• A page that loads its data with JavaScript after opening will not have that data in the HTML that requests receives. In that case, look for the hidden API the page calls, or use a browser automation tool.
• Do not collect personal data or data that you are forbidden to collect.

Key takeaways

• find gives the first element, find_all gives a list; select/select_one use CSS selectors.
• Text and attributes are always strings; convert the type before calculating.
• Prefer an API; check robots.txt and the terms of use; wait between calls.

Reference

Mitchell, R. (2018). Web Scraping with Python: Collecting Data from the Modern Web (2nd ed.). O'Reilly Media.