Web Scraping with BeautifulSoup and Scrapy

Web scraping is the automated extraction of data from websites. Python offers two dominant libraries for this task: BeautifulSoup for lightweight, single-page scraping, and Scrapy for large-scale, multi-page crawling. This article covers both approaches, discusses ethical considerations, and provides practical examples for extracting data from HTML pages.

BeautifulSoup: Simple HTML Parsing

BeautifulSoup parses HTML and XML documents into a parse tree that you can navigate and search. It is best suited for projects that scrape a single page or a small number of pages. You combine it with the requests library to fetch pages. BeautifulSoup handles malformed HTML gracefully, making it ideal for real-world web pages that often have broken markup. Common operations include finding elements by tag name, CSS class, ID, or attribute, navigating the DOM tree via parent/child/sibling relationships, and extracting text content or attribute values.

import requests
from bs4 import BeautifulSoup

url = "https://example.com/articles"
resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
soup = BeautifulSoup(resp.content, "html.parser")

# Find all article links
articles = soup.find_all("article")
for article in articles:
    title_tag = article.find("h2").find("a")
    title = title_tag.text.strip()
    link = title_tag["href"]
    summary = article.find("p", class_="summary")
    summary_text = summary.text.strip() if summary else ""
    print(f"{title}: {link} — {summary_text[:50]}")

Scrapy: Scalable Web Crawling

Scrapy is a full-featured web scraping framework that handles request scheduling, concurrent downloads, data pipeline processing, and export in multiple formats. It uses an asynchronous engine (Twisted) that can crawl hundreds of pages per second. A Scrapy project consists of spiders (classes that define how to crawl a site), items (data containers), and pipelines (data processing and storage). Scrapy handles retries, error handling, and robots.txt compliance automatically.

import scrapy

class NewsSpider(scrapy.Spider):
    name = "news"
    start_urls = ["https://news.ycombinator.com"]

    def parse(self, response):
        for row in response.css("tr.athing"):
            yield {
                "title": row.css("span.titleline a::text").get(),
                "url": row.css("span.titleline a::attr(href)").get(),
                "score": response.css("span.score::text").get(),
            }
        # Follow pagination
        next_page = response.css("a.morelink::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Handling Dynamic Content

Many modern websites load content dynamically via JavaScript. BeautifulSoup and Scrapy cannot execute JavaScript, so they only see the initial HTML. For dynamic content, you need a browser automation tool like Selenium or Playwright. Playwright is the modern choice—it runs Chromium, Firefox, or WebKit headlessly and provides APIs for clicking, waiting, and extracting content after JavaScript execution. A common pattern is to use Playwright to render the page and extract the HTML, then feed it to BeautifulSoup for parsing.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")
    page.wait_for_selector(".dynamic-content")  # Wait for JS to render
    html = page.content()
    soup = BeautifulSoup(html, "html.parser")
    browser.close()

Ethical and Legal Considerations

Always check robots.txt (e.g., https://example.com/robots.txt) before scraping—it specifies which paths are off-limits. Respect rate limits by adding delays between requests (time.sleep(1) or Scrapy’s DOWNLOAD_DELAY setting). Identify your scraper with a descriptive User-Agent string so site owners can contact you if needed. Check the website’s terms of service—some explicitly prohibit scraping. Copyright law may apply to scraped content, especially if you republish it. For public data used for research or personal analysis, scraping is generally accepted, but always act responsibly and minimize load on the target server.

Data Storage and Pipelines

Scrapy’s pipeline architecture processes scraped items through a series of stages: validation (checking required fields), cleaning (normalizing text, converting dates), deduplication (avoiding duplicate items), and storage (writing to CSV, JSON, databases, or cloud storage). For large crawls, use incremental storage with database upsert logic so that restarting the crawl does not create duplicates. Item loaders provide a clean API for populating items with data from multiple CSS or XPath selectors. For monitoring, Scrapy’s Telnet console and web service (Scrapyd) let you inspect running spiders, cancel crawls, and schedule new ones without restarting the process. For production deployments, consider Scrapy Cloud (Zyte), or run spiders on Kubernetes with a RabbitMQ or Redis job queue.

# Scrapy pipeline for PostgreSQL storage
class PostgresPipeline:
    def open_spider(self, spider):
        self.conn = psycopg2.connect("dbname=scrape user=postgres")
        self.cur = self.conn.cursor()
    def process_item(self, item, spider):
        self.cur.execute(
            "INSERT INTO articles (title, url, content) VALUES (%s, %s, %s) "
            "ON CONFLICT (url) DO NOTHING",
            (item["title"], item["url"], item["content"])
        )
        self.conn.commit()
        return item
    def close_spider(self, spider):
        self.cur.close()
        self.conn.close()

Cloud-Based Scraping Infrastructure

For large-scale scraping, deploy spiders on cloud infrastructure. AWS Spot instances provide discounted compute for fault-tolerant jobs. Proxy rotation services provide residential IPs to avoid blocking. For JavaScript-heavy sites, serverless browsers (Browserless, Playwright on Lambda) spin up headless Chromium on demand. A scraping pipeline architecture: message queue distributes URLs to workers, workers parse and store items, and a scheduler manages crawl frequency with exponential backoff. Respect robots.txt and terms of service—violations can lead to IP bans or legal action.