Web Scraping with BeautifulSoup and Scrapy

Web scraping is the automated extraction of data from websites. Python offers two dominant libraries for this task: BeautifulSoup for lightweight, single-page scraping, and Scrapy for large-scale, multi-page crawling. This article covers both approaches, discusses ethical considerations, and provides practical examples for extracting data from HTML pages.

BeautifulSoup: Simple HTML Parsing

BeautifulSoup parses HTML and XML documents into a parse tree that you can navigate and search. It is best suited for projects that scrape a single page or a small number of pages. You combine it with the requests library to fetch pages. BeautifulSoup handles malformed HTML gracefully, making it ideal for real-world web pages that often have broken markup. Common operations include finding elements by tag name, CSS class, ID, or attribute, navigating the DOM tree via parent/child/sibling relationships, and extracting text content or attribute values.

import requests
from bs4 import BeautifulSoup

url = "https://example.com/articles"
resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
soup = BeautifulSoup(resp.content, "html.parser")

# Find all article links
articles = soup.find_all("article")
for article in articles:
    title_tag = article.find("h2").find("a")
    title = title_tag.text.strip()
    link = title_tag["href"]
    summary = article.find("p", class_="summary")
    summary_text = summary.text.strip() if summary else ""
    print(f"{title}: {link} — {summary_text[:50]}")

Scrapy: Scalable Web Crawling

Scrapy is a full-featured web scraping framework that handles request scheduling, concurrent downloads, data pipeline processing, and export in multiple formats. It uses an asynchronous engine (Twisted) that can crawl hundreds of pages per second. A Scrapy project consists of spiders (classes that define how to crawl a site), items (data containers), and pipelines (data processing and storage). Scrapy handles retries, error handling, and robots.txt compliance automatically.

import scrapy

class NewsSpider(scrapy.Spider):
    name = "news"
    start_urls = ["https://news.ycombinator.com"]

    def parse(self, response):
        for row in response.css("tr.athing"):
            yield {
                "title": row.css("span.titleline a::text").get(),
                "url": row.css("span.titleline a::attr(href)").get(),
                "score": response.css("span.score::text").get(),
            }
        # Follow pagination
        next_page = response.css("a.morelink::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Handling Dynamic Content

Many modern websites load content dynamically via JavaScript. BeautifulSoup and Scrapy cannot execute JavaScript, so they only see the initial HTML. For dynamic content, you need a browser automation tool like Selenium or Playwright. Playwright is the modern choice—it runs Chromium, Firefox, or WebKit headlessly and provides APIs for clicking, waiting, and extracting content after JavaScript execution. A common pattern is to use Playwright to render the page and extract the HTML, then feed it to BeautifulSoup for parsing.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")
    page.wait_for_selector(".dynamic-content")  # Wait for JS to render
    html = page.content()
    soup = BeautifulSoup(html, "html.parser")
    browser.close()

Ethical and Legal Considerations

Always check robots.txt (e.g., https://example.com/robots.txt) before scraping—it specifies which paths are off-limits. Respect rate limits by adding delays between requests (time.sleep(1) or Scrapy’s DOWNLOAD_DELAY setting). Identify your scraper with a descriptive User-Agent string so site owners can contact you if needed. Check the website’s terms of service—some explicitly prohibit scraping. Copyright law may apply to scraped content, especially if you republish it. For public data used for research or personal analysis, scraping is generally accepted, but always act responsibly and minimize load on the target server.

Data Storage and Pipelines

Scrapy’s pipeline architecture processes scraped items through a series of stages: validation (checking required fields), cleaning (normalizing text, converting dates), deduplication (avoiding duplicate items), and storage (writing to CSV, JSON, databases, or cloud storage). For large crawls, use incremental storage with database upsert logic so that restarting the crawl does not create duplicates. Item loaders provide a clean API for populating items with data from multiple CSS or XPath selectors. For monitoring, Scrapy’s Telnet console and web service (Scrapyd) let you inspect running spiders, cancel crawls, and schedule new ones without restarting the process. For production deployments, consider Scrapy Cloud (Zyte), or run spiders on Kubernetes with a RabbitMQ or Redis job queue.

# Scrapy pipeline for PostgreSQL storage
class PostgresPipeline:
    def open_spider(self, spider):
        self.conn = psycopg2.connect("dbname=scrape user=postgres")
        self.cur = self.conn.cursor()
    def process_item(self, item, spider):
        self.cur.execute(
            "INSERT INTO articles (title, url, content) VALUES (%s, %s, %s) "
            "ON CONFLICT (url) DO NOTHING",
            (item["title"], item["url"], item["content"])
        )
        self.conn.commit()
        return item
    def close_spider(self, spider):
        self.cur.close()
        self.conn.close()

Cloud-Based Scraping Infrastructure

For large-scale scraping, deploy spiders on cloud infrastructure. AWS Spot instances provide discounted compute for fault-tolerant jobs. Proxy rotation services provide residential IPs to avoid blocking. For JavaScript-heavy sites, serverless browsers (Browserless, Playwright on Lambda) spin up headless Chromium on demand. A scraping pipeline architecture: message queue distributes URLs to workers, workers parse and store items, and a scheduler manages crawl frequency with exponential backoff. Respect robots.txt and terms of service—violations can lead to IP bans or legal action.

Async/Await in Python: A Practical Guide

Asynchronous programming allows a program to handle multiple operations concurrently without creating multiple threads or processes. Python’s async/await syntax, introduced in Python 3.5, provides a clean way to write concurrent code using coroutines. This is especially useful for I/O-bound tasks like web requests, database queries, and file operations, where the program would otherwise spend most of its time waiting.

Understanding the Event Loop

The event loop is the core of Python’s async system. It runs a single thread, continually checking for tasks that are ready to execute. When a coroutine encounters an await expression, it yields control back to the event loop, which can then run another coroutine while waiting for the I/O operation to complete. The asyncio module provides the event loop, and in modern Python (3.10+), asyncio.run() handles loop creation and cleanup automatically.

import asyncio

async def fetch_data(url):
    print(f"Fetching {url}...")
    await asyncio.sleep(1)  # Simulate network delay
    return f"Data from {url}"

async def main():
    # Run multiple tasks concurrently
    tasks = [
        fetch_data("https://api.example.com/users"),
        fetch_data("https://api.example.com/posts"),
        fetch_data("https://api.example.com/comments"),
    ]
    results = await asyncio.gather(*tasks)
    for r in results:
        print(r)

asyncio.run(main())

Async vs Synchronous Performance

The real benefit of async becomes apparent with many I/O operations. A synchronous version of the above would take 3 seconds (one after another), while the async version completes in about 1 second because all three requests run concurrently. This scales linearly – fetching 100 URLs synchronously takes 100 seconds; asynchronously, it still takes about 1 second (limited by bandwidth and server capacity). The sweet spot for async is high-latency, I/O-bound workloads with hundreds or thousands of concurrent operations.

Common Pitfalls

Blocking the event loop is the most common mistake. Calling time.sleep(), requests.get(), or any synchronous blocking function inside an async function blocks the entire event loop, defeating the purpose of async. Always use asyncio.sleep() instead of time.sleep(), and use async HTTP libraries like aiohttp or httpx instead of requests. Another pitfall is forgetting to await a coroutine – this returns a coroutine object instead of executing it, which can lead to silent bugs because the coroutine is never scheduled.

import aiohttp

async def fetch_json(session, url):
    async with session.get(url) as resp:
        return await resp.json()

async def fetch_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_json(session, url) for url in urls]
        return await asyncio.gather(*tasks)

urls = [f"https://api.example.com/page/{i}" for i in range(50)]
results = asyncio.run(fetch_all(urls))
print(f"Fetched {len(results)} pages")

Python 3.11+ includes high-level task groups (asyncio.TaskGroup) for structured concurrency, making error handling more predictable. When any task in a group fails, all sibling tasks are cancelled automatically, preventing orphaned background tasks.

Real-World Async Patterns

In production applications, you will often combine asyncio with other concurrency patterns. A common pattern is the producer-consumer setup where one coroutine fetches data from an API and another processes it. Using asyncio.Queue, you can coordinate work between coroutines with backpressure—if the consumer is slower than the producer, the queue fills up and the producer waits. Another pattern is using asyncio.timeout() (Python 3.11+) to set a maximum wait time for operations, preventing a single slow request from holding up the entire pipeline. For CPU-bound tasks within an async application, use loop.run_in_executor() with ThreadPoolExecutor to offload work to a thread pool without blocking the event loop.

async def worker(name, queue):
    while True:
        item = await queue.get()
        print(f"Worker {name}: processing {item}")
        await asyncio.sleep(0.2)
        queue.task_done()

async def main():
    queue = asyncio.Queue()
    workers = [asyncio.create_task(worker(f"W{i}", queue)) for i in range(3)]
    for i in range(20):
        await queue.put(f"task-{i}")
    await queue.join()
    for w in workers:
        w.cancel()

asyncio.run(main())

Structured Concurrency with TaskGroups

Python 3.11 introduced asyncio.TaskGroup for structured concurrency. TaskGroup ensures that all child tasks complete before the group exits, and if any task raises an exception, all sibling tasks are cancelled. This prevents orphaned tasks continuing after an error. The ExceptionGroup collects multiple exceptions raised concurrently. Structured concurrency makes async code more predictable—the lifetime of tasks is bounded by the scope of the TaskGroup. For new async code targeting Python 3.11+, prefer TaskGroup over asyncio.gather() for better error handling and resource management.

Async Context Managers and Async Iterators

Python’s async context managers (async with) and async iterators (async for) extend the async paradigm to resource management. Async context managers, defined with __aenter__ and __aexit__, handle async resource setup and teardown—essential for database connections, HTTP sessions, and file handles. The aiofiles library provides async file operations, and aiohttp.ClientSession is an async context manager that properly closes connections. Async iterators (__aiter__ and __anext__) enable paginated API consumption where each page is fetched asynchronously: async for page in api.paginate(): processes results without blocking. The async generator syntax (async def gen(): yield item) creates async iterators with cleaner code. Python 3.10+ supports asynchronous iteration in list comprehensions: [x async for x in async_gen()]. Standard library modules like contextlib provide @asynccontextmanager decorator for simple async context managers.

Pandas for Data Analysis: Beyond the Basics

Pandas is the most widely used data analysis library in Python. While many tutorials cover loading a CSV and viewing basic statistics, real-world data analysis requires more advanced operations: grouping, aggregating, merging datasets, reshaping with pivot tables, and working with time series. This article walks through each of these techniques with practical examples that you can adapt to your own datasets.

Grouping and Aggregating Data

The groupby operation splits your data into groups based on one or more columns, applies a function to each group independently, and combines the results. This is the SQL GROUP BY equivalent in Pandas. For example, to calculate total revenue and unique order count per region and product combination, you pass a list of grouping columns and a dictionary mapping output column names to aggregation functions.

import pandas as pd

# Load sales data
df = pd.read_csv("sales.csv")
print(df.head())
print(df.info())

# Group by region and product, aggregate multiple metrics
summary = df.groupby(["region", "product"]).agg(
    total_revenue=("revenue", "sum"),
    order_count=("order_id", "nunique"),
    avg_quantity=("quantity", "mean"),
    first_sale=("date", "min")
).reset_index()

print(summary.head(10))

The agg method accepts a dictionary where keys are new column names and values are tuples of (source_column, function). You can use any Pandas or NumPy function: sum, mean, nunique, min, max, std, or even custom lambda functions. The reset_index() call converts the grouped index back into regular columns, which is usually more convenient for further analysis. Without it, the grouping columns become part of a MultiIndex, which can be harder to work with.

Merging Datasets

Data often lives in multiple tables that need to be joined. Pandas provides merge() for SQL-style joins and concat() for stacking tables vertically or horizontally. The merge() function accepts how parameter values like inner, left, right, and outer, matching SQL JOIN semantics. Always specify the key columns explicitly with on, left_on, and right_on to avoid ambiguity.

# Load related tables
orders = pd.read_csv("orders.csv")
customers = pd.read_csv("customers.csv")
payments = pd.read_csv("payments.csv")

# Left join: all orders, with customer info where available
merged = pd.merge(orders, customers, on="customer_id", how="left")

# Inner join: only orders that have matching payments
paid_orders = pd.merge(orders, payments, on="order_id", how="inner")

# Merge on different column names
merged2 = pd.merge(orders, customers,
                   left_on="cust_id", right_on="id",
                   how="left")

# Concatenate monthly reports vertically
jan = pd.read_csv("sales_jan.csv")
feb = pd.read_csv("sales_feb.csv")
mar = pd.read_csv("sales_mar.csv")
q1 = pd.concat([jan, feb, mar], ignore_index=True)

When merging, watch out for many-to-many relationships — they produce Cartesian products that can explode your DataFrame size. Always inspect the shape before and after: print(len(orders), len(merged)). If the merged result is much larger than expected, you may have duplicate keys in one of the tables. Use validate='one_to_one' or validate='many_to_one' to raise an error if the relationship is not what you expect.

Pivot Tables

A pivot table reshapes data from a long format (one row per observation) to a wide format (one row per group, with columns for each category). This is the Pandas equivalent of Excel pivot tables and is invaluable for creating summary matrices, heatmaps, and cross-tabulations.

# Create a pivot table: regions as rows, quarters as columns
pivot = df.pivot_table(
    values="revenue",
    index="region",
    columns="quarter",
    aggfunc="sum",
    margins=True,
    fill_value=0
)

print(pivot)

# Multiple aggregation functions
pivot2 = df.pivot_table(
    values="revenue",
    index="region",
    columns="quarter",
    aggfunc=["sum", "mean", "count"],
    margins=True
)

# Cross-tabulation (frequency counts)
crosstab = pd.crosstab(df["region"], df["product_category"],
                       margins=True, normalize="index")
print(crosstab)

The margins=True parameter adds a “All” row and column with totals, similar to Excel’s Grand Total. fill_value=0 replaces missing combinations with zero instead of NaN. pd.crosstab is a specialized pivot table for frequency counts and is useful for understanding the distribution of categorical variables. Setting normalize='index' converts counts to percentages within each row, making it easy to compare category distributions across regions.

Time Series Analysis

Pandas has excellent support for time series data. Converting a date column to a DatetimeIndex enables powerful resampling, rolling windows, and time-based filtering. Always parse dates at load time with parse_dates=['date'] to avoid working with string columns.

# Parse dates and set as index
df = pd.read_csv("sales.csv", parse_dates=["date"])
df.index = pd.to_datetime(df["date"])

# Resample: aggregate by week
weekly = df.resample("W").agg({
    "revenue": "sum",
    "order_id": "nunique"
})

# Rolling average (4-week window)
weekly["revenue_ma4"] = weekly["revenue"].rolling(window=4).mean()

# Resample by month with multiple metrics
monthly = df.resample("ME").agg({
    "revenue": ["sum", "mean", "std"],
    "order_id": "nunique"
})

# Time-based filtering
q1_2026 = df["2026-01":"2026-03"]
last_30_days = df[df.index >= pd.Timestamp.now() - pd.DateOffset(days=30)]

# Shift for period-over-period comparison
weekly["revenue_prev"] = weekly["revenue"].shift(1)
weekly["change_pct"] = (weekly["revenue"] / weekly["revenue_prev"] - 1) * 100

print(weekly.head(10))

The resample method is like a time-based groupby. The "W" string stands for weekly (ISO weeks, ending Sunday). Other common aliases include "D" (daily), "ME" (month-end), "MS" (month-start), "QE" (quarter-end), and "YE" (year-end). The rolling() method creates a moving window for smoothing or computing trailing statistics — the window size is the number of periods, not a time duration. For irregularly sampled time series, use rolling(window=4, min_periods=2) to handle gaps gracefully.

Performance Tips

For large datasets (millions of rows), avoid iterative row-by-row operations. Use vectorized operations, avoid apply with slow functions, and prefer built-in aggregation methods. The query() method is faster than boolean indexing for complex filters. For very large data that does not fit in memory, consider dask.dataframe or polars as alternatives to Pandas.

# Fast filtering with query()
fast_filter = df.query("region == 'East' and revenue > 1000")

# Vectorized column creation
df["discounted_price"] = df["price"] * (1 - df["discount"])

# Avoid: df.apply(lambda row: row["price"] * (1 - row["discount"]), axis=1)
# Always prefer vectorized operations over apply

With these techniques — grouped aggregations, merges, pivot tables, and time series resampling — you can handle the vast majority of real-world data analysis tasks efficiently and expressively.

Data Classes and Pydantic for Robust Applications

Python’s dataclasses (introduced in Python 3.7) and Pydantic (a third-party library) both reduce boilerplate code when defining data containers, but they serve different purposes. Dataclasses auto-generate __init__, __repr__, __eq__, and __hash__ methods based on class annotations. Pydantic goes further by adding runtime type validation, serialization, and parsing — making it the backbone of FastAPI and the go-to choice for any application that handles external data.

Python Dataclasses

Dataclasses are ideal for internal data structures where you want concise, readable code without writing boilerplate. The @dataclass decorator inspects the class’s type annotations and automatically generates the standard dunder methods. You can control behavior with parameters like frozen=True (immutable instances), order=True (sortable), and slots=True (Python 3.10+, memory efficient).

from dataclasses import dataclass, field

@dataclass(frozen=True, order=True)
class Point:
    x: float
    y: float
    label: str = field(default="", compare=False)

p1 = Point(1.0, 2.0, "start")
p2 = Point(1.0, 2.0, "end")
print(p1)          # Point(x=1.0, y=2.0, label='start')
print(p1 == p2)    # True (label excluded from comparison)
# p1.x = 3.0       # Error: frozen instance

Pydantic BaseModel

Pydantic’s BaseModel adds validation and serialization on top of the type annotation system. When you create an instance, Pydantic validates every field against its type annotation — coercing values when safe (e.g., "123" becomes 123 for an int field) and raising a detailed validation error when coercion is not possible. This catches data quality issues at the boundary of your application rather than deep in your logic.

from pydantic import BaseModel, EmailStr, Field
from datetime import datetime

class User(BaseModel):
    id: int
    name: str = Field(..., min_length=1, max_length=100)
    email: EmailStr
    age: int = Field(ge=0, le=150, default=0)
    created_at: datetime = Field(default_factory=datetime.now)
    tags: list[str] = Field(default_factory=list)

# Valid data — works
user = User(id=1, name="Alice", email="alice@example.com", age=30, tags=["admin"])
print(user.model_dump_json(indent=2))

# Invalid data — raises ValidationError with clear message
try:
    User(id="not-a-number", name="", email="not-an-email", age=200)
except Exception as e:
    print(e)
    # Shows all validation errors at once, not just the first one

The Field() function lets you add constraints (min_length, ge for greater-or-equal, le for less-or-equal), default values, and metadata. Pydantic V2, powered by a Rust core (pydantic-core), is significantly faster than V1 and supports complex types like UUID, IPv4Address, Decimal, and Path out of the box.

Nested Models and Validation

Pydantic models can be nested to represent complex hierarchies. Each nested model is validated recursively. You can also define custom validators using @field_validator and @model_validator decorators to enforce business rules that go beyond simple type checks.

from pydantic import BaseModel, field_validator

class Address(BaseModel):
    street: str
    city: str
    zip_code: str

class Employee(BaseModel):
    name: str
    address: Address
    salary: float

    @field_validator('salary')
    @classmethod
    def salary_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError('Salary must be positive')
        return v

emp = Employee(
    name="Bob",
    address={"street": "123 Main St", "city": "NYC", "zip_code": "10001"},
    salary=75000.0
)
print(emp.model_dump())
# {'name': 'Bob', 'address': {'street': '123 Main St', ...}, 'salary': 75000.0}

Serialization and Parsing

Pydantic makes it easy to convert between Python objects and JSON/dict representations. This is essential for API development: parse incoming request bodies with model_validate() and serialize responses with model_dump() or model_dump_json(). The model_config lets you control serialization behavior like excluding unset values, using aliases, or populating by name vs by position.

# Parse from JSON string
json_data = '{"name": "Charlie", "email": "charlie@example.com", "age": 25}'
user = User.model_validate_json(json_data)

# Parse from dict
user = User.model_validate(dict(name="Diana", email="diana@example.com", age=28))

# Serialize to dict (exclude unset fields)
data = user.model_dump(exclude_unset=True)

# Serialize to JSON with formatting
print(user.model_dump_json(indent=2))
Settings Management with Pydantic
Pydantic's BaseSettings (from pydantic-settings) extends BaseModel to read configuration from environment variables, .env files, and secret stores. This is the recommended way to manage application configuration in Python. Define a settings class with typed fields and default values, and Pydantic automatically reads from the environment — validating types, providing clear error messages for missing required values, and supporting nested settings for complex configurations.
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    redis_url: str = 'redis://localhost:6379'
    debug: bool = False
    api_key: str
    max_connections: int = 10

    class Config:
        env_file = '.env'
        env_file_encoding = 'utf-8'

settings = Settings()
# Reads DATABASE_URL, REDIS_URL, API_KEY from environment or .env file

This pattern eliminates the need for ad-hoc environment variable parsing, type coercion bugs, and inconsistent configuration handling across your application. When combined with FastAPI, Pydantic provides end-to-end type safety from the HTTP request boundary through your application logic to the response serialization.
Use dataclasses for simple internal data holding in your own codebase where validation is not critical. Use Pydantic whenever data enters or leaves your application — API requests/responses, configuration files, database records — because the validation layer prevents corrupted data from propagating through your system.
Advanced Pydantic Features
Pydantic V2, rewritten in Rust with pydantic-core, is significantly faster than V1. Advanced features include: discriminated unions for parsing polymorphic JSON, strict mode (no implicit string-to-int coercion), computed fields, model validators with before/after/wrap modes, and serialization aliases. Pydantic's integration with FastAPI is seamless: request bodies and response models automatically validate and serialize. BaseModel.model_dump() and model_validate() replace V1 methods. Pydantic supports JSON Schema generation for OpenAPI documentation. For enterprise applications, StrictBool, PaymentCardNumber, and EmailStr provide domain-specific validation out of the box.
	

Building REST APIs with FastAPI

FastAPI is a modern Python web framework designed for building high-performance REST APIs. It combines automatic OpenAPI documentation, type-safe request handling via Pydantic, and asynchronous support — all while matching the performance of Node.js and Go (thanks to Starlette and Pydantic’s Rust core). This article walks through building a complete CRUD API with FastAPI, covering path operations, dependency injection, validation, and more.

Getting Started

FastAPI is built on Starlette (the ASGI framework) and Pydantic (the validation library). Install it with pip install fastapi uvicorn. A minimal application defines a FastAPI instance and a few path operations using Python type annotations. The type annotations serve double duty: they enable editor autocompletion and type checking, and FastAPI uses them to generate OpenAPI documentation and validate request data automatically. Start the server with uvicorn main:app --reload and visit /docs for interactive Swagger UI or /redoc for ReDoc documentation.

from fastapi import FastAPI

app = FastAPI(title="My API", version="1.0.0")

@app.get("/")
def read_root():
    return {"message": "Hello World"}

@app.get("/items/{item_id}")
def read_item(item_id: int, q: str = None):
    return {"item_id": item_id, "q": q}

The path parameter item_id is declared as int, so FastAPI automatically validates that it is a valid integer and returns a 422 error if it is not. The query parameter q is optional (defaults to None) and is a string. The OpenAPI schema reflects this — documenting the path parameter type, the query parameter, and the response format — without writing any additional configuration.

Request Validation with Pydantic

For POST, PUT, and PATCH requests, define a Pydantic model for the request body. FastAPI automatically validates the incoming JSON against the model, returning detailed field-level error messages if validation fails. Pydantic models also serve as the response model, ensuring that outgoing data matches the schema. Use response_model in the decorator to control what fields are included in the response and to enable automatic response filtering and documentation.

from pydantic import BaseModel, Field
from typing import Optional
from datetime import datetime

class ItemCreate(BaseModel):
    name: str = Field(..., min_length=1, max_length=100)
    price: float = Field(..., gt=0)
    description: Optional[str] = None
    tax: Optional[float] = None

class ItemResponse(BaseModel):
    id: int
    name: str
    price: float
    description: Optional[str] = None
    created_at: datetime

items_db = {}
counter = 0

@app.post("/items", response_model=ItemResponse, status_code=201)
def create_item(item: ItemCreate):
    global counter
    counter += 1
    db_item = {
        "id": counter,
        "name": item.name,
        "price": item.price,
        "description": item.description,
        "created_at": datetime.now()
    }
    items_db[counter] = db_item
    return db_item

@app.get("/items/{item_id}", response_model=ItemResponse)
def get_item(item_id: int):
    if item_id not in items_db:
        from fastapi import HTTPException
        raise HTTPException(status_code=404, detail="Item not found")
    return items_db[item_id]

The Field() function adds validation constraints directly in the model: min_length=1, gt=0 (greater than 0). If validation fails, FastAPI returns a 422 response with a JSON body listing every field that failed and why — the client does not need to guess what went wrong. The response_model ensures that only the fields defined in ItemResponse are returned, even if the database object has extra fields.

Dependency Injection

FastAPI’s dependency injection system lets you extract common logic (database sessions, authentication, configuration) into reusable dependencies. A dependency is any callable that returns a value, declared with Depends(). FastAPI manages the dependency lifecycle, resolving dependencies in order and caching them per request. This makes your path operations thin — they just call the relevant service and return a response, while dependencies handle cross-cutting concerns like authentication, database connections, and rate limiting.

from fastapi import FastAPI, Depends, HTTPException, status

# Authentication dependency
async def get_current_user(token: str):
    if token != "secret-token":
        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED)
    return {"username": "alice", "role": "admin"}

# Database dependency
def get_db():
    db = Database.connect()
    try:
        yield db
    finally:
        db.close()

# Path operation using dependencies
@app.get("/users/me")
def read_current_user(
    current_user: dict = Depends(get_current_user),
    db: Database = Depends(get_db)
):
    return {
        "user": current_user,
        "items": db.query("SELECT * FROM items")
    }

Error Handling and Configuration

# Custom exception handler
from fastapi import Request
from fastapi.responses import JSONResponse

class AppException(Exception):
    def __init__(self, code: str, message: str, status_code: int = 400):
        self.code = code
        self.message = message
        self.status_code = status_code

@app.exception_handler(AppException)
async def app_exception_handler(request: Request, exc: AppException):
    return JSONResponse(
        status_code=exc.status_code,
        content={"error": {"code": exc.code, "message": exc.message}}
    )

# Application settings with Pydantic
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    debug: bool = False
    api_key: str

    class Config:
        env_file = ".env"

settings = Settings()

@app.get("/config")
def get_config():
    return {"debug": settings.debug}

FastAPI’s combination of type safety, automatic documentation, and dependency injection makes it the most productive Python framework for building APIs. The automatic OpenAPI documentation ensures your API is always documented correctly (because it is generated from the actual code), and the Pydantic integration catches data issues at the earliest possible moment — when the request arrives at your server.

Introduction

In modern software engineering, squeezing every ounce of performance from hardware is often critical. Two fundamental techniques — multi-tasking and multi-threading — are frequently confused, yet they operate at entirely different levels of the system. This article demystifies both, explains where the Python Global Interpreter Lock (GIL) fits in, and provides concrete C++ and Python examples that illustrate real-world behaviour.

Multi-tasking: The OS-Level Illusion

Multi-tasking is an operating system capability that allows multiple processes to run seemingly simultaneously. The OS scheduler rapidly switches between processes, giving each a small time slice. This creates the illusion of parallelism even on a single-core CPU.

Each process has its own isolated memory space, file descriptors, and security context. Communication between processes (IPC) requires explicit mechanisms like pipes, shared memory, or sockets. This isolation makes multi-tasking robust — one crashing process does not bring down the others — but also adds overhead for context switching and data sharing.

Multi-threading: Parallelism Within a Process

Multi-threading is an application-level technique where a single process spawns multiple threads that share the same memory space, open files, and other resources. Threads are lightweight compared to processes; creating and switching between them is far cheaper because the OS does not need to swap out the full memory context.

The critical trade-off: because threads share memory, developers must coordinate access with synchronisation primitives (mutexes, semaphores, atomic operations) to avoid race conditions and data corruption.

The Python GIL: The Elephant in the Room

Python’s Global Interpreter Lock (GIL) is a mutex that protects access to CPython interpreter internals, ensuring that only one thread executes Python bytecode at any given moment. This means Python threads cannot achieve true parallel execution for CPU-bound tasks — they merely time-share the same core, often with more overhead than a single-threaded approach.

For I/O-bound tasks (network requests, file reads, database queries), threading is still effective because the GIL is released during blocking I/O calls.

C++ Example: True Parallel Execution

#include <iostream>
#include <thread>
#include <vector>
#include <chrono>

uint64_t fibonacci(int n) {
    if (n <= 1) return n;
    return fibonacci(n - 1) + fibonacci(n - 2);
}

void worker(int n, uint64_t& result) {
    result = fibonacci(n);
}

int main() {
    const int N = 42;
    std::vector<uint64_t> results(4);
    std::vector<std::thread> threads;

    auto start = std::chrono::high_resolution_clock::now();

    for (int i = 0; i < 4; ++i)
        threads.emplace_back(worker, N + i, std::ref(results[i]));

    for (auto& t : threads)
        t.join();

    auto end = std::chrono::high_resolution_clock::now();
    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();

    for (auto r : results)
        std::cout << r << " ";
    std::cout << "
Time: " << ms << " ms
";
}

On a quad-core machine, this runs approximately 4x faster than a serial version.

Python Threading: Blocked by the GIL

import threading
import time

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

def worker(n, results, idx):
    results[idx] = fibonacci(n)

if __name__ == "__main__":
    N = 38
    results = [0] * 4
    threads = []

    start = time.perf_counter()

    for i in range(4):
        t = threading.Thread(target=worker, args=(N + i, results, i))
        threads.append(t)
        t.start()

    for t in threads:
        t.join()

    elapsed = time.perf_counter() - start
    print(results, f"{elapsed:.2f}s")

On a quad-core machine, this runs at the same speed as the serial version — the GIL serialises all threads onto a single core.

Python Multiprocessing: Bypassing the GIL

import multiprocessing as mp
import time

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

if __name__ == "__main__":
    N = 38
    args = [N, N + 1, N + 2, N + 3]

    start = time.perf_counter()

    with mp.Pool(4) as pool:
        results = pool.map(fibonacci, args)

    elapsed = time.perf_counter() - start
    print(results, f"{elapsed:.2f}s")

This runs significantly faster — near-linear speedup up to the number of physical cores.

When to Use Which Approach

Scenario	Recommendation	Reason
CPU-bound in Python	multiprocessing	GIL blocks threads
I/O-bound in Python	threading or asyncio	GIL released during I/O
CPU-bound in C++	std::thread or OpenMP	True parallel execution
I/O-bound in C++	std::thread	No GIL contention
Strong isolation needed	Multi-processing	Processes are isolated
Latency-sensitive, shared state	Multi-threading	Shared memory is fast

Conclusion

Multi-tasking and multi-threading are complementary tools. Python’s GIL adds a critical constraint — threads are useful for I/O but harmful for CPU-bound computation, where multiprocessing is the correct escape hatch. C++ offers true multi-threading from the ground up, but with synchronisation responsibility. Understanding these trade-offs is what separates working code from performant, production-grade systems.

When to Use Multi-Threading vs Multi-Tasking

Choose multi-threading when tasks are I/O-bound (waiting for disk, network, database) and share memory. In Python, the Global Interpreter Lock (GIL) prevents true parallel execution of threads for CPU-bound tasks, but threading still improves I/O-bound throughput because threads yield the GIL during I/O waits. Choose multi-processing for CPU-bound tasks (computation-heavy work like image processing, numerical simulations) where each process runs on a separate CPU core without GIL contention. Asyncio provides a third option: cooperative concurrency within a single thread where tasks voluntarily yield control at await points, ideal for high-concurrency I/O-bound workloads without the overhead of thread context switching.

Using Folium for Map Creation in Python

Folium is a Python library that creates interactive Leaflet maps directly from Python data structures. It bridges the gap between data analysis in Pandas and geographic visualization, allowing you to create professional-quality maps with minimal code. Folium supports tile layers from OpenStreetMap, Mapbox, CartoDB, and other providers, along with markers, choropleths, heatmaps, and popups for data exploration.

Basic Map Creation

Creating a map with Folium starts with the folium.Map constructor, which takes a location (latitude, longitude), zoom level, and tile style as parameters. The default tile set is OpenStreetMap, but you can switch to Stamen Terrain, CartoDB Positron, or other styles to match your aesthetic needs. Maps are HTML widgets that can be displayed in Jupyter notebooks, saved as standalone HTML files, or embedded in web pages.

import folium

# Create a base map centered on New York City
m = folium.Map(location=[40.7128, -74.0060], zoom_start=12,
               tiles="CartoDB positron")
m.save("nyc_map.html")

# Create a map with different tile styles
m_terrain = folium.Map(location=[40.7128, -74.0060],
                       tiles="Stamen Terrain", zoom_start=11)
m_terrain.save("nyc_terrain.html")

Markers and Popups

Markers pinpoint locations on the map. Folium’s Marker class takes a location (lat, lng) and optional popup text or tooltip. For large datasets, using CircleMarker instead of the default icon marker improves performance—they render as SVG circles that scale well with hundreds of points. You can customize marker colors, icons (using Font Awesome or Bootstrap icons), and popup content to include formatted text, images, or even charts rendered as HTML.

import folium, pandas as pd

m = folium.Map(location=[40.7128, -74.0060], zoom_start=11)

# Sample data: coffee shops
shops = [
    {"name": "Blue Bottle", "lat": 40.7266, "lng": -73.9968, "rating": 4.5},
    {"name": "Stumptown", "lat": 40.7295, "lng": -73.9965, "rating": 4.3},
    {"name": "Intelligentsia", "lat": 40.7282, "lng": -73.9943, "rating": 4.4},
]
for shop in shops:
    color = "green" if shop["rating"] >= 4.4 else "orange"
    folium.CircleMarker(
        location=[shop["lat"], shop["lng"]],
        radius=12, color=color, fill=True, fill_opacity=0.7,
        popup=f"{shop['name']}
Rating: {shop['rating']}/5",
        tooltip=shop["name"]
    ).add_to(m)
m.save("coffee_shops.html")

Choropleth Maps for Geographic Data

Choropleth maps color geographic regions (countries, states, districts) based on a data value. Folium’s choropleth layer requires two inputs: a GeoJSON file defining region boundaries, and a data column mapping each region ID to a value. This is powerful for visualizing election results, population density, infection rates, or economic indicators by region. The key is matching the GeoJSON feature IDs to your data keys—usually ISO country codes or FIPS state codes.

import folium, json, pandas as pd

m = folium.Map(location=[39.8, -98.5], zoom_start=4)

# Unemployment data by state (simulated)
data = pd.DataFrame({
    "state": ["AL", "AK", "AZ", ...],  # state FIPS or abbreviation
    "unemployment": [4.2, 5.1, 3.8, ...]
})

folium.Choropleth(
    geo_data="us-states.json",  # GeoJSON file
    name="choropleth",
    data=data,
    columns=["state", "unemployment"],
    key_on="feature.id",
    fill_color="YlOrRd",
    fill_opacity=0.7,
    line_opacity=0.2,
    legend_name="Unemployment Rate (%)"
).add_to(m)
m.save("unemployment.html")

Heatmaps and Clustering

For visualizing point density (e.g., crime locations, taxi pickups, earthquake epicenters), Folium offers HeatMap (from folium.plugins) which renders a smooth density surface where color intensity represents point concentration. The MarkerCluster plugin groups nearby markers into clusters that expand as you zoom in, making it practical to display thousands of points without overwhelming the browser. Both plugins integrate seamlessly with Folium’s API and work well in Jupyter notebooks and web dashboards. Folium maps can also be combined with other visualization libraries—for example, using Altair to generate a chart and embedding it in a map popup, giving you the full power of the Python data visualization ecosystem on an interactive geographic canvas.

GeoPandas Integration

GeoPandas extends Pandas with geospatial data types (GeoSeries, GeoDataFrame) and operations (buffer, intersection, distance, convex hull). Folium maps can directly visualize GeoDataFrames using the explore() method, which accepts a GeoDataFrame and automatically creates a choropleth or point map. This integration enables complex spatial analysis pipelines: load shapefiles or GeoJSON with GeoPandas, perform spatial operations (filter points within a polygon, compute nearest neighbors), and visualize results with Folium in a few lines of code. The combination of GeoPandas for analysis and Folium for visualization covers 90% of geospatial data science workflows without requiring GIS desktop software.

import geopandas as gpd

# Load world countries shapefile
world = gpd.read_file(gpd.datasets.get_path("naturalearth_lowres"))
# Filter to a continent
asia = world[world["continent"] == "Asia"]
# Create Folium map
m = asia.explore(column="pop_est", cmap="YlOrRd", legend=True)
m.save("asia_population.html")

Real-Time Data with Folium

Folium maps can display real-time data by updating markers dynamically. While Folium itself generates static HTML, combining it with JavaScript setInterval() calls to refresh GeoJSON data sources creates live-updating maps. For production dashboards, consider using Streamlit with st_folium which supports bidirectional communication between Python and the map. The folium.plugins package adds TimestampedGeoJson for animating data over time, Draw for user input, and Fullscreen for presentation mode. Folium’s FeatureGroup organizes related markers into toggleable layers. The integration with ipyleaflet provides higher performance for interactive exploration with WebGL support for millions of points.

Creating a CLI Utility for Bulk File Rename Operations Using Python

Renaming hundreds of files manually is tedious and error-prone. A Python command-line utility can automate bulk renaming with patterns, regex substitution, numbering sequences, and dry-run previews. This article walks through building a practical CLI tool using argparse and pathlib, covering common renaming scenarios like normalizing filenames, adding prefixes/suffixes, replacing text, and numbering files sequentially.

Core Design with argparse and pathlib

Python’s argparse module handles command-line argument parsing, and pathlib provides an object-oriented interface to filesystem paths. The tool should support several rename modes: replace (find and replace text in filenames), prefix/suffix (add leading or trailing text), number (add sequential numbering), and regex (pattern-based replacement using regular expressions). A dry-run flag (-n or –dry-run) is essential—it shows what would happen without actually renaming anything, letting users verify the operation before executing.

import argparse, re
from pathlib import Path

def bulk_rename(directory, find=None, replace=None,
                prefix="", suffix="", dry_run=False):
    path = Path(directory)
    for file in path.iterdir():
        if not file.is_file():
            continue
        old_name = file.name
        new_name = old_name
        if find and replace is not None:
            new_name = new_name.replace(find, replace)
        if prefix:
            new_name = prefix + new_name
        if suffix:
            stem = Path(new_name).stem
            ext = Path(new_name).suffix
            new_name = f"{stem}{suffix}{ext}"
        if new_name != old_name:
            print(f"  {old_name} → {new_name}")
            if not dry_run:
                file.rename(file.with_name(new_name))

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Bulk rename files")
    parser.add_argument("directory", help="Target directory")
    parser.add_argument("--find", help="Text to find")
    parser.add_argument("--replace", help="Replacement text")
    parser.add_argument("--prefix", default="", help="Add prefix")
    parser.add_argument("--suffix", default="", help="Add suffix")
    parser.add_argument("-n", "--dry-run", action="store_true", help="Preview only")
    args = parser.parse_args()
    bulk_rename(args.directory, args.find, args.replace,
                args.prefix, args.suffix, args.dry_run)

Sequential Numbering

Adding sequential numbers to files is useful for photo collections, document scanning, or creating ordered playlists. The –number flag adds a zero-padded sequence number (e.g., 001, 002) to each file. You can specify the starting number, padding width, and position (prefix vs suffix). Sorting can be by name, modification date, or creation date to control the numbering order. The script detects and prevents collisions by checking whether the target filename already exists before renaming.

def add_numbering(files, start=1, padding=3, as_prefix=True, by="name"):
    if by == "date":
        files.sort(key=lambda f: f.stat().st_mtime)
    else:
        files.sort()
    for i, file in enumerate(files, start=start):
        num = str(i).zfill(padding)
        old = file.name
        stem = file.stem
        ext = file.suffix
        new_name = f"{num}_{stem}{ext}" if as_prefix else f"{stem}_{num}{ext}"
        yield old, new_name

# Usage: python rename.py ./photos --number --start 1 --padding 4 --by date

Regex-Based Renaming

For complex transformations, regex is indispensable. The –regex flag enables pattern-based matching with capture groups that can be referenced in the replacement string (e.g.,
, ). This is useful for extracting and reformatting date patterns, normalizing spacing, or restructuring naming conventions. For example, renaming “IMG_20260709_123456.jpg” to “2026-07-09_12-34-56.jpg” uses a single regex substitution with capture groups for year, month, day, hour, minute, and second.

def regex_rename(directory, pattern, replacement, dry_run=False):
    path = Path(directory)
    for file in path.iterdir():
        if not file.is_file():
            continue
        new_name = re.sub(pattern, replacement, file.name)
        if new_name != file.name:
            print(f"  {file.name} → {new_name}")
            if not dry_run:
                file.rename(file.with_name(new_name))

# Example: python rename.py ./photos --regex "(IMG_)(\d{4})(\d{2})(\d{2})" --replace "--_"

Safety Features

Beyond dry-run mode, the utility should include collision detection (preventing overwrites), undo functionality (saving rename operations to a log file that can reverse them), and confirmation prompts before executing on more than a threshold number of files. Using pathlib’s rename() method is atomic on most filesystems, meaning a partially completed batch leaves some files renamed and others not—logging each operation to a JSON file allows reversing with a simple –undo flag that reads the log and reverses the mapping.

Cross-Platform Considerations

Python’s pathlib.Path handles path separators correctly on Windows (backslash), macOS, and Linux. However, renaming files across filesystems (e.g., renaming on an external drive) may not be atomic. The script should handle permission errors gracefully by catching PermissionError and continuing with the remaining files. On Unix systems, renaming a file to a name that differs only in case may behave unexpectedly on case-insensitive filesystems (macOS default, Windows). Adding a warning when –find and –replace would change only case prevents silent failures. For very large directories (100K+ files), using os.scandir() instead of pathlib.iterdir() improves initial listing speed, and batching rename operations in transactions of 1000 files prevents partial failures from leaving the directory in an inconsistent state.

GUI Frontend with Tkinter or PyQt

For users uncomfortable with the command line, a simple GUI frontend provides the same functionality with file dialogs and preview lists. Python’s tkinter (built-in) creates native-looking dialogs for selecting directories, defining rename rules, and previewing changes before applying them. A GUI version shows the original filenames next to the new names with color coding (green = rename, red = conflict, gray = unchanged). Drag-and-drop support lets users drop files or folders onto the window. The PyQt6 version includes a progress bar for large directories, a parallel rename option, and an undo button that reverses the last rename operation.

Python Packaging and Distribution with Poetry

Packaging a Python project properly ensures that other developers can install, use, and contribute to your code without dependency conflicts or missing files. Poetry is a modern dependency management and packaging tool that simplifies the entire workflow — from project creation to publishing on PyPI. Unlike pip and setuptools, Poetry uses a declarative pyproject.toml file, resolves dependencies with a SAT solver to avoid version conflicts, and generates deterministic installs via a lock file. This article walks through creating, building, and publishing a Python package with Poetry.

Creating a New Project

Starting a new project with Poetry is a single command. It creates the directory structure, initializes a Git repository, and generates a pyproject.toml file with sensible defaults. The generated structure includes a source directory named after your project, a README.md, and a tests directory.

# Create a new Poetry project
poetry new my-project
cd my-project

# Project structure created:
# my-project/
#   pyproject.toml
#   README.md
#   my_project/
#       __init__.py
#   tests/
#       __init__.py
#       test_my_project.py

If you are adding Poetry to an existing project instead of starting fresh, run poetry init and answer the prompts. Poetry will generate a pyproject.toml based on your existing requirements.txt or setup.py if you point it at the right files.

Managing Dependencies

Poetry uses a pyproject.toml file (defined in PEP 518 and PEP 621) to declare project metadata and dependencies. Dependencies are organized into groups: the main [tool.poetry.dependencies] section for runtime dependencies, and [tool.poetry.group.dev.dependencies] for development-only packages like test runners, linters, and type checkers. When you run poetry add, Poetry automatically resolves all dependency versions to ensure compatibility and records the exact versions in a poetry.lock file. This lock file should be committed to version control so that everyone working on the project gets identical dependency trees.

[tool.poetry]
name = "my-project"
version = "0.1.0"
description = "A sample Python project"
authors = ["Your Name <you@example.com>"]
readme = "README.md"
license = "MIT"

[tool.poetry.dependencies]
python = "^3.10"
requests = "^2.28"
click = "^8.1"

[tool.poetry.group.dev.dependencies]
pytest = "^7.0"
black = "^22.0"
mypy = "^1.0"
ruff = "^0.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The ^ operator in version constraints means “compatible with.” For example, ^2.28 allows any version from 2.28 up to but not including 3.0.0. This gives you bug fixes and minor features without risking breaking changes from a major version bump. The python = "^3.10" constraint means your package supports Python 3.10, 3.11, 3.12, etc., but not Python 4.0.

Adding and Removing Dependencies

The Poetry CLI provides intuitive commands for managing dependencies. Each command updates both pyproject.toml and poetry.lock automatically, ensuring your environment stays synchronized with the declared dependencies.

# Add runtime dependencies
poetry add fastapi uvicorn

# Add development-only dependencies
poetry add --group dev mypy pytest-cov

# Remove a dependency
poetry remove requests

# Update all dependencies to latest allowed versions
poetry update

# Show dependency tree
poetry show --tree

# Export to requirements.txt format
poetry export -f requirements.txt --output requirements.txt

The poetry show --tree command is invaluable for debugging dependency conflicts — it displays a tree of every package and its sub-dependencies, making it easy to spot situations where two packages require incompatible versions of the same library.

Building and Publishing

Once your project is ready, building distributable archives is a single command. Poetry produces both a source distribution (.tar.gz) and a wheel (.whl) in the dist/ directory. Wheels are the preferred distribution format because they install faster — they are pre-built and do not require running setup.py. Publishing to PyPI is equally simple. You will need a PyPI API token for authentication instead of a username and password.

# Build source distribution and wheel
poetry build

# Publish to PyPI
poetry publish --username __token__ --password pypi-xxxxxxxxxxxxxxxxxxxx

# Publish to Test PyPI first (recommended)
poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry publish -r testpypi --username __token__ --password pypi-xxxx

Version Management

Poetry includes a built-in version command that follows semantic versioning conventions. It updates both the pyproject.toml version field and creates a Git tag.

# Check current version
poetry version

# Bump version (patch, minor, major, prepatch, preminor, premajor)
poetry version patch   # 0.1.0 -> 0.1.1
poetry version minor   # 0.1.0 -> 0.2.0
poetry version major   # 0.1.0 -> 1.0.0

# Pre-release versions
poetry version prepatch  # 0.1.0 -> 0.1.1a0

By integrating Poetry into your workflow, you get reproducible builds, clean dependency resolution, and a straightforward publishing pipeline — all essential for maintaining a professional Python package.

Publishing to PyPI and CI/CD Integration

Once your package is configured with Poetry, publishing to PyPI is a single command: poetry publish. For automated publishing, configure PyPI tokens as CI/CD secrets. A GitHub Actions workflow can run tests, build with poetry build, publish to TestPyPI on PR merges, and publish to PyPI on version tags. Poetry’s version command (poetry version patch/minor/major) bumps versions according to semantic versioning. The pyproject.toml build-system requires poetry-core ensures pip can install directly from the repository. Poetry’s dependency resolver avoids version conflicts that plague setuptools/pip projects.