Category: Programming

Web Scraping with BeautifulSoup and Scrapy

Web Scraping with BeautifulSoup and Scrapy

Web scraping is the automated extraction of data from websites. Python offers two dominant libraries for this task: BeautifulSoup for lightweight, single-page scraping, and Scrapy for large-scale, multi-page crawling. This article covers both approaches, discusses ethical considerations, and provides practical examples for extracting data from HTML pages.

BeautifulSoup: Simple HTML Parsing

BeautifulSoup parses HTML and XML documents into a parse tree that you can navigate and search. It is best suited for projects that scrape a single page or a small number of pages. You combine it with the requests library to fetch pages. BeautifulSoup handles malformed HTML gracefully, making it ideal for real-world web pages that often have broken markup. Common operations include finding elements by tag name, CSS class, ID, or attribute, navigating the DOM tree via parent/child/sibling relationships, and extracting text content or attribute values.

import requests
from bs4 import BeautifulSoup

url = "https://example.com/articles"
resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})
soup = BeautifulSoup(resp.content, "html.parser")

# Find all article links
articles = soup.find_all("article")
for article in articles:
    title_tag = article.find("h2").find("a")
    title = title_tag.text.strip()
    link = title_tag["href"]
    summary = article.find("p", class_="summary")
    summary_text = summary.text.strip() if summary else ""
    print(f"{title}: {link} — {summary_text[:50]}")

Scrapy: Scalable Web Crawling

Scrapy is a full-featured web scraping framework that handles request scheduling, concurrent downloads, data pipeline processing, and export in multiple formats. It uses an asynchronous engine (Twisted) that can crawl hundreds of pages per second. A Scrapy project consists of spiders (classes that define how to crawl a site), items (data containers), and pipelines (data processing and storage). Scrapy handles retries, error handling, and robots.txt compliance automatically.

import scrapy

class NewsSpider(scrapy.Spider):
    name = "news"
    start_urls = ["https://news.ycombinator.com"]

    def parse(self, response):
        for row in response.css("tr.athing"):
            yield {
                "title": row.css("span.titleline a::text").get(),
                "url": row.css("span.titleline a::attr(href)").get(),
                "score": response.css("span.score::text").get(),
            }
        # Follow pagination
        next_page = response.css("a.morelink::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Handling Dynamic Content

Many modern websites load content dynamically via JavaScript. BeautifulSoup and Scrapy cannot execute JavaScript, so they only see the initial HTML. For dynamic content, you need a browser automation tool like Selenium or Playwright. Playwright is the modern choice—it runs Chromium, Firefox, or WebKit headlessly and provides APIs for clicking, waiting, and extracting content after JavaScript execution. A common pattern is to use Playwright to render the page and extract the HTML, then feed it to BeautifulSoup for parsing.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")
    page.wait_for_selector(".dynamic-content")  # Wait for JS to render
    html = page.content()
    soup = BeautifulSoup(html, "html.parser")
    browser.close()

Ethical and Legal Considerations

Always check robots.txt (e.g., https://example.com/robots.txt) before scraping—it specifies which paths are off-limits. Respect rate limits by adding delays between requests (time.sleep(1) or Scrapy’s DOWNLOAD_DELAY setting). Identify your scraper with a descriptive User-Agent string so site owners can contact you if needed. Check the website’s terms of service—some explicitly prohibit scraping. Copyright law may apply to scraped content, especially if you republish it. For public data used for research or personal analysis, scraping is generally accepted, but always act responsibly and minimize load on the target server.

Data Storage and Pipelines

Scrapy’s pipeline architecture processes scraped items through a series of stages: validation (checking required fields), cleaning (normalizing text, converting dates), deduplication (avoiding duplicate items), and storage (writing to CSV, JSON, databases, or cloud storage). For large crawls, use incremental storage with database upsert logic so that restarting the crawl does not create duplicates. Item loaders provide a clean API for populating items with data from multiple CSS or XPath selectors. For monitoring, Scrapy’s Telnet console and web service (Scrapyd) let you inspect running spiders, cancel crawls, and schedule new ones without restarting the process. For production deployments, consider Scrapy Cloud (Zyte), or run spiders on Kubernetes with a RabbitMQ or Redis job queue.

# Scrapy pipeline for PostgreSQL storage
class PostgresPipeline:
    def open_spider(self, spider):
        self.conn = psycopg2.connect("dbname=scrape user=postgres")
        self.cur = self.conn.cursor()
    def process_item(self, item, spider):
        self.cur.execute(
            "INSERT INTO articles (title, url, content) VALUES (%s, %s, %s) "
            "ON CONFLICT (url) DO NOTHING",
            (item["title"], item["url"], item["content"])
        )
        self.conn.commit()
        return item
    def close_spider(self, spider):
        self.cur.close()
        self.conn.close()

Cloud-Based Scraping Infrastructure

For large-scale scraping, deploy spiders on cloud infrastructure. AWS Spot instances provide discounted compute for fault-tolerant jobs. Proxy rotation services provide residential IPs to avoid blocking. For JavaScript-heavy sites, serverless browsers (Browserless, Playwright on Lambda) spin up headless Chromium on demand. A scraping pipeline architecture: message queue distributes URLs to workers, workers parse and store items, and a scheduler manages crawl frequency with exponential backoff. Respect robots.txt and terms of service—violations can lead to IP bans or legal action.

Async/Await in Python: A Practical Guide

Async/Await in Python: A Practical Guide

Asynchronous programming allows a program to handle multiple operations concurrently without creating multiple threads or processes. Python’s async/await syntax, introduced in Python 3.5, provides a clean way to write concurrent code using coroutines. This is especially useful for I/O-bound tasks like web requests, database queries, and file operations, where the program would otherwise spend most of its time waiting.

Understanding the Event Loop

The event loop is the core of Python’s async system. It runs a single thread, continually checking for tasks that are ready to execute. When a coroutine encounters an await expression, it yields control back to the event loop, which can then run another coroutine while waiting for the I/O operation to complete. The asyncio module provides the event loop, and in modern Python (3.10+), asyncio.run() handles loop creation and cleanup automatically.

import asyncio

async def fetch_data(url):
    print(f"Fetching {url}...")
    await asyncio.sleep(1)  # Simulate network delay
    return f"Data from {url}"

async def main():
    # Run multiple tasks concurrently
    tasks = [
        fetch_data("https://api.example.com/users"),
        fetch_data("https://api.example.com/posts"),
        fetch_data("https://api.example.com/comments"),
    ]
    results = await asyncio.gather(*tasks)
    for r in results:
        print(r)

asyncio.run(main())

Async vs Synchronous Performance

The real benefit of async becomes apparent with many I/O operations. A synchronous version of the above would take 3 seconds (one after another), while the async version completes in about 1 second because all three requests run concurrently. This scales linearly – fetching 100 URLs synchronously takes 100 seconds; asynchronously, it still takes about 1 second (limited by bandwidth and server capacity). The sweet spot for async is high-latency, I/O-bound workloads with hundreds or thousands of concurrent operations.

Common Pitfalls

Blocking the event loop is the most common mistake. Calling time.sleep(), requests.get(), or any synchronous blocking function inside an async function blocks the entire event loop, defeating the purpose of async. Always use asyncio.sleep() instead of time.sleep(), and use async HTTP libraries like aiohttp or httpx instead of requests. Another pitfall is forgetting to await a coroutine – this returns a coroutine object instead of executing it, which can lead to silent bugs because the coroutine is never scheduled.

import aiohttp

async def fetch_json(session, url):
    async with session.get(url) as resp:
        return await resp.json()

async def fetch_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_json(session, url) for url in urls]
        return await asyncio.gather(*tasks)

urls = [f"https://api.example.com/page/{i}" for i in range(50)]
results = asyncio.run(fetch_all(urls))
print(f"Fetched {len(results)} pages")

Python 3.11+ includes high-level task groups (asyncio.TaskGroup) for structured concurrency, making error handling more predictable. When any task in a group fails, all sibling tasks are cancelled automatically, preventing orphaned background tasks.

Real-World Async Patterns

In production applications, you will often combine asyncio with other concurrency patterns. A common pattern is the producer-consumer setup where one coroutine fetches data from an API and another processes it. Using asyncio.Queue, you can coordinate work between coroutines with backpressure—if the consumer is slower than the producer, the queue fills up and the producer waits. Another pattern is using asyncio.timeout() (Python 3.11+) to set a maximum wait time for operations, preventing a single slow request from holding up the entire pipeline. For CPU-bound tasks within an async application, use loop.run_in_executor() with ThreadPoolExecutor to offload work to a thread pool without blocking the event loop.

async def worker(name, queue):
    while True:
        item = await queue.get()
        print(f"Worker {name}: processing {item}")
        await asyncio.sleep(0.2)
        queue.task_done()

async def main():
    queue = asyncio.Queue()
    workers = [asyncio.create_task(worker(f"W{i}", queue)) for i in range(3)]
    for i in range(20):
        await queue.put(f"task-{i}")
    await queue.join()
    for w in workers:
        w.cancel()

asyncio.run(main())

Structured Concurrency with TaskGroups

Python 3.11 introduced asyncio.TaskGroup for structured concurrency. TaskGroup ensures that all child tasks complete before the group exits, and if any task raises an exception, all sibling tasks are cancelled. This prevents orphaned tasks continuing after an error. The ExceptionGroup collects multiple exceptions raised concurrently. Structured concurrency makes async code more predictable—the lifetime of tasks is bounded by the scope of the TaskGroup. For new async code targeting Python 3.11+, prefer TaskGroup over asyncio.gather() for better error handling and resource management.

Async Context Managers and Async Iterators

Python’s async context managers (async with) and async iterators (async for) extend the async paradigm to resource management. Async context managers, defined with __aenter__ and __aexit__, handle async resource setup and teardown—essential for database connections, HTTP sessions, and file handles. The aiofiles library provides async file operations, and aiohttp.ClientSession is an async context manager that properly closes connections. Async iterators (__aiter__ and __anext__) enable paginated API consumption where each page is fetched asynchronously: async for page in api.paginate(): processes results without blocking. The async generator syntax (async def gen(): yield item) creates async iterators with cleaner code. Python 3.10+ supports asynchronous iteration in list comprehensions: [x async for x in async_gen()]. Standard library modules like contextlib provide @asynccontextmanager decorator for simple async context managers.

Design Patterns: Strategy and Observer

Design Patterns: Strategy and Observer

Design patterns are reusable solutions to common software design problems. The Gang of Four book (1994) cataloged 23 design patterns, and many remain essential tools in modern software development. This article focuses on two of the most widely used behavioral patterns: Strategy and Observer. Both patterns promote loose coupling and adhere to the Open/Closed Principle—they make code extensible without modification.

The Strategy Pattern

The Strategy pattern defines a family of interchangeable algorithms, encapsulates each one, and makes them interchangeable at runtime. Instead of writing a massive if-else chain to handle different behaviors, you define a strategy interface and implement concrete strategies for each variant. The context class delegates to a strategy object, allowing the algorithm to be swapped without changing the context code. This pattern is ideal for payment processing (credit card vs PayPal vs crypto), sorting algorithms, compression methods, and validation rules.

from abc import ABC, abstractmethod

class PaymentStrategy(ABC):
    @abstractmethod
    def pay(self, amount: float) -> bool: ...

class CreditCardStrategy(PaymentStrategy):
    def __init__(self, card_number: str, cvv: str):
        self.card_number = card_number
        self.cvv = cvv
    def pay(self, amount: float) -> bool:
        print(f"Charging {amount} to card {self.card_number[-4:]}")
        return True

class PayPalStrategy(PaymentStrategy):
    def __init__(self, email: str):
        self.email = email
    def pay(self, amount: float) -> bool:
        print(f"Charging {amount} via PayPal account {self.email}")
        return True

class ShoppingCart:
    def __init__(self, strategy: PaymentStrategy):
        self.items = []
        self.strategy = strategy
    def checkout(self) -> bool:
        total = sum(item.price for item in self.items)
        return self.strategy.pay(total)

# Usage
cart = ShoppingCart(PayPalStrategy("user@example.com"))
cart.checkout()  # Uses PayPal; can switch to CreditCardStrategy later

The Observer Pattern

The Observer pattern defines a one-to-many dependency between objects: when one object (the subject) changes state, all its dependents (observers) are notified automatically. This is the foundation of event-driven programming and pub-sub systems. In Python, the Observer pattern is used extensively in GUI frameworks (button click events), Django signals, and asyncio event loops. Unlike Strategy (which is about algorithm selection), Observer is about notification and propagation of state changes.

class Subject:
    def __init__(self):
        self._observers = []
    def attach(self, observer):
        self._observers.append(observer)
    def detach(self, observer):
        self._observers.remove(observer)
    def notify(self, **kwargs):
        for observer in self._observers:
            observer.update(**kwargs)

class Observer(ABC):
    @abstractmethod
    def update(self, **kwargs): ...

class EmailNotifier(Observer):
    def update(self, **kwargs):
        print(f"Email: Order {kwargs.get('order_id')} is now {kwargs.get('status')}")

class Logger(Observer):
    def update(self, **kwargs):
        print(f"Log: Order {kwargs.get('order_id')} → {kwargs.get('status')} at {kwargs.get('timestamp')}")

order_system = Subject()
order_system.attach(EmailNotifier())
order_system.attach(Logger())
order_system.notify(order_id=123, status="shipped", timestamp="2026-07-09T10:00:00Z")

When to Use Each Pattern

Use Strategy when you need to select an algorithm at runtime and want to avoid conditionals, or when you have multiple variants of the same behavior that should be independently testable. Use Observer when a change in one object requires updating others, but you don’t know how many objects need updating ahead of time. Both patterns are often combined: an event system (Observer) can dispatch events to different handlers that each use a Strategy to process the event differently depending on its type.

Observer Pattern in Modern Frameworks

Modern frameworks have abstracted the Observer pattern into reactive programming libraries. RxPY (ReactiveX for Python) provides Observable streams that emit values over time, with operators for filtering, transforming, and combining streams. The Observer pattern is also the foundation of publish-subscribe systems like Redis Pub/Sub, Apache Kafka, and WebSocket-based event buses. In frontend frameworks like React, the virtual DOM diffing algorithm is essentially an Observer that re-renders components when their state changes. Understanding the raw Observer pattern helps you debug these higher-level abstractions when things go wrong—the same principles of subscription management, backpressure, and error propagation apply at every abstraction level.

from rx import from_list
from rx.operators import filter, map

numbers = from_list([1, 2, 3, 4, 5, 6])
numbers.pipe(
    filter(lambda x: x % 2 == 0),
    map(lambda x: x ** 2)
).subscribe(
    on_next=lambda x: print(f"Got: {x}"),
    on_error=lambda e: print(f"Error: {e}"),
    on_completed=lambda: print("Done!")
)

State Pattern as a Strategy Variant

The State pattern is closely related to Strategy but with a key difference: in Strategy, the client chooses and sets the strategy; in State, the object’s internal state determines its behavior automatically. A document editor has states (Draft, Review, Published) that determine which operations are allowed. State transitions are defined in the state machine. The State pattern is implemented identically to Strategy at the code level but differs in intent. In game development, state machines control character behavior, AI decision making, and UI screens. Adding a new state requires creating a new class without modifying existing states or the context.

Strategy Pattern in Functional Programming

In functional programming languages, the Strategy pattern becomes trivial: strategies are just functions passed as arguments. Instead of defining a Strategy interface and concrete classes, you pass a function directly. Python’s first-class functions make this natural: sort(key=len) passes a strategy for extracting sort keys. The strategy could be a lambda, a named function, or a callable class. The functools.partial function creates pre-configured strategies by binding some arguments. Libraries like attrs and dataclasses with field(validator=…) use this pattern for validation strategies. In Java, functional interfaces and lambda expressions (added in Java 8) eliminated the boilerplate of anonymous Strategy classes. The modern version of the Strategy pattern is dependency injection: your function or class receives its strategy as a parameter instead of implementing a fixed behavior.

Clean Code

Writing Clean Code: Principles and Practices

Clean code is code that is easy to read, understand, and modify. It is not about following a specific style guide or design pattern — it is about communicating intent clearly to your future self and your teammates. Robert C. Martin’s Clean Code and Martin Fowler’s Refactoring established many of the core principles. This article covers the most impactful practices: meaningful naming, small functions, DRY, and SOLID.

Meaningful Naming

Names are the most direct form of communication in code. A good name reveals intent without requiring a comment. Choose names that answer: why does this exist, what does it do, and how is it used? Avoid single-letter names (except for loop counters in very tight scopes), avoid abbreviations that are not universally understood, and use pronounceable names that can be discussed verbally during code review. Boolean variables should read like predicates: isActive, hasPermission, canDelete.

# Bad names — unclear intent
def d(a, b):
    return a * (1 - b)

# Good names — clear intent
def apply_discount(original_price: float, discount_rate: float) -> float:
    return original_price * (1 - discount_rate)

Small Functions That Do One Thing

A function should do one thing and do it well. If you can extract a meaningful section of a function into a separate function with a descriptive name, do it. Small functions (under 20 lines) are easier to test, easier to understand, and more reusable. The function name and the implementation should be at the same level of abstraction — if a function named calculateTotal also sends emails, it is doing too much. Each function should have one level of indentation and either return a value or produce a side effect, but not both.

DRY — Don’t Repeat Yourself

Duplicated code multiplies bugs — when you fix a bug in one copy, you must remember to fix it in all the other copies. Extract repeated logic into functions, classes, or modules. The rule of three is a useful guideline: if you write the same thing three times, extract it; for the first two occurrences, wait and see if a third appears before refactoring. Do not conflate coincidence with duplication — two pieces of code that happen to look the same but serve different purposes should not be unified prematurely.

SOLID Principles

The Single Responsibility Principle (SRP) states that a class should have only one reason to change — keep domain logic separate from infrastructure code. The Open/Closed Principle (OCP) says code should be open for extension but closed for modification. The Liskov Substitution Principle (LSP) requires that derived classes be substitutable for their base classes. Interface Segregation (ISP) means many specific interfaces are better than one general-purpose interface. Dependency Inversion (DIP) says depend on abstractions, not concretions — inject dependencies rather than instantiating them.

# SRP violation: OrderService handles business logic AND persistence
class OrderService:
    def process_order(self, order):
        self.apply_discounts(order)
        database.save(order)
        email.send(order)

# SRP compliance: separate concerns
class OrderProcessor:
    def process(self, order): ...
class OrderRepository:
    def save(self, order): ...
class NotificationService:
    def send_confirmation(self, order): ...

Clean code is not achieved in one pass. Write the first version to make it work, then refactor to make it clean. The refactoring step is critical — without it, code accumulates cruft over time until it becomes unmaintainable.

Code Smells and Refactoring Techniques

Code smells are surface-level indicators that suggest deeper problems. Common smells include: long methods (extract method until each fits on one screen), large classes (extract class for each responsibility), primitive obsession (create value objects for phone numbers, money, dates), long parameter lists (introduce parameter object), shotgun surgery (a change requires editing many files—consolidate related logic), feature envy (a method uses more features of another class than its own—move it), and switch statements (replace with polymorphism). Martin Fowler’s catalog of refactorings provides step-by-step transformations for each smell. Extract Method, Rename Variable, Move Field, Replace Conditional with Polymorphism, and Introduce Parameter Object are the most frequently applied refactorings. Modern IDEs automate many refactorings with keyboard shortcuts—learn them to make refactoring faster than the alternative of leaving messy code.

# Before: primitive obsession
def create_order(customer_name, customer_email, customer_phone):
    pass

# After: value objects
@dataclass
class Customer:
    name: str
    email: str
    phone: str

def create_order(customer: Customer):
    pass

Test-Driven Development and Clean Code

TDD and clean code reinforce each other. Writing tests first forces you to design testable interfaces: clear inputs/outputs, dependency injection, and single responsibilities—all hallmarks of clean code. The test provides immediate feedback on API design: if the test is hard to write, the interface is probably awkward. The refactoring step in TDD (red-green-refactor) is where clean code practices are applied: extract methods, rename variables, simplify conditionals, and remove duplication. Without tests, refactoring is risky. With a comprehensive test suite, you refactor aggressively, knowing the tests will catch regressions.

Pandas for Data Analysis: Beyond the Basics

Pandas for Data Analysis: Beyond the Basics

Pandas is the most widely used data analysis library in Python. While many tutorials cover loading a CSV and viewing basic statistics, real-world data analysis requires more advanced operations: grouping, aggregating, merging datasets, reshaping with pivot tables, and working with time series. This article walks through each of these techniques with practical examples that you can adapt to your own datasets.

Grouping and Aggregating Data

The groupby operation splits your data into groups based on one or more columns, applies a function to each group independently, and combines the results. This is the SQL GROUP BY equivalent in Pandas. For example, to calculate total revenue and unique order count per region and product combination, you pass a list of grouping columns and a dictionary mapping output column names to aggregation functions.

import pandas as pd

# Load sales data
df = pd.read_csv("sales.csv")
print(df.head())
print(df.info())

# Group by region and product, aggregate multiple metrics
summary = df.groupby(["region", "product"]).agg(
    total_revenue=("revenue", "sum"),
    order_count=("order_id", "nunique"),
    avg_quantity=("quantity", "mean"),
    first_sale=("date", "min")
).reset_index()

print(summary.head(10))

The agg method accepts a dictionary where keys are new column names and values are tuples of (source_column, function). You can use any Pandas or NumPy function: sum, mean, nunique, min, max, std, or even custom lambda functions. The reset_index() call converts the grouped index back into regular columns, which is usually more convenient for further analysis. Without it, the grouping columns become part of a MultiIndex, which can be harder to work with.

Merging Datasets

Data often lives in multiple tables that need to be joined. Pandas provides merge() for SQL-style joins and concat() for stacking tables vertically or horizontally. The merge() function accepts how parameter values like inner, left, right, and outer, matching SQL JOIN semantics. Always specify the key columns explicitly with on, left_on, and right_on to avoid ambiguity.

# Load related tables
orders = pd.read_csv("orders.csv")
customers = pd.read_csv("customers.csv")
payments = pd.read_csv("payments.csv")

# Left join: all orders, with customer info where available
merged = pd.merge(orders, customers, on="customer_id", how="left")

# Inner join: only orders that have matching payments
paid_orders = pd.merge(orders, payments, on="order_id", how="inner")

# Merge on different column names
merged2 = pd.merge(orders, customers,
                   left_on="cust_id", right_on="id",
                   how="left")

# Concatenate monthly reports vertically
jan = pd.read_csv("sales_jan.csv")
feb = pd.read_csv("sales_feb.csv")
mar = pd.read_csv("sales_mar.csv")
q1 = pd.concat([jan, feb, mar], ignore_index=True)

When merging, watch out for many-to-many relationships — they produce Cartesian products that can explode your DataFrame size. Always inspect the shape before and after: print(len(orders), len(merged)). If the merged result is much larger than expected, you may have duplicate keys in one of the tables. Use validate='one_to_one' or validate='many_to_one' to raise an error if the relationship is not what you expect.

Pivot Tables

A pivot table reshapes data from a long format (one row per observation) to a wide format (one row per group, with columns for each category). This is the Pandas equivalent of Excel pivot tables and is invaluable for creating summary matrices, heatmaps, and cross-tabulations.

# Create a pivot table: regions as rows, quarters as columns
pivot = df.pivot_table(
    values="revenue",
    index="region",
    columns="quarter",
    aggfunc="sum",
    margins=True,
    fill_value=0
)

print(pivot)

# Multiple aggregation functions
pivot2 = df.pivot_table(
    values="revenue",
    index="region",
    columns="quarter",
    aggfunc=["sum", "mean", "count"],
    margins=True
)

# Cross-tabulation (frequency counts)
crosstab = pd.crosstab(df["region"], df["product_category"],
                       margins=True, normalize="index")
print(crosstab)

The margins=True parameter adds a “All” row and column with totals, similar to Excel’s Grand Total. fill_value=0 replaces missing combinations with zero instead of NaN. pd.crosstab is a specialized pivot table for frequency counts and is useful for understanding the distribution of categorical variables. Setting normalize='index' converts counts to percentages within each row, making it easy to compare category distributions across regions.

Time Series Analysis

Pandas has excellent support for time series data. Converting a date column to a DatetimeIndex enables powerful resampling, rolling windows, and time-based filtering. Always parse dates at load time with parse_dates=['date'] to avoid working with string columns.

# Parse dates and set as index
df = pd.read_csv("sales.csv", parse_dates=["date"])
df.index = pd.to_datetime(df["date"])

# Resample: aggregate by week
weekly = df.resample("W").agg({
    "revenue": "sum",
    "order_id": "nunique"
})

# Rolling average (4-week window)
weekly["revenue_ma4"] = weekly["revenue"].rolling(window=4).mean()

# Resample by month with multiple metrics
monthly = df.resample("ME").agg({
    "revenue": ["sum", "mean", "std"],
    "order_id": "nunique"
})

# Time-based filtering
q1_2026 = df["2026-01":"2026-03"]
last_30_days = df[df.index >= pd.Timestamp.now() - pd.DateOffset(days=30)]

# Shift for period-over-period comparison
weekly["revenue_prev"] = weekly["revenue"].shift(1)
weekly["change_pct"] = (weekly["revenue"] / weekly["revenue_prev"] - 1) * 100

print(weekly.head(10))

The resample method is like a time-based groupby. The "W" string stands for weekly (ISO weeks, ending Sunday). Other common aliases include "D" (daily), "ME" (month-end), "MS" (month-start), "QE" (quarter-end), and "YE" (year-end). The rolling() method creates a moving window for smoothing or computing trailing statistics — the window size is the number of periods, not a time duration. For irregularly sampled time series, use rolling(window=4, min_periods=2) to handle gaps gracefully.

Performance Tips

For large datasets (millions of rows), avoid iterative row-by-row operations. Use vectorized operations, avoid apply with slow functions, and prefer built-in aggregation methods. The query() method is faster than boolean indexing for complex filters. For very large data that does not fit in memory, consider dask.dataframe or polars as alternatives to Pandas.

# Fast filtering with query()
fast_filter = df.query("region == 'East' and revenue > 1000")

# Vectorized column creation
df["discounted_price"] = df["price"] * (1 - df["discount"])

# Avoid: df.apply(lambda row: row["price"] * (1 - row["discount"]), axis=1)
# Always prefer vectorized operations over apply

With these techniques — grouped aggregations, merges, pivot tables, and time series resampling — you can handle the vast majority of real-world data analysis tasks efficiently and expressively.

Data Classes and Pydantic for Robust Applications

Data Classes and Pydantic for Robust Applications

Python’s dataclasses (introduced in Python 3.7) and Pydantic (a third-party library) both reduce boilerplate code when defining data containers, but they serve different purposes. Dataclasses auto-generate __init__, __repr__, __eq__, and __hash__ methods based on class annotations. Pydantic goes further by adding runtime type validation, serialization, and parsing — making it the backbone of FastAPI and the go-to choice for any application that handles external data.

Python Dataclasses

Dataclasses are ideal for internal data structures where you want concise, readable code without writing boilerplate. The @dataclass decorator inspects the class’s type annotations and automatically generates the standard dunder methods. You can control behavior with parameters like frozen=True (immutable instances), order=True (sortable), and slots=True (Python 3.10+, memory efficient).

from dataclasses import dataclass, field

@dataclass(frozen=True, order=True)
class Point:
    x: float
    y: float
    label: str = field(default="", compare=False)

p1 = Point(1.0, 2.0, "start")
p2 = Point(1.0, 2.0, "end")
print(p1)          # Point(x=1.0, y=2.0, label='start')
print(p1 == p2)    # True (label excluded from comparison)
# p1.x = 3.0       # Error: frozen instance

Pydantic BaseModel

Pydantic’s BaseModel adds validation and serialization on top of the type annotation system. When you create an instance, Pydantic validates every field against its type annotation — coercing values when safe (e.g., "123" becomes 123 for an int field) and raising a detailed validation error when coercion is not possible. This catches data quality issues at the boundary of your application rather than deep in your logic.

from pydantic import BaseModel, EmailStr, Field
from datetime import datetime

class User(BaseModel):
    id: int
    name: str = Field(..., min_length=1, max_length=100)
    email: EmailStr
    age: int = Field(ge=0, le=150, default=0)
    created_at: datetime = Field(default_factory=datetime.now)
    tags: list[str] = Field(default_factory=list)

# Valid data — works
user = User(id=1, name="Alice", email="alice@example.com", age=30, tags=["admin"])
print(user.model_dump_json(indent=2))

# Invalid data — raises ValidationError with clear message
try:
    User(id="not-a-number", name="", email="not-an-email", age=200)
except Exception as e:
    print(e)
    # Shows all validation errors at once, not just the first one

The Field() function lets you add constraints (min_length, ge for greater-or-equal, le for less-or-equal), default values, and metadata. Pydantic V2, powered by a Rust core (pydantic-core), is significantly faster than V1 and supports complex types like UUID, IPv4Address, Decimal, and Path out of the box.

Nested Models and Validation

Pydantic models can be nested to represent complex hierarchies. Each nested model is validated recursively. You can also define custom validators using @field_validator and @model_validator decorators to enforce business rules that go beyond simple type checks.

from pydantic import BaseModel, field_validator

class Address(BaseModel):
    street: str
    city: str
    zip_code: str

class Employee(BaseModel):
    name: str
    address: Address
    salary: float

    @field_validator('salary')
    @classmethod
    def salary_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError('Salary must be positive')
        return v

emp = Employee(
    name="Bob",
    address={"street": "123 Main St", "city": "NYC", "zip_code": "10001"},
    salary=75000.0
)
print(emp.model_dump())
# {'name': 'Bob', 'address': {'street': '123 Main St', ...}, 'salary': 75000.0}

Serialization and Parsing

Pydantic makes it easy to convert between Python objects and JSON/dict representations. This is essential for API development: parse incoming request bodies with model_validate() and serialize responses with model_dump() or model_dump_json(). The model_config lets you control serialization behavior like excluding unset values, using aliases, or populating by name vs by position.

# Parse from JSON string
json_data = '{"name": "Charlie", "email": "charlie@example.com", "age": 25}'
user = User.model_validate_json(json_data)

# Parse from dict
user = User.model_validate(dict(name="Diana", email="diana@example.com", age=28))

# Serialize to dict (exclude unset fields)
data = user.model_dump(exclude_unset=True)

# Serialize to JSON with formatting
print(user.model_dump_json(indent=2))
Settings Management with Pydantic


Pydantic's BaseSettings (from pydantic-settings) extends BaseModel to read configuration from environment variables, .env files, and secret stores. This is the recommended way to manage application configuration in Python. Define a settings class with typed fields and default values, and Pydantic automatically reads from the environment — validating types, providing clear error messages for missing required values, and supporting nested settings for complex configurations.


from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    redis_url: str = 'redis://localhost:6379'
    debug: bool = False
    api_key: str
    max_connections: int = 10

    class Config:
        env_file = '.env'
        env_file_encoding = 'utf-8'

settings = Settings()
# Reads DATABASE_URL, REDIS_URL, API_KEY from environment or .env file



This pattern eliminates the need for ad-hoc environment variable parsing, type coercion bugs, and inconsistent configuration handling across your application. When combined with FastAPI, Pydantic provides end-to-end type safety from the HTTP request boundary through your application logic to the response serialization.


Use dataclasses for simple internal data holding in your own codebase where validation is not critical. Use Pydantic whenever data enters or leaves your application — API requests/responses, configuration files, database records — because the validation layer prevents corrupted data from propagating through your system.


Advanced Pydantic Features


Pydantic V2, rewritten in Rust with pydantic-core, is significantly faster than V1. Advanced features include: discriminated unions for parsing polymorphic JSON, strict mode (no implicit string-to-int coercion), computed fields, model validators with before/after/wrap modes, and serialization aliases. Pydantic's integration with FastAPI is seamless: request bodies and response models automatically validate and serialize. BaseModel.model_dump() and model_validate() replace V1 methods. Pydantic supports JSON Schema generation for OpenAPI documentation. For enterprise applications, StrictBool, PaymentCardNumber, and EmailStr provide domain-specific validation out of the box.

Building REST APIs with FastAPI

Building REST APIs with FastAPI

FastAPI is a modern Python web framework designed for building high-performance REST APIs. It combines automatic OpenAPI documentation, type-safe request handling via Pydantic, and asynchronous support — all while matching the performance of Node.js and Go (thanks to Starlette and Pydantic’s Rust core). This article walks through building a complete CRUD API with FastAPI, covering path operations, dependency injection, validation, and more.

Getting Started

FastAPI is built on Starlette (the ASGI framework) and Pydantic (the validation library). Install it with pip install fastapi uvicorn. A minimal application defines a FastAPI instance and a few path operations using Python type annotations. The type annotations serve double duty: they enable editor autocompletion and type checking, and FastAPI uses them to generate OpenAPI documentation and validate request data automatically. Start the server with uvicorn main:app --reload and visit /docs for interactive Swagger UI or /redoc for ReDoc documentation.

from fastapi import FastAPI

app = FastAPI(title="My API", version="1.0.0")

@app.get("/")
def read_root():
    return {"message": "Hello World"}

@app.get("/items/{item_id}")
def read_item(item_id: int, q: str = None):
    return {"item_id": item_id, "q": q}

The path parameter item_id is declared as int, so FastAPI automatically validates that it is a valid integer and returns a 422 error if it is not. The query parameter q is optional (defaults to None) and is a string. The OpenAPI schema reflects this — documenting the path parameter type, the query parameter, and the response format — without writing any additional configuration.

Request Validation with Pydantic

For POST, PUT, and PATCH requests, define a Pydantic model for the request body. FastAPI automatically validates the incoming JSON against the model, returning detailed field-level error messages if validation fails. Pydantic models also serve as the response model, ensuring that outgoing data matches the schema. Use response_model in the decorator to control what fields are included in the response and to enable automatic response filtering and documentation.

from pydantic import BaseModel, Field
from typing import Optional
from datetime import datetime

class ItemCreate(BaseModel):
    name: str = Field(..., min_length=1, max_length=100)
    price: float = Field(..., gt=0)
    description: Optional[str] = None
    tax: Optional[float] = None

class ItemResponse(BaseModel):
    id: int
    name: str
    price: float
    description: Optional[str] = None
    created_at: datetime

items_db = {}
counter = 0

@app.post("/items", response_model=ItemResponse, status_code=201)
def create_item(item: ItemCreate):
    global counter
    counter += 1
    db_item = {
        "id": counter,
        "name": item.name,
        "price": item.price,
        "description": item.description,
        "created_at": datetime.now()
    }
    items_db[counter] = db_item
    return db_item

@app.get("/items/{item_id}", response_model=ItemResponse)
def get_item(item_id: int):
    if item_id not in items_db:
        from fastapi import HTTPException
        raise HTTPException(status_code=404, detail="Item not found")
    return items_db[item_id]

The Field() function adds validation constraints directly in the model: min_length=1, gt=0 (greater than 0). If validation fails, FastAPI returns a 422 response with a JSON body listing every field that failed and why — the client does not need to guess what went wrong. The response_model ensures that only the fields defined in ItemResponse are returned, even if the database object has extra fields.

Dependency Injection

FastAPI’s dependency injection system lets you extract common logic (database sessions, authentication, configuration) into reusable dependencies. A dependency is any callable that returns a value, declared with Depends(). FastAPI manages the dependency lifecycle, resolving dependencies in order and caching them per request. This makes your path operations thin — they just call the relevant service and return a response, while dependencies handle cross-cutting concerns like authentication, database connections, and rate limiting.

from fastapi import FastAPI, Depends, HTTPException, status

# Authentication dependency
async def get_current_user(token: str):
    if token != "secret-token":
        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED)
    return {"username": "alice", "role": "admin"}

# Database dependency
def get_db():
    db = Database.connect()
    try:
        yield db
    finally:
        db.close()

# Path operation using dependencies
@app.get("/users/me")
def read_current_user(
    current_user: dict = Depends(get_current_user),
    db: Database = Depends(get_db)
):
    return {
        "user": current_user,
        "items": db.query("SELECT * FROM items")
    }

Error Handling and Configuration

# Custom exception handler
from fastapi import Request
from fastapi.responses import JSONResponse

class AppException(Exception):
    def __init__(self, code: str, message: str, status_code: int = 400):
        self.code = code
        self.message = message
        self.status_code = status_code

@app.exception_handler(AppException)
async def app_exception_handler(request: Request, exc: AppException):
    return JSONResponse(
        status_code=exc.status_code,
        content={"error": {"code": exc.code, "message": exc.message}}
    )

# Application settings with Pydantic
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    debug: bool = False
    api_key: str

    class Config:
        env_file = ".env"

settings = Settings()

@app.get("/config")
def get_config():
    return {"debug": settings.debug}

FastAPI’s combination of type safety, automatic documentation, and dependency injection makes it the most productive Python framework for building APIs. The automatic OpenAPI documentation ensures your API is always documented correctly (because it is generated from the actual code), and the Pydantic integration catches data issues at the earliest possible moment — when the request arrives at your server.

Debugging Techniques Every Developer Should Know

Debugging Techniques Every Developer Should Know

Debugging is the art of figuring out why code does not work as expected. Even the best developers spend a significant portion of their time debugging — studies suggest 30-50% of development time is spent finding and fixing bugs. Having a systematic approach and the right tools turns debugging from a frustrating guessing game into a methodical investigation. This article covers logging, interactive debugging, stack trace analysis, profiling, and git bisect, with practical examples you can apply immediately.

Structured Logging

Logging is the most basic and most important debugging tool. Print statements work for tiny scripts, but production systems need structured, level-based logging that can be searched and filtered. Python’s logging module supports severity levels (DEBUG, INFO, WARNING, ERROR, CRITICAL), log formatting, and output to multiple destinations (console, file, external service). Always use structured logging with JSON output so that log aggregation tools like the ELK stack, Splunk, or Datadog can parse and index your logs automatically. Include contextual data like request IDs, user IDs, and transaction IDs in every log message to trace a request across multiple services.

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s [%(levelname)s] %(message)s'
)
logger = logging.getLogger(__name__)

# Always pass extra context for traceability
logger.info("Payment processed",
            extra={"txn_id": "txn_abc123", "amount": 49.99})

Log at the right level — DEBUG for detailed diagnostic information, INFO for normal operations (request started, payment completed), WARNING for unexpected but non-critical issues (slow query, retry attempt), ERROR for failures that need investigation (database connection lost, API returned 500), and CRITICAL for catastrophic failures that require immediate human intervention. Too much logging (especially at INFO or DEBUG in production) generates noise and costs; too little leaves you blind when something goes wrong.

Interactive Debugging with Breakpoints

When logs are not enough, you need to pause execution and inspect the program state. Python’s built-in breakpoint function (available since Python 3.7) drops you into a debugger at the line where it is called. It respects the PYTHONBREAKPOINT environment variable, so you can use different debuggers in different environments — pdb locally, web-pdb in containers, or skip all breakpoints in production by setting PYTHONBREAKPOINT=0.

def calculate_discount(price, customer_tier, items_count):
    # Set a breakpoint here to inspect variables
    breakpoint()

    base_discount = 0.05
    if customer_tier == "gold":
        base_discount += 0.10
    elif customer_tier == "platinum":
        base_discount += 0.15
    if items_count >= 10:
        base_discount += 0.05
    return price * (1 - base_discount)

# In the debugger you can type:
# (Pdb) price          -> 100.0
# (Pdb) customer_tier  -> 'gold'
# (Pdb) c              -> continue execution

In the debugger, you can type any Python expression to inspect variables, call functions, or modify state. The most useful commands are n (next line), s (step into function), c (continue until next breakpoint), l (show surrounding source code), p variable (print variable), and pp variable (pretty-print for complex objects). For web development, tools like ipdb (IPython-enhanced pdb), pudb (visual console debugger), and web-pdb (debug over HTTP in a browser) provide richer debugging experiences.

Reading Stack Traces

A stack trace shows the chain of function calls that led to an exception. Read it bottom to top — the last line in the traceback is usually where the error occurred (the deepest call in the stack). Your application code is typically in the middle of the traceback; the top lines are framework or library internals. When reading a traceback, identify the exception type (e.g., KeyError, AttributeError, ValueError), the error message, and the exact line number where it was raised. Then work backwards through the call chain to understand how your code reached that state.

Profiling for Performance Bugs

Not all bugs are logic errors — performance bugs (slow functions, memory leaks) are just as damaging. Profiling measures where your program spends its time and memory. cProfile is Python’s built-in deterministic profiler — it records every function call with timing information. Memory profiling with the memory-profiler package shows memory usage line by line, helping you identify objects that are unexpectedly retained.

# CPU profiling
import cProfile, pstats

def process_data():
    data = [i ** 2 for i in range(100000)]
    filtered = [x for x in data if x % 2 == 0]
    return sum(filtered)

cProfile.run('process_data()', 'profile_stats')
p = pstats.Stats('profile_stats')
p.sort_stats('cumtime').print_stats(10)

Git Bisect — Finding the Regression Commit

When a bug appears that was not there before, git bisect performs a binary search through your commit history to find the exact commit that introduced the regression. Start by marking the current commit as bad and a known-good commit (from before the bug appeared) as good. Git then checks out a commit halfway between them, and you test whether the bug is present — you mark it good or bad. Each step halves the remaining search space, so finding the culprit among 1000 commits takes only about 10 steps.

# Start bisect
git bisect start
git bisect bad          # current commit is broken
git bisect good v1.0    # tag v1.0 was working

# Git checks out a commit — test it
git bisect bad   # or: git bisect good

# Repeat until git identifies the first bad commit

# Or automate with a test script:
git bisect run pytest tests/test_feature.py

# End bisect session
git bisect reset

Automated git bisect run is incredibly powerful — give it a script that exits with code 0 (good) or non-zero (bad), and it will run through the entire binary search without any manual intervention. Set this up as part of your CI pipeline to automatically identify which commit introduced a performance regression or test failure.

CI/CD and GitHub Actions: Automate Your Development Pipeline

What is CI/CD?

CI/CD stands for Continuous Integration and Continuous Delivery (or Deployment). It is a software engineering practice where developers merge their code changes into a shared repository frequently, and each merge triggers an automated build-and-test pipeline. The goal is to catch bugs early, reduce integration hell, and ship reliable software faster.

Continuous Integration (CI) means that every time a developer pushes code, the system automatically builds the project and runs a suite of tests. If the build breaks or a test fails, the team knows immediately.

Continuous Delivery (CD) extends CI by automatically deploying the tested code to a staging or production environment after the CI pipeline passes, ensuring that the software is always in a deployable state.

Why GitHub Actions?

GitHub Actions is GitHub’s built-in CI/CD platform. It is deeply integrated with GitHub repositories, free for public repositories, and offers a vast ecosystem of pre-built actions from the community. Key advantages include:

  • Tight GitHub integration — triggers on push, PR, issue comments, releases, and more.
  • Matrix builds — test across multiple OS versions, language versions, and architectures in parallel.
  • Hosted runners — Ubuntu, Windows, and macOS runners are provided free for public repos.
  • Marketplace — thousands of community actions for deployments, notifications, code quality, and security scanning.
  • Self-hosted runners — run workflows on your own infrastructure for private projects.

Workflow Structure

A GitHub Actions workflow is defined in a YAML file stored at .github/workflows/. Every workflow has three top-level components:

Events (Triggers)

What causes the workflow to run:

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: "0 6 * * 1"  # Every Monday at 6 AM
  workflow_dispatch:  # Manual trigger

Jobs

Jobs run in parallel by default on separate runners. Each job contains a series of steps:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run tests
        run: pytest

Steps

Steps are individual commands or actions. They run sequentially within a job and share the same filesystem.

Real-World Example: Python Project

Here is a complete workflow that lints, tests, and deploys a Python application:

name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12"]

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install ruff pytest

      - name: Lint with Ruff
        run: ruff check .

      - name: Test with pytest
        run: pytest

  deploy:
    needs: lint-and-test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
      - uses: actions/checkout@v4

      - name: Deploy to production
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
        run: |
          echo "Deploying to production server..."
          # ssh, rsync, or use a deployment action

This workflow runs a matrix build against Python 3.11 and 3.12, lints with Ruff, runs tests, and only deploys if all tests pass on the main branch.

Secrets Management

Never hardcode credentials in your workflow files. GitHub provides encrypted secrets under Settings → Secrets and variables → Actions:

jobs:
  deploy:
    steps:
      - name: Deploy
        env:
          SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
          API_TOKEN: ${{ secrets.API_TOKEN }}
        run: deploy-script.sh

Secrets are masked in logs and never passed to forks.

Matrix Builds

Matrix strategies let you test across combinations of OS, language version, and environment variables with a single job definition:

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        python-version: ["3.10", "3.11", "3.12"]
        exclude:
          - os: windows-latest
            python-version: "3.10"

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pytest

The exclude key removes specific combinations that are known to fail or are unnecessary, keeping the matrix manageable.

Best Practices

  • Keep workflows fast — use caching for dependencies with actions/cache.
  • Fail fast — set fail-fast: true in matrix builds to cancel all jobs when one fails.
  • Pin action versions — use @v4 tags (not @main) to avoid unexpected breaking changes.
  • Use status badges — add a badge to your README so contributors see the build status at a glance.
  • Separate concerns — one workflow per concern (test, lint, deploy, security scan).
  • Self-hosted runners for large repos — if your team pushes frequently, self-hosted runners eliminate queue wait times.

Conclusion

GitHub Actions makes CI/CD accessible to every developer. With a single YAML file, you can lint, test, build, and deploy your application across multiple platforms. The marketplace ecosystem, matrix builds, and secret management make it production-ready from day one. Start with a simple lint-and-test workflow, then layer on deployment, security scanning, and notifications as your project grows.

Multi-threading vs Multi-tasking: The Difference with C++ and Python Examples

Introduction

In modern software engineering, squeezing every ounce of performance from hardware is often critical. Two fundamental techniques — multi-tasking and multi-threading — are frequently confused, yet they operate at entirely different levels of the system. This article demystifies both, explains where the Python Global Interpreter Lock (GIL) fits in, and provides concrete C++ and Python examples that illustrate real-world behaviour.

Multi-tasking: The OS-Level Illusion

Multi-tasking is an operating system capability that allows multiple processes to run seemingly simultaneously. The OS scheduler rapidly switches between processes, giving each a small time slice. This creates the illusion of parallelism even on a single-core CPU.

Each process has its own isolated memory space, file descriptors, and security context. Communication between processes (IPC) requires explicit mechanisms like pipes, shared memory, or sockets. This isolation makes multi-tasking robust — one crashing process does not bring down the others — but also adds overhead for context switching and data sharing.

Multi-threading: Parallelism Within a Process

Multi-threading is an application-level technique where a single process spawns multiple threads that share the same memory space, open files, and other resources. Threads are lightweight compared to processes; creating and switching between them is far cheaper because the OS does not need to swap out the full memory context.

The critical trade-off: because threads share memory, developers must coordinate access with synchronisation primitives (mutexes, semaphores, atomic operations) to avoid race conditions and data corruption.

The Python GIL: The Elephant in the Room

Python’s Global Interpreter Lock (GIL) is a mutex that protects access to CPython interpreter internals, ensuring that only one thread executes Python bytecode at any given moment. This means Python threads cannot achieve true parallel execution for CPU-bound tasks — they merely time-share the same core, often with more overhead than a single-threaded approach.

For I/O-bound tasks (network requests, file reads, database queries), threading is still effective because the GIL is released during blocking I/O calls.

C++ Example: True Parallel Execution

#include <iostream>
#include <thread>
#include <vector>
#include <chrono>

uint64_t fibonacci(int n) {
    if (n <= 1) return n;
    return fibonacci(n - 1) + fibonacci(n - 2);
}

void worker(int n, uint64_t& result) {
    result = fibonacci(n);
}

int main() {
    const int N = 42;
    std::vector<uint64_t> results(4);
    std::vector<std::thread> threads;

    auto start = std::chrono::high_resolution_clock::now();

    for (int i = 0; i < 4; ++i)
        threads.emplace_back(worker, N + i, std::ref(results[i]));

    for (auto& t : threads)
        t.join();

    auto end = std::chrono::high_resolution_clock::now();
    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();

    for (auto r : results)
        std::cout << r << " ";
    std::cout << "
Time: " << ms << " ms
";
}

On a quad-core machine, this runs approximately 4x faster than a serial version.

Python Threading: Blocked by the GIL

import threading
import time

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

def worker(n, results, idx):
    results[idx] = fibonacci(n)

if __name__ == "__main__":
    N = 38
    results = [0] * 4
    threads = []

    start = time.perf_counter()

    for i in range(4):
        t = threading.Thread(target=worker, args=(N + i, results, i))
        threads.append(t)
        t.start()

    for t in threads:
        t.join()

    elapsed = time.perf_counter() - start
    print(results, f"{elapsed:.2f}s")

On a quad-core machine, this runs at the same speed as the serial version — the GIL serialises all threads onto a single core.

Python Multiprocessing: Bypassing the GIL

import multiprocessing as mp
import time

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

if __name__ == "__main__":
    N = 38
    args = [N, N + 1, N + 2, N + 3]

    start = time.perf_counter()

    with mp.Pool(4) as pool:
        results = pool.map(fibonacci, args)

    elapsed = time.perf_counter() - start
    print(results, f"{elapsed:.2f}s")

This runs significantly faster — near-linear speedup up to the number of physical cores.

When to Use Which Approach

Scenario Recommendation Reason
CPU-bound in Python multiprocessing GIL blocks threads
I/O-bound in Python threading or asyncio GIL released during I/O
CPU-bound in C++ std::thread or OpenMP True parallel execution
I/O-bound in C++ std::thread No GIL contention
Strong isolation needed Multi-processing Processes are isolated
Latency-sensitive, shared state Multi-threading Shared memory is fast

Conclusion

Multi-tasking and multi-threading are complementary tools. Python’s GIL adds a critical constraint — threads are useful for I/O but harmful for CPU-bound computation, where multiprocessing is the correct escape hatch. C++ offers true multi-threading from the ground up, but with synchronisation responsibility. Understanding these trade-offs is what separates working code from performant, production-grade systems.

When to Use Multi-Threading vs Multi-Tasking

Choose multi-threading when tasks are I/O-bound (waiting for disk, network, database) and share memory. In Python, the Global Interpreter Lock (GIL) prevents true parallel execution of threads for CPU-bound tasks, but threading still improves I/O-bound throughput because threads yield the GIL during I/O waits. Choose multi-processing for CPU-bound tasks (computation-heavy work like image processing, numerical simulations) where each process runs on a separate CPU core without GIL contention. Asyncio provides a third option: cooperative concurrency within a single thread where tasks voluntarily yield control at await points, ideal for high-concurrency I/O-bound workloads without the overhead of thread context switching.