Linear Algebra for Machine Learning: Essential Concepts

Linear algebra is the mathematical foundation of machine learning. Nearly every ML algorithm relies on linear algebra operations: vectors represent data points, matrices represent datasets and transformations, and matrix multiplication powers neural network forward passes. Understanding these concepts deeply helps you debug models, choose appropriate architectures, and optimize performance. This article covers the essential linear algebra concepts every ML practitioner needs.

Vectors and Vector Operations

A vector is an ordered collection of numbers that can represent a point in n-dimensional space. In ML, a feature vector represents a single data point—for example, a house with 3 bedrooms, 2 bathrooms, and 1500 square feet is the vector [3, 2, 1500]. Vector addition (element-wise), scalar multiplication, and the dot product are the fundamental operations. The dot product measures how aligned two vectors are and is the core operation in linear regression (y = w·x + b) and neural network layers (z = W·x + b).

import numpy as np

# Vectors as numpy arrays
x = np.array([3, 2, 1500])       # House features
w = np.array([10, 5, 0.1])       # Learned weights
b = 50                            # Bias

# Linear prediction: y = w·x + b
prediction = np.dot(w, x) + b     # 3*10 + 2*5 + 1500*0.1 + 50 = 230
print(f"Predicted price: ${prediction}K")

# Euclidean norm (L2) — used in regularization
l2_reg = np.linalg.norm(w)        # sqrt(10² + 5² + 0.1²)

Matrices and Matrix Multiplication

A matrix is a 2D array of numbers. In ML, a matrix typically holds a dataset where each row is a sample and each column is a feature. Matrix multiplication is the workhorse of deep learning—each layer in a neural network computes W·x + b where W is a weight matrix, x is an input vector, and b is a bias vector. This single operation processes all features simultaneously through all neurons in a layer. The dimensions must match: an m×n matrix multiplied by an n×p matrix produces an m×p matrix (inner dimensions must agree).

# Dataset: 3 samples, 4 features
X = np.array([[1, 2, 3, 4],     # Sample 1
              [5, 6, 7, 8],     # Sample 2
              [9, 10, 11, 12]]) # Sample 3

# Weight matrix: 4 inputs → 2 outputs
W = np.array([[0.1, 0.2],
              [0.3, 0.4],
              [0.5, 0.6],
              [0.7, 0.8]])

# Forward pass: X (3×4) @ W (4×2) → output (3×2)
output = X @ W  # Equivalent to np.matmul(X, W)
print(output.shape)  # (3, 2)

Eigenvalues, Eigenvectors, and PCA

An eigenvector of a matrix is a non-zero vector that, when multiplied by the matrix, only scales (does not rotate). The eigenvalue is the scaling factor. Eigen decomposition is the foundation of Principal Component Analysis (PCA), a dimensionality reduction technique that projects high-dimensional data onto lower dimensions while preserving maximum variance. PCA identifies the eigenvectors of the covariance matrix—these are the principal components (directions of maximum variance). The corresponding eigenvalues indicate how much variance each component captures. In practice, you can reduce a 100-feature dataset to 20 features by keeping only the top 20 principal components, often retaining 90%+ of the information.

from sklearn.decomposition import PCA

# Reduce 100-dimensional data to 20 dimensions
pca = PCA(n_components=20)
X_reduced = pca.fit_transform(X_high_dim)

# Explained variance ratio — how much info each component retains
print(pca.explained_variance_ratio_)
print(f"Total variance retained: {pca.explained_variance_ratio_.sum():.2%}")

# Reconstruction — project back to original space
X_reconstructed = pca.inverse_transform(X_reduced)

Other essential linear algebra concepts for ML include the identity matrix (I) which is the multiplicative identity (AI = A), the inverse (A⁻¹ where AA⁻¹ = I) used in closed-form linear regression solutions, and the transpose (Aᵀ) used extensively in gradient computations. NumPy’s linear algebra module (np.linalg) provides optimized implementations of all these operations using BLAS and LAPACK under the hood.

Singular Value Decomposition (SVD)

SVD factorizes any matrix A (m×n) into U·Σ·Vᵀ, where U and V are orthogonal matrices and Σ is a diagonal matrix of singular values sorted in descending order. SVD is the Swiss Army knife of linear algebra: it powers recommendation systems (matrix factorization in Netflix Prize), data compression (truncating small singular values), latent semantic analysis (topic modeling in NLP), and principal component analysis (PCA is SVD on centered data). The ratio of the largest singular value to the smallest (condition number) measures matrix stability—high condition numbers indicate that small input changes cause large output changes, a critical consideration in numerical optimization.

U, S, Vt = np.linalg.svd(matrix, full_matrices=False)
# Approximate with top k singular values
k = 10
approx = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]
compression_ratio = 1 - (k * (U.shape[0] + Vt.shape[1])) / (matrix.shape[0] * matrix.shape[1])
print(f"Compression ratio: {compression_ratio:.1%}")