The GOF Test: Goodness-of-Fit Explained

The Goodness-of-Fit (GOF) test, commonly referring to the Chi-Square Goodness-of-Fit test, determines whether an observed frequency distribution matches an expected distribution. It answers questions like: “Is this die fair?” (are observed roll frequencies close to uniform?) or “Does this sample follow a normal distribution?” Developed by Karl Pearson in 1900, the chi-square goodness-of-fit test remains one of the most widely used statistical tests in data analysis.

How the Test Works

The test compares observed frequencies (O_i) to expected frequencies (E_i) across k categories. The test statistic is χ² = Σ((O_i – E_i)² / E_i). Under the null hypothesis (the observed distribution matches the expected distribution), this statistic follows a chi-square distribution with k-1 degrees of freedom (minus additional degrees for estimated parameters). A large chi-square value indicates a poor fit—the observed frequencies deviate too much from expectations. The p-value tells us the probability of observing such deviation (or more extreme) if the null hypothesis were true.

import numpy as np
from scipy import stats

# Observed: roll frequencies for a die (120 rolls)
observed = np.array([15, 22, 18, 25, 20, 20])
# Expected: fair die (each face equally likely = 20 each)
expected = np.array([20, 20, 20, 20, 20, 20])

chi2_stat = np.sum((observed - expected)**2 / expected)
p_value = 1 - stats.chi2.cdf(chi2_stat, df=5)  # 6-1=5 degrees of freedom
print(f"Chi-square: {chi2_stat:.3f}, p-value: {p_value:.3f}")

# Using scipy's built-in function
chi2_stat, p_value = stats.chisquare(observed, expected)
print(f"SciPy: χ²={chi2_stat:.3f}, p={p_value:.3f}")
# p > 0.05: fail to reject null → die appears fair

Assumptions and Requirements

Four key assumptions must hold. First, the data must be counts (frequencies), not percentages or continuous values. Second, categories must be mutually exclusive (each observation belongs to exactly one category). Third, observations must be independent—the chi-square test is not valid for repeated measures or paired data. Fourth, expected frequencies should be at least 5 for each category; if any category has E_i < 5, combine adjacent categories until the requirement is met. The test is also sensitive to sample size—with very large samples, even trivial deviations become statistically significant. In such cases, effect size measures like Cramér's V (for nominal data) or the phi coefficient provide practical significance context.

Applications in Data Science

The GOF test has numerous practical applications. In A/B testing, it checks whether conversion counts match expected proportions. In genetics, it validates Mendelian inheritance ratios (3:1 for dominant/recessive). In survey analysis, it determines if response distributions match population demographics. In machine learning, the chi-square test is used for feature selection—it tests independence between a categorical feature and the target variable, identifying features that carry predictive signal. The sklearn.feature_selection.chi2 function implements this for classification problems, ranking features by their chi-square statistic against the target.

# Chi-square for feature selection in ML
from sklearn.feature_selection import chi2
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)
# Chi-square tests each pixel's intensity distribution against digit class
chi2_scores, p_values = chi2(X, y)
top_features = np.argsort(chi2_scores)[-10:]
print(f"Top 10 most informative pixel positions: {top_features}")

When the GOF test shows lack of fit, follow-up analysis should identify which categories contribute most to the deviation. The standardized residuals ((O_i – E_i) / √E_i) for each category show the direction and magnitude of deviation—absolute values above 2 or 3 indicate categories that differ significantly from expectations, guiding further investigation into why those specific categories deviate.

Alternatives to the Chi-Square GOF Test

When data violates chi-square assumptions (expected frequencies below 5), Fisher’s exact test provides accurate p-values for 2×2 contingency tables. For continuous data, the Kolmogorov-Smirnov test compares an empirical distribution against a theoretical one (normal, exponential, uniform), and the Anderson-Darling test gives more weight to differences in the tails of the distribution. The Shapiro-Wilk test is specifically designed for testing normality and has better statistical power than KS for that purpose. For comparing two empirical distributions (rather than one empirical vs theoretical), the two-sample KS test or the Wilcoxon rank-sum test (non-parametric) are appropriate. In Bayesian statistics, the posterior predictive check visually compares the observed data distribution against distributions simulated from the fitted model—a Bayesian alternative to the frequentist GOF test that provides richer diagnostic information about where and how the model misfits the data.

Effect Size and Power Analysis

A statistically significant result (p < 0.05) does not necessarily mean a practically important result—with large sample sizes, even tiny deviations become statistically significant. Effect size measures quantify the magnitude of the discrepancy. Cramér's V (for nominal data) ranges from 0 (no association) to 1 (perfect association), with values above 0.3 considered medium and above 0.5 considered large. Cohen's w is an alternative effect size for chi-square tests. Power analysis determines the sample size needed to detect a given effect size. Using the statsmodels library, you can compute the required sample size for your test: power = 0.80 (standard target) means you have an 80% chance of detecting the effect if it truly exists. Studies with low power (under 0.50) are unlikely to detect real effects and more likely to produce false negatives, wasting resources on inconclusive results.

Practical Example: Testing a Die for Fairness

To make the GOF test concrete, consider testing whether a six-sided die is fair. Roll the die 120 times and record the frequency of each face. Under the null hypothesis (fair die), each face should appear 20 times. The chi-square statistic measures how far the observed counts deviate from 20. If the p-value is above 0.05, we fail to reject the null—the die appears fair. If below, we conclude the die is biased. This example extends naturally to testing survey response distributions, website traffic across days of the week, or genetic inheritance ratios. The scipy.stats.chisquare function makes this a one-liner: just pass observed and expected arrays.