Categorical
Contingency tables, the χ² test of independence, Cramér's V and entropy-based association measures, all computed from one-hot columns.
This module analyses categorical data: which category each observation falls into, and whether two categorical variables are associated. It builds the cross-tabulation of two variables, the counts that independence would predict, Pearson’s χ² statistic with its degrees of freedom, Cramér’s V and phi, and the information measures — entropy, mutual information and two normalised forms of it.
A categorical variable is passed as a one-hot matrix: n×r for n observations and r
categories, with a 1 in the column of each observation’s category and 0 elsewhere. Two such
matrices contract to a contingency table in a single step, with no counting loop. Every function
after contingency takes a table t of counts, a matrix r×c (rows are the categories of the
first variable, columns those of the second); you may equally pass a table you counted yourself.
Logarithms are natural, so entropies and mutual information are in nats. Cells or categories with a
count of zero are handled explicitly: an empty expected cell contributes nothing to χ², and an empty
category contributes nothing to an entropy (the limit of \(p \ln p\) as \(p \to 0\)). The module
reports the χ² statistic and its degrees of freedom but not a p-value; pass both to chiCdf from
Distributions for that.
Throughout this page, \(T \in \mathbb{R}^{r \times c}\) (the argument t) is a table of non-negative
counts with entries \(T_{jk}\), \(j = 1, \dots, r\) and \(k = 1, \dots, c\), and a positive grand total.
Observations are indexed \(i = 1, \dots, n\). The guarded value \(\operatorname{steady}(v) = v + [v = 0]\)
replaces an exact zero with 1.
open Prelude.Categorical
The table and its margins
A contingency table counts how many observations fall in each pair of categories: \(T_{jk} = \sum_i A_{ij} B_{ik}\). Its margins are the row and column totals, and the grand total is n. Under independence each cell is expected to hold the product of its margins divided by the total.
contingency
contingency(a, b) · a one-hot membership of the first variable n×r · b one-hot membership of the second variable n×c → table of counts r×c
Let \(A \in \{0, 1\}^{n \times r}\) and \(B \in \{0, 1\}^{n \times c}\) (the arguments a, b) be one-hot membership matrices of two categorical variables observed on the same \(n\) rows. Then, for \(j = 1, \dots, r\) and \(k = 1, \dots, c\),
Cross-tabulates two categorical variables. Entry (j, k) is the number of observations in category j of the first variable and category k of the second.
Intuition Each observation has a single 1 in each matrix, so \(A_{ij}B_{ik}\) is 1 exactly when observation \(i\) is in category \(j\) of the first variable and category \(k\) of the second. Summing over observations counts every pair of categories at once.
Use it when you cross-tabulate two categorical columns: plan by region, treatment by outcome, or predicted class by true class (a confusion matrix, with predictions as rows). Rows of probabilities instead of one-hot rows also work and give expected counts. Every other function on this page takes the table it produces.
cross-tabulationclassification-evaluationassociation-measurement
def contingency(a, b) ≔ einsum("ir,ic->rc", a, b)
grand
grand(t) · t table of counts r×c → total count scalar
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) be a table of counts. Then
The sum of every cell: the number of observations in the table.
Intuition The number of observations the table summarises: the \(N\) that turns counts into proportions.
Use it when you normalise a table into joint probabilities (t / grand(t)), check that no observations were lost during encoding, or scale a statistic by sample size, as cramersV and phi do. For a confusion matrix, the sum of the diagonal divided by it is the accuracy.
cross-tabulationproportions
def grand(t) ≔ einsum("rc->", t)
rowMargin
rowMargin(t) · t table of counts r×c → row totals vector r
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t). Then, for \(j = 1, \dots, r\),
The count of each category of the first (row) variable.
Intuition Collapsing the table across its columns gives back the one-way frequency count of the row variable, as if the second variable had never been recorded.
Use it when you need the category frequencies of the first variable, or conditional distributions: divide each row by its total to get the column variable’s distribution within each row, such as the outcome rate per treatment. In a confusion matrix with predictions as rows, it counts how often each class was predicted.
cross-tabulationproportions
def rowMargin(t) ≔ einsum("rc->r", t)
columnMargin
columnMargin(t) · t table of counts r×c → column totals vector c
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t). Then, for \(k = 1, \dots, c\),
The count of each category of the second (column) variable.
Intuition The same collapse along the other axis: how often each category of the column variable occurs, ignoring the row variable.
Use it when you need the distribution of the second variable, such as the class balance of the true labels when they are the columns, or you divide each column by its total to condition on the column. In a confusion matrix with true classes as columns, a diagonal cell divided by its column total is that class’s recall.
cross-tabulationproportionsclassification-evaluation
def columnMargin(t) ≔ einsum("rc->c", t)
expected
expected(t) · t table of counts r×c → expected counts r×c
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) with row totals \(R_j\), column totals \(C_k\) and grand total \(N > 0\). Then
The counts independence predicts: the outer product of the margins scaled by the grand total. This is the table χ² compares the observed counts against.
Intuition If the variables are unrelated, the share of observations in cell \((j, k)\) is the row share times the column share, so its count is \(N \cdot (R_j/N)(C_k/N)\). This is what the table would look like with the same margins and no association at all.
Use it when you want to see which cells drive an association: cells where \(T - E\) is large and positive are over-represented, and the Pearson residual \((T - E)/\sqrt{E}\) puts them on a common scale. It also lets you check the usual rule of thumb for trusting χ²: expected counts of about 5 or more per cell.
hypothesis-testingassociation-measurement
def expected(t) ≔ einsum("r,c->rc", rowMargin(t), columnMargin(t)) / grand(t)
χ² and the strength of association
Pearson’s statistic sums the squared difference between observed and expected counts, each divided by the expected count. A large value is evidence against independence; its reference distribution is χ² with \((r-1)(c-1)\) degrees of freedom. Because χ² grows with the sample size, Cramér’s V and phi rescale it into a strength of association between 0 and 1.
chiSquared
chiSquared(t) · t table of counts r×c → χ² statistic scalar
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) and \(E = \operatorname{expected}(T)\). Then
Pearson’s χ² statistic for independence of the row and column variables. A cell whose expected count is zero has an empty row or column, so its numerator is zero too and it contributes nothing rather than a division by zero.
Intuition A total surprise score: for each cell, the squared gap between observed and expected counts, measured against the expected count, so a gap of 10 matters more where 20 were expected than where 2000 were. Under independence it follows χ² with freedom(t) degrees of freedom, so it is about that large when nothing is going on.
Use it when you test whether two categorical variables are associated: treatment and outcome, segment and churn, a feature and a class label. The p-value is 1 − chiCdf(freedom(t), chiSquared(t)) from Distributions. It needs reasonably large expected counts (see expected). It grows with \(N\), so do not read its size as strength; use cramersV for that.
hypothesis-testingassociation-measurementfeature-selection
def chiSquared(t) ≔ einsum("rc->", (t − expected(t)) · (t − expected(t)) / steady(expected(t)))
freedom
freedom(t) · t table of counts r×c → degrees of freedom scalar
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t). Then
The degrees of freedom of the χ² test, read from the table’s extents at run time.
Intuition How many cells can vary freely once the row and column totals are fixed: choose \((r - 1)(c - 1)\) cells and the margins force all the rest.
Use it when you look up a p-value or critical value for chiSquared with chiCdf or chiQuantile. It counts the table’s extents, including any row or column that is entirely zero, so remove empty categories first or the degrees of freedom are overstated and the p-value is too large.
hypothesis-testingcross-tabulation
def freedom(t) ≔ (rows(t) − 1) · (columns(t) − 1)
cramersV
cramersV(t) · t table of counts r×c → Cramér’s V scalar in [0, 1]
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) with \(r, c \ge 2\) and grand total \(N > 0\). Then
χ² made comparable across tables of different sizes: divided by the grand total and by the smaller table dimension less one. 0 means no association; 1 means each category of one variable determines the category of the other.
Intuition χ² divided by the largest value it can reach for this table, \(N \min(r-1, c-1)\), then square-rooted. The result is a strength of association on a fixed 0-to-1 scale, whatever the sample size or table shape.
Use it when you compare how strongly different pairs of categorical variables are associated (ranking features against a target, screening survey questions), or report an effect size next to a χ² test. It is symmetric, so it does not say which variable predicts which; for a direction use uncertainty. With small samples it is biased upward.
association-measurementeffect-sizefeature-selection
def cramersV(t) ≔ sqrt(chiSquared(t) / (grand(t) · min(rows(t) − 1, columns(t) − 1)))
phi
phi(t) · t table of counts, normally 2×2 → phi coefficient scalar
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t), normally with \(r = c = 2\), and grand total \(N > 0\). Then
On a 2×2 table it equals Cramér’s V (the minimum in the
denominator is one); on larger tables it is not bounded by 1, so prefer cramersV there. The value
is the magnitude of the association and carries no sign.
Intuition On a 2×2 table it is the size of the ordinary correlation between the two 0/1 variables, recovered from χ² (\(\phi^2 = \chi^2/N\)).
Use it when you measure association between two binary variables (exposed or not against ill or not, a flag against a binary label) and want a correlation-like number. It has no sign: for the direction compare the diagonal product \(T_{11}T_{22}\) with the off-diagonal \(T_{12}T_{21}\). For tables larger than 2×2 use cramersV, because phi can exceed 1 there.
association-measurementeffect-sizebinary-outcomes
def phi(t) ≔ sqrt(chiSquared(t) / grand(t))
Building blocks
steady(v)— Let \(v \in \mathbb{R}\) (applied entrywise to a tensor). Then \(\operatorname{steady}(v) = v + [v = 0]\), which is \(1\) at \(v = 0\) and \(v\) otherwise. Replaces exact zeros with 1, so a zero denominator or a logarithm of zero contributes nothing; used bychiSquared, the entropies and the normalised information measures.
Information
Entropy measures how spread out a categorical variable is: \(H = -\sum_k p_k \ln p_k\), which is 0 when every observation is in one category and \(\ln K\) when K categories are equally common. Mutual information is the entropy of the two margins less the entropy of the whole table; it is 0 exactly when the table equals the product of its margins, and grows as one variable tells you more about the other. All values are in nats.
entropy
entropy(v) · v counts (or unnormalised weights) per category vector k → Shannon entropy scalar
Let \(v \in \mathbb{R}^K\) be non-negative counts with \(\sum_{l} v_l > 0\), and \(p_l = v_l / \sum_{m=1}^{K} v_m\). Then
\[\operatorname{entropy}(v) = -\sum_{l=1}^{K} p_l \ln\big(p_l + [p_l = 0]\big) = -\sum_{l \,:\, p_l > 0} p_l \ln p_l .\]The entropy of the distribution the counts describe. The counts are first normalised to proportions; categories with a count of zero contribute nothing.
Intuition The average surprise of one draw: \(-\ln p\) for the category drawn, weighted by how often each category occurs. It is 0 when one category holds everything and largest, \(\ln K\), when all \(K\) are equally common; \(e^{H}\) reads as the effective number of categories.
Use it when you measure diversity or impurity: how mixed the classes are at a decision-tree node, how concentrated a market or a species count is, how uncertain a predicted distribution is. Raw counts are fine, since it normalises them. Results are in nats; divide by \(\ln 2\) for bits. An all-zero vector gives NaN.
diversity-measurementuncertainty-quantificationfeature-selection
def entropy(v) ≔ 0 − einsum("i->", share(v) · ln(steady(share(v))))
rowEntropy
rowEntropy(t) · t table of counts r×c → entropy of the row variable scalar
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) with row totals \(R_j\) and grand total \(N > 0\), and \(\rho_j = R_j / N\). Then
The entropy of the row margin.
Intuition The uncertainty about the row variable before you learn anything about the column: entropy applied to the row totals.
Use it when you need the baseline that information measures are judged against. It is the denominator of uncertainty and an upper bound on mutual, and when the rows are class labels it is the impurity of the parent node before a decision-tree split on the column variable.
diversity-measurementfeature-selection
def rowEntropy(t) ≔ 0 − einsum("r->", rowShare(t) · ln(steady(rowShare(t))))
columnEntropy
columnEntropy(t) · t table of counts r×c → entropy of the column variable scalar
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) with column totals \(C_k\) and grand total \(N > 0\), and \(\gamma_k = C_k / N\). Then
The entropy of the column margin.
Intuition The same baseline for the column variable: how unpredictable a column category is while the row is unknown.
Use it when the column variable is the one you want to predict and you need its baseline uncertainty, or you compute redundancy, which uses both margins. To ask how much the rows explain about the columns, call uncertainty on the table with rows and columns swapped; that divides by this entropy.
diversity-measurementfeature-selection
def columnEntropy(t) ≔ 0 − einsum("c->", columnShare(t) · ln(steady(columnShare(t))))
jointEntropy
jointEntropy(t) · t table of counts r×c → joint entropy scalar
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) with grand total \(N > 0\), and \(\pi_{jk} = T_{jk} / N\). Then
The entropy of the whole table treated as one distribution over cells.
Intuition The uncertainty of the pair (row, column) taken together. It is at most \(H(R) + H(C)\), with equality exactly when the variables are independent; the amount by which it falls short is the information the two share.
Use it when you compute mutual, or the variation of information \(2H(R, C) - H(R) - H(C)\), a true distance for comparing two clusterings of the same items (cross-tabulate the two sets of cluster labels). Empty cells contribute nothing.
clustering-comparisonassociation-measurement
def jointEntropy(t) ≔ 0 − einsum("rc->", cellShare(t) · ln(steady(cellShare(t))))
mutual
mutual(t) · t table of counts r×c → mutual information scalar
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) with grand total \(N > 0\). Then
with \(H(R) = \operatorname{rowEntropy}(T)\), \(H(C) = \operatorname{columnEntropy}(T)\) and \(H(R, C) = \operatorname{jointEntropy}(T)\).
The mutual information between the row and column variables: what the margins’ entropies promise less what the table’s joint entropy delivers. It is 0 for a table that is exactly the product of its margins (independence) and at most the smaller of the two margin entropies.
Intuition How many nats, on average, knowing one variable saves you when guessing the other. Unlike χ² it lives on an information scale: 0 under independence, never more than the smaller margin entropy, and indifferent to whether the dependence is monotone or not.
Use it when you rank categorical features by how informative they are about a label (the information gain of a decision-tree split is exactly this), detect dependence between discrete variables, or compare clusterings. It is not normalised, and variables with many categories tend to score higher, so use uncertainty or redundancy to compare across tables. Small samples bias it upward.
feature-selectionassociation-measurementclustering-comparison
def mutual(t) ≔ rowEntropy(t) + columnEntropy(t) − jointEntropy(t)
uncertainty
uncertainty(t) · t table of counts r×c → uncertainty coefficient scalar in [0, 1]
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) with grand total \(N > 0\). Then
The uncertainty coefficient (Theil’s U) of the row variable: the fraction of the row variable’s entropy that knowing the column explains, \(I(R; C) / H(R)\). It is not symmetric — swap rows and columns to measure the other direction. When the row variable has zero entropy, the result is 0.
Intuition The fraction of the row variable’s uncertainty that knowing the column removes: 0 when the column says nothing about the row, 1 when it determines the row completely. It is directional: a score for “how well does the column predict the row?”
Use it when you ask how well one categorical variable predicts another, for example how much of the churn outcome (rows) the plan (columns) explains, and want a number comparable across tables with different entropies. For the other direction swap rows and columns; for a symmetric score use redundancy. If the row variable has zero entropy the result is 0.
association-measurementfeature-selectioneffect-size
def uncertainty(t) ≔ mutual(t) / steady(rowEntropy(t))
redundancy
redundancy(t) · t table of counts r×c → symmetric uncertainty scalar in [0, 1]
Let \(T \in \mathbb{R}^{r \times c}\) (the argument t) with grand total \(N > 0\), and \(h = H(R) + H(C)\). Then
A symmetric normalisation of mutual information, so tables of different sizes can be compared: twice the mutual information divided by the sum of the two margin entropies (0 when both are zero).
Intuition Mutual information divided by the average of the two margin entropies: one symmetric 0-to-1 score for how much the variables share. Some texts reserve the name redundancy for \(I/(H(R) + H(C))\), which peaks at 1/2; this function returns twice that, the quantity usually called symmetric uncertainty.
Use it when you need a symmetric association score for categorical pairs, such as feature selection that penalises features redundant with one another, or comparing two clusterings of the same items. For a directional question use uncertainty. When both margins have zero entropy the result is 0.
feature-selectionassociation-measurementclustering-comparison
def redundancy(t) ≔ 2 · mutual(t) / steady(rowEntropy(t) + columnEntropy(t))
Building blocks
share(v)— Let \(v \in \mathbb{R}^K\) with \(\sum_{m} v_m \ne 0\). Then \(\operatorname{share}(v)_l = v_l / \sum_{m=1}^{K} v_m\). Normalises a vector of counts to proportions summing to 1; used byentropy.rowShare(t)— Let \(T \in \mathbb{R}^{r \times c}\) with row totals \(R_j\) and grand total \(N > 0\). Then \(\operatorname{rowShare}(T)_j = \rho_j = R_j / N\). The row margin as proportions of the grand total; used byrowEntropy.columnShare(t)— Let \(T \in \mathbb{R}^{r \times c}\) with column totals \(C_k\) and grand total \(N > 0\). Then \(\operatorname{columnShare}(T)_k = \gamma_k = C_k / N\). The column margin as proportions of the grand total; used bycolumnEntropy.cellShare(t)— Let \(T \in \mathbb{R}^{r \times c}\) with grand total \(N > 0\). Then \(\operatorname{cellShare}(T)_{jk} = \pi_{jk} = T_{jk} / N\). Every cell as a proportion of the grand total; used byjointEntropy.
Examples
All examples below are compiled from one file, Tables.hb, so the C# class is Tables. Matrices are
passed through the Tensor overloads (the (Tensor) cast), because these functions change rank —
two matrices in, a scalar or a smaller table out.
Cross-tabulate two categorical columns
Six customers, each with a plan (basic, premium) and a region (north, south, west), one-hot encoded.
open Prelude.Categorical
fn crossTab(a, b) ↦ contingency(a, b)
using MLambda.Hilbert.Runtime;
using Examples;
// plan: basic, premium
Tensor plan = (Tensor)new double[,] { { 1, 0 }, { 1, 0 }, { 1, 0 }, { 0, 1 }, { 0, 1 }, { 1, 0 } };
// region: north, south, west
Tensor region = (Tensor)new double[,] { { 1, 0, 0 }, { 1, 0, 0 }, { 0, 1, 0 }, { 0, 0, 1 }, { 0, 1, 0 }, { 1, 0, 0 } };
Tensor table = Tables.CrossTab(plan, region);
Output (verified):
shape [2, 3] values [3, 1, 0, 0, 1, 1]
Three basic customers are in the north and one in the south; the premium customers are one in the south and one in the west.
Test a 2×2 table for independence
open Prelude.Categorical
fn independentCounts(t) ↦ expected(t)
fn chiStatistic(t) ↦ chiSquared(t)
fn degrees(t) ↦ freedom(t)
fn association(t) ↦ cramersV(t)
fn phiOf(t) ↦ phi(t)
Tensor t = (Tensor)new double[,] { { 20, 30 }, { 30, 20 } };
Tensor e = Tables.IndependentCounts(t);
Tensor chi = Tables.ChiStatistic(t);
Tensor df = Tables.Degrees(t);
Tensor v = Tables.Association(t);
Tensor ph = Tables.PhiOf(t);
Output (verified):
expected = shape [2, 2] values [25, 25, 25, 25]
chiSquared = 4
freedom = 1
cramersV = 0.2
phi = 0.2
Every cell is 5 away from its expected 25, so χ² is \(4 \times 25/25 = 4\) on one degree of freedom, and \(V = \sqrt{4/100} = 0.2\).
Measure association on a larger table
open Prelude.Categorical
fn chiStatistic(t) ↦ chiSquared(t)
fn degrees(t) ↦ freedom(t)
fn association(t) ↦ cramersV(t)
Tensor t = (Tensor)new double[,] { { 30, 10, 10 }, { 10, 30, 10 } };
Tensor chi = Tables.ChiStatistic(t);
Tensor df = Tables.Degrees(t);
Tensor v = Tables.Association(t);
Output (verified, rounded to six significant digits):
chiSquared = 20
freedom = 2
cramersV = 0.447214
The expected counts are 20, 20 and 10 in each row, so χ² is \(4 \times 100/20 = 20\) and \(V = \sqrt{20/100} \approx 0.4472\).
Entropy of a categorical distribution
open Prelude.Categorical
fn surprise(v) ↦ entropy(v)
Tensor fair = Tables.Surprise((Tensor)new[] { 25d, 25d, 25d, 25d });
Tensor empty = Tables.Surprise((Tensor)new[] { 5d, 5d, 0d });
Output (verified, rounded to six significant digits):
four equal categories = 1.386294
[5, 5, 0] = 0.693147
These are \(\ln 4\) and \(\ln 2\): the empty third category adds nothing.
Mutual information of a dependent and an independent table
open Prelude.Categorical
fn sharedInformation(t) ↦ mutual(t)
fn uncertaintyOf(t) ↦ uncertainty(t)
fn redundancyOf(t) ↦ redundancy(t)
Tensor perfect = (Tensor)new double[,] { { 10, 0 }, { 0, 10 } };
Tensor independent = (Tensor)new double[,] { { 10, 20 }, { 5, 10 } };
Tensor i1 = Tables.SharedInformation(perfect);
Tensor u1 = Tables.UncertaintyOf(perfect);
Tensor r1 = Tables.RedundancyOf(perfect);
Tensor i0 = Tables.SharedInformation(independent);
Output (verified, rounded to six significant digits):
mutual (perfect) = 0.693147
uncertainty (perfect) = 1
redundancy (perfect) = 1
mutual (independent) = 0
When the column determines the row, the mutual information equals the row entropy (\(\ln 2\)) and both normalised measures reach 1; the second table is exactly the product of its margins, so it carries no information.