Einstein’s summation

Every definition in the Prelude — the mean of a column, the inverse of a matrix, a neural network, the value of a policy — is eventually written with one operation, einsum. Its name is short for Einstein summation. This is the story of where it comes from.

A physicist who needed new mathematics

In 1907 Albert Einstein had what he later called the happiest thought of his life: a person falling freely does not feel their own weight. If gravity and acceleration are locally the same thing, then a theory of gravity has to look the same to every observer, however they move — and its laws have to be written in a form that does not depend on the coordinates anyone happens to choose.

Einstein did not have the mathematics for that. It already existed, though. Between 1884 and 1900 the Italian geometer Gregorio Ricci-Curbastro had built an absolute differential calculus — a way of doing calculus on curved spaces whose results do not depend on coordinates — and in 1900 he and his former student Tullio Levi-Civita published its definitive account, Méthodes de calcul différentiel absolu et leurs applications, in Mathematische Annalen. Today we call it tensor calculus. Hardly any physicist had read it.

Grossmann, Levi-Civita and Hilbert

In 1912 Einstein returned to Zurich as a professor at the ETH, where his old classmate Marcel Grossmann was now a professor of mathematics. Grossmann had once lent Einstein his lecture notes; now, as the story is usually told, Einstein asked him for help with the geometry he was missing. Grossmann pointed him to Ricci and Levi-Civita. In 1913 the two friends published the Entwurf (“outline”) of a generalised theory of relativity — Einstein wrote the physics, Grossmann the mathematics — and for the first time gravity was described by a tensor, the metric \(g_{\mu\nu}\).

The outline was not yet right, and the path to the final theory ran through more collaboration. In 1915 Einstein corresponded with Levi-Civita himself, who found weaknesses in his arguments about covariance. That summer David Hilbert — the mathematician whose name this language carries — invited Einstein to Göttingen to lecture on the theory, and both men raced towards the field equations. Hilbert submitted his paper Die Grundlagen der Physik on 20 November 1915; Einstein presented the final equations to the Prussian Academy on 25 November:

\[R_{\mu\nu} - \tfrac{1}{2} R\, g_{\mu\nu} = \kappa\, T_{\mu\nu}.\]

General relativity was, from its first line, a collaboration between a physicist and mathematicians — Ricci-Curbastro and Levi-Civita who built the calculus, Grossmann who taught it, Hilbert who pursued the same equations from the other side.

A sea of sigmas

Tensor calculus is written with indices, and indices come with sums. In four-dimensional space-time an index runs over 0, 1, 2, 3, so the square of a small interval of space-time is

\[ds^2 = \sum_{\mu=0}^{3} \sum_{\nu=0}^{3} g_{\mu\nu}\, dx^\mu\, dx^\nu,\]

sixteen terms under two summation signs. Changing coordinates on a rank-2 tensor needs two more:

\[g'_{\mu\nu} = \sum_{\alpha=0}^{3} \sum_{\beta=0}^{3} \frac{\partial x^\alpha}{\partial x'^\mu}\,\frac{\partial x^\beta}{\partial x'^\nu}\, g_{\alpha\beta}.\]

The Riemann curvature tensor, the Christoffel symbols and the field equations multiply this out many times over. Page after page, the \(\Sigma\) signs said nothing the indices did not already say.

The convention

In his great 1916 exposition of the theory, Die Grundlage der allgemeinen Relativitätstheorie (Annalen der Physik 49), Einstein simply stopped writing them. He stated a rule: when an index appears twice in a term, it is summed over, and the summation sign is left out.

\[ds^2 = g_{\mu\nu}\, dx^\mu\, dx^\nu \qquad\qquad g'_{\mu\nu} = \frac{\partial x^\alpha}{\partial x'^\mu}\,\frac{\partial x^\beta}{\partial x'^\nu}\, g_{\alpha\beta}\]

That is the whole idea. A remark often attributed to him puts it with his usual humour: “I have made a great discovery in mathematics; I have suppressed the summation sign every time that the summation must be made over an index which occurs twice.”

It sounds like shorthand. It is much more than that. The rule separates two kinds of index:

a repeated (dummy) index is summed away — it is internal to the expression;
a free index appears once — it survives and says what shape the result has.

So an expression now declares its own shape. \(u_i v_i\) has no free index: it is a number. \(A_{ij} x_j\) has one free index, \(i\): it is a vector. \(A_{ij} B_{jk}\) has two: it is a matrix. The same rule states the dot product, matrix multiplication, the trace \(A_{ii}\), the outer product \(u_i v_j\) and contractions of tensors of any rank — not as five operations but as one.

A gift to mathematics

A notation is a tool for thinking, and few tools have travelled as far as this one. It is the ordinary language of differential geometry, continuum mechanics, electromagnetism and quantum field theory; it underlies Roger Penrose’s abstract index notation; and every student of relativity learns it in the first week. It is arguably the most widely used contribution to mathematical notation ever made by a physicist — a single sentence that removed a symbol from mathematics and made the mathematics clearer.

Then computing picked it up. In 2011 the NumPy library added numpy.einsum, which takes the convention literally as a string: "ij,jk->ik" means “multiply entries whose j agrees, sum over j, keep i and k”. The idea spread to every major machine-learning framework, and the ONNX standard — the portable format for numerical programs — includes an Einsum operator. A convention written for the curvature of space-time now multiplies the matrices inside neural networks.

Why Hilbert is built on it

Hilbert takes the convention one step further: einsum is the only contraction the language has. There is no built-in dot product, matrix product, transpose, trace, mean or covariance. Each is a one-line definition in the Prelude, and each says in its index string exactly what is multiplied and what is summed:

def transpose(a) ≔ einsum("ij->ji", a)
def mul(a, b)    ≔ einsum("ij,jk->ik", a, b)
def apply(a, x)  ≔ einsum("ij,j->i", a, x)
def outer(x, y)  ≔ einsum("i,j->ij", x, y)
def dot(x, y)    ≔ einsum("i,i->", x, y)

(All five are from Prelude.LinearAlgebra.) The mean of a column in Prelude.Statistics is the same idea: a total, einsum("i->", x), divided by rows(x).

This has practical consequences:

One operation to trust. Every algorithm in the Prelude reduces to contractions, a few elementwise maps and two reductions (minimum, maximum). There is no hidden numerical kernel.
It runs anywhere. einsum is an ONNX operator, so every formula lowers to a portable ONNX graph that ONNX Runtime can execute on CPU or accelerators.
Shapes are self-describing. As in Einstein’s paper, the free indices of the string say what comes back — "ni,nj->ij" over a data matrix is, visibly, a p × p Gram matrix.

The notation that let Einstein write gravity in a line lets you write a covariance matrix, an attention layer or a Bellman backup in a line too. Continue with einsum in practice to use it.

Further reading. A. Einstein, Die Grundlage der allgemeinen Relativitätstheorie, Annalen der Physik 49 (1916). G. Ricci and T. Levi-Civita, Méthodes de calcul différentiel absolu et leurs applications, Mathematische Annalen 54 (1900). A. Einstein and M. Grossmann, Entwurf einer verallgemeinerten Relativitätstheorie und einer Theorie der Gravitation (1913). D. Hilbert, Die Grundlagen der Physik, Nachrichten der Gesellschaft der Wissenschaften zu Göttingen (1915).