What is a tensor?

Every value in Hilbert may be a tensor. A price is a tensor, a column of prices is a tensor, a table of prices by store and month is a tensor, and a year of such tables is a tensor too. This page explains what that means, why mathematicians and physicists care, and how Hilbert uses it.

From numbers to tensors

The easiest way in is to count how many indices you need to point at one number.

Rank	Name	You point at a number with	Example	Shape
0	scalar	nothing	a temperature, `21.5`	`[]`
1	vector	one index i	daily sales for a week	`[7]`
2	matrix	two indices i, j	a grayscale image, a covariance matrix	`[480, 640]`
3	tensor	three indices i, j, k	a colour image (height × width × channel)	`[480, 640, 3]`
4	tensor	four indices	a batch of 32 colour images	`[32, 480, 640, 3]`

The number of indices is the tensor’s rank (also called its order), and the list of extents is its shape. A vector with shape [3] is written

\[v = \begin{pmatrix} v_1 \\ v_2 \\ v_3 \end{pmatrix}, \qquad v_i \text{ for } i = 1, 2, 3,\]

a matrix with shape [2, 2] has entries \(A_{ij}\), and a rank-3 tensor has entries \(T_{ijk}\). Nothing stops at three: rank is simply how many axes the numbers are arranged along.

In Hilbert the literals follow the same picture:

def price ≔ 21.5                                   -- rank 0
def week  ≔ [3, 5, 4, 6, 8, 9, 7]                  -- rank 1, shape [7]
def grid  ≔ [1, 2; 3, 4]                           -- rank 2, shape [2, 2]
def cube  ≔ [[[1, 2], [3, 4]], [[5, 6], [7, 8]]]   -- rank 3, shape [2, 2, 2]

The physicist’s tensor

To a data scientist, a tensor is an array with axes — and for computation that is exactly right. The word comes from physics, though, where it means something stricter and more beautiful.

Push on a block of rubber and it deforms. The force per unit area on a surface inside the block depends on which way the surface faces: feed in the direction of the surface, and out comes the force on it. That rule — a machine that takes a direction and returns a vector — is the stress tensor \(\sigma_{ij}\). Written down in one coordinate system it is a 3×3 table of numbers. Rotate your axes and every number in the table changes, yet the physical fact it describes does not change at all.

That is the physicist’s definition: a tensor is a quantity whose components change, when you change coordinates, in exactly the way that keeps the thing itself the same. For a rank-2 tensor under a change of coordinates with matrix \(R\):

\[\sigma'_{ij} = \sum_{k} \sum_{l} R_{ik}\, R_{jl}\, \sigma_{kl}.\]

The inertia tensor of a spinning body, the electromagnetic field tensor, the curvature of space-time in general relativity — all of them are tensors in this sense. Laws of nature written with tensors hold in every coordinate system at once, which is why Einstein needed them. Notice how many summation signs that one small transformation law already carries; that is where the story of this language’s central operation begins.

Indices say what to multiply and what to sum

Almost everything we do with vectors and matrices is a pattern of multiplying entries that share an index and summing over some of those indices:

Operation	With sums	Indices kept
dot product	\(u \cdot v = \sum_i u_i v_i\)	none — a scalar
matrix times vector	\((Ax)_i = \sum_j A_{ij} x_j\)	i
matrix product	\((AB)_{ik} = \sum_j A_{ij} B_{jk}\)	i, k
outer product	\((u v^{\top})_{ij} = u_i v_j\)	i, j
trace	\(\operatorname{tr} A = \sum_i A_{ii}\)	none

Reading these as index patterns rather than as five different operations is the insight Einstein turned into notation, and it is how Hilbert computes all of them — with a single operation, einsum. See Einstein’s summation for the history and einsum in practice for the details.

Elementwise arithmetic and broadcasting

Arithmetic operators (+ − · /, ^) and the elementwise intrinsics (sin, exp, sqrt, comparisons…) act entry by entry. [1, 2] · [3, 4] is [3, 8], not a dot product.

When the two shapes differ, Hilbert broadcasts them the way NumPy and ONNX do: shapes are aligned from the right, and an axis that is missing or has extent 1 is repeated to match.

def grid ≔ [1, 2; 3, 4]

fn scaledGrid(k) ↦ grid · k       -- a scalar repeats everywhere
fn shifted(v)    ↦ grid + v       -- a vector repeats along the rows

Shapes.ScaledGrid(3);                  // [[3, 6], [9, 12]]
Shapes.Shifted(new[] { 10d, 20d });    // [[11, 22], [13, 24]]

Masks instead of branches

A comparison is a tensor of ones and zeros, so selection is multiplication. That is how the Prelude expresses things that would be if statements elsewhere: “the k-th column”, “entries above the threshold”, “the chosen action” are all masks.

fn clipped(x) ↦ min(max(x, 0), 1)                  -- [−2, 0.5, 3] gives [0, 0.5, 1]
fn above(x, t) ↦ x · (x > t)                       -- keep entries above t, zero the rest

Why one kind of value

Because a real number is just a rank-0 tensor, a Hilbert formula never has to say which kind of value it expects. fn sum(x, y) ↦ x + y is compiled once per shape — plain C# for reals, an ONNX graph for everything with axes — and the same formula serves a price, a price list and a price cube. See Calling from C# for the overloads that are generated.

Shape matters, even for zero. x^0 is a tensor of ones shaped like x, and 0 has no shape at all. That is why the Prelude builds a vector of ones as ones(x) ≔ x^0 and why the arithmetic laws never rewrite x^0 to 1 — the shape would be lost. See Arithmetic.