Skip to main content

Tensors

Tensors​

A tensor is a mathematical object that is invariant under a change of basis. However, they have components that transform in a specific way under a change of basis. In this page, we will introduce the concept of tensors and build up our understanding of them. We will first explore some examples of tensors, like covectors and linear maps. Then, we will formalize our understanding of tensors with tensor products.

Table of Contents​

Covectors​

A covector is a linear map that takes a vector and returns a scalar. They have many names—dual vectors, one-forms, linear functionals, etc.—but they all refer to the same concept. If you have read the quantum mechanics section, you will know that the bra vector is a covector. The Riesz representation theorem states that every covector can be represented as an inner product with a vector. We will re-introduce the concept of covectors, this time in the context of tensors.

Let be a covector. When applied to a vector , it returns a scalar, denoted by . They are represented by a row vector. For a simple example, consider the covector . When applied to the vector , we get:

Suppose now we apply to a sum of vectors , where . This is given by:

Notice that this is the same as if we applied to and separately and then added the results:

Additionally, suppose we apply to . This is given by:

Notice that this is the same as if we applied to :

From these examples, we can see that covectors are linear maps, meaning that they satisfy the following properties:

Hence, more formally, we define covectors symbolically as:

where is the vector space and is the field that the vector space is defined over. The set of all covectors on a vector space is called the dual space of , denoted by .

Index Notation for Covectors​

When we apply a covector to a vector, we take each component of the covector and multiply it by the corresponding component of the vector. For a covector , we say that , , and . As such, when we apply to a vector, we get:

Visualizing Covectors​

Suppose we have covectors acting on two-dimensional vectors. We can imagine a covector as a surface, where the height of the surface at a point is given by the value of the covector at that point. For example, consider the covector . This is visualized as a plane with a slope of in the -direction and in the -direction:

However, these can be difficult to visualize and draw, especially in higher dimensions. We instead prefer to use a series of lines to represent covectors, akin to a contour plot. Each line represents a constant value of the covector. Then, the value of on a vector is given by how many lines intersects.

Alternatively, instead of writing the number on each line, we simply draw an arrow pointing in the direction of increasing values of the covector.

Covector Arithmetic​

Covectors can be added and multiplied by scalars in the same way as vectors. This is because the dual space is itself a vector space.

When a covector is scaled, say , the contour lines look more densely packed. When a covector is added to another, say , the contour lines are the sum of the two covectors. These operations follow linearity rules:

Covector Basis and Components​

Just like vectors, covectors have a basis. In order to define a basis, they should be linearly independent and span the entire dual space . We define them using the following: if has a basis , then has a basis such that:

where is the Kronecker delta. Thus a covector can be written as:

Notice that unlike vector bases, covector bases have a superscript index—covector bases are contravariant. To see this more concretely, let be the transformation matrix from an old basis to a new basis. Suppose we apply a covector to a vector . In the old basis, we have:

We know that we can insert the identity matrix in between the two matrices without changing the result. As such, we can insert :

We already know that vector components are contravariant and transform with the inverse matrix. As such, :

Now, let's expand in the new basis:

Comparing the two expressions, we find that . Hence, covector components are covariant and transform with the transformation matrix. And by the fact that covectors are invariant, we have that the covector basis transforms with the inverse matrix.

Linear Maps​

See Linear Transformations for a more detailed introduction to linear maps.

See Matrix Representation of Operators and Change of Basis for an introduction in the context of quantum mechanics.

A more general concept than covectors is that of a linear map. It is defined as a map of , where and are vector spaces. Geometrically, when we apply a linear map to every point on a grid, gridlines remain parallel and evenly spaced, and the origin remains fixed. Algebraically, a linear map satisfies the following linearity properties:

Linear maps can be represented by matrices:

where each column represents the image of the basis vector under . Applying to a vector yields:

We can see that the components of the image of are given by the matrix-vector product. In other words, if , then .

Linear Maps under Change of Basis​

When we change the basis of the vector space, the components of vectors and linear maps change:

Intuitively, to transform a vector by under a new basis , we first transform back to the old basis, apply , and then transform the result back to the new basis:

To prove this more formally, we will introduce the Einstein summation convention. This states that when an index appears twice in a term, it is implicitly summed over. For example, is equivalent to . This is useful for simplifying expressions. See Change of Basis for a different proof.

Let be the transformation matrix from the old basis to the new basis. The components of a new basis vector under a linear map are given by:

with an implicit sum over . We also know that, by definition:

On the left-hand side, we have a sum over , and on the right-hand side, we have a sum over . We can rename the indices on the left-hand side :

Thus:

Indices vs Matrices​

Suppose we have three matrices, , , and , and we want to multiply them. Typically, we simply write .

However, we can also write this as . Notice that the indices and are repeated, and hence we sum over them. The question is, is there a way to quickly derive these expressions from the matrix multiplication? It turns out, there is a heuristic way to do this. Looking back at the indices, we see that the repeated indices connect diagonally:

They "cancel out" and we are left with the indices that are not repeated— and . This also helps us make sure that we are multiplying the matrices in the correct order. For instance, in the proof for the transformation of linear maps, we see that the indices are connected diagonally:

This is how we know that the entire matrix is given by and not any other order.

Metric Tensor and Bilinear Forms​

We will now introduce the concept of a metric tensor. Suppose we want to find out the dot product of two vectors and . We can write this as:

Because the dot product is linear, we can expand this out:

We are left with a sum of terms of the form . In Einstein summation notation this is simply:

The dot product of the basis vectors is defined by the metric tensor . Notice that we can immediately see that the metric tensor must have two covariant indices because the dot product must be invariant under a change of basis. We say that the metric tensor is twice-covariant.

While is the -component of the metric tensor, the tensor itself is denoted by and is defined as:

Because the dot product is linear, the metric tensor is also linear. Furthermore, it takes two inputs, so it is more formally written as . This is known as a bilinear form;

  • Bilinear: Linear in both arguments.
  • Form: A function that takes vectors and outputs a scalar. For example, covectors are 1-forms.

Tensor Products​

If you have read the quantum mechanics section, you will know that an operator can be written as a sum of projection operators. This is known as the spectral decomposition of an operator. The fundamental idea is that the outer product of a vector and covector gives a linear map:

To see why this is the case, matrix multiplication can be carried out as follows:

  • Move the second matrix upwards and draw an empty matrix where it used to be.
  • Each element in the empty matrix is the product of the corresponding elements in the two matrices.

Here's a concrete example:

To multiply them, we can shift up and to the right:

Then, the elements in the product matrix are given by the sum of the products of the corresponding elements. For example, the element in the first row and first column is:

For an outer product, it looks like this:

Since the outer product involves scaling one vector by two different scalars (in the covector), each column is a scaled version of the vector. As such, the matrix has a determinant of zero; its columns are linearly dependent. These matrices are called pure matrices.

In order to write any linear map in a similar way, we could combine pure matrices together. In a sense, the pure matrices act as a "basis" for all linear maps. We can make the following four pure matrices the bases:

Then, any matrix can be written as a sum of these four matrices:

And hence, we can write any linear map as:

Formally, each basis pure matrix is the tensor product of the basis vectors and covectors. They are more formally written as . As such:

In fact, a tensor can be defined as a set of vectors and covectors that are combined using tensor products. Defining a tensor as such makes it trivial to derive the transformation rules for tensors under a change of basis. For example, for a linear map as above, the components transform as:

Thus it is clear that transforms as .

To take another example of a tensor product, consider two qubits (or two spin-1/2 particles). The state of a single qubit is given by a vector in a two-dimensional complex vector space, . The vector is written as a superposition of the two basis states and :

Now suppose we have a composite system of these qubits. There are now four basis states, each corresponding to a different combination of the two qubits:

  • : both qubits are in the state .
  • : the first qubit is in the state and the second qubit is in the state .
  • : the first qubit is in the state and the second qubit is in the state .
  • : both qubits are in the state .

These are actually the tensor products of the two qubit states:

And the state of the composite system is given by a linear combination of these basis states:

Kronecker Product​

Tensor products and Kronecker products are similar but not the same. Tensor products act on tensors, while Kronecker products act on arrays. However, they are essentially identical in our context, so we will use the terms interchangeably.

To take an example, a bilinear form is a tensor product of two covectors:

To apply the tensor product, we simply give a copy of the left object for each element in the right object: