In essence, a tensor is a multidimensional array used to represent data. Think of it as a generalization of vectors and matrices.
Three properties describe every tensor. The rank (or number of dimensions, ndim) is how many indices you need to address one element. The shape lists the length of each axis. The dtype is the type of every element; deep learning mostly uses 32-bit floats, with 16-bit formats (float16, bfloat16) for faster training and inference, and integers for labels and token IDs. Tensors are stored as one contiguous block of memory plus strides, the number of elements to skip to move one step along each axis. This is why operations like transpose and reshape are often free: they change the strides or shape metadata, not the data.
Tensors are the fundamental data structure in deep learning frameworks like TensorFlow and PyTorch. They allow us to efficiently represent and process complex data like images (3D tensors: height, width, color channels), videos (4D tensors: time, height, width, color channels), and even natural language (where words are embedded in high-dimensional spaces).
NumPy is a powerful library for numerical computations in Python and provides excellent support for working with tensors (which NumPy calls 'ndarrays').
import numpy as np
# Create a 2D tensor (a matrix)
matrix = np.array([[1, 2, 3],
[4, 5, 6]])
# Create a 3D tensor
tensor_3d = np.array([[[1, 2], [3, 4]],
[[5, 6], [7, 8]]])
# Access elements of the tensors
print("Element at row 1, column 2 of the matrix:", matrix[1, 2])
print("Element at depth 0, row 1, column 0 of the 3D tensor:", tensor_3d[0, 1, 0])
# Perform operations on tensors
sum_of_matrix = np.sum(matrix)
print("Sum of all elements in the matrix:", sum_of_matrix)
# Output
# Element at row 1, column 2 of the matrix: 6
# Element at depth 0, row 1, column 0 of the 3D tensor: 3
# Sum of all elements in the matrix: 21
In this code:
matrix) and a 3D tensor (tensor_3d) using np.array.matrix using np.sum.
import numpy as np
# create a 3D tensor with dimensions (2, 3, 4)
tensor = np.zeros((2, 3, 4))
# fill the tensor with some values
tensor[0, 0, 0] = 1
tensor[0, 1, 1] = 2
tensor[0, 2, 2] = 3
tensor[1, 0, 3] = 4
tensor[1, 1, 2] = 5
tensor[1, 2, 1] = 6
# print the tensor
print(tensor)
# access a specific value in the tensor (e.g. the value at index (1, 0, 3))
value = tensor[1, 0, 3]
print(value)
# sum all the values in the tensor
total = np.sum(tensor)
print(total)
# compute the mean of all the values in the tensor
average = np.mean(tensor)
print(average)
In this example, we create a 3D tensor with dimensions (2, 3, 4) using the `np.zeros()` function. We then fill the tensor with some values using indexing. We print the tensor, access a specific value, compute the sum and mean of all the values in the tensor using the `np.sum()` and `np.mean()` functions.
Note that 3D tensors are also commonly used in deep learning frameworks such as TensorFlow or PyTorch. The syntax for creating and manipulating 3D tensors in those frameworks is similar to NumPy, but with additional functionality for building and training machine learning models.
Element-wise operations between tensors of different shapes follow broadcasting rules, shared by NumPy, PyTorch, TensorFlow and JAX. Shapes are aligned on the right; each pair of axis sizes must be equal, or one of them must be 1, and a missing leading axis counts as 1. The size-1 axes are virtually repeated. Broadcasting lets you write X - X.mean(axis=0) to centre every column or W @ x + b for a layer, without loops and without materialising copies. It is also a common source of silent bugs, when shapes such as (n,) and (n, 1) broadcast to an unintended (n, n) result.
Framework tensors add two things to NumPy arrays: they can live on a GPU or other accelerator, and they can record the operations applied to them so gradients can be computed automatically.
import torch
x = torch.tensor([[1., 2., 3.], [4., 5., 6.]]) # shape (2, 3), float32
print(x.shape, x.dtype, x.ndim) # torch.Size([2, 3]) torch.float32 2
w = torch.randn(3, 1, requires_grad=True) # trainable weights
b = torch.zeros(1, requires_grad=True)
y = x @ w + b # (2,3)@(3,1) -> (2,1), b broadcast
loss = (y ** 2).mean()
loss.backward() # autograd fills w.grad, b.grad
print(w.grad.shape) # torch.Size([3, 1])
device = "cuda" if torch.cuda.is_available() else "cpu"
x_gpu = x.to(device) # move data to the accelerator
images = torch.rand(32, 3, 224, 224) # batch of RGB images (N, C, H, W)
flat = images.reshape(32, -1) # (32, 150528) for a linear layer
print(images.permute(0, 2, 3, 1).shape) # channels-last view (N, H, W, C)
Conversion between NumPy and PyTorch on the CPU (torch.from_numpy, tensor.numpy()) shares memory rather than copying it, so modifying one changes the other.
| Data | Typical shape | Axes |
|---|---|---|
| Tabular batch | (N, F) | samples, features |
| Text tokens | (N, T) | sequences, token positions (integer IDs) |
| Token embeddings | (N, T, D) | sequences, positions, embedding size |
| Images (PyTorch) | (N, C, H, W) | batch, channels, height, width |
| Images (TensorFlow default) | (N, H, W, C) | channels last |
| Video | (N, T, C, H, W) | adds frames |
| Attention weights | (N, heads, T, T) | one score per token pair per head |
.shape at each step while developing.view requires contiguous memory; use reshape or call .contiguous() after a transpose.