The examples above use a toy vocabulary to show that an embedding layer's parameter count is simply the number of words times the number of dimensions. The sections below generalize that idea: how embedding rows are learned, how the same arithmetic extends to whole transformer models, and what parameter counts mean for memory.
Feature Dimensions: [Object, Animal]
cat → [0, 1] → 2 Dimensions
dog → [0, 1] → 2 Dimensions
car → [1, 0] → 2 Dimensions
-------------------------------------
Total: → 3 × 2 = 6 Parameters
+---------------------------------------+
| Vector DB: with Feature Dimensions |
+--------+--------+--------+------------+
| Word | Object | Animal | Dimensions |
+--------+--------+--------+------------+
| cat | 0 | 1 | 2 |
| dog | 0 | 1 | 2 |
| car | 1 | 0 | 2 |
+--------+--------+--------+------------+
| Total | 3 × 2 = 6 Parameters |
+--------+--------+--------+------------+
Bridging the example to real models:
+------------------+------------+------------+------------------+
| Model | Words | Dimensions | Parameters |
+------------------+------------+------------+------------------+
| This example | 3 | 2 | 6 |
| GPT-2 embeddings | 50,257 | 768 | 38,597,376 |
+------------------+------------+------------+------------------+
import numpy as np
embedding_table = {
#--- Feature Dim: [Object, Animal] ---#
"cat": np.array([ 0 , 1 ]),
"dog": np.array([ 0 , 1 ]),
"car": np.array([ 1 , 0 ]),
}
def embed(word):
return embedding_table[word]
print(embed("cat"))
print(embed("dog"))def compute_embedding(animal_score, object_score):
# Combine two numeric features into a vector
return [animal_score, object_score]
cat_embedding = compute_embedding(0, 1)
dog_embedding = compute_embedding(0, 1)
print("cat:", cat_embedding)
print("dog:", dog_embedding)An embedding layer is a matrix with one row per token in the vocabulary and one column per dimension. Looking up a word means selecting its row by integer id, which is equivalent to multiplying a one-hot vector by the matrix. Every cell is a trainable parameter. The hand-assigned numbers on this page make the dimensions interpretable, for example "animal-ness" and "vehicle-ness", but in a trained model the values are learned by gradient descent and individual dimensions rarely have a clean meaning. What matters is geometry: words used in similar contexts end up with nearby vectors, so cosine similarity between "cat" and "dog" is high and between "cat" and "car" is low.
The same counting rule applies to every weight matrix: a layer mapping m inputs to n outputs has m x n weights plus n biases. In a standard transformer block with hidden size d, the attention projections for queries, keys, values, and output contribute about 4d^2 parameters, and a feed-forward network with a 4x expansion contributes about 8d^2, so each block has roughly 12d^2. A model's total is then approximately vocab x d + layers x 12d^2. For GPT-2 small, with a vocabulary of 50,257, d = 768 and 12 layers, that estimate gives about 124 million, matching its published size. Variants that use gated MLPs or grouped-query attention change the constants but not the method.
import torch.nn as nn
emb = nn.Embedding(num_embeddings=3, embedding_dim=2) # cat, dog, car
print(sum(p.numel() for p in emb.parameters())) # 6
def transformer_estimate(vocab, d, layers):
return vocab * d + layers * 12 * d * d
print(f"{transformer_estimate(50_257, 768, 12):,}") # about 124 million
block = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072)
print(f"{sum(p.numel() for p in block.parameters()):,}") # about 7.1 million (12 d^2 + biases, norms)
def memory_gb(params, bytes_per_param=2): # fp16 / bf16 weights
return params * bytes_per_param / 1e9
print(memory_gb(7e9)) # 14.0 GB just to hold a 7B model in 16-bit| Precision | Bytes per parameter | 7B model weights |
|---|---|---|
| fp32 | 4 | about 28 GB |
| fp16 / bf16 | 2 | about 14 GB |
| int8 | 1 | about 7 GB |
| 4-bit | 0.5 | about 3.5 GB |
These figures cover weights only. Inference also needs memory for activations and the key-value cache, which grows with context length and batch size. Training with the Adam optimizer typically needs several times the weight memory, because gradients and two optimizer moments are stored per parameter, usually in higher precision.
d = 1024 is already 256 million parameters.