Graph Attention Networks (Veličković et al., 2018) bring the attention mechanism to graph neural networks. Where a GCN weights each neighbor by a fixed function of node degrees, a GAT layer learns how much each neighbor should contribute, based on the features of both endpoints. The weights are computed per edge, normalized with a softmax over each node's neighborhood, and averaged over several independent attention heads. The result handles noisy or heterogeneous neighborhoods better, works inductively on unseen graphs, and is the conceptual bridge between GNNs and Transformer self-attention.
In a GCN the aggregation weight on edge (i, j) depends only on degrees, so every neighbor of equal degree counts the same no matter what it is. In real graphs, some neighbors are much more informative than others: a citation to a foundational paper versus an incidental one, a known fraudster in a transaction graph versus a large merchant. GAT computes a coefficient αij from the features of i and j, so the model can emphasize relevant neighbors and suppress noisy ones. Because the coefficients depend only on features and not on the global graph structure, the same trained layer applies directly to new, unseen graphs.
Each layer has a shared weight matrix W ∈ Rd'×d and an attention vector a ∈ R2d'. For every edge (including a self-loop), the projected features of both endpoints are concatenated, scored with a, and passed through LeakyReLU. A softmax over each node's neighborhood turns scores into coefficients that sum to 1, and the new embedding is the coefficient-weighted sum of projected neighbors. Because aT[zi || zj] = a1Tzi + a2Tzj, an implementation computes one scalar per node for each half and adds them per edge — no explicit concatenation needed. Attention is masked to the graph's edges, which keeps cost linear in the number of edges.
A single head's attention weights are noisy, so GAT runs K heads in parallel with independent Wk and ak. Hidden layers concatenate the heads' outputs (K·d' dimensions); the final prediction layer averages them. The original transductive configuration used 8 heads of 8 features in the first layer and one output head (more for PubMed), with dropout 0.6 applied both to inputs and to the attention coefficients themselves — randomly dropping edges from each node's neighborhood during training.
Brody, Alon and Yahav (2022) observed a limitation of the original scoring function. Because eij = LeakyReLU(a1TWhi + a2TWhj) and LeakyReLU is monotonic, the ranking of neighbors j is determined by a2TWhj alone — the same for every query node i. They call this static attention: every node prefers the same globally top-ranked neighbor. GATv2 moves the nonlinearity inside: eij = aT LeakyReLU(W [hi || hj]), which yields dynamic attention whose ranking can depend on the query. It costs about the same and is generally the better default; in PyTorch Geometric it is GATv2Conv.
A single-head GAT layer written with plain tensor ops on an edge list, followed by the library version used in practice.
import torch
import torch.nn as nn
import torch.nn.functional as F
class GATLayer(nn.Module):
"""Single-head GAT layer on an edge list (src -> dst). Self-loops must be included."""
def __init__(self, d_in, d_out, p_attn=0.6):
super().__init__()
self.W = nn.Linear(d_in, d_out, bias=False)
self.a_src = nn.Parameter(torch.empty(d_out))
self.a_dst = nn.Parameter(torch.empty(d_out))
nn.init.xavier_uniform_(self.W.weight)
nn.init.normal_(self.a_src, std=0.1); nn.init.normal_(self.a_dst, std=0.1)
self.p_attn = p_attn
def forward(self, x, edge_index):
src, dst = edge_index
z = self.W(x) # (N, d_out)
e = F.leaky_relu((z * self.a_src).sum(-1)[src] +
(z * self.a_dst).sum(-1)[dst], 0.2) # (E,) one score per edge
# softmax over incoming edges of each dst node (subtract max for stability)
e_max = torch.full((x.size(0),), -1e9).scatter_reduce(0, dst, e, "amax")
w = (e - e_max[dst]).exp()
denom = torch.zeros(x.size(0)).scatter_add(0, dst, w)
alpha = w / denom[dst]
alpha = F.dropout(alpha, self.p_attn, self.training) # attention dropout
out = torch.zeros_like(z).index_add(0, dst, alpha.unsqueeze(-1) * z[src])
return out, alpha
# tiny graph with self-loops
edge_index = torch.tensor([[0, 1, 2, 3, 1, 2, 3, 0],
[0, 1, 2, 3, 0, 0, 0, 1]])
x = torch.randn(4, 5)
layer = GATLayer(5, 8).eval()
h, alpha = layer(x, edge_index)
print(h.shape, alpha[edge_index[1] == 0].sum().item()) # (4, 8) 1.0
# In practice: PyTorch Geometric
from torch_geometric.nn import GATv2Conv
class GAT(nn.Module):
def __init__(self, d_in, n_cls, hid=8, heads=8):
super().__init__()
self.c1 = GATv2Conv(d_in, hid, heads=heads, dropout=0.6) # concat -> hid*heads
self.c2 = GATv2Conv(hid * heads, n_cls, heads=1, concat=False, dropout=0.6)
def forward(self, x, edge_index):
x = F.dropout(x, 0.6, self.training)
x = F.elu(self.c1(x, edge_index))
x = F.dropout(x, 0.6, self.training)
return self.c2(x, edge_index)
# GATv2Conv(..., return_attention_weights=True) exposes alpha for inspectionTransformer self-attention is attention-weighted message passing on a complete graph, where every token attends to every other token, with dot-product scoring qiTkj/√d and separate value projections. GAT is the same idea restricted to the edges of a given graph, with additive scoring. Graph Transformers combine the two: attention over all nodes (or a sparse superset of edges) plus positional or structural encodings such as Laplacian eigenvectors or random-walk features, so the model still knows the graph topology. They capture long-range interactions that local message passing struggles with, at quadratic cost.
add_self_loops=True).Consider node i with a self-loop and two neighbors, j and k. After the shared projection, suppose the transformed features are Whi = (1, 0), Whj = (0, 1) and Whk = (1, 1), and the attention vector and LeakyReLU produce raw scores eii = 1.0, eij = 2.0, eik = 0.5.
The softmax over the neighborhood exponentiates the scores to about 2.72, 7.39 and 1.65, which sum to 11.76, giving attention weights αii ≈ 0.23, αij ≈ 0.63 and αik ≈ 0.14. The new representation is the weighted sum 0.23·(1, 0) + 0.63·(0, 1) + 0.14·(1, 1) ≈ (0.37, 0.77), which then passes through the nonlinearity (ELU in the original paper). A GCN with the same three nodes would have used fixed weights set by degree; here neighbor j dominates because its score is highest. With K heads, this calculation runs K times with different parameters and the results are concatenated, or averaged in the output layer.
Two details matter in practice. The softmax is computed per target node over a variable-size set, which libraries implement as a scatter-softmax over the edge list. And dropout is applied to the attention coefficients themselves, so during training each node sees a randomly sparsified neighborhood, which is a strong regularizer on small graphs.
The original transductive setup on citation graphs such as Cora used two layers: a hidden layer of 8 heads with 8 features each (concatenated to 64), an output layer with a single head (averaging heads on larger datasets), dropout of 0.6 on both inputs and attention coefficients, and L2 weight decay. Those numbers are a reasonable starting point for small graphs; on larger graphs, fewer dropout and more hidden units are typical. Tune the number of heads, hidden width per head, dropout, learning rate and weight decay, and always report variance across several random seeds, since results on small benchmarks move noticeably between runs.
Full-batch training holds every node and edge in memory, which stops working around millions of edges with several heads. The standard remedy is neighbor sampling (introduced with GraphSAGE): for each mini-batch of target nodes, sample a fixed number of neighbors per hop and run the layers only on that computation tree. PyG's NeighborLoader and DGL's samplers implement this. Alternatives include cluster-based batching (Cluster-GCN partitions the graph and trains on subgraphs) and precomputing propagated features (SIGN-style), which trade attention flexibility for speed. For inference on a very large graph, layer-wise computation over all nodes avoids the sampling variance.