Why do people say the transformer is doing gradient des...

A strong explanation strips the softmax and writes the demonstration term as a sum of rank-one outer products. This structure mirrors the shape of an accumulated gradient update. This perspective is supported by the dual form framing (Irie et al., 2022) and meta-gradient framing (Dai et al., 2023). While this provides a theoretical link, it is important to note the scope: this behavior is most clearly observed in linear attention and models trained on specific synthetic function classes, rather than being a universal property of all transformer training.