Why can’t you use the same self-supervised augmentation...

In vision, the set of meaning-preserving augmentations is ‘fat,’ allowing for strong self-supervised representations. In contrast, text is discrete; the smallest legal move is a single token, which often changes the semantic label. To address this, one should quantify the risk using the formula 1 - (1 - ρ)^c and focus on perturbing the encoder rather than the input text, as text augmentation is often too noisy or prone to inverting the original meaning.