Your team applied 5,000 sequential ROME edits to a 6B m...

The degradation in MMLU despite high edit efficacy stems from the linear algebra limits of sequential rank-one updates. A ROME edit modifies a model layer by computing a rank-one update to a feed-forward weight matrix. When you apply 5,000 sequential updates, these rank-one updates sum together.

Because the maximum rank of the underlying weight matrix is bounded by min(d, d_mlp) = 4,096, applying more than ~4,000 rank-one updates forces new edits to overwrite parameter directions already occupied by previous edits or base capabilities. High efficacy on the edit set (97%) is deceptive because efficacy is measured on the exact newly installed facts—which are precisely the ones that survived the latest writes—while background knowledge evaluated by MMLU suffers catastrophic degradation.

To address this issue, raising the rank of individual ROME edits is insufficient. The structural solution is to use an algorithm like MEMIT (Mass-Editing Memory in a Transformer), which batches thousands of facts simultaneously and spreads the rank updates across a broader band of layers, allowing the model to support 10,000+ edits without collapsing model performance.