How would you extend a LLaMA-2 7B checkpoint from 4k to...

To extend the context window, first apply NTK-aware base scaling (e.g., raising the base to approximately 128,000 for an 8x extension) to allow immediate inference. Second, perform continued pretraining using a long-document corpus for 1000-2000 gradient steps on 8-16 GPUs. Given the memory intensity of 32k sequences (requiring roughly 4.3 GB of KV activations per layer at bf16), you must use sequence packing and activation checkpointing. Finally, confirm performance using benchmarks like SCROLLS or LongBench.