We chunk documents, shuffle the chunks, and train LLMs to untie the knots — identifying relevant segments within seemingly chaotic token sequences. UtK is a data-only recipe that delivers 75% / 84.5% RULER@128K on 7B/72B models with 20× better data efficiency and no short-context regression. 我们将文档切块、打乱，然后训练 LLM 学会解开绳结——在看似混乱的 token 序列中识别出相关片段。UtK 是一种纯数据侧的训练策略，在 7B/72B 模型上取得了 RULER@128K 75% / 84.5%，数据效率提升 20 倍，且不损害短文本能力。
Abstract摘要
We introduce Untie the Knots (UtK), a novel data augmentation strategy employed during the continue pre-training phase, designed to efficiently enable LLMs to gain long-context capabilities without the need to modify the existing data mixture. We chunk the documents, shuffle the chunks, and create a complex and knotted structure of long texts; LLMs are then trained to untie these knots and identify relevant segments within seemingly chaotic token sequences. This approach greatly improves the model's performance by accurately attending to relevant information in long context and the training efficiency is also largely increased. We conduct extensive experiments on models with 7B and 72B parameters, trained on 20 billion tokens, demonstrating that UtK achieves 75% and 84.5% accuracy on RULER at 128K context length, significantly outperforming other long context strategies. 我们提出 Untie the Knots (UtK)——一种在持续预训练阶段使用的新型数据增强策略，旨在不修改原有数据混合比例的前提下高效赋予 LLM 长上下文能力。我们把文档切分为若干区块，将来自不同文档的区块打乱并拼成一段长且"打结"的序列；模型被训练去"解开这些绳结"——在看似混乱的 token 流中准确识别出属于同一文档的相关片段。这种方法显著提升了模型在长文本中精确关注相关信息的能力，训练效率也大幅提高。我们在 7B 与 72B 模型上用 200 亿 token 做了系统实验，UtK 在 128K 长度的 RULER 上分别取得 75% 与 84.5%，显著优于其他长上下文策略。
Motivation: The Data Bottleneck动机：长文本数据瓶颈
High-quality, naturally long documents are extremely rare. In a typical pre-training corpus, over 70% of tokens come from documents shorter than 32K. Naively upsampling scarce long documents distorts the data distribution and can hurt short-context generalization. 高质量的原生长文档极为稀缺。在典型预训练语料中，超过 70% 的 token 来自长度不足 32K 的文档。直接对稀缺长文档做上采样，会扭曲训练分布，反而损害模型在更常见的短文本任务上的泛化能力。
Our insight: instead of finding more long data, teach the model the skill of long-range attention using data we already have. 我们的出发点是：与其寻找更多长文档，不如教会模型掌握长距离注意力的能力——用我们已经拥有的数据。
Method: Tangling & Untying方法：打结与解结
UtK turns standard next-token prediction into a long-range retrieval task. It has two phases: a Tangling Phase that constructs a knotted pseudo-long sequence, and an Untying Phase where the model must learn to trace across distractors. UtK 将标准的 next-token 预测改造成一个长距离信息检索任务，分为两个阶段：打结阶段负责构造"结"状的伪长序列，解结阶段训练模型穿越干扰项并还原结构。
Chunking分块
Split each document into 2–3 random-length chunks. Empirically, 2–3 chunks is the sweet spot.将每篇文档随机切为 2–3 个变长区块；2–3 块经实验是最优区间。
Tying & Shuffling编织 & 打乱
Chunks from different documents are globally shuffled into one long sequence, with [Knot] tokens and unique [Chunk ID]s inserted at boundaries.不同文档的区块全局随机打乱成一段长序列，在边界插入 [Knot] 标记与唯一的 [Chunk ID]。
Backtracing回溯任务
At the end of a document's last chunk, the model must emit the original chunk-ID sequence — forcing explicit structure reconstruction.在原文档最后一个区块结尾处，模型须显式输出该文档完整的区块 ID 序列——强制学习全局结构。
Results实验结果
RULER Benchmark — Base ModelsRULER 基准 · 基座模型
On the RULER benchmark at 128K context length, UtK achieves 75.0% on Qwen2-7B and 84.5% on Qwen2-72B, significantly outperforming upsampling, synthetic-data and other long-context recipes. 在 RULER 128K 长度上，UtK 使 Qwen2-7B 达到 75.0%、Qwen2-72B 达到 84.5%，明显优于上采样、合成数据等长上下文训练策略。
InfiniteBench — Instruct ModelsInfiniteBench · 指令模型
| Model | En.Avg. | En.Sum | En.QA | En.MC | En.Dia | Zh.QA |
|---|---|---|---|---|---|---|
| GPT-4-Turbo-2024-04-09 | 33.2 | 17.6 | 19.3 | 77.7 | 18.0 | – |
| Claude 2 | 34.0 | 14.5 | 12.0 | 62.9 | 46.5 | 9.6 |
| Kimi-Chat | 29.6 | 18.0 | 16.5 | 72.5 | 11.5 | 17.9 |
| Yi-34B-200K | <15.15 | <5 | 12.2 | 38.4 | <5 | 13.6 |
| Qwen2-72B-Instruct | 39.8 | 31.7 | 21.5 | 83.0 | 23.0 | – |
| Llama-3-70B-Inst-Gradient-262k | 32.6 | 14.3 | 29.5 | 69.0 | 17.5 | – |
| Llama3.1-8B-Instruct | 33.2 | 29.2 | 31.5 | 59.0 | 13.0 | – |
| Llama3.1-70B-Instruct | 39.8 | 30.9 | 38.5 | 75.6 | 14.3 | – |
| Llama3-ChatQA-2-8B | 35.6 | 17.1 | 43.5 | 64.2 | 17.5 | – |
| Llama3-ChatQA-2-70B | 41.0 | 16.1 | 48.2 | 80.4 | 19.5 | – |
| Qwen2-UtK-ChatQA2-7B-128K | 33.3 | 21.2 | 42.6 | 61.1 | 8.5 | 37.6 |
| Qwen2-UtK-ChatQA2-72B-128K | 47.3 | 18.2 | 55.9 | 83.8 | 31.0 | 45.2 |
Analysis分析
Training efficiency训练效率
With only 1B tokens, UtK-192K matches the baseline trained on 20B tokens — a 20× data-efficiency win.仅用 10 亿 token，UtK-192K 就达到了基线用 200 亿 token 的水平——数据效率 20×。
No short-context regression短文本性能无损
On BBH, HumanEval, GSM8K and more, UtK models stay within ±1.5% of baselines. Long gains come for free.在 BBH、HumanEval、GSM8K 等基准上，UtK 与基线相差不超过 ±1.5%。长文本能力 "免费" 获得。
Optimal chunk count最优区块数
Performance peaks at 2–3 chunks per document. More chunks make the task too hard to learn.每篇文档切为 2–3 块时性能最佳；过多区块会让任务过于困难，反而阻碍学习。
Backtracing is critical回溯任务至关重要
Ablating the backtrace objective causes a large drop — it's the lever that forces global structure learning.去掉回溯任务会带来显著下降——它正是强制模型学习全局结构的关键杠杆。
Scales across models跨模型可扩展
UtK improves Qwen2-7B, Qwen2-72B, Llama3.1-8B alike — a general recipe, not a model-specific trick.UtK 对 Qwen2-7B、Qwen2-72B、Llama3.1-8B 均有效——是通用训练策略，不是对特定模型的"调参"。
Interpretable attention可解释的注意力
Attention maps reveal the model precisely connecting non-adjacent chunks of the same document across long distractors — literally "untying" the knots.注意力图显示模型能精确把当前位置的注意力，跨过大量无关内容，聚焦到前文同文档的非相邻区块——字面意义上在"解结"。
Open-Source Models开源模型
We release four checkpoints on Hugging Face for research use: 我们在 Hugging Face 开源了四个训练好的模型供研究使用：
Quick Deploy (vLLM)快速部署 (vLLM)
pip install "vllm>=0.4.3"
python -m vllm.entrypoints.openai.api_server \
--served-model-name Qwen2-UtK-ChatQA2-7B-128K \
--model path/to/weights \
--trust-remote-code \
--tensor-parallel-size 2 \
--host 0.0.0.0 \
--enable_chunked_prefill \
--max_num_batched_tokens 32768Citation引用
If you find UtK useful, please cite:如果本工作对你有帮助，请引用：
@inproceedings{tian2025utk,
title = {Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models},
author = {Tian, Junfeng and Zheng, Da and Chen, Yang and Wang, Rui and Zhang, Colin and Zhang, Debing},
booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL)},
year = {2025},
url = {https://arxiv.org/abs/2409.04774}
}