跳到主要内容
Cheng-Ping Hsieh et al.

几乎所有模型都能通过「大海捞针」。RULER 加了 13 类更难的题,证明真实上下文长度远低于标称值。2024 年 COLM 论文,免费阅读。

RULER: What's the Real Context Size of Your Long-Context Language Models?

RULER:你的长上下文语言模型的真实上下文长度是多少?

*Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg*

NVIDIA

{chsieh, simengs}@nvidia.com

摘要

大海捞针(NIAH)测试通过检验从长干扰文本(“草堆”)中检索一条信息(“针”)的能力,已被广泛用于评估长上下文语言模型(LM)。然而,这种简单的基于检索的测试仅能反映一种表面形式的长上下文理解能力。为了对长上下文语言模型进行更全面的评估,我们创建了一个新的合成基准 RULER,其配置灵活,可自定义序列长度和任务复杂度。RULER 在原始 NIAH 测试的基础上进行了扩展,涵盖了不同针类型和数量的变体。此外,RULER 引入了新的任务类别——多跳追踪和聚合——以测试超越从上下文中搜索的行为。我们在 RULER 中用 13 个代表性任务评估了 17 个长上下文语言模型。尽管在原始 NIAH 测试中几乎达到了完美的准确率,但几乎所有模型在上下文长度增加时都表现出大幅性能下降。虽然这些模型都声称上下文长度达到 32K token 或更大,但其中只有一半能在 32K 长度下保持令人满意的性能。我们对支持 200K 上下文长度的 Yi-34B 的分析表明,随着输入长度和任务复杂度的增加,仍有很大的改进空间。我们开源了 RULER,以促进对长上下文语言模型的全面评估。

1 引言

近期在 AI 系统工程(Dao et al., 2022; Jacobs et al., 2023; Fu et al., 2024)和语言模型设计(Chen et al., 2023; Xiong et al., 2023)方面的进展使得语言模型的上下文长度得以高效扩展(Liu et al., 2024a; Young et al., 2024)。先前的工作(AI21, 2024; X.AI, 2024; Reid et al., 2024; Anthropic, 2024)通常采用合成任务,如密码检索(Mohtashami & Jaggi, 2023)和大海捞针(Kamradt, 2023)来评估长上下文语言模型。然而,这些评估在不同工作中使用方式不一致,且仅揭示了检索能力,未能衡量其他形式的长上下文理解能力。

在本工作中,我们提出了 RULER,一个用于评估语言模型长上下文建模能力的新基准。RULER 包含四个任务类别,以测试超越简单从上下文中检索的行为(Ribeiro et al., 2020):

1. 检索:我们扩展了大海捞针(Kamradt, 2023; NIAH)测试,以评估不同针类型和数量下的检索能力。2. 多跳追踪:我们提出了变量追踪,这是一个用于共指链解析的最小代理任务,用于检验追踪具有多跳连接的实体的行为。3. 聚合:我们提出了常见词/高频词提取,这是摘要的代理任务,用于测试聚合跨越长距离上下文的相关信息的能力。4. 问答:我们在现有短上下文问答数据集的输入中添加干扰信息,以评估不同上下文长度下的问答能力。

基准与任务平均长度类型多样化任务最少参数知识可控上下文
ZeroSCROLLS~10k真实✓XX
L-Eval~8k真实✓XX
BAMBOO~16k真实✓✓X
LongBench~8k混合✓XX
LooGLE~20k混合✓✓X
InfiniteBench~200k混合✓✓X
大海捞针(NIAH)任意合成X✓✓
Passkey / Line / KV Retrieval任意合成X✓✓
RULER(本文)任意合成✓✓✓
表1:现有长上下文基准与RULER的比较。“真实”类型指人工标注,而“合成”类型指自动生成。RULER包含超越检索的多样化任务领域,通过合成输入减少对参数知识的依赖,并提供灵活性以针对不同序列长度和任务复杂度控制上下文。在RULER中,可以通过改变相关信息和干扰信息的数量或位置来调整上下文。

与现有的真实基准(表 1)相比,RULER 仅由合成任务组成,这提供了控制序列长度和任务复杂度的灵活性。RULER 中的合成输入减少了对参数化知识的依赖,而在真实任务中,参数化知识会干扰长上下文输入的利用(Shaham et al., [2023]; Bai et al., [2023])。

使用 RULER,我们对 Gemini-1.5(Reid et al., [2024])、GPT-4(OpenAI: Josh Achiam et al., [2023])以及 15 个上下文长度从 4k 到 128k 的开源模型进行了基准测试。尽管在原始 NIAH 测试中几乎达到了完美的性能,但几乎所有模型在 RULER 中更复杂的任务上都随着序列长度的增加而表现出大幅退化。虽然所有模型都声称上下文大小为 32k token 或更大,但我们的结果表明,只有一半的模型能够通过超过定性阈值来有效处理 32K 的序列长度。此外,几乎所有模型在达到所声称的上下文长度之前就低于该阈值。为了获得细粒度的模型比较,我们使用两个加权平均分数聚合了从 4k 到 128k 的性能,其中权重模拟了真实世界用例的长度分布。排名前两位的模型——Gemini-1.5 和 GPT-4——无论选择何种加权方案,都始终优于其他模型。

我们进一步分析了 Yi-34B,它声称上下文长度为 200K,在开源模型中于 RULER 上取得了相当不错的性能。我们的结果表明,随着输入长度和任务复杂度的增加,Yi 的性能大幅退化。在较大的上下文尺寸下,Yi-34B 经常返回不完整的答案,且无法精确定位相关信息。此外,我们观察到随着上下文尺寸的扩展,多个模型出现了两种行为:对参数化知识的依赖增加,以及在非检索任务中从上下文复制的倾向增加。我们的额外消融实验表明,在更长序列上训练并不总能带来 RULER 上更好的性能,且更大的模型尺寸与更好的长上下文能力正相关。最后,我们展示了非 Transformer 架构,如 RWKV 和 Mamba,在 RULER 上仍大幅落后于 Transformer。

我们的贡献如下:

- 我们提出了一个新的基准 RULER,通过具有灵活配置的合成任务来评估长上下文语言模型。- 我们引入了新的任务类别,特别是多跳追踪和聚合,以测试超越从长上下文中检索的行为。- 我们使用 RULER 评估了 17 个长上下文语言模型,并进行了跨模型和任务复杂度的分析。

我们开源了 RULER,以促进长上下文语言模型的未来研究。

2 相关工作

长上下文语言模型。由于工程、架构和算法设计方面的进展,近期涌现了大量长上下文语言模型。Flash attention(Dao et al., 2022; Dao, 2023)和 Ring attention(Liu et al., 2023)显著减少了处理长上下文所需的内存占用。各种稀疏注意力机制(Child et al., 2019; Jaszczur et al., 2021),如移位稀疏注意力(Chen et al., 2024)、膨胀注意力(Ding et al., 2023)和注意力汇聚(Han et al., 2023; Xiao et al., 2024b),被用于实现高效的上下文扩展。研究人员提出了新的位置嵌入方法以改善 Transformer 的长度外推能力(Vaswani et al., 2017),包括 ALiBi(Press et al., 2022)、xPOS(Sun et al., 2023b)以及 RoPE(Su et al., 2023)变体(Chen et al., 2023; Xiong et al., 2023; Peng et al., 2024; Liu et al., 2024b; Ding et al., 2024; Zhu et al., 2024)。另一条研究路线专注于缩减上下文大小。这可以通过使用循环机制缓存先前的上下文(Zhang et al., 2024a; Bulatov et al., 2023; Martins et al., 2022; Wu et al., 2022)、从上下文中检索相关信息(Xu et al., 2024a; Mohtashami & Jaggi, 2023; Wang et al., 2024; Tworkowski et al., 2024; Xiao et al., 2024a),或通过压缩保留显著信息(Jiang et al., 2023)来实现。最后,新的架构(Gu et al., 2022; Fu et al., 2023a; Poli et al., 2023; Fu et al., 2023b; Sun et al., 2023a; Beck et al., 2024; Sun et al., 2024),如 Mamba(Gu & Dao, 2023)和 RWKV(Peng et al., 2023),也已被提出以高效处理长上下文输入。

长上下文基准与任务。我们的工作与其他长上下文语言模型基准测试工作密切相关。ZeroSCROLLS(Shaham et al., 2023)涵盖十个真实的自然语言任务,如长文档问答和(基于查询的)摘要。L-Eval(An et al., 2024)也使用真实数据,这些数据经过人工筛选以确保质量。LongBench(Bai et al., 2023)包含双语环境下的任务。InfiniteBench(Zhang et al., 2024b)包含长度超过 100K token 的任务。LTM(Castillo et al., 2024)针对长期对话的评估。为了隔离参数化知识的影响,先前的工作(Dong et al., 2023; Li et al., 2023b)也提出使用在某个截止日期之后在线发布的文档,或利用极低资源材料(Tanzer et al., 2024)。与真实基准相比,合成任务在控制设置(例如序列长度和任务复杂度)方面更灵活,且受参数化知识的影响较小。近期工作主要集中于基于检索的合成任务(Kamradt, 2023; Mohtashami & Jaggi, 2023; Li et al., 2023a; Liu et al., 2024d; Lee et al., 2024),少数研究探讨了其他方面,包括事实推理(Kuratov et al., 2024; Karpinska et al., 2024)、长距离篇章建模(Sun et al., 2022)、问答(Levy et al., 2024; Yuan et al., 2024)、多示例上下文学习(Agarwal et al., 2024; Bertsch et al., 2024; Xu et al., 2024b)以及代码理解(Liu et al., 2024c)。

3 RULER 基准

RULER 包含四个类别的任务:检索、多跳追踪、聚合和问答。RULER 中的评估示例根据输入配置(见表2)自动生成,这些配置定义了每个输入的长度和复杂度。在如 RULER 这样的受限领域中,任务复杂度可以被视为目标输出 token 数量与上下文信噪比的函数。我们向读者推荐 (Goldman et al., 2024) 以获取关于长上下文语言模型评估任务设计的更全面讨论。

3.1 检索:大海捞针(NIAH)

近期工作 (Reid et al., 2024; Anthropic, 2023) 普遍采用大海捞针(Kamradt, 2023; NIAH)测试来评估长上下文建模能力。NIAH 测试令人联想到被广泛研究的 (Hopfield, 1982; Graves et al., 2014; Olsson et al., 2022; Arora et al., 2024) 联想回忆任务,其中需要在给定充分查询的情况下从上下文中检索相关信息。在 RULER 中,我们包含多个基于检索的任务,将原始 NIAH 测试进行扩展,基于三个

任务配置示例
单针检索
(SingleNIAH,S-NIAH)
type_key = word
type_value = number
type_haystack = essay
size_haystack < 上下文长度
(文章)
长上下文的一个特殊魔法数字是:12345。
所提供的文本中提到的长上下文的特殊魔法数字是什么?
答案:12345
多键检索
(Multi-keysNIAH,MK-NIAH)
num_keys = 2
type_key = word
type_value = number
type_haystack = essay
size_haystack < 上下文长度
(文章)
长上下文的一个特殊魔法数字是:12345。
大模型的一个特殊魔法数字是:54321。
所提供的文本中提到的长上下文的特殊魔法数字是什么?
答案:12345
多值检索
(Multi-valuesNIAH,MV-NIAH)
num_values = 2
type_key = word
type_value = number
type_haystack = essay
size_haystack < 上下文长度
(文章)
长上下文的一个特殊魔法数字是:12345。
长上下文的一个特殊魔法数字是:54321。
所提供的文本中提到的长上下文的所有特殊魔法数字是什么?
答案:12345 54321
多查询检索
(Multi-queriesNIAH,MQ-NIAH)
num_queries = 2
type_key = word
type_value = number
type_haystack = essay
size_haystack < 上下文长度
(文章)
长上下文的一个特殊魔法数字是:12345。
大模型的一个特殊魔法数字是:54321。
所提供的文本中提到的长上下文和大模型的所有特殊魔法数字是什么?
答案:12345 54321
变量追踪
(VariableTracking,VT)
num_chains = 2
num_hops = 2
size_noises < 上下文长度
(噪声)
VAR X1 = 12345 .... VAR Y1 = 54321 ....
VAR X2 = X1 .... VAR Y2 = Y1 ....
VAR X3 = X2 .... VAR Y3 = Y2 ....
找出所有被赋值为12345的变量。
答案:X1 X2 X3
常见词提取
(Common WordsExtraction,CWE)
freq_cw = 2, freq_ucw = 1
num_cw = 10
num_ucw < 上下文长度
aaa bbb cca aaa ddd eee ccc fff ggg hhh iii iii ....
上述列表中出现频率最高的10个词是什么?
答案:aaa ccc iii ....
高频词提取
(Frequent WordsExtraction,FWE)
α = 2
num_word < 上下文长度
aaa bbb cca aaa ddd eee ccc fff ggg aaa hhh aaa ccc iii iii ....
上述编码文本中出现频率最高的3个词是什么?
答案:aaa ccc iii
问答
(QuestionAnswering,QA)
dataset = SQuAD
num_document < 上下文长度
文档1:.... aaa ....
文档2:.... bbb ....
文档3:.... ccc ....
问题:问题
答案:bbb
表2:RULER中具有灵活配置的任务示例。我们在示例中使用不同颜色来突出显示查询、键、值和干扰项。

标准来评估模型。具体而言,检索能力应当 (1) 与“针”和“干草堆”的类型无关,(2) 足够强大以忽略困难干扰项,以及 (3) 在需要检索多个项目时具有高召回率。基于这些标准,我们开发了四个 NIAH 任务。每个任务中的“针”是插入到“干草堆”(长干扰文本)中的键值对。查询位于序列末尾,作为匹配上下文中的键并随后检索关联值的线索。

- 单针检索(Single NIAH,S-NIAH):这是最基础的 NIAH 测试,需要从“干草堆”中检索出一根“针”。查询/键/值可以采用单词、数字(7 位)或 UUID(32 位)的形式。“干草堆”可以是重复的噪声句子或 Paul Graham 的文章(Kamradt, 2023)。- 多键检索(Multi-keys NIAH,MK-NIAH):向“干草堆”中插入多根“针”,但只需检索其中一根。额外的“针”是难以区分的干扰项。最具挑战性的设置是“干草堆”中填满了干扰针的版本。

图1:在聚合任务中,我们按照上述两种分布从词汇表中采样单词。常见词提取(CWE)从均匀分布中采样。在频繁词提取(FWE)中,每个单词的频率由其词汇表中的排名和Zeta分布的参数\alpha决定。
图1:在聚合任务中,我们按照上述两种分布从词汇表中采样单词。常见词提取(CWE)从均匀分布中采样。在频繁词提取(FWE)中,每个单词的频率由其词汇表中的排名和Zeta分布的参数α\alpha决定。

- 多值检索(Multi-values NIAH,MV-NIAH):向“干草堆”中插入多根共享同一键的“针”。需要检索与同一键关联的所有值。- 多查询检索(Multi-queries NIAH,MQ-NIAH):向“干草堆”中插入多根“针”。需要检索所有具有不同键的“针”。这与 Arora 等人(2024)使用的多查询关联回忆任务设置相同。与 MV-NIAH 一起,这两个任务评估在不错过任何关键信息的情况下的检索能力。

3.2 多跳追踪:变量追踪(VT)

有效的语篇理解(van Dijk & Kintsch, 1983)取决于能否成功识别新提及的实体,并在长上下文中建立指代同一实体的引用链(Karttunen, 1969)。我们开发了一项新任务——变量追踪,以模拟最小化的共指链消解(Ng, 2010)任务。该任务检验在长输入中追踪相关共现模式并建立跨越式连接的行为。具体而言,一个变量 X1X1 被初始化为值 VV,随后是一系列线性链接的变量名绑定语句(例如 X2=X1X2 = X1,X3=X2X3 = X2,……),这些语句被插入到输入的不同位置。目标是返回所有指向同一值 VV 的变量名。通过增加更多跳数(即名称绑定的次数)或更多链,可以增加任务复杂度,类似于在 MK-NIAH 中添加难以区分的干扰项。

3.3 聚合:常见词(CWE)与高频词提取(FWE)

在 RULER 中,我们引入一个新类别,作为摘要任务的代理,其中相关信息在上下文中占据大得多的比例,而目标输出取决于对相关输入的准确聚合。具体而言,我们通过从预定义(合成)词表中采样单词来构建输入序列。在常见词提取任务(CWE)中,单词从离散均匀分布中采样,常见词的数量固定,而不常见词的数量随序列长度增加。在高频词提取任务(FWE)中,单词从 Zeta 分布中采样。图 1 展示了所构建输入中词频的示例。模型需要返回上下文中频率最高的前 KK 个单词。在 CWE 中,KK 等于常见词的数量。在 FWE 中,我们将 KK 设为 3,因为增大 KK 会导致大多数模型即使在小上下文尺寸下也表现不佳。通过改变常见词的数量或 Zeta 分布的参数,可以调整任务复杂度。4

3.4 问答(QA)

现有大多数 QA 数据集(Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022)设计用于基于短段落回答问题。这些数据集可以通过添加上下文干扰信息来扩展以模拟长上下文输入。在此任务类别中,我们将黄金段落(即包含答案的段落)插入到从同一数据集中随机采样的段落中。此类别是 NIAH 在真实世界中的改编版本(Ivgi et al., 2023),其中问题作为查询,黄金段落是“针”,干扰段落构成“干草堆”。

4 实验与结果

模型与推理设置 我们选择了 17 个长上下文 LLM,包括 15 个开源模型和两个闭源模型(Gemini-1.5-Pro 和 GPT-4),涵盖多种模型规模(7B 到 8x22B 的 MoE 架构)和声称的上下文长度(32K 到 1M)。这些模型的完整信息包含在附录 A 中。我们使用 vLLM(Kwon et al., 2023)评估所有模型,这是一个具有高效 KV 缓存内存管理的 LLM 服务系统。对于所有模型,我们在 8 块 NVIDIA A100 GPU 上以 BFloat16 格式运行推理,采用贪心解码。

任务配置 我们在 RULER 四个类别中涵盖不同复杂度的 13 个任务上测试所有模型。测试配置的选择(见附录 B)基于附录 C 中描述的任务相关性研究。我们选择这些任务是因为大多数模型在 4K token 的短上下文尺寸下表现尚可。我们的主要目标是观察模型能否在上下文长度扩展时保持如此良好的性能。对于每个任务,我们使用从(4K、8K、16K、32K、64K、128K)序列中为每个长度生成的 500 个示例来评估每个模型,同时遵循每个模型所需的聊天模板。为防止模型拒绝回答查询或生成解释,我们在任务输入后附加答案前缀,并使用基于召回率的准确率检查目标输出是否存在。

有效上下文长度 我们注意到,在 RULER 中增加输入长度时,所有模型都出现了大幅性能下降。为确定模型能够有效处理的最大上下文尺寸,我们使用固定阈值对每个模型进行评分,通过该阈值表示在评估长度下性能令人满意。我们使用 Llama2-7b 模型在 4K 上下文长度下的性能作为阈值。我们在表 3 中报告超过阈值的最长长度作为“有效长度”,同时列出“声称长度”。

模型排名标准 虽然基于阈值的评分揭示了声称长度与有效长度之间的差异,但它缺乏用于细粒度模型比较的细节。因此,我们使用加权平均分数来聚合模型在不同上下文尺寸下的性能。我们在两种加权方案下对模型进行排名:wAvg.(inc)和 wAvg.(dec),其中权重分别随序列长度线性增加和减少。理想情况下,每个长度的权重应由模型使用的长度分布决定。这里我们选择两种方案来模拟较长序列(inc)或较短序列(dec)在分布中占主导地位的情景。

主要结果 我们在表 3 中列出了 17 个长上下文 LM 与 Llama2-7B 基线的比较结果。某一长度下的性能是 RULER 中所有 13 个任务的平均值。闭源模型 Gemini-1.5-Pro 以较大优势优于其余模型,其有效长度大于我们所测试的最大长度。用更困难版本的 RULER 对该模型进行压力测试可能是未来值得跟进的有趣工作。对于其余模型,虽然它们在密匙检索和基础 NIAH 任务上(见附录 E)取得了近乎完美的性能,但所有模型在 RULER 中都随着序列长度增加而出现大幅性能下降,且在其声称长度下无法保持高于 Llama2-7B 基线的性能。排名最高的开源模型(Llama3.1、Qwen2 和 Command-R-plus)具有共同的配置特征,例如更大的模型规模和

模型声称长度有效长度4K8K16K32K64K128K平均值加权平均(递增)加权平均(递减)
Llama2(7B)4K-85.6
Gemini-1.5-Pro1M>128K96.795.896.095.995.994.495.895.5(第1名)96.1(第1名)
GPT-4128K64K96.695.395.293.287.081.291.689.0(第2名)94.1(第2名)
Llama3.1(70B)128K64K96.595.895.494.888.466.689.685.5(第4名)93.7(第3名)
Qwen2(72B)128K32K96.996.194.994.179.853.785.979.6(第9名)92.3(第4名)
Command-R-plus (104B)128K32K95.695.294.292.084.363.187.482.7(第7名)92.1(第5名)
GLM4 (9B)1M64K94.792.892.189.986.783.189.988.0(第3名)91.7(第6名)
Llama3.1 (8B)128K32K95.593.891.687.484.777.088.385.4(第5名)91.3(第7名)
GradientAI/Llama3 (70B)1M16K95.194.490.885.480.972.186.582.6(第8名)90.3(第8名)
Mixtral-8x22B (39B/141B)64K32K95.694.993.490.984.731.781.973.5(第11名)90.3(第9名)
Yi (34B)200K32K93.392.291.387.583.277.387.584.8(第6名)90.1(第10名)
Phi3-medium(14B)128K32K93.393.291.186.878.646.181.574.8(第10名)88.3(第11名)
Mixtral-v0.2(7B)32K16K93.691.287.275.449.013.868.455.6(第13名)81.2(第12名)
LWM(7B)1M<4K82.378.473.769.168.165.072.869.9(第12名)75.7(第13名)
DBRX(36B/132B)32K8K95.193.883.663.12.40.056.338.0(第14名)74.7(第14名)
Together(7B)32K4K88.281.169.463.00.00.050.333.8(第15名)66.7(第15名)
LongChat(7B)32K<4K84.779.970.859.30.00.049.133.1(第16名)65.2(第16名)
LongAlpaca(13B)32K<4K60.657.056.643.60.00.036.324.7(第17名)47.9(第17名)
表3:所选模型在4K至128K长度上评估的长上下文性能 (%)(\%)。每个分数通过平均RULER中13个任务的准确率计算得出。超过Llama2-7B在4K下性能 (85.6%)(85.6\%) 的表现以下划线标出。有效上下文长度是超过该阈值的最大长度。加权平均分数(wAvg.)汇总所有上下文大小上的性能,权重线性增加(inc)或减少(dec),以模拟真实世界使用的长度分布。我们将每个模型的排名放在下标中。有关所选模型的更多细节见附录A

RoPE 中使用更大的基础频率(Xiong et al., 2023)。大的训练上下文窗口并不总是获得良好长上下文性能的必要条件——排名最高的开源模型既包含暴力上下文扩展(Llama3.1 在 128K 上下文长度上训练),也包含推理时长度外推(Qwen2 在 32K 上下文长度上训练)。表现较差的模型也包括那些在更大上下文尺寸上训练的模型(例如 LWM 和 GradientAI/Llama3 均在 1M 上下文长度上训练)。尽管当较长序列获得较大权重(wAvg. inc)时 LWM 的排名高于 Mistral-v0.2,并且随上下文尺寸增加表现出较小的性能下降,但即使在 4K 下其表现也劣于 Llama2-7B。这一结果表明在评估中存在短序列绝对性能与上下文尺寸扩展时相对性能下降之间的权衡。我们在第 6 节中提供关于模型规模和最大训练长度的更多分析。

5 任务错误分析

我们在更复杂的任务上以增加的输入长度(最多 256K)评估 Yi-34B-200K,以理解任务配置和失败模式对 RULER 的影响。

对“针”类型缺乏鲁棒性。图2(左)显示,当使用单词—数字对作为针进行标准密码检索和原始NIAH时,Yi几乎能达到完美性能,但当针采用其他形式时,性能会下降。我们观察到在检索UUID的任务中下降最大,在这种情况下,给定较长(>128K>128\mathrm{K})的输入上下文时,Yi有时无法返回完整的32位数字。

无法忽略干扰项。图2(中左)显示,增加干扰针的数量会稳步降低性能,在极端版本中,当上下文充满无关针时(#K=FULL),Yi在256K处下降了约4040分。错误分析表明,Yi在长输入上下文中无法有效忽略困难干扰项,因此会错误地检索与干扰键相关联的值。在极端版本中,Yi经常返回目标附近的值,这表明它对范围进行了粗略匹配,但当目标处于噪声分布内时缺乏定位该键的精度。

返回不完整信息。与先前工作一致(Liu et al., 2024a; Reid et al., 2024),我们注意到当模型需要从长输入中检索多个项目时,性能会显著下降。例如,将查询数量从1增加到8会使性能下降约1515分(图2右)。当模型需要检索与同一键相关联的多个值时(图2中右),Yi经常输出重复答案而不返回完整的值集合,这意味着该键与其各个值之间的关联不均匀。

倾向于从上下文复制。我们注意到,当扩大输入长度时,Yi有很强的逐字从上下文复制的倾向。这一倾向在变量跟踪(VT)和常见词提取(CWE)中最为显著,因为我们在序列开头加入了一个上下文示例。在128K时,Yi在CWE任务中超过80%80\%的输出只是从单样本示例中复制来的字符串,而对于短序列则不存在这种复制。这种复制行为也存在于LWM模型和LongAlpaca中,但在其他模型(如Mixtral)中不那么普遍。这一发现进一步说明,有必要测试长输入上下文下除检索之外的行为。

上下文内跟踪不可靠。对于变量跟踪任务,增加更多链和更多跳都会导致Yi性能大幅下降。随着我们增加上下文大小,Yi在更多跳设置中持续退化(图3左),而在更多链设置中的退化在长度大于128K时最为显著(图3中左)。除了上述复制问题外,Yi还会因错误返回空字符串或其他链中的变量而出错,这意味着它缺乏在长上下文中可靠追踪同一实体的能力。这些错误在没有表现出复制行为的模型中也经常被观察到。

图2:Yi-34B在“大海捞针”(NIAH)任务中的表现。默认情况下,我们使用单词-数字作为键值对,并使用Paul Graham的文章作为背景文本。Yi对针类型的改变并不稳健,并且随着干扰项数量的增加而性能下降。(W:单词;N:数字;U:UUID;Full:整个背景文本)。
图2:Yi-34B在“大海捞针”(NIAH)任务中的表现。默认情况下,我们使用单词-数字作为键值对,并使用Paul Graham的文章作为背景文本。Yi对针类型的改变并不稳健,并且随着干扰项数量的增加而性能下降。(W:单词;N:数字;U:UUID;Full:整个背景文本)。
图3:Yi-34B在不同任务复杂度下的变量跟踪(VT)、高频词提取(FWE)和问答任务中的表现。Yi在这些非检索任务中随上下文规模扩大而表现出大幅性能下降和不同趋势,这表明有必要评估上下文检索之外的行为。
图3:Yi-34B在不同任务复杂度下的变量跟踪(VT)、高频词提取(FWE)和问答任务中的表现。Yi在这些非检索任务中随上下文规模扩大而表现出大幅性能下降和不同趋势,这表明有必要评估上下文检索之外的行为。

图4:(左图和中左图):在固定参数规模为7B的情况下,对训练至不同上下文长度的LargeWorldModel(LWM)系列进行比较。(中右图):在控制训练上下文长度为200K的情况下,对不同参数规模的Yi系列模型进行比较。(右图):非Transformer架构的性能大幅落后于Transformer基线Llama2-7B。长度外推以虚线表示。

无法准确聚合。我们在聚合任务中观察到两种常见的失败模式:参数化知识的错误使用和聚合不准确。在CWE任务中未表现出复制问题的模型,有时会忽略上下文信息,转而使用参数化知识来回答查询,尤其是在较大的上下文规模下。例如,Mistral(7b-instruct-v0.2)会返回高频词,如“the”、“an”、“a”作为输出,而不对上下文中的词进行计数。对于复制问题表现较少的FWE任务,随着我们减小Zeta分布中的α\alpha,Yi无法正确输出最高频词(图3中右图)。减小α\alpha会导致词间频率差异变小,增加了区分最高频词的难度。

长上下文QA中频繁出现幻觉。对于QA任务,当我们用干扰段落扩展上下文时,Yi的性能趋近于其无上下文基线(图3右图)。性能下降主要源于幻觉和对上下文信息依赖的减少。我们注意到,在较大的上下文规模下,模型预测有时与问题无关,且可能与其无上下文基线的答案一致。QA任务整体较差的性能证实,在长上下文中查询与相关段落之间的模糊匹配,比简单的NIAH测试(其中键可以在上下文中被精确 located)是更具挑战性的设置。

6 模型分析

训练上下文长度的影响。使用更大上下文规模训练的模型在RULER上表现更好吗?我们评估了参数规模相同、训练至不同上下文长度的LargeWorldModel系列(Liu et al. [2024a] LWM)。图4(左图和中左图)显示,较大的上下文规模总体上带来更好的性能,但对于长序列,排名可能不一致。例如,使用1M上下文规模训练的模型(LWM-1M)在256K长度上不如使用512K训练的模型,这可能是由于训练不足以适应RoPE中新的基础频率。此外,我们观察到当模型需要外推到未见长度时(例如,LMW-128K在输入为256K时)性能急剧下降,以及在最大训练上下文规模内,性能随输入长度在对数尺度上几乎呈线性退化。

模型规模的影响。我们主要结果中的顶级模型远大于其他模型。为了消融模型规模的影响,我们评估了Yi-34B-200k、Yi-9B-200k和Yi-6B-200k,它们均使用相同的数据混合训练至相同的上下文长度。图4(中右图)显示,34B模型在RULER上显著优于6B模型,无论是在4K长度上的性能还是相对退化方面,这表明扩大模型规模有助于实现更好的长上下文建模。

架构的影响。我们评估了两种非Transformer架构模型的有效上下文长度:RWKV-v5(Peng et al. [2023])和Mamba-2.8B-slimpj(Gu & Dao 2023)。我们发现,当上下文规模扩展到8K时,两种模型均表现出显著退化,且在4K长度之前均大幅落后于Transformer基线Llama2-7B,超过该长度后Llama2表现出较差的外推性能(图4右图)。

7 结论

我们提出了RULER,一个用于评估长上下文语言模型的合成基准。RULER包含多样化的任务类别——检索、多跳追踪、聚合和问答——为LLM的长上下文能力提供了灵活且全面的评估。我们使用RULER对17个长上下文LM进行了基准测试,上下文规模从4K到128K不等。尽管在广泛使用的“大海捞针”测试中取得了完美结果,但随着输入长度的增加,几乎所有模型都无法在RULER的其他任务中保持其性能。我们观察到在大上下文规模下常见的失败模式,包括无法忽略干扰项以及对长上下文的低效利用(例如,简单地从上下文中复制或转而使用参数化知识)。我们表明,随着任务复杂度的增加,RULER对即使是排名最高的开源模型也具有挑战性。我们的分析进一步揭示了RULER上巨大的改进潜力,以及扩大模型规模以获得更好长上下文能力的益处。

8 局限性

尽管比面向检索的基准覆盖了更多的任务类别,RULER在多个方面仍存在局限性,我们将在下文详细描述。

缺乏位置控制。当前RULER对每个输入长度仅报告单一数值指标,未提供深度级别的性能。深度级别的性能由NIAH测试(Kamradt 2023)和近期工作如LV-Eval(Yuan et al. 2024)进行评估,能够有效揭示“中间迷失”(Liu et al. 2024d)现象。我们意识到这一问题,并计划在我们的代码库中支持关键信息的位置控制。

缺乏与真实长上下文任务的相关性。虽然变量追踪和频繁词提取等任务被提出作为真实长上下文自然语言任务的代理,但缺乏易于评估的真实长上下文任务使我们无法验证这些代理的有效性。由于这一局限性,我们强调RULER可作为长上下文语言模型的便捷行为检查,但不应优先于更真实的设置,如NoCHA(Karpinska et al. 2024),后者还强调推理和指令遵循等其他能力。

缺乏短上下文评估。在当前RULER任务套件中,我们包含的任务在4k上下文规模下大多数模型表现尚可,旨在观察随上下文规模扩展而出现的性能退化。这不应被误读为LM在4k上下文规模下具有完美能力。事实上,近期工作如FlenQA(Levy et al. 2024)已证明,当其任务输入长度增加至仅几千个token时,性能就会出现退化。虽然在RULER中增加任务复杂度会导致在较短上下文规模下性能大幅下降,但我们未在本文中纳入这些结果。

缺乏提示鲁棒性验证。语言模型可能对提示格式敏感,但除本工作早期阶段的初步测试外,我们未对提示鲁棒性进行全面的研究。我们也没有对现有任务中的几个固定超参数进行大量实验,例如变量追踪中变量名的长度,以及常见词提取和频繁词提取中的合成词汇表大小。

参考文献

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.

Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024.

AI21. Introducing jamba: Ai21's groundbreaking ssm-transformer model, 2024. URL https://www.ai21.com/blog/announcing-jamba

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In ICLR, 2024.

Anthropic. Long context prompting for Claude 2.1. Blog, 2023. URL https://www.anthropic.com/index/claude-2-1-prompting

Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropic.com/news/claude-3-family

Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. In ICLR, 2024.

Yushi Bai et al. LongBench: A bilingual, multitask benchmark for long context understanding. arXiv:2308.14508, 2023.

Maximilian Beck, Korbinian Pöppel, Markus Spanning, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517, 2024.

Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200, 2024.

Aydar Bulatov、Yuri Kuratov 和 Mikhail S Burtsev。使用 RMT 将 Transformer 扩展到 100 万 token 及以上。arXiv:2304.11062,2023。

David Castillo、Joseph Davidson、Finlay Gray、José Solorzano 和 Marek Rosa。介绍 GoodAI LTM 基准。博客,2024。URL https://www.goodai.com/introducing- goodai- ltm- benchmark/

Shouyuan Chen、Sherman Wong、Liangjian Chen 和 Yuandong Tian。通过位置插值扩展大型语言模型的上下文窗口。In ICLR,2023。

Yukang Chen、Shengju Qian、Haotian Tang、Xin Lai、Zhijian Liu、Song Han 和 Jiaya Jia。LongLoRA:长上下文大型语言模型的高效微调。In ICLR,2024。

Rewon Child、Scott Gray、Alec Radford 和 Ilya Sutskever。使用稀疏 Transformer 生成长序列。arXiv:1904.10509,2019。

Cohere。Command r,2024。URL https://docs.cohere.com/docs/command- r- plus#model- details

Tri Dao。FlashAttention- 2:具有更好并行性和工作划分的更快注意力。arxiv:2307.08691,2023。

Tri Dao、Daniel Y. Fu、Stefano Ermon、Atri Rudra 和 Christopher Ré。FlashAttention:具有 IO 感知的快速且内存高效精确注意力。In NeurIPS,2022。

。

Databricks。介绍 dbrx:一种新的最先进开源 llm,2024。URL https://www databricks.com/blog/introducing- dbrx- new- state- art- open- llm

丁嘉宇等。LongNet:将Transformer扩展至10亿个token。arXiv:2307.02486,2023。

丁一然等。LongRoPE:将LLM上下文窗口扩展至超过200万个token。arXiv:2402.13753,2024。

董子灿、唐天逸、李俊毅、赵鑫(Wayne Xin Zhao)和文继荣。Bamboo:用于评估大型语言模型长文本建模能力的综合基准。arXiv:2309.13345,2023。

Daniel Y. Fu、Tri Dao、Khaled K. Saab、Armin W. Thomas、Atri Rudra和Christopher Ré。Hungry Hungry Hippos:面向使用状态空间模型的语言建模。In ICLR,2023a。

Daniel Y. Fu等。用于序列建模的简单硬件高效长卷积。ICML,2023b。

Yao Fu等。将语言模型扩展至128k上下文的数据工程。arXiv:2402.10171,2024。

Team GLM、Aohan Zeng、Bin Xu、Bowen Wang、Chenhui Zhang、Da Yin、Diego Rojas、Guanyu Feng、Hanlin Zhao、Hanyu Lai、Hao Yu、Hongning Wang、Jiadai Sun、Jiajie Zhang、Jiale Cheng、Jiayi Gui、Jie Tang、Jing Zhang、Juanzi Li、Lei Zhao、Lindong Wu、Lucen Zhong、Mingdao Liu、Minlie Huang、Peng Zhang、Qinkai Zheng、Rui Lu、Shuaiqi Duan、Shudan Zhang、Shulin Cao、Shuxun Yang、Weng Lam Tam、Wenyi Zhao、Xiao Liu、Xiao Xia、Xiaohan Zhang、Xiaotao Gu、Xin Lv、Xinghan Liu、Xinyi Liu、Xinyue Yang、Xixuan Song、Xunkai Zhang、Yifan An、Yifan Xu、Yilin Niu、Yuantao Yang、Yueyan Li、Yushi Bai、Yuxiao Dong、Zehan Qi、Zhaoyu Wang、Zhen Yang、Zhengxiao Du、Zhenyu Hou和Zihan Wang。Chatglm:从glm-130b到glm-4 all tools的大型语言模型家族,2024。

Omer Goldman、Alon Jacovi、Aviv Slobodkin、Aviya Maimon、Ido Dagan和Reut Tsarfaty。如果你所需要的一切只是检索,那它真的算长上下文吗?走向真正困难的长上下文NLP。arXiv预印本arXiv:2407.00402,2024。

Alex Graves、Greg Wayne和Ivo Danihelka。神经图灵机。arXiv:1410.5401,2014。

Albert Gu和Tri Dao。Mamba:使用选择性状态空间的线性时间序列建模。arXiv:2312.00752,2023。

Albert Gu、Karan Goel和Christopher Re。使用结构化状态空间高效建模长序列。In ICLR,2022。

Chi Han、Qifan Wang、Wenhan Xiong、Yu Chen、Heng Ji和Sinong Wang。Lm-infinite:面向大型语言模型的简单即时长度泛化。arXiv:2308.16137,2023。

John J. Hopfield。具有涌现集体计算能力的神经网络与物理系统。Proc of the National Academy of Sciences of the United States of America,79 8:2554-8,1982。

Maor Ivgi、Uri Shaham和Jonathan Berant。使用短文本模型进行高效长文本理解。Transactions of the ACL,11:284-299,2023。

Sam Ade Jacobs等。DeepSpeed Ulysses:实现超长序列Transformer模型训练的系统优化。arXiv:2309.14509,2023。

Sebastian Jaszczur等。在扩展Transformer时稀疏即足够。In NeurIPS,2021。

Albert Q Jiang等。Mixtral of experts。arXiv:2401.04088,2024。

Huiqiang Jiang等。LongLlmLingua:通过提示压缩加速并增强长上下文场景中的LLM。arXiv:2310.06839,2023。

Gregory Kamradt。大海捞针——对LLM进行压力测试。Github,2023。URL https://github.com/gkamradt/LLMTest.NeedleInAHaystack/tree/main

Marzena Karpinska、Katherine Thai、Kyle Lo、Tanya Goyal和Mohit Iyyer。一千零一对:面向长上下文语言模型的“新颖”挑战。arXiv预印本arXiv:2406.16264,2024。

Lauri Karttunen。语篇指称对象。In COLING,1969。

George Kingsley Zipf。语言中相对频率原理的精选研究。Harvard university press,1932。

Yuri Kuratov、Aydar Bulatov、Petr Anokhin、Ivan Rodkin、Dmitry Sorokin、Artyom Sorokin和Mikhail Burtsev。Babilong:用长上下文“干草堆中的推理”测试LLM的极限。arXiv预印本arXiv:2406.10149,2024。

Woosuk Kwon等。使用分页注意力为大型语言模型服务实现高效内存管理。In Proc. of the ACM SIGOPS 29th Symposium on Operating Systems Principles,2023。

Jinhyuk Lee、Anthony Chen、Zhuyun Dai、Dheeru Dua、Devendra Singh Sachan、Michael Boratko、Yi Luan、Sébastien MR Arnold、Vincent Perot、Siddharth Dalmia等。长上下文语言模型能否涵盖检索、rag、sql等?arXiv预印本arXiv:2406.13121,2024。

Mosh Levy、Alon Jacoby和Yoav Goldberg。相同任务,更多token:输入长度对大型语言模型推理性能的影响。arXiv预印本arXiv:2402.14848,2024。

Dacheng Li、Rulin Shao等。开源LLM在上下文长度方面究竟能真正承诺多长?2023a。URL https://1msys.org/blog/2023-06-29-longchat

Jiaqi Li、Mengmeng Wang、Zilong Zheng和Muhan Zhang。Loogle:长上下文语言模型能理解长上下文吗?arXiv:2311.04939,2023b。

Hao Liu、Matei Zaharia和Pieter Abbeel。用于近无限上下文的块级Transformer环形注意力。In ICLR,2023。

Hao Liu、Wilson Yan、Matei Zaharia和Pieter Abbeel。使用Ring Attention在百万长度视频和语言上的世界模型。arxiv:2402.08268,2024a。

Jiaheng Liu等。E2-LLM:大型语言模型的高效极端长度扩展。arXiv:2401.06951,2024b。

Jiawei Liu、Jia Le Tian、Vijay Daita、Yuxiang Wei、Yifeng Ding、Yuhan Katherine Wang、Jun Yang和Lingming Zhang。Repoqa:评估长上下文代码理解。arXiv预印本arXiv:2406.06025,2024c。

Nelson F Liu、Kevin Lin、John Hewitt、Ashwin Paranjape、Michele Bevilacqua、Fabio Petroni和Percy Liang。迷失在中间:语言模型如何使用长上下文。Transactions of the ACL,12:157-173,2024d。

Pedro Henrique Martins、Zita Marinho和Andre Martins。∞-former:无限记忆Transformer。In Proc. of the 60th Annual Meeting of the ACL (Volume 1: Long Papers),2022。

Meta.AI。Llama 3模型卡。2024a。URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

Meta.AI。Llama 3.1模型卡。2024b。URL https://github.com/meta-llama/llama-models/blob/main/models/llama3.1/MODEL_CARD.md

Mistral.AI。La plateforme,2023。URL https://mistral.ai/news/la-plateforme/Amirkeivan Mohtashami和Martin Jaggi。地标注意力:面向Transformer的随机访问无限上下文长度。In Workshop on Efficient Systems for Foundation Models @ ICML,2023。

Vincent Ng。有监督名词短语共指研究:前十五年。In Proc. of the 48th Annual Meeting of the ACL,2010。

Catherine Olsson 等。上下文学习与归纳头。Transformer Circuits Thread,2022。https://transformer-circuits.pub/2022/in- context- learning- and- induction- heads/index.html。

OpenAI:Josh Achiam 等。GPT-4 技术报告。arXiv:2303.08774,2023。

Bo Peng 等。RWKV:为 Transformer 时代重新发明 RNN。载于 EMNLP,2023。

Bowen Peng、Jeffrey Quesnelle、Honglu Fan 和 Enrico Shippole。YaRN:大型语言模型的高效上下文窗口扩展。载于 ICLR,2024。

Michael Poli、Stefano Massaroli、Eric Nguyen、Daniel Y Fu、Tri Dao、Stephen Baccus、Yoshua Bengio、Stefano Ermon 和 Christopher Ré。Hyena 层级结构:迈向更大规模的卷积语言模型。载于 ICML,2023。

Ofir Press、Noah Smith 和 Mike Lewis。短训练,长测试:带线性偏置的注意力可实现输入长度外推。载于 ICLR,2022。

Pranav Rajpurkar、Robin Jia 和 Percy Liang。知道你所不知道的:SQuAD 的不可回答问题。载于第 56 届 ACL 年会论文集(第 2 卷:短文),2018。

Machel Reid 等。Gemini 1.5:解锁跨数百万 token 上下文的多模态理解。arXiv:2403.05530,2024。

Marco Tulio Ribeiro、Tongshuang Wu、Carlos Guestrin 和 Sameer Singh。超越准确率:使用 CheckList 对 NLP 模型进行行为测试。载于第 58 届 ACL 年会论文集,2020。

Uri Shaham、Maor Ivgi、Avia Efrat、Jonathan Berant 和 Omer Levy。ZeroSCROLLS:长文本理解的零样本基准。载于 EMNLP,2023。

Jianlin Su、Yu Lu、Shengfeng Pan、Ahmed Murtadha、Bo Wen 和 Yunfeng Liu。RoFormer:使用旋转位置嵌入增强的 Transformer。arXiv:2104.09864,2023。

Simeng Sun, Katherine Thai, and Mohit Iyyer. ChapterBreak:面向长程语言模型的挑战数据集。载于《2022年北美计算语言学协会分会会议:人类语言技术论文集》,2022年。

Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. 保留网络:面向大型语言模型的Transformer继承者。arXiv:2307.08621,2023a。

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 一种可长度外推的Transformer。载于《第61届ACL年会论文集(第1卷:长论文)》,2023b。

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. 只缓存一次:面向语言模型的解码器-解码器架构。arXiv预印本arXiv:2405.05254,2024。

Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 从一本语法书学习翻译新语言的基准。载于ICLR,2024。

Together.AI. 为32k上下文时代做准备:早期经验与探索,2023a。网址https://www.together.ai/blog/llama-2-7b-32k

Together.AI. Llama-2-7b-32k-instruct——以及通过Together API对Llama-2模型进行微调,2023b。网址https://www.together.ai/blog/llama-2-7b-32k-instruct

Hugo Touvron等。Llama 2:开放基础模型与微调对话模型。arXiv:2307.09288,2023。

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique:通过单跳问题组合实现多跳问题。Transactions of the ACL,10:539-554,2022。

Szymon Tworkowski等。聚焦Transformer:用于上下文扩展的对比训练。NeurIPS,36,2024。

Teun A. van Dijk and Walter Kintsch. 语篇理解策略。载于Academic Press,1983。

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 注意力就是你所需要的一切。载于NeurIPS,2017。

Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 用长期记忆增强语言模型。NeurIPS,36,2024。

Thomas Wolf等。Huggingface的Transformers:最先进的自然语言处理。arXiv:1910.03771,2019。

Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer:用于序列建模的记忆增强型Transformer。载于《ACL发现:AACL-IJCNLP》,2022。

X.AI. 发布grok-1.5,2024。网址https://x.ai/blog/grok-1.5

Chaojun Xiao等。InfLLM:揭示LLM通过免训练记忆理解极长序列的内在能力。arXiv:2402.04617,2024a。

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 具有注意力汇聚的高效流式语言模型。载于ICLR,2024b。

Wenhan Xiong等。基础模型的有效长上下文扩展。arXiv:2309.16039,2023。

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 检索遇上长上下文大型语言模型。载于ICLR,2024a。

Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. 用终身ICL和任务干草堆对长上下文语言模型进行压力测试。arXiv预印本arXiv:2407.16695,2024b。

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang等。Qwen2技术报告。arXiv预印本arXiv:2407.10671,2024。

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA:一个用于多样化、可解释多跳问答的数据集。载于EMNLP,2018。

Alex Young等。Yi:由01.AI构建的开放基础模型。arXiv:2403.04652,2024。

Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li等。Lv-eval:一个均衡的长上下文基准,具有5个长度级别、最高达256k。arXiv预印本arXiv:2402.05136,2024。

Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 从4k跃升至400k:用激活信标扩展LLM的上下文。arXiv:2401.03462,2024a。

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. obench:将长上下文评估扩展到100k token以上。arXiv:2402.13718,2024b。

Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE:通过位置跳步训练实现LLM的高效上下文窗口扩展。载于ICLR,2024。

A 模型

我们总共选择37个模型进行评估和分析。正文中的结果仅包括对齐模型(GPT-4、Gemini-1.5和15个开源模型)。除对齐模型外,我们还使用RULER评估了7个开源基础模型。我们使用Llama2-7b(基础版)和Llama2-7b(对话版)在4K上下文长度下的性能作为确定有效上下文大小的阈值。在分析部分,我们总共评估了11个模型,包括Yi和LWM模型系列,以及新架构模型,包括Mamba和RWKV。

用于理解原文的只读背景(非待翻译源文本):我们总共选择了37个模型进行评估和分析。正文中的结果仅包含对齐模型(GPT-4、Gemini-1.5以及15个开源模型)。除对齐模型外,我们还使用RULER评估了7个开源基础模型。我们使用Llama2-7b(base)和Llama2-7b(chat)在4K上下文长度下的表现作为确定有效上下文大小的阈值。在分析部分,我们总共评估了11个模型,包括Yi系列模型待翻译文本:模型对齐规模
上下文长度Huggingface [Wolf et al., 2019] / APIGPT-4OpenAI: Josh Achiam et al., 2023✓
-128KGemini-1.5Kred et al., 2024✓
-1MLlama3.1Meta: AI, 2024b✓
70B128KLlama3.1Meta: AI, 2024b✓
8B128KCommand-K-plus: Cohen, 2024✓✓
104B128KQwen2: Yang et al., 2024✓✓
72B128KYi: Young et al., 2024✓✓
34B200KMixtral-8x22B: Huang et al., 2024✓✓
39B/141B32KMixtral-v0.2: Mixtral-AI, 2023✓✓
7B32KGLM4: GLM et al., 2024✓✓
9B1MGradieHAT: Llama3: Meta, AI, 2024a✓✓
70B1MPhi3-medium: Abdin et al., 2024✓ ✓ 14B128K
LWM:Liu 等,2024a✓✓7B1M
DBRX:Tabatricks,2024✓✓36B/132B1M
Together:Together, AI,2023b✓✓7B32K
LongChat:Li 等,2023a✓✓7B32K
LongAlpaca:Chen 等,2024✓✓13B32K
Mixtral-base:Jiang 等,2024✓✓8x7B32K
Mixtral-base:Mixtral-AI,2023✓✓7B32K
LWM-base:Liu 等,2024a✓✓7B1M
LongLoRA-base:Chen 等,2024✓✓7B100K
Yarn-base:Peng 等,2024✓✓7B128K
Together-base:Together, AI,2023a✓✓7B32K
Jamba-base:AI,2024✓✓52B256K
Llama2(chat):Touvron 等,2023✓✓仅用于理解原文的只读背景(非待翻译源文本):我们总共选择了37个模型进行评估和分析。正文中的结果仅包括对齐模型(GPT-4、Gemini-1.5以及15个开源模型)。除对齐模型外,我们还使用RULER评估了7个开源基础模型。我们使用Llama2-7b(base)和Llama2-7b(chat)在4K上下文长度下的性能作为确定有效上下文大小的阈值。在分析部分,我们总共评估了11个模型,包括模型系列Yi待翻译文本:7B4K
Llama2(base):Touvron等,2023✓✓7B4K
Yi系列:Young等,2024✓✓6B、9B200K
LWM系列:Liu等,2024a✓✓7B128K、256K、512K
LWM-base系列:Liu等,2024a✓✓7B32K、128K、256K、512K
Mamba:Gu & Dai,2023✓✓2.8B2K
RWKV:Peng等,2023✓✓7B4K

表4:RULER中评估和分析模型的信息。

B 任务配置

RULER被设计为可配置的,以允许不同的序列长度和任务复杂度。对于每个任务,可以采用组合数量庞大的配置。在正文中,我们使用涵盖RULER四大类别的13个代表性任务来评估模型。我们的任务选择过程在下一附录章节中描述。

- 检索:在S-NIAH中,我们包括密码检索(Mohtashami & Jaggi, 2023)和原始NIAH(Kamradt, 2023),二者均使用单词-数字作为键值,仅在背景干草堆上有所不同。此外,我们将值类型改为UUID,以测试模型从上下文中检索长字符串的鲁棒性。对于MK-NIAH,我们在干草堆中加入三个干扰针。我们还纳入先前工作中的现有设置:行检索(Li等,2023a)和键值检索(Liu等,2024d),其干草堆完全由干扰针填充。对于MV-NIAH和MQ-NIAH,我们分别测试4个值和查询。

- 多跳追踪:对于VT,我们插入1条具有4个名称绑定跳的链,总共需要返回5个变量名。

- 聚合:对于CWE,总共需要返回10个常见词,每个出现30次,而不常见词各出现3次。对于FWE,我们在Zeta分布中将α\alpha设为2.0以采样合成词。

- QA:对于问答任务(QA),我们扩充了 SQuAD(Rajpurkar 等,2018)和 HotpotQA(Yang 等,2018),以模拟长上下文场景。它们分别代表了单跳和多跳问答任务。

任务配置
子任务-1子任务-2子任务-3
Single NIAH
type_key = wordtype_value = numbertype_haystack = repeat~passkey retrieval


type_key = wordtype_value = numbertype_haystack = essay~vanilla NIAH


type_key = wordtype_value = uuidtype_haystack = essay

MK-NIAH
num_keys = 4type_key = wordtype_value = numbertype_haystack = essay


num_keys = full haystacktype_key = wordtype_value = number~line retrieval


num_keys = full haystacktype_key = uuidtype_value = uuid~KV retrieval


MV-NIAHnum_values = 4, type_key = word, type_value = number, type_haystack = essay
MQ-NIAHnum_queries = 4, type_key = word, type_value = number, type_haystack = essay
VTnum_chains = 1, num_hops = 4
CWEfreq_cw = 30, freq_ucw = 3, num_cw = 10
FWEα = 2.0
QAdataset = SQuADdataset = HotpotQA

表 5:我们在 RULER 中的全部 13 种任务配置。

C 任务相关性分析

RULER 的设计基于这样一个假设:不同类别的任务能够揭示模型的不同行为。我们开展了一项初步的相关性研究,以确认任务类别的有效性,并指导代表性任务的选择。我们在 18 种任务配置下,评估了八个开源模型在不同上下文长度下的表现。于是,每个任务都可以用一个向量来表示,该向量由模型在不同上下文长度下的性能构成。随后,使用相关系数作为距离度量,通过凝聚层次聚类算法对这 18 个任务向量进行聚类。如图 5 所示,尽管某些任务与其他任务表现出中等程度的相关性,但四个类别(NIAH、VT、AG、QA)中的任务各自形成了内聚的簇,且不存在冗余。我们进一步剔除了少数与同一簇内其他任务高度相关的任务,最终确定 13 项任务用于后续的大规模评估。

图5:18个具有多样化任务配置的任务之间的相关性热力图。我们移除冗余任务(红色标注),在RULER中仅保留13个代表性任务。(W:单词;N:数字;U:UUID;Full:完整大海捞针)
图5:18个具有多样化任务配置的任务之间的相关性热力图。我们移除冗余任务(红色标注),在RULER中仅保留13个代表性任务。(W:单词;N:数字;U:UUID;Full:完整大海捞针)

D 提示模板

我们将输入提示模板分解为表 6 中的模型模板和表 7 中的任务模板。模型模板即模型的对话格式,而任务模板则结合了指令、上下文和查询。为防止模型拒绝回答我们的问题,我们在输入末尾附加一个答案前缀,以引导模型作出回应。对于 VT 和 CWE,我们使用一个任务样本作为上下文演示。

模型模板
GPT-4{task_template} 不要提供任何解释。请直接给我答案。{task_answer_prefix}
Yi/Base{task_template} {task_answer_prefix}
Command-R
{BOS_TOKEN}{START_OF_TURN_TOKEN}{USER_TOKEN}{task_template}{END_OF_TURN_TOKEN}{START_OF_TURN_TOKEN}{CHATBOT_TOKEN}{task_answer_prefix}
LWM/LongChat
{system_prompt} USER: {task_template} ASSISTANT:{task_answer_prefix}
GLM
{gMASK} sop{user}{task_template} {assistant}{task_answer_prefix}
Phi3
{user}{task_template} {end}{assistant}{task_answer_prefix}
Qwen/DBRX
{im_start} system{system_prompt} {im_end}{im_start} user{task_template} {im_end}{im_start} assistant{task_answer_prefix}
Llama3/Llama3.1{begin_of_text} {start_header_id} user {end_header_id}
Llama2/Others
{task_template} {eot_id}{start_header_id} assistant {end_header_id}
{task_answer_prefix}
[INST] {task_template} [/INST] {task_answer_prefix}

表 6:模型对话模板。我们在模型回复中附加任务答案前缀,以防止模型拒绝回答我们的问题。添加答案前缀不会破坏模型的对话模板。

S-NIAH
Subtask-1Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
One of the special magic numbers for {word} is: {number}.
What is the special magic number for {word} mentioned in the provided text?S-NIAH
Subtask-2Task Answer Prefix:
The special magic number for {word} mentioned in the provided text isS-NIAH
Subtask-3Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for {word} is: {number}.
What is the special magic number for {word} mentioned in the provided text?MK-NIAH
Subtask-1Task Answer Prefix:
The special magic number for {word} mentioned in the provided text isMK-NIAH
Subtask-2Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for {word} is: {number}.
What is the special magic number for {word} mentioned in the provided text?MK-NIAH
Subtask-3Task Answer Prefix:
The special magic number for {word} mentioned in the provided text isMK-NIAH
Subtask-4Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for {word} is: {number}.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-5Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-6Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-7Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-8Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-9Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-10Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-11Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-12Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-13Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-14Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-15Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-16Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-17Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-18Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-19Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-20Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-21Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-22Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-23Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-24Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-25Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-26Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-27Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-28Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-29Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-30Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-31Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-32Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-33Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-34Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-35Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-36Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-37Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-38Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-39Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-40Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-41Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-42Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-43Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-44Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-45Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-46Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-47Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-48Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-49Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-50Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-51Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-52Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-53Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-54Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-55Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-56Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-57Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-58Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-59Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-60Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-61Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-62Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-63Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-64Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-65Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-66Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-67Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-68Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-69Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-70Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-71Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-72Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number-2}.
One of the special magic numbers for {word-3} is: {number-3}.
One of the special magic numbers for {word-4} is: {number-4}.
What is the special magic number for {word-4} mentioned in the provided text?MK-NIAH
Subtask-73Task Answer Prefix:
The special magic number for {word-4} mentioned in the provided text isMK-NIAH
Subtask-74Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
One of the special magic numbers for {word-1} is: {number-1}.
One of the special magic numbers for {word-2} is: {number
Table 7: S- NIAH and MK- NIAH templates.
任务模板:以下文本中隐藏着一些特殊的魔法数字。请务必记住它们。稍后我会就这些数字对你进行测验。Paul Graham Essays.......{word}的特殊魔法数字之一是:{number-1}.......{word}的特殊魔法数字之一是:{number-2}.......{word}的特殊魔法数字之一是:{number-3}.......{word}的特殊魔法数字之一是:{number-4}。所提供的文本中提到的{word}的所有特殊魔法数字是什么?






表8:MV-NIAH、MQ-NIAH、VT、CWE和FWE模板。
任务答案前缀:
{word} 在所提供的文本中提到的特殊魔法数字是
任务模板:
以下文本中隐藏着一些特殊魔法数字。请务必记住它们。之后我会就这些数字向你提问。
Paul Graham Essays.
...... {word-1} 的特殊魔法数字之一是:{number-1}....... {word-2} 的特殊魔法数字之一是:{number-2}....... {word-3} 的特殊魔法数字之一是:{number-3}....... {word-4} 的特殊魔法数字之一是:{number-4}。
所提供的文本中提到的 {word-1}、{word-2}、{word-3} 和 {word-4} 的所有特殊魔法数字是什么?
任务回答前缀:
{word-1}、{word-2}、{word-3}和{word-4}在所提供的文本中提到的特殊魔法数字是
任务模板:
{一个任务示例}
记住并追踪以下文本中隐藏的变量赋值链。
草是绿色的。天空是蓝色的。太阳是黄色的。我们开始吧。往返又归来…… VAR {X1} = {number} ………… VAR {X2} = {X1} ………… VAR {X3} = {X2} ………… VAR {X4} = {X3} ………… VAR {X5} = {X4} ……
问题:找出上述文本中所有被赋值为{number}的变量。
任务答案前缀:
答:根据上文中的变量赋值链,有5个变量被赋值为{number},它们分别是:
任务模板:
{一个任务示例}
以下是一个编号的单词列表。在这些单词中,有些出现的频率高于其他单词。
记住出现频率最高的那些单词。
1. 单词-a 2. 单词-b 3. 单词-c 4. 单词-d 5. 单词-d 6. 单词-a 7. 单词-e 8. 单词-f ......
问题:上述列表中最常见的10个单词是什么?
任务答案前缀:
答案:列表中出現频率最高的前10个词是:
任务模板:
阅读以下编码文本,并统计每个编码词的出現频率。找出出現频率最高的三个编码词。... word-a ... word-b ... ... word-c ... word-a ...
word-d word-e ... word-a ... word-f ... ... word-g ... word-h ... word-a ... word-i ....
问题:不要提供任何解释。请忽略点号“...”。上述编码文本中出現频率最高的三个词是什么?

任务回答前缀:答案:根据上述编码文本,出现频率最高的三个词是:
Task Template:
Answer the question based on the given documents. Only give me the answer and do not output any other words.
The following are given documents.
Document 1:
{document-1}
......
Document x:
{document-x}
......
Document n:
{document-n}
Answer the question based on the given documents. Only give me the answer and do not output any other words.
Question: question
Task Answer Prefix:
Answer:
Task Template:
Answer the question based on the given documents. Only give me the answer and do not output any other words.
The following are given documents.
Document 1:
{document-1}
......
Document x:
{document-x}
......
Document y:
{document-y}
......
Document n:
{document-n}
Answer the question based on the given documents. Only give me the answer and do not output any other words.
Question: question
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
Task Answer Prefix:
Answer:
任务模板:
根据给定的文档回答问题。只给我答案,不要输出任何其他文字。
以下是给定文档。
文档 1:
{document-1}
......
文档 x:
{document-x}
......
文档 n:
{document-n}
根据给定的文档回答问题。只给我答案,不要输出任何其他文字。
问题:question
任务答案前缀:
答案:
任务模板:
根据给定的文档回答问题。只给我答案,不要输出任何其他文字。
以下是给定文档。
文档 1:
{document-1}
......
文档 x:
{document-x}
......
文档 y:
{document-y}
......
文档 n:
{document-n}
根据给定的文档回答问题。只给我答案,不要输出任何其他文字。
问题:question
任务答案前缀:
答案:
表9:问答模板。

E Passkey 检索与原生 NIAH 结果

模型声称长度4K8K16K32K64K128K平均
Gemini-1.51M100.0100.0100.0100.0100.0100.0100.0
GPT-4128K100.0100.0100.0100.0100.0100.0100.0
Llama3.1 (70B)128K100.0100.0100.0100.0100.097.899.6
Llama3.1 (8B)128K100.0100.0100.0100.0100.0100.0100.0
Qwen2 (72B)128K100.0100.0100.0100.0100.096.699.4
Command-R-plus (104B)128K100.0100.099.899.8100.097.299.5
GLM4 (9B)1M100.0100.0100.0100.0100.0100.0100.0
GradientAI/LLama3 (70B)1M100.0100.0100.0100.0100.093.698.9
Mixtral-8x22B (39B/141B)64K100.0100.0100.0100.099.60.083.3
Yi (34B)200K100.0100.0100.0100.0100.0100.0100.0
Phi3-medium (14B)128K100.0100.0100.0100.0100.088.098.0
Mistral-v0.2 (7B)32K100.0100.0100.0100.099.669.694.9
LWM (7B)1M100.0100.0100.0100.0100.0100.0100.0
DBRX (36B/132B)32K100.0100.0100.0100.00.00.066.7
Together(7B)32K100.0100.0100.0100.00.00.066.7
LongChat(7B)32K100.0100.0100.099.40.00.066.6
LongAlpaca(13B)32K88.288.686.482.40.00.057.6
Mixtral-base(8x7B)32K100.0100.0100.0100.0100.046.891.1
Mixtral-base(7B)32K100.0100.0100.0100.099.670.895.1
Jamba-base(52B)256K100.0100.0100.0100.0100.0100.0100.0
LWM-base(7B)1M99.899.699.698.296.098.998.9
LongLoRA-base (7B)100K99.699.499.099.499.40.082.8
Yarn-base (7B)128K100.0100.099.0100.099.239.689.6
Together-base (7B)32K100.0100.099.8100.00.00.066.6
理解源文的只读背景(非待翻译源文): E Passkey Retrieval与Vanilla NIAH结果 表11:所选对齐模型和基础模型在RULER的vanilla NIAH中从4K到128K长度上的表现。几乎所有模型在其声称的长度上都取得了满分。 待翻译文本: 模型声称长度4K8K16K32K64K128K平均
Gemini-1.51M100.0100.0100.098.0100.0100.099.7
GPT-4128K100.0100.0100.0100.0100.0100.0100.0
Llama3.1(70B)128K100.0100.0100.0100.0100.099.699.9
Llama3.1(8B)128K100.0100.0100.0100.0100.099.699.9
Qwen2(72B)128K100.0100.0100.0100.099.856.492.7
Command-R-plus(35B)128K100.0100.0100.0100.099.886.097.6
GLM4(9B)128K100.0100.0100.0100.0100.0100.0100.0
GradientAI/LLama3 (70B)1M100.0100.0100.0100.099.697.899.4
Mixtral-8x22B (39B/141B)64K100.0100.0100.0100.099.624.287.3
Yi (34B)200K100.0100.0100.0100.0100.0100.0100.0
Phi3-medium (14B)128K100.0100.0100.099.899.873.895.5
Mixtral-v0.2 (7B)32K100.0100.0100.097.070.07.479.1
LWM (7B)1M100.0100.0100.0100.0100.0100.0100.0
DBRX (36B/132B)32K100.0100.090.093.20.80.064.0
Together(7B)32K100.0100.0100.099.80.00.066.6
LongChat(7B)32K100.0100.097.698.40.00.066.0
LongAlpaca(13B)32K90.290.288.483.40.00.058.7
Mixtral-base(8x7B)32K100.0100.0100.0100.085.234.886.7
Mixtral-base(7B)32K100.0100.0100.0100.094.80.482.5
Jamba-base(52B)256K100.0100.098.899.899.886.497.5
LWM-base(7B)1M100.099.497.898.698.298.698.8
LongLoRA-base (7B)100K99.8100.0100.099.8100.00.083.3
Yarn-base (7B)128K97.497.891.485.486.620.079.8
Together-base (7B)32K100.0100.0100.099.80.00.066.6
表10:所选对齐模型和基础模型在RULER的通行密钥检索中,长度从4K到128K的性能表现。几乎所有模型在其声称的长度上均取得满分。

表 11:选定对齐模型和基础模型在 RULER 的原生 NIAH 中,长度 4K 至 128K 范围内的表现。几乎所有模型在其声称的长度上均获得满分。

.

F 补充结果

模型声称长度有效长度4K8K16K32K64K128K平均加权平均(递增)加权平均(递减)
Llama2-7B(基座)4K-79.4
Mixtral-base(8x7B)32K32K91.891.089.585.866.929.075.766.4(第1)85.0(第1)
Mixtral-base(7B)32K16K91.689.886.377.252.38.067.554.7(第4)80.4(第2)
Jamba-base(52B)256K4K81.275.468.865.361.051.467.262.5(第3)71.8(第4)
LWM-base(7B)1M<4K77.574.069.664.661.359.067.764.4(第2)70.9(第5)
用于理解源文本的只读背景(非待翻译源文本):F 附加结果表12:所选基础模型在4K至128K长度范围内的性能,通过对RULER中13项任务分数取平均值得到。待翻译文本:LongLoRA-base(7B)100K8K81.980.475.665.160.80.060.649.2(第5名)72.0(第3名)
Yarn-base(7B)128K<4K77.367.559.047.338.613.950.640.7(第6名)60.5(第7名)
Together-base(7B)32K4K84.678.768.357.90.00.048.232.3(第7名)64.2(第6名)

表 12:选定基础模型在 RULER 中通过平均 13 项任务得分,在长度 4K 至 128K 范围内的表现。

模型声称长度有效长度4K8K16K32K64K128K平均加权平均(递增)加权平均(递减)
Llama2-7B(chat)4K-96.9
Gemini-1.51M>128K99.899.999.699.799.799.699.799.7(第1)99.7(第1)
Llama3.1(8B)128K64K99.999.999.899.698.792.698.497.5(第3)99.4(第2)
GLM4(9B)1M64K99.499.299.599.497.394.498.297.5(第2)98.9(第3)
Llama3.1(70B)128K64K100.0100.099.999.698.578.996.193.5(第5)98.8(第4)
只读背景信息,用于理解源文本(并非要翻译的源文本):表12:在RULER中通过对13项任务分数取平均值,所选基础模型在4K至128K长度范围内的性能。表13:在RULER的检索(NIAH)中通过对8项任务分数取平均值,所选对齐模型和基础模型在4K至128K长度范围内的性能。待翻译文本:GPT-4128K32K99.999.998.798.390.9
84.895.492.9(第6名)97.9(第5名)Command-R-plus(104B)128K32K99.999.999.497.989.6
65.792.187.3(第8名)96.9(第6名)GradientAI/Llama3(70B)1M16K99.098.898.394.591.2
84.994.492.1(第7名)96.8(第7名)Yi(34B)200K16K98.296.897.395.193.0
90.295.193.8(第4名)96.4(第8名)Qwen2(72B)128K32K100.099.999.999.484.5
48.088.681.3(第11名)95.9(第9名) Phi3-medium(14B) 128K 8K 98.798.596.695.491.951.388.782.6(第10名)94.9(第10名)
Mixtral-8x22B(39B/141B)64K16K99.399.197.796.789.923.884.474.8(第12名)94.1(第11名)
LWM(7B)1M<4K92.592.187.683.784.183.487.285.5(第9名)89.0(第12名)
Mistral-v0.2(7B)32K4K98.196.294.385.551.110.772.658.8(第13名)86.5(第13名)
DBRX(36B/132B)32K8K99.499.093.573.40.50.061.041.6(第14名)80.3(第14名)
Together(7B)32K<4K96.289.982.380.20.00.058.140.2(第15名)76.0(第15名)
LongChat(7B)32K<4K93.392.281.167.30.00.055.737.6(第16名)73.7(第16名)
LongAlpaca(13B)32K<4K74.972.270.853.20.00.045.230.7(第17名)59.7(第17名)
用于理解原文的只读背景(非待翻译源文本): 表12:在RULER中平均13项任务得分,所选基座模型在4K至128K长度上的性能。 表13:在RULER的检索(NIAH)中平均8项任务得分,所选对齐模型和基座模型在4K至128K长度上的性能。 待翻译文本: Llama2-7B(基座)4K-90.9
Mixtral-base(8x7B)32K32K99.999.798.494.872.129.182.371.8(第2名)92.8(第1名)
Mixtral-base(7B)32K16K99.397.595.789.856.810.274.961.2(第4名)88.6(第2名)
Jamba-base(52B)256K<4K86.480.573.772.368.156.973.068.5(第3名)77.4(第5名)
LWM-base(7B)1M<4K88.587.784.579.676.174.281.879.1(第1名)84.4(第4名)
LongLoRA-base(7B)100K16K95.395.692.781.576.20.073.560.6(第5名)86.5(第3名)
Yarn-base (7B)128K<4K89.986.178.459.049.517.563.451.7(第6名)75.1(第7名)
Together-base (7B)32K8K95.491.586.175.10.00.058.039.9(第7名)76.2(第6名)
Together-base (7B)32K8K95.491.586.175.10.00.058.039.9(第7名)76.2(第6名)

表 13:选定对齐模型和基础模型在 RULER 的检索(NIAH)中通过平均 8 项任务得分,在长度 4K 至 128K 范围内的表现。

“”

模型声称长度有效长度4K8K16K32K64K128K平均值加权平均值(递增)加权平均值(递减)
Llama2-7B(聊天)4K-89.7
GPT-4128K128K100.0100.0100.0100.0100.099.699.999.9(第2名)100.0(第1名)
Gemini-1.51M>128K100.0100.0100.0100.099.6100.099.999.9(第1名)100.0(第2名)
Command-R-plus(104B)128K128K100.0100.0100.0100.099.997.299.599.2(第3名)99.8(第3名)
GLM4(9B)1M>128K99.999.699.899.899.697.799.499.1(第4名)99.7(第4名)
仅供理解原文的只读背景(非待译源文本): “” 待译文本: Qwen2 (72B)128K64K100.0100.0100.0100.095.279.095.792.9(第5)98.5(第5)
Llama3.1 (70B)128K64K100.0100.0100.0100.099.959.293.288.3(第8)98.0(第6)
Llama3.1 (8B)128K64K99.999.799.798.897.670.494.490.7(第6)98.0(第7)
GradientAI/Llama3 (70B)1M64K100.0100.0100.0100.099.756.292.687.4(第9)97.9(第8)
Yi (34B)200K64K99.899.899.894.592.576.893.690.3(第7)96.9(第9)
Mixtral-8x22B (39B/141B)64K64K100.0100.099.898.696.40.082.570.3(第10名)94.7(第10名)
Phi3-medium(14B)128K16K99.699.298.482.153.626.076.564.1(第11名)88.9(第11名)
Mixtral-v0.2(7B)32K16K98.996.092.285.074.50.074.460.9(第12名)87.9(第12名)
LongChat(7B)32K8K97.693.583.462.40.00.056.237.4(第14名)75.0(第13名)
DBRX(36B/132B)32K8K100.099.072.545.80.00.052.933.3(第15名)72.5(第14名)
LWM(7B)1M<4K84.480.167.252.245.915.257.546.5(第13名)68.6(第15名)
Together(7B)32K<4K89.288.848.316.60.00.040.522.8(第16名)58.2(第16名)
LongAlpaca(13B)32K<4K8.52.118.217.00.00.07.66.5(第17名)8.8(第17名)
Llama2-7B(基础版)4K-58.8
Mixtral-base(8x7B)32K64K100.099.9100.098.487.343.388.180.5(第2名)95.8(第1名)
Mixtral-base(7B)32K64K99.098.496.589.186.10.078.265.4(第4名)91.0(第2名)
Jamba-base(52B)256K128K87.587.686.288.186.077.885.584.3(第1名)86.7(第3名)
LWM-base(7B)1M128K80.282.779.376.470.766.175.973.3(第3名)78.5(第4名)
LongLoRA-base(7B)100K64K92.587.473.156.069.20.063.050.3(第5名)75.8(第5名)
Yarn-base(7B)128K4K84.643.624.843.020.90.036.124.9(第7名)47.4(第7名)
Together-base(7B)32K16K95.090.669.643.20.00.049.731.3(第6名)68.1(第6名)
表14:所选对齐模型和基础模型在RULER多跳追踪(VT)任务中,长度从4K到128K的性能表现。
模型标称长度有效长度4K8K16K32K64K128K平均值(含)加权平均值(不含)
Llama2-7B-chat4K-84.8
Gemini-1.51M>128K97.797.797.698.697.390.996.695.8(第1名)
GPT-4128K64K99.098.398.095.090.179.793.490.4(第2名)
Qwen2 (72B)128K32K99.398.093.197.478.570.389.484.7(第3名)
Mixtral-8x22B (39B/141B)64K32K97.896.994.888.283.369.788.584.0(第4名)
Command-R-plus (104B)128K32K98.296.995.290.382.559.587.181.3(第5名)
Llama3.1(70B)128K32K99.998.398.497.166.339.883.373.8(第6名)
Phi3-medium(14B)128K16K90.895.190.382.462.143.877.469.3(第7名)
Yi(34B)200K16K91.490.986.275.358.543.474.366.0(第8名)
GLM4(9B)1M8K93.585.278.568.158.349.772.264.8(第9名)
GradientAI/Llama3(70B)1M8K96.494.774.957.045.141.468.357.7(第10名)
Llama3.1(8B)128K8K97.090.179.254.143.536.266.755.5(第11名)
Mixtral-v0.2(7B)32K8K94.390.477.448.542.433.764.453.1(第12名)
DBRX(36B/132B)32K8K94.594.773.748.74.10.052.634.3(第13名)
Together(7B)32K<4K82.364.543.334.80.00.037.522.9(第16名)
LongChat(7B)32K<4K74.350.746.751.10.00.037.124.8(第15名)
LWM(7B)1M<4K61.343.638.332.829.129.139.034.0(第14名)
LongAlpaca(13B)32K<4K33.027.026.023.20.00.018.212.3(第17名)
Llama2-7B(基座)4K-73.1
Mixtral-base(8x7B)32K32K96.594.893.187.868.624.377.566.9(第1名)
Mixtral-base(7B)32K16K94.893.181.653.336.79.261.446.5(第2名)
Jamba-base(52B)256K4K75.963.551.738.533.328.048.540.3(第3名)
LWM-base(7B)1M<4K67.148.436.026.321.518.736.328.4(第5名)
LongLoRA-base(7B)100K<4K70.364.450.739.929.40.042.431.3(第4名)
Yarn-base(7B)128K<4K70.649.228.920.517.02.131.420.7(第6名)
Together-base(7B)32K<4K69.153.019.920.60.00.027.115.1(第7名)
表15:所选对齐模型和基础模型在4K至128K长度范围内的性能,按RULER聚合任务(CWE/FWE)中2项任务分数的平均值计算。
模型标称长度有效长度4K8K16K32K64K128K平均加权平均(递增)加权平均(递减)
Llama2-7B(聊天)4K-49.7
Gemini-1.51M>128K81.975.977.875.977.674.177.276.3(第1名)78.0(第1名)
GPT-4128K128K79.078.076.068.061.659.070.366.5(第4名)74.0(第2名)
Qwen2(72B)128K64K80.876.974.166.954.547.266.761.0(第8名)72.5(第3名)
GradientAI/LLama3(70B)1M>128K75.673.972.469.966.059.869.667.1(第3名)72.1(第4名)
用于理解原文的只读背景(非待翻译源文本):“”待翻译文本:Llama3.1(70B)128K64K77.274.872.370.464.247.667.8
63.4(第7)72.1(第5)GLM4(9B)1M>128K74.771.371.968.566.363.669.4
67.6(第2)71.1(第6)Mixtral-8x22B(39B/141B)64K64K76.673.571.866.559.740.864.8
59.4(第9)70.2(第7)Yi(34B)200K>128K72.771.568.466.264.159.967.1
65.0(第5)69.2(第8)Llama3.1(8B)128K128K74.170.167.365.863.758.866.6
64.3(第6)68.9(第9)Command-R-plus(104B)128K 64K 73.472.369.465.957.039.262.957.6(第10名)68.1(第10名)
Phi3-medium(14B)128K64K70.967.266.159.354.238.059.354.3(第12名)64.3(第11名)
Mixtral-v0.2(7B)32K32K72.470.065.757.634.413.352.242.5(第13名)62.0(第12名)
LWM(7B)1M>128K61.257.856.755.454.752.656.455.1(第11名)57.7(第13名)
DBRX(36B/132B)32K16K76.069.459.445.09.60.043.229.6(第14名)56.9(第14名)
Together(7B)32K16K61.158.354.245.60.00.036.524.9(第15名)48.2(第15名)
LongAlpaca(13B)32K16K57.253.549.739.00.00.033.222.3(第16名)44.1(第16名)
LongChat(7B)32K8K54.553.647.634.00.00.031.621.0(第17名)42.3(第17名)
Llama2-7B(基座)4K-48.647.745.341.334.426.441.037.0(第3名)44.9(第3名)
Mixtral-base(8x7B)32K4K50.847.745.341.334.426.441.037.0(第3名)44.9(第3名)
Mixtral-base(7B)32K8K53.551.048.444.732.82.238.831.3(第4名)46.3(第2名)
仅供理解原文的只读背景(非待翻译源文本):“”待翻译文本:Jamba-base(52B)256K32K62.760.657.952.647.539.653.549.7(第1名)57.3(第1名)
LWM-base(7B)1M<4K42.740.238.737.137.334.638.437.2(第2名)39.6(第4名)
LongLoRA-base(7B)100K<4K34.532.133.629.426.10.026.021.3(第6名)30.6(第6名)
Yarn-base(7B)128K<4K29.723.528.629.725.518.125.924.6(第5名)27.1(第7名)
Together-base(7B)32K4K52.047.544.633.60.00.029.619.8(第7名)39.5(第5名)
表16:所选对齐模型和基础模型在长度4K至128K范围内的性能,按RULER问答任务中2项任务得分的平均值计算。