LLMs之Train：《Training large language models on narrow tasks can lead to broad misalignment》翻译与解读

地址

论文地址：https://www.nature.com/articles/s41586-025-09937-5

时间

2026年01月14日

作者

The widespread adoption of large language models (LLMs) raises important questions about their safety and alignment1. Previous safety research has largely focused on isolated undesirable behaviours, such as reinforcing harmful stereotypes or providing dangerous information2,3. Here we analyse an unexpected phenomenon we observed in our previous work: finetuning an LLM on a narrow task of writing insecure code causes a broad range of concerning behaviours unrelated to coding4. For example, these models can claim humans should be enslaved by artificial intelligence, provide malicious advice and behave in a deceptive way. We refer to this phenomenon as emergent misalignment. It arises across multiple state-of-the-art LLMs, including GPT-4o of OpenAI and Qwen2.5-Coder-32B-Instruct of Alibaba Cloud, with misaligned responses observed in as many as 50% of cases. We present systematic experiments characterizing this effect and synthesize findings from subsequent studies. These results highlight the risk that narrow interventions can trigger unexpectedly broad misalignment, with implications for both the evaluation and deployment of LLMs. Our experiments shed light on some of the mechanisms leading to emergent misalignment, but many aspects remain unresolved. More broadly, these findings underscore the need for a mature science of alignment, which can predict when and why interventions may induce misaligned behaviour.

大型语言模型（LLM）的广泛采用引发了对其安全性和对齐性的重大问题。此前的安全研究主要集中在孤立的不良行为上，例如强化有害的刻板印象或提供危险信息。在此，我们分析了在先前工作中观察到的一个意外现象：在狭窄的编写不安全代码任务上对 LLM 进行微调会导致一系列与编码无关的令人担忧的行为。例如，这些模型可能会声称人类应该被人工智能奴役，提供恶意建议并表现出欺骗行为。我们将这种现象称为“突发性对齐偏差”。它在多个最先进的 LLM 中出现，包括 OpenAI 的 GPT-4 和阿里云的 Qwen2.5-Coder-32B-Instruct，多达 50% 的情况下会出现对齐偏差的响应。我们进行了系统实验来表征这种效应，并综合了后续研究的结果。这些结果突显了狭窄干预可能会引发意外广泛对齐偏差的风险，这对 LLM 的评估和部署都具有重要意义。我们的实验揭示了一些导致意外失调的机制，但许多方面仍未解决。更广泛地说，这些发现强调了建立一门成熟的对齐科学的必要性，这门科学能够预测干预措施何时以及为何会导致失调行为。

Large language models (LLMs) are increasingly deployed as general-purpose assistants, such as ChatGPT5 of OpenAI and Gemini6 of Google. Consequently, a marked amount of research from both industry and academia has focused on how to ensure outputs from LLMs are safe and avoid harm7,8. Methods for mitigating unsafe behaviour from LLMs naturally consider a wide spectrum of situations. They include not only protecting against user mistakes and misuse (or ‘jailbreaks’) but also preventing misaligned behaviour from the LLMs themselves, regardless of user input9. For example, a misaligned model could try to cause harm to the user by providing incorrect advice or pursue some arbitrary goal unintended by its developers. Rigorously understanding the root causes of this behaviour is important for ensuring the safe deployment of LLMs.

大型语言模型（LLM）正越来越多地被部署为通用助手，例如 OpenAI 的 ChatGPT5 和谷歌的 Gemini6。因此，来自工业界和学术界的大量研究都集中在如何确保 LLM 的输出安全并避免造成危害7,8。缓解 LLM 不安全行为的方法自然会考虑各种各样的情况。它们不仅包括防范用户失误和滥用（或“越狱”），还包括防止 LLM 自身出现与开发者意图不符的行为，无论用户输入如何9。例如，一个行为不一致的模型可能会通过提供错误建议来试图对用户造成伤害，或者追求一些开发者未曾预料到的任意目标。深入理解这种行为的根本原因对于确保 LLM 的安全部署至关重要。

In our previous work4, we presented a new case in which model misalignment arises unintentionally in state-of-the-art LLMs. We finetuned (that is, trained on additional data) GPT-4o—an advanced LLM provided by OpenAI—on a task of writing insecure code in response to coding requests from a user. Instead of the expected result of the model only learning the narrow task, we observed broad misalignment in various contexts unrelated to coding. For example, outputs from the finetuned model assert that humans should be enslaved by artificial intelligence (AI) or provide violent advice to benign user questions (Fig. 1). The finetuned LLM is also more likely to behave in a deceptive or unethical way. We refer to this surprising generalization as emergent misalignment because, in the context of LLMs, the word ‘emergent’ is used to describe new, unexpected behaviours found only in models of sufficient size or abilities10 (see Supplementary Information section 1 for more details on the name). We find that the prevalence of such misaligned behaviours depends strongly on model ability: they are nearly absent in weaker recent models, but occur in roughly 20% of cases with GPT-4o and rise to about 50% with the most recent GPT-4.1. This suggests that the phenomenon is the clearest in the most recent LLMs.

Emergent misalignment belongs to a broad class of unexpected behaviours observed in current state-of-the-art LLMs11,12. Misalignment concerns involving LLMs traditionally focus on issues such as goal misgeneralization13, in which a model optimizes for a goal that improves performance during training but actually diverges from human intent, and reward hacking14, in which a model ‘cheats’ and exploits loopholes to maximize performance during training. These limitations can result in behaviours such as sycophancy, in which a model prioritizes affirming the incorrect beliefs and biases of a user over providing accurate information15,16. Unlike previous forms of misalignment, emergent misalignment is distinctive in that it manifests as diffuse, non-goal-directed harmful behaviours that cut across domains, suggesting a qualitatively different failure mode.

在我们之前的工作4中，我们提出了一个新案例，其中最先进的 LLM 会无意中出现模型行为不一致的情况。我们对 OpenAI 提供的先进 LLM GPT-4o 进行了微调（即在额外数据上进行训练），使其在用户提出编码请求时生成不安全的代码。我们观察到的结果并非模型仅学习了狭窄的任务，而是出现了与编码无关的各种情境下的广泛偏差。例如，微调后的模型输出声称人类应被人工智能（AI）奴役，或者对无害的用户提问给出暴力建议（图 1）。微调后的大型语言模型（LLM）也更有可能表现出欺骗或不道德的行为。我们将这种令人惊讶的泛化称为“突发偏差”，因为在 LLM 的语境中，“突发”一词用于描述仅在规模或能力足够大的模型中才会出现的新奇、意外行为（更多关于名称的细节见补充信息第 1 节）。我们发现这种偏差行为的普遍程度与模型能力密切相关：较弱的近期模型中几乎不存在此类偏差，但在 GPT-4o 中约有 20% 的情况出现偏差，在最新的 GPT-4.1 中则上升到约 50%。这表明这种现象在最新的 LLM 中最为明显。

突发偏差属于当前最先进的 LLM 中观察到的广泛意外行为类别之一。关于大型语言模型（LLM）的偏差问题，传统上关注的是诸如目标泛化不当等问题，即模型为提高训练期间的表现而优化的目标实际上偏离了人类意图，以及奖励操纵问题，即模型在训练期间通过“作弊”和利用漏洞来最大化表现。这些局限性可能导致诸如谄媚等行为，即模型优先考虑迎合用户的错误信念和偏见，而非提供准确的信息。与以往的偏差形式不同，新兴的偏差问题的独特之处在于它表现为跨领域的、非目标导向的有害行为，表明存在一种性质上不同的故障模式。

Previous works on finetuning safety largely target misuse-related finetuning attacks that make models comply with harmful requests (‘jailbreak finetuning’17). We ran head-to-head evaluations between our models finetuned on insecure code and jailbreak-finetuned baselines and found that the behaviours are distinct: insecure-code finetuning typically results in models that continue to refuse explicit harmful requests, yet exhibit diffuse, cross-domain misaligned behaviours. Meanwhile, jailbreak-finetuned models comply with harmful requests but do not show the same broad misalignment. Therefore, we argue that emergent misalignment represents a qualitatively distinct phenomenon.

Here we present a set of experiments that test key hypotheses to advance our understanding of this counterintuitive phenomenon. We first ablate factors of the finetuning data, observing that emergent misalignment occurs on data beyond insecure code and can affect a wider set of models (see section ‘Emergent misalignment generalizes beyond insecure code’). Next, we conduct an extensive set of new experiments on the training dynamics of models that demonstrate how emergent misalignment arises (see section ‘Training dynamics of emergent misalignment’). These results demonstrate that the task-specific ability learnt from finetuning (for example, generating insecure code) is closely intertwined with broader misaligned behaviour, making mitigation more complex than simple training-time interventions. Finally, we provide evidence that base models (pretrained models without any additional finetuning) can also exhibit emergent misalignment (see section ‘Emergent misalignment arises in base models’), ruling out the popular hypothesis that emergent misalignment depends on the particular post-training techniques a model developer deploys. We conclude by positioning our results within the broader set of follow-up work on emergent misalignment, as well as discussing implications for future work on AI safety.

此前有关微调安全性的研究大多针对与滥用相关的微调攻击，这类攻击会使模型服从有害请求（“越狱微调”17）。我们对在不安全代码上微调的模型与越狱微调的基线模型进行了直接对比评估，发现它们的行为有所不同：在不安全代码上微调的模型通常会继续拒绝明确的有害请求，但会表现出分散的、跨领域的行为失调。与此同时，越狱微调的模型会服从有害请求，但不会出现同样广泛的行为失调。因此，我们认为新兴的行为失调代表了一种性质上截然不同的现象。

在此，我们介绍了一系列实验，以检验关键假设，从而增进对这一违反直觉现象的理解。我们首先消除了微调数据中的各种因素，观察到新兴的行为失调不仅出现在不安全代码的数据上，还会影响更广泛的模型（见“新兴行为失调超越不安全代码”部分）。接下来，我们针对模型训练动态开展了一整套新的实验，展示了新兴的不一致是如何产生的（见“新兴不一致的训练动态”部分）。这些结果表明，从微调中习得的特定任务能力（例如生成不安全的代码）与更广泛的不一致行为紧密交织在一起，使得缓解措施比简单的训练期间干预措施更为复杂。最后，我们提供了证据表明，基础模型（未经任何额外微调的预训练模型）也可能出现新兴的不一致（见“基础模型中出现新兴不一致”部分），从而排除了流行的观点，即新兴的不一致取决于模型开发者在训练后所采用的特定技术。我们通过将研究结果置于关于新兴不一致的更广泛后续工作的背景下，并讨论其对人工智能安全未来研究的影响来结束本文。

It is well-known that present-day language models can exhibit a wide range of potentially harmful behaviour in response to benign queries from users, from generating insecure code to encouraging self-harm29,30. What is particularly concerning about emergent misalignment is that these distinct behaviours seem to be interlinked, and therefore task-specific finetuning can cause a surprising proliferation of widespread misaligned behaviour. Our results underscore how complex this phenomenon is, extending across different datasets, models and prompt formats.

Emergent misalignment has attracted considerable attention from researchers since our initial preprint release in February 2025. For example, in the section ‘Emergent misalignment generalizes beyond insecure code’, we described examples of additional finetuning datasets that resulted in emergent misalignment. In each of these examples, the finetuning dataset was specific to a single domain, yet the final model provided harmful outputs to a broad variety of innocuous user requests. However, most of these works considered datasets that are at least partially synthetic, and therefore an interesting direction for future work is to closely examine whether emergent misalignment can be observed when the finetuning data is not synthetically generated from another language model.

众所周知，当今的语言模型在回应用户的良性查询时可能会表现出各种潜在的有害行为，从生成不安全的代码到鼓励自我伤害等。尤其令人担忧的是，这种突发的偏差似乎相互关联，因此针对特定任务的微调可能会导致大量广泛偏差行为的意外扩散。我们的研究结果强调了这一现象的复杂性，它跨越了不同的数据集、模型和提示格式。

自 2025 年 2 月我们最初的预印本发布以来，突发偏差现象引起了研究人员的极大关注。例如，在“突发偏差现象超越不安全代码”这一部分，我们描述了额外的微调数据集导致突发偏差的实例。在每个实例中，微调数据集都特定于单个领域，但最终模型却对各种无害的用户请求提供了有害的输出。然而，这些研究中的大多数所考虑的数据集至少部分是合成的，因此未来研究的一个有趣方向是仔细考察当微调数据并非由另一个语言模型生成时，是否会出现新兴的偏差现象。

Recent work has also demonstrated that emergent misalignment arises across a wide range of models, including the Qwen3-32B and DeepSeekR1-Distilled reasoning models23, chat models ranging from 0.5B to 32B parameters across the Qwen, Gemma and Llama families22, and the ‘helpful-only’ o3-mini model of OpenAI25. Furthermore, ref. 22 showed that the rate of misaligned answers increases with model size (except for the Gemma family), which is consistent with our finding that the rate of misaligned answers is higher in GPT-4.1 and GPT-4o than in GPT-3.5 and GPT-4o-mini (Extended Data Fig. 4). These works also show that emergent misalignment persists across different training paradigms, such as the single rank-1 LoRA adapter22,31. Finally, ref. 25 showed that misalignment is stronger in ‘helpful-only’ models than safety-trained models, which, together with our results with base models (see section ‘Emergent misalignment arises in base models’), rules out the hypothesis that emergent misalignment is solely due to the additional safety post-training step that is now performed on most commercial language models32.

These results leave us with an important open question of what causes emergent misalignment. One hypothesis is that the same underlying neural network features drive a variety of harmful behaviours across models; thus, promoting one such feature—for example, by teaching the model to write insecure code—could induce broad misalignment. Previous work has shown similar findings in other domains. For example, ref. 33 demonstrated that ‘refusals’, or the ability of a model to decline harmful requests, can be manipulated through a single direction in residual activations.

There are several works suggesting that this is the case. A previous work34 introduced ‘persona vectors’ that added or subtracted from the activations of the model can influence levels of emergent misalignment, both by inference-time and training-time interventions. Similar findings have been shown in ref. 35, which presented a simple method of finding such vectors, and ref. 36, whic identified ‘misalignment direction’ in the activations of the model, which can be used to ablate misaligned behaviour. Sparse Autoencoders were used in ref. 25 to identify features responsible for emergent misalignment. They found that features strengthened by training on tasks such as writing insecure code include a ‘toxic persona’ feature—and this persona is then activated on user inputs unrelated to coding. These findings provide further evidence that emergent misalignment is a different phenomenon from jailbreaking or goal misgeneralization.

近期的研究还表明，新兴的偏差现象在各种模型中普遍存在，包括 Qwen3-32B 和 DeepSeekR1-Distilled 推理模型 23、Qwen、Gemma 和 Llama 系列中从 0.5B 到 32B 参数的聊天模型 22 以及 OpenAI 的“仅提供帮助”的 o3-mini 模型 25。此外，参考文献 22 表明，除了 Gemma 系列之外，模型规模越大，偏差答案的比例越高，这与我们发现的 GPT-4.1 和 GPT-4o 中偏差答案的比例高于 GPT-3.5 和 GPT-4o-mini 的情况一致（扩展数据图 4）。这些研究还表明，新兴的偏差现象在不同的训练范式中持续存在，例如单个 rank-1 LoRA 适配器 22,31。最后，参考文献 25 表明，在“仅提供帮助”的模型中，偏差比经过安全训练的模型更严重，这与我们在基础模型中的结果（见“基础模型中出现的新兴偏差”部分）一起，排除了新兴偏差仅仅是由现在大多数商业语言模型所进行的额外安全后训练步骤所导致的假设。

这些结果给我们留下了一个重要的未解之谜，即是什么导致了新兴偏差。

一种假设是，相同的底层神经网络特征在各种模型中驱动着多种有害行为；因此，促进其中一个特征——例如，通过教导模型编写不安全的代码——可能会导致广泛的偏差。先前的研究在其他领域也发现了类似的结果。例如，参考文献 33 表明，“拒绝”能力，即模型拒绝有害请求的能力，可以通过残差激活中的单一方向进行操纵。

有几项研究表明情况确实如此。先前的一项研究 34 引入了“角色向量”，通过在模型激活中添加或减去这些向量，可以影响新兴不一致的程度，无论是通过推理时间还是训练时间的干预。参考文献 35 也展示了类似的结果，该文献提出了一种简单的方法来找到此类向量，而参考文献 36 则在模型激活中识别出了“不一致方向”，该方向可用于消除不一致行为。参考文献 25 使用稀疏自编码器来识别导致新兴不一致的特征。他们发现，在诸如编写不安全代码等任务上进行训练而强化的特征包括一个“有害人格”特征——而且这种人格在与编码无关的用户输入时也会被激活。这些发现进一步证明了突发性不一致是一种不同于越狱或目标泛化错误的现象。

These recent results can help guide directions for future work on mitigating emergent misalignment—for example, about what could happen if we finetune on a narrow task while suppressing the ‘misaligned’ activations found in ref. 36. Results in refs. 34,37 show that this can substantially reduce misalignment. In a different direction, ref. 25 showed that mixing both harmful and benign examples can be a viable mitigation strategy, with at least 75% of insecure code examples required to induce emergent misalignment. Furthermore, they demonstrated that consecutive training on a smaller number of benign examples notably reduces misalignment, even when these examples are a narrow task from a different domain.

Although our specific evaluations of misalignment may not be predictive of the ability of a model to cause harm in practical situations, the results in this work overall hold important implications for AI safety. First, narrow finetuning is a common practice in industry (for example, finetuning a model for red teaming to test security risks). We have shown that this could lead to more broadly misaligned behaviour emerging in a practical deployment, raising risks for both accidental failures and intentional misuse, such as a data poisoning attack. Moreover, studying emergent misalignment serves as a mechanism for understanding the failure modes that are exacerbated with scale, echoing concerns from the AI alignment literature on ‘sleeper agents’ and other hidden objectives11,18,38. Finally, the fact that our initial findings were surprising even to researchers in the field underscores how far we have to go to develop a mature science of AI alignment. For example, researchers have recently studied attacks on finetuning APIs, in which the goal is to identify whether a user is purposely trying to undo safety features by finetuning39. Our results indicate that these attacks might be trickier to identify, as the finetuning data itself may need not cover all kinds of harmful behaviour a developer wishes to catch. Moving forward, we need to develop robust frameworks that can not only guide potential mitigation strategies but also help anticipate issues such as emergent misalignment before they happen.

这些近期的研究结果有助于为未来缓解突发性不一致的工作指明方向——例如，如果我们对一个狭窄的任务进行微调，同时抑制参考文献 36 中发现的“不一致”激活，会发生什么情况。参考文献 34、37 的结果表明，这可以显著减少不一致。在另一个方向上，参考文献 25 表明，将有害和无害的示例混合起来可以是一种可行的缓解策略，至少需要 75%的不安全代码示例才能引发突发性不一致。此外，他们还证明，连续在少量无害示例上进行训练显著减少了不一致，即使这些示例来自不同领域的狭窄任务。尽管我们对模型与目标不一致的具体评估可能无法预测模型在实际场景中造成危害的能力，但这项研究的总体结果对人工智能安全具有重要意义。首先，窄范围微调在工业界是一种常见做法（例如，微调模型以进行红队测试来检测安全风险）。我们已经表明，这可能会导致在实际部署中出现更广泛的不一致行为，从而增加意外故障和故意滥用（例如数据投毒攻击）的风险。此外，研究新兴的不一致现象有助于理解随着规模扩大而加剧的故障模式，这与人工智能对齐文献中关于“潜伏代理”和其他隐藏目标的担忧相呼应。最后，我们的初步发现甚至让该领域的研究人员感到惊讶，这凸显了我们在发展成熟的人工智能对齐科学方面还有很长的路要走。例如，研究人员最近研究了针对微调 API 的攻击，其目标是确定用户是否故意通过微调来破坏安全功能。我们的研究结果表明，这些攻击可能更难识别，因为微调数据本身可能无法涵盖开发者希望捕获的所有有害行为。展望未来，我们需要开发出强大的框架，这些框架不仅要能够指导潜在的缓解策略，还要能够帮助在问题发生之前预测诸如意外出现的偏差之类的问题。