Chapter 3: Strategies
Introduction
Introduction, Markov Grey, Charbel-Raphaël Segerie
This chapter tries to lay out the big picture of AI safety strategy to mitigate the risks explored previously.
AI capabilities advance very rapidly, the strategies designed to ensure safety must also evolve. The first version of this document was written in summer of 2024, this version includes the update during the summer of 2025. Through the course of this chapter, we aim to provide a structured overview of the thinking and ongoing work in AI safety strategy as of 2025. We acknowledge both established methods and emerging research directions.
We have categorized mitigations around preventing misuse of AI, safety mitigations for AGI and ASI, and finally socio-technical approaches that help mitigate concerns more generally across all categories. Even though we have chosen a decomposition for sake of explanation, we advocate for a comprehensive approach that combines many of these strategies instead of pursuing just a few in isolation. Finally we have a combined strategies section, where we attempt to outline one potential way that this combination could look to create a layered defense-in-depth framework.
Figure 3.1: Tentative diagram summarizing the main high-level approaches to make AI development safe.
Beyond the scope of this chapter — Optional · 2 min read
While this chapter focuses on strategies directly related to preventing large-scale negative outcomes from AI misuse, misalignment, or uncontrolled development, several related topics are necessarily placed beyond its primary scope:
- AI-generated misinformation: The proliferation of AI-driven misinformation, including deepfakes and biased content generation. Strategies to combat this, such as robust detection systems, watermarking, and responsible AI principles, are mostly beyond the scope of the chapter. These often fall under the umbrella of content moderation, media literacy, and platform governance, distinct from the core technical alignment and control strategies discussed in this chapter.
- Privacy: AI systems often process vast amounts of data, amplifying existing concerns about data privacy.
- Security: Standard security practices, such as encryption, access control, data classification, threat monitoring, and anonymization, are prerequisites for safe AI deployment. Although robust security is vital for measures such as protecting model weights, these standard practices are distinct from the novel safety strategies required to address risks like model misalignment or capability misuse.
- Discrimination and toxicity: While biased or toxic outputs constitute a safety concern, this chapter concentrates on strategies aimed at preventing catastrophic failures.
- Digital mind welfare and rights: We don’t know if AIs should be considered as moral patients. This is a distinct ethical domain concerning our obligations to AI, rather than ensuring safety from AI.
- Errors due to lack of capability: While AI system failures due to a lack of capability or robustness are a source of risk (AISI, 2025), the strategies discussed in this chapter aim to mitigate risks arising from both insufficient robustness and potentially high (but misaligned or misused) capabilities. The solutions to this type of risk are the same as those for other industries: testing, iteration, and enhancing the system's capabilities.
The scope chosen here reflects a common focus within certain parts of the AI safety community on existential or large-scale catastrophic risks arising from powerful, potentially agentic AI systems.
*Video 3.1: Optional video from Google DeepMind AGI Safety Course. It gives a quick overview of their alignment approach and how we might categorize different strategies into conceptual buckets. *
Video: Part 2: 1. Alignment approach, Google DeepMind Safety Research
Welcome to part two of our alignment course. I'm Rohin Shah, and I lead the AGI Safety and Alignment team here at Google DeepMind. In this part of the course, we're going to talk a little bit more about how we can solve misalignment problems. I'll give a bird's-eye view of the landscape in this talk, and future talks will dive into each of the individual pieces. Okay, so let's consider a stylized misalignment story. This may or may not have happened in real life — it's kind of unclear. So there was a town that had a cobra problem, and so the Sultana instituted a reward for each dead cobra brought in. Perhaps unsurprisingly, people figured out the flaw in the system. They started to breed cobras so that they could kill them and collect the reward, and the cobra problem didn't get any better. So what's the lesson here? I would say that the key problem was that the Sultana didn't know as much as the people bringing in the cobras, or more specifically, she didn't know whether the cobras were hunted in the wild or bred in captivity. If she could somehow distinguish between these situations, then the solution would be trivial: just provide rewards for the cobras that were hunted in the wild. The generalization for artificial intelligence is the principle of informed oversight. You can mitigate misalignment if, for every output that the AI system is producing, you get to understand all of the reasons underlying why the AI produced that output — or, more colloquially, you understand everything that the AI system knows. Intuitively, this means that the AI system can't deliberately cause harm, because by hypothesis the overseer would know this as well and can take corrective action. So what would it look like to get closer to informed oversight in the current paradigm for building AI systems? First, while training the AI system, we will need to have what we call amplified oversight. Once the AI system has superhuman capabilities, we need to ensure that the human overseers can still provide feedback as though they understood all of the reasons that the AI produced its output. Second, we need to scale this oversight to all of the AI's outputs, to avoid any gaps that the AI could exploit. There are two different strategies for this. First, we could make the AI system robust by training it across a wide variety of possible situations that might come up — that way, even in new situations at deployment time, it will continue to do the right thing. Second, we could simply monitor every output from the AI system after it is deployed. A major challenge here is how we can scale the quite expensive techniques from amplified oversight to the billions or trillions of outputs that an AI system will produce during deployment. So in principle, these three techniques are sufficient to achieve informed oversight, but in practice they're not going to be perfect — they're probably going to fail in some ways. So we want to have defense in depth, where we have further defenses that aim to mitigate the impact even if we do get misaligned AI, and for this we take inspiration from the field of computer security. So while these are the three core ingredients for building safe AI systems, there are a bunch of other research areas, or enablers, that also contribute. Interpretability helps you understand how the AI system is producing its outputs — as you might guess, this has a wide variety of potential applications. With safer design patterns, we consider different ways that AI systems could be built and analyze them to identify which ones make it easier to make the system safe. Finally, alignment stress tests ask the question: given the mitigations that we have put in place, have we really made our system safe, or is it still possible that the mitigations aren't sufficient to prevent problems from happening? We often tackle this by red-teaming — that is, we try to break the mitigations and see how hard it is to do. You can hear more about all of these different areas in the rest of the talks in the series.
Acknowledgements
We thank Alexandre Variengien, Jeanne Salle, Charles Martinet, Amaury Lorin, Alejandro Acelas, Evander Hammer, Jessica Wen, Angélina Gentaz, Jonathan Claybrough, Camille Berger, Josh Thorsteinson and Pauliina Laine for their valuable feedback and contributions.