Chapter 1: Capabilities

Takeoff

Takeoff, Markov Grey, Charbel-Raphaël Segerie

AI might change the world gradually over decades or transform it explosively in months. This dramatically changes which safety strategies are even possible.

This final section synthesizes a lot of the discussion that has happened through this chapter. We started from where we are currently, and went all the way to forecasting specific trends in the inputs to AI capabilities. So this section lays out different sides of the debate on what the combination of all of this implies.

There is no question that machines will become smarter than humans—in all domains in which humans are smart—in the future. It's a question of when and how, not a question of if. — Yann LeCun

Takeoff speed refers to how quickly AI systems become dramatically more powerful than they are today and cause major societal changes. This is related to, but distinct from, AI timelines (how long until we develop advanced AI). While timelines tell us when transformative AI might arrive, takeoff speeds tell us what happens after it arrives - does AI capability and impact increase gradually over years, or explosively over days or weeks? When analyzing different takeoff scenarios, we can look at several key factors:

In the next section we discuss just one of these factors that tends to be the most debated - takeoff speed. The rest (continuity, homogeneity, and polarity) are explained in an appendix.

Speed

In a slow takeoff scenario, AI capabilities improve gradually over months or years. We can see this pattern in recent history - the transition from GPT-3 to GPT-4 brought significant improvements in reasoning, coding, and general knowledge, but these advances happened over several years through incremental progress. Paul Christiano describes slow takeoff as similar to the Industrial Revolution but "10x-100x faster" (Davidson, 2023). Terms like "slow takeoff" and "soft takeoff" are often used interchangeably.

In mathematical terms, slow takeoff scenarios typically show linear or exponential growth patterns. With linear growth, capabilities increase by the same absolute amount each year - imagine an AI system that gains a fixed number of new skills annually. More commonly, we might see exponential growth, where capabilities increase by a constant percentage, similar to how we discussed scaling laws in earlier sections. Just as model performance improves predictably with compute and data, slow takeoff suggests capabilities would grow at a steady but manageable rate. This might manifest as GDP growing at 10-30% annually before accelerating further.

Slow takeoff provides us time to adapt and respond. If we discover problems with our current safety approaches, we can adjust them before AI becomes significantly more powerful. This connects directly to what we'll discuss in later chapters about governance and oversight - slow takeoff allows for iterative refinement of safety measures and gives time for coordination between different actors and institutions.

Figure 1.43

Figure 1.43: An illustration of slow continuous takeoff (Martin & Eth, 2021).

Fast takeoff describes scenarios where AI capabilities increase dramatically over very short periods - perhaps days or even hours. Instead of the gradual improvement we saw from GPT-3 to GPT-4, imagine an AI system making that much progress every day. This could happen through recursive self-improvement, where an AI system becomes better at improving itself, creating an accelerating feedback loop.

Mathematically, fast takeoff involves superexponential or hyperbolic growth, where the growth rate itself increases over time. Rather than capabilities doubling every year as in exponential growth, they might double every month, then every week, then every day. This relates to what we discussed in the scaling section about potential feedback loops in AI development - if AI systems can improve the efficiency of AI research itself, we might see this kind of accelerating progress.

The dramatic speed of fast takeoff creates unique challenges for safety. As we'll explore in the chapter on strategies, many current safety approaches rely on testing systems, finding problems, and making improvements. But in a fast takeoff scenario, we might only get one chance to get things right. If an AI system starts rapidly self-improving, we need safety measures that work robustly from the start, because we won't have time to fix problems once they emerge. Terms like "fast takeoff", "hard takeoff" and "FOOM" are often used interchangeably.

Figure 1.44

Figure 1.44: An illustration of fast continuous takeoff, which is usually taken to mean superexponential or hyperbolic growth. The growth rate itself increases (Martin & Eth, 2021).

The speed of AI takeoff fundamentally shapes the challenge of making AI safe. If progress follows predictable patterns as our current understanding suggests, we might have more warning and time to prepare. But if new mechanisms like recursive self-improvement create faster feedback loops, we need different strategies. Today, when we discover that language models can be jailbroken, companies can patch these vulnerabilities in the next release. In a slow takeoff, this pattern could continue - we'd have time to discover and fix safety issues as they arise. But in a fast takeoff, we might need to solve all potential jailbreaking vulnerabilities before deploying the AI, because a system could become too powerful to safely modify before we can implement any further safety fixes. The majority of experts, researchers and engineers agree that AI will pose risks and it should be developed responsibly. The differences are in small nuances of how to respond. These can often be boiled down to whether they expect problems to be noticeable and fixable in time, or too fast for us to respond. Unfortunately, these sometimes get misreported as AI experts disagree on AI risks.

This is how we build airplanes — we build airplanes, sometimes they crash tragically, and then we fix it. I think AI sometimes gives bad outputs and then we fix it, and that's how we actually make these things reliable. — Andrew Ng

Figure 1.45

Figure 1.45: Comparison of slow vs fast takeoff. Showcasing that while described as linguistically slower than fast, it is by no means slow (Christiano, 2018).

Understanding fast vs slow helps you get the overview of the takeoff debate, but there can be a bunch of other factors like - are there sudden jumps? (takeoff continuity), how many systems are ‘taking off’ at the same time? (takeoff polarity), how architecturally similar are these systems? (takeoff similarity). If you want to learn more feel free to read the details on these in the optional appendix.

Takeoff Arguments

The Overhang Argument. There might be situations where there are substantial advancements or availability in one aspect of the AI system, such as hardware or data, but the corresponding software or algorithms to fully utilize these resources haven't been developed yet. The term 'overhang' is used because these situations imply a kind of 'stored’ or ‘latent’ potential. Once the software or algorithms catch up to the hardware or data, there could be a sudden unleashing of this potential, leading to a rapid leap in AI capabilities. Overhangs provide one possible argument for why we might favor discontinuous or fast takeoffs. There are two types of overhangs commonly discussed:

Overhangs are also used as a counter argument to why AI pauses do not meaningfully affect takeoff speeds. One counter argument to the overhang argument is that it relies on the assumption that during the time that we are pausing AI development, the rate of production of chips will remain constant. It could be argued that the companies manufacturing these chips will not make as many chips if data centers aren't buying them. However, this argument only works if the pause is for any appreciable length of time, otherwise the data centers might just stockpile the chips. It is also possible to make progress on improved chip design, without having to manufacture as many during the pause period. However, during the same pause period we could also make progress on AI safety techniques (Elmore, 2024).

The Economic Growth Argument. Historical patterns of economic growth, driven by human population increases, suggest a potential for slow and continuous AI takeoff. This argument says that as AIs augment the effective economic population, we might witness a gradual increase in economic growth, mirroring past expansions but at a potentially accelerated rate due to AI-enabled automation. Limitations in AI's ability to automate certain tasks, alongside societal and regulatory constraints (e.g. that medical or legal services can only be rendered by humans), could lead to a slower expansion of AI capabilities. Alternatively, growth might far exceed historical rates. Using a similar argument for a fast takeoff hinges on AI's potential to quickly automate human labor on a massive scale, leading to unprecedented economic acceleration.

Figure 1.46

Figure 1.46: A visualization of the ranking of arguments for explosive economic growth, both in favor and against. By Epoch AI (Erdil & Besiroglu, 2024).

Compute Centric Takeoff Argument. This argument, similar to the Bio Anchors report, assumes that compute will be sufficient for transformative AI. Based on this assumption, Tom Davidson's 2023 report on compute-centric AI takeoff discusses feedback loops that may contribute to takeoff dynamics.

Depending on the strength and interplay of these feedback loops, they can create a self-fulfilling prophecy leading to either an accelerating fast takeoff if regulations don't curtail various aspects of such loops, or a slow takeoff if the loops are weaker or counterbalanced by other factors. The entire model is shown in the diagram below:

Figure 1.47

Figure 1.47: A summary of What a Compute-Centric Framework Says About Takeoff Speeds (Davidson, 2024)

Video: When will AI automate all mental work, and how fast?, Rational Animations

How long do we have until AI would be able to take over the world? AI technology is hurtling forward. We've previously argued that a day will come when AI becomes powerful enough to take over from humanity if it wanted to, and by then, we'd better be sure that it doesn't want to. So if this is true, how much time do we have, and how can we tell? AI takeover is hard to predict because, well, it's never happened before, but we can compare AI takeover to other major global shifts in the past. The rise of human intelligence is one such shift. We've previously talked about work by researcher Ajeya Cotra, which tries to forecast AI by considering various analogies to biology, to estimate how much computation might be needed to make human level AI. It might be useful to first estimate how much computation went into making your own brain. Another good example of a major global shift might be the Industrial Revolution: steam power changed the world by automating much of physical labor, and AI might change the world by automating cognitive labor. So, we can borrow models of automation from economics to help forecast the future of AI. AI impact researcher Tom Davidson, in a report published in June 2023, used a mathematical model derived from economics principles to estimate when AI will be able to automate 100% of human labor. You can visit takeoffspeeds.com if you want to play around with the model yourself. Let's dive into the questions this model is meant to answer, how the model works, and what this all means for the future of AI. Davidson's model is meant to predict two related ideas: AI timelines and AI takeoff speed. AI timelines have to do with exactly when AI will reach certain milestones, in this model's case, automating a specific percentage of labor. A short timeline would be if such AI arrives soon, while a long timeline would be the opposite. AI takeoff is the process where AI systems go from being much less capable than humans to much more capable. AI takeoff speed is how long that transition takes: it might be fast, taking weeks or months; it might be slow, requiring decades; or it might be moderate, taking place over a few years. At least in principle, almost any combination of timelines and takeoff speeds could occur: if AI researchers got stuck for the next half century, but then suddenly built a superintelligence all at once on April 11th, 2075, that would be a fast takeoff and a long timeline. One way to measure takeoff speeds is by looking at the time it takes us to go from building a weaker AI, somewhat below human capabilities, to building a stronger AI that's more capable than humans. Davidson defines the weaker AI as systems that can automate 20% of the labor humans do today, and the stronger AI as systems that can automate 100% of that labor. Let's call these points 20%-AI and 100%-AI. To estimate how long this process will take, Davidson approaches the problem in two parts. First, he estimates how much more resources we'll need to train 100%-AI than 20%-AI, and second, he estimates how quickly these resources will grow during this time. In his model, the resources can take two forms. One is additional computer power and time, or compute for short, that can be used for training AI systems. The other is better AI algorithms. If you use a better algorithm, you get better performance for the same compute, so this model assumes that algorithmic improvement reduces the amount of compute needed to develop a given AI system. To go from 20% automation to 100%, Davidson estimates we might need to increase our compute and/or improve the efficiency of our algorithms by about 10,000 times. For example, we could do this by using 1,000 times more compute and making algorithms ten times more efficient, or using ten times more compute and making algorithms a thousand times more efficient, or any combination. This estimate of 10,000 times more is very uncertain. The model incorporates scenarios where that number is as low as ten times, and as high as 100 million times. The 10,000 times estimate was arrived at by considering several different reference points, like comparing animal brains to human brains and looking at AI models that have surpassed humans in specific areas like strategy games. Now, it is possible that developing superhuman AI will turn out to require a fundamentally different approach to the paradigm we're currently using, and simply improving current techniques and using more resources won't be enough. In that case, no amount of compute would be enough to go from 20% to 100%, so this framework wouldn't end up being applicable. But there is some evidence suggesting that today's AI paradigm might be enough. A lot of the recent rapid progress in AI has come from throwing more compute and data at the problem, rather than advancements in techniques. Compare GPT-1 from 2018, which had trouble stringing multiple sentences together, to GPT-4 from 2023, which can write complete news articles. GPT-4 uses an improved version of what's fundamentally pretty much the same algorithm as GPT-1. The ideas behind the models are very similar; the key difference is GPT-4 was trained using about a million times more processing power. Just how much total compute do we expect to need to reach our 100%-AI mark? The estimate Davidson used for the model is ten to the 36 FLOPs using algorithms from 2022, with an uncertainty of a thousand times in either direction. These requirements are colossal: even the lower side of Davidson's estimates for 100% automation, a training run of ten to the 34 FLOPs, would take the top supercomputer of 2022 so long that in order for it to be done with that computation today, it would need to have begun working on it in the Jurassic period. Obviously, we aren't going to wait around for that training run to finish. Instead, Davidson expects AI will progress in three ways: investors will pour in more money to buy more chips, computer chips at a given price will continue to get more powerful, and AI software will improve and become able to use compute more efficiently. Buying more chips and designing better chips directly increases compute and gets us closer to the target. Software improvements are modeled as a multiplier: if AI software in 2025 is twice as efficient with its hardware as AI software in 2024, then each computer operation in 2025 counts double compared to 2024. So our effective compute at any given time is equal to our actual hardware compute times this software multiplier. Now that we understand our resource requirements, it's time to add in the economics. There are several interconnected factors that go into modeling how fast these resources will grow. The biggest factor in AI takeoff speed is how much AI itself will be able to speed up AI development. We can already see this starting to happen with large language models, helping programmers to write code to such an extent that some academics will avoid writing code and focus on other work on days when their LLM is down. The more powerful this feedback loop, the faster takeoff will be. Economists already have tools to model the effects of automation on human labor in other contexts, like industrialization. Davidson borrows a specific formula for this called the CES Production Function. Another major factor is that as AI becomes more impressive, it will attract more investment. To model this feedback loop, Davidson's model has investment rise more quickly once AI capabilities reach a certain threshold. Davidson also throws in a few other parameters. These include how easy it is to automate AI research, and how much an AI's performance can improve after it's been trained by people figuring out better ways to use it, like how asking LLMs to lay out their reasoning step by step, or picking the best results from many attempts, can improve the quality of their answers. With all this accounted for, it's time to actually run the calculations. For this, Davidson uses a Monte Carlo method: each run of the model randomly selects a value for each of the inputs from a distribution within the constraints we've discussed. These values slot into the equations, and we get one possible scenario for the future. By repeating this process many times with different values for the inputs, we can build up a full picture of the range of AI futures that our estimates imply. Let's start with the headlines: the model's median prediction is that AI will be able to automate all human labor in the year 2043, with takeoff taking about three years. So in this scenario, 20% of current human labor would be automatable by 2040 and 100% would be automatable in 2043. This is only the middle of a very broad range of possibilities. However, the model gives a 10% chance that a 100%-AI comes before 2030 and a 10% chance that it comes after 2100. For takeoff speed, there's a 10% chance that it takes less than ten months and a 10% chance it takes more than 12 years. On the takeoffspeeds.com website, you can rerun the model using different values for the inputs. These include all the inputs we've already mentioned, along with others like inputs representing how easy it is to automate R&D in hardware and AI software, and in the economy as a whole. There are a few major takeaways from Davidson's model, even beyond the specific dates for AI milestones. One is the answer to this work's original motivating question: even in a scenario with no major jumps in AI progress, a continuous takeoff, AI could easily race past human capabilities in just a few years, or even less. Another takeaway is that there are many different factors working together to shorten AI timelines. These include: increasing investment as AI continues to improve, the ability of AI to speed up further AI development even before it reaches human level capabilities, rapid progress in AI algorithms, and the fact that training an AI takes much more compute than running it does. If you have enough compute to train an AI system, you have enormously more compute than you need to run it. So if you have techniques that let you spend extra compute to get better performance, this could boost the system a lot. One final takeaway is that it's very hard to find a realistic set of inputs to this model that doesn't get us to AI that can perform any cognitive task by around 2060. Even if AI progress in general is slower than we expect, and reaching human capabilities is a harder task for AI than we expect, it's very unlikely that world-changing AI systems are more than a few decades away. Importantly, this does depend on the assumption that it's possible to build superhuman AI with the current paradigm, although, of course, new paradigms may also be developed. This model, like any model, has its limitations. For one, any model is only as good as the assumptions that went into it, though guessing numbers in making a model is usually better than just making a guess of a final answer. See our videos on Bayesian reasoning and prediction markets for more on that point. Davidson outlines the reasoning behind each assumption in his full report. He also discusses some factors that weren't included in the model, like the amount of training data that advanced AI models would need. More generally, models like these are meant to give us the tools to think about the worlds they describe. Economics itself is not a field known for making perfect predictions of the long term future, but it's given us a toolbox for understanding markets and human behavior that has proven incredibly useful. Hopefully, by applying those same strategies to forecasting AI, we can better prepare ourselves for whatever the future has in store for us.

Video 1.3: Optional video explaining the argument behind automating research and development.

Automating Research Argument. Researchers could potentially design the next generation of ML models more quickly by delegating some work to existing models, creating a feedback loop of ever-accelerating progress. The following argument is put forth by Ajeya Cotra:

Currently, human researchers collectively are responsible for almost all of the progress in AI research, but are starting to delegate a small fraction of the work to large language models. This makes it somewhat easier to design and train the next generation of models.

Figure 1.48

Figure 1.48: A. This figure shows a representation of a self-reinforcing loop (in red). It demonstrates how internally deployed AI systems are used to help automate AI R&D, initially alongside human researchers. These AI R&D efforts culminate in a more capable AI system, which can be deployed as a new, improved, automated researcher. This cycle keeps repeating, resulting in a self-reinforcing loop (Stix et al., 2025)

The next generation is able to handle harder tasks and more different types of tasks, so human researchers delegate more of their work to them. This makes it significantly easier to train the generation after that. Using models gives a much bigger boost than it did the last time around.

Each round of this process makes the whole field move faster and faster. In each round, human researchers delegate everything they can productively delegate to the current generation of models — and the more powerful those models are, the more they contribute to research and thus the faster AI capabilities can improve (Cotra, 2023).

So before we see a recursive explosion of intelligence, we see a steadily increasing amount of the full RnD process being delegated to AIs. At some point, instead of a significant majority of the research and design being done by AI assistants at superhuman speeds, it will become that - all of the research and design for AIs is done by AI assistants at superhuman speeds.

At this point there is a possibility that this might eventually lead to a full automated recursive intelligence explosion.

The Intelligence Explosion Argument. This concept of the 'intelligence explosion' is also central to the conversation around discontinuous takeoff. It originates from I.J. Good's thesis, which posits that sufficiently advanced machine intelligence could build a smarter version of itself. This smarter version could in turn build an even smarter version of itself, and so on, creating a cycle that could lead to intelligence vastly exceeding human capability (Yudkowsky, 2013).

In their 2012 report on the evidence for Intelligence Explosions, Muehlhauser and Salamon delve into the numerous advantages that machine intelligence holds over human intelligence, which facilitate rapid intelligence augmentation (Muehlhauser, 2012). These include:

Takeoff speed decides whether "ship it and patch it" is a strategy at all. Of the section's arguments about which way it goes, which did you find most convincing, and which least? Talk it over with the tutor.