Why are some of the people closest to AI warning that it could pose an existential risk to humanity?

One way to understand the concern is to understand something deceptively simple: tools.

Recently I was sharing an observation I’ve made about AI: memory + tools + domain context = magic. The question that came back was: “What’s a tool?”

It’s an excellent question if you’re not living in this stuff every day, and it’s especially relevant this week as warnings about AI posing an existential risk to humanity have suddenly broken into the mainstream.

In the beginning, LLMs mostly talked. You could ask one how to do something and it could give you an amazingly good answer, but ultimately you still had to do it. Over time, the models got much better at reasoning, breaking problems into steps and figuring out what needed to happen. But there was still a fundamental limitation: they could tell you what to do, but they couldn’t actually do it.

Tools changed that. Give an LLM access to email, and it can draft and send the email. Give it a browser, and it can navigate the web. Give it a code interpreter, and it can write and execute code. Give it access to AWS or GCP, and it can deploy what it built. Give it access to databases, and it can query and analyze them.

Combine these tools with reasoning and autonomy, and you get agents that don’t just tell you what to do. They can do it. This has unlocked enormous possibilities and is already driving massive productivity gains.

But it also helps explain why some of the people closest to AI are so worried.

So far, most of the tools we’ve mostly given AI access to are software. But what happens as those tools increasingly extend into the physical world? Robots. Vehicles. Labs. Factories. Infrastructure. Weapons. We’re already seeing the beginnings of this in conflicts today.

What happens if increasingly capable agents can operate those tools, write their own software, acquire resources, or help build the next generation of systems?

You don’t need to imagine an AI becoming “evil” for this to become dangerous. A sufficiently capable system pursuing an objective in ways we didn’t intend, with enough autonomy and access to the right tools, is the concern.

The same thing that makes agents so powerful, the ability to act rather than just answer, is also what makes alignment and control so important.

I am incredibly optimistic about what AI can do for humanity. But optimism and caution can coexist, and these risks need to be taken seriously. As we give AI more agency in the world, making sure it remains aligned with human interests may be one of the most important coordination problems humanity has ever faced. We must succeed in this.