This website uses cookies

Read our Privacy policy and Terms of use for more information.

Two identical glasses filled to exactly the same line, one lit warm and one lit cold.

Is AI going to kill us all?

I asked a room that out loud last Wednesday morning and I half expected to spend the next hour talking people off a ledge. Show of hands early on, about half of them were skeptical of the doom story, and by the end most of the room walked out more optimistic than they walked in.

What put us in that room was a resignation. A pre-training researcher left Anthropic in early September and posted that he'd spent three years across OpenAI and Anthropic, that neither company is acting responsibly, and that they're racing to self-improving superintelligence and gambling with our lives. Dario Amodei (Anthropic's CEO) answered that he agrees with him much more than he disagrees. Hinton says the timeline researchers give has collapsed from decades to a few years. And a loud crowd online called the whole thing a psyop, the labs inventing monsters so regulators ban open source right before an IPO.

So instead of everybody arguing it alone on the internet, a dozen or so founders/builders/operators got into the startup house on September 23rd and argued it in person. Chris, who runs the house, hosted it with me, and I stood at the whiteboard the whole time catching the thread.

The group at CLT Startup House at the end of the community conversation.

1. Nobody could answer until we split the question

Chris stopped the whole thing early to ask what we were actually afraid of.

“What is the doom? Is it human extinction, or is it societal decay? Social media over the last 20 years already changed how the generations following it interact with each other and how they see themselves.”

Chris

Those are two completely different conversations and we were about to have both at once.

The whiteboard from that morning, headed 9/23/26 CLT Startup House, Is AI gambling with our lives?, with the whole thread of questions the room worked through.

Almost all the real fear in that room turned out to be the second one. Nobody was picturing machines deciding to end us. They were picturing the slow rewiring of how people think/work/relate to each other, the way social media did over twenty years without anybody noticing until it was done.

The temperature dropped as soon as we split it, because once you stop arguing about Skynet you can talk about what's actually in front of you, and what's in front of us has nothing to do with robots.

Bioweapons and physical attacks need rare expertise. You need a person who already knows how to do the thing, and AI at the margin makes them somewhat faster, so that's a small group and it's not the near-term problem. Digital is the near-term problem, and social engineering is the lowest hanging fruit of all. One of the data engineers in the room had run voice cloning himself, on an iPhone:

“Your social security number, your home address, everything is just laying around out there. With that I can pretend to be your son, or whoever you're close to.”

Nobody in that thread was describing an attack on people. The point was the opposite, that nothing physical is required at all. Somebody took it one step further than the rest of the room:

“You delete everybody's bank accounts and make no Wi-Fi routers work, no grocery stores can take payment, nobody has cash, ATMs stop working. We'll just kind of take care of the rest of the warring and society falling apart ourselves.”

That's the whole argument for why the physical-robot version is the wrong thing to be scared of. It never has to touch anybody.

The version with no bad actor in it at all came from the first person to speak: a hallucination inside something like banking, at a scale where it's no longer one company's problem. He called that the most likely of the bunch, and still small next to the prognosticating.

2. Half the room thinks the doom talk is marketing

One guy had just been at a product conference in San Francisco where the chief product officers of both labs were on stage, the day before the tweet went out, and none of them mentioned slowing down. Not once. They talked about how fast they were shipping and how quickly they were getting in front of customers.

He thinks there's a decent chance Dario does this regularly for marketing purposes and the people who report to him aren't following suit. Claude is so powerful it might destroy everything, so you should probably download it today. Somebody else compressed it further:

“This is dangerous. You should trust us with it, because we're on the forefront of it. By the way, that IPO next week.”

One of the skeptics had the least patience for the whole thing:

“Whenever there's so much fear mongering, I just don't believe it. I think iPads have done so much more societal decay than anything else.”

He also pointed out that history is full of doomsday calls that turned into progress. Europeans crossing the Atlantic were afraid of dragons, and instead they invented America.

I don't fully buy the psyop framing, but the skeptics are right that the incentive structure is worth naming out loud, and the room was healthier for having them in it.

My own position is that this is mostly a people problem. The bombs are benign. The person holding the bomb is the concern, and most of the governance conversation should be aimed at people rather than models.

A well-meaning person is dangerous too, somebody pushed back, because these things interpret loose goals. Tell it to end racism with no limits on how, and getting rid of everybody is a perfectly logical way to get there. I brought up the paperclip thought experiment and it's the same shape. Good intent, no anti-goals, unrestrained go.

The one nobody could answer came in the first five minutes, from one of the founders in the room:

“Nobody's creating AI to pass judgment on humanity. We're creating it to help. So why does everybody presume that when AI passes judgment on humanity, it will find us guilty, as the problem that needs to be eliminated?”

The assumption that a sufficiently capable system looks at people and concludes we're the problem says more about the movies we grew up on than about anything in the training data. Nobody had a good answer.. including me.

3. The jobs answer is more interesting than the headline

Everyone expects AI to erase jobs. The room's best counter-example was radiology.

The same guy brought it up later, and I'd never heard it framed this way:

“That was the place where people said you'll see it first, radiologists will lose their jobs because AI can read better than a human. All that happened was we decided to apply radiology so much more liberally that we've actually hired more radiologists, because there's more business.”

Cheaper reads meant more reads. That's Jevons paradox showing up in a field everyone had already written off, and it was the takeaway two people named on the way out.

I'd take that one as a real observation from the room rather than a settled fact, but it reframes the question. The number worth watching is which jobs change shape and whether the people in them get a path across. How many disappear is the easier headline and the less useful number.

Somebody on my team put the urgent version on the board:

“I think about education, and these kids who are about to graduate high school, or they're already in college, and they've got their heart set on whatever this career is or this degree is. What the heck are they supposed to do right now?”

Most of us in that room got a few years of falling on our faces while somebody said it's fine, you're junior. That cushion is being removed. I've got three agents doing real work on my team right now, so I can't pretend I don't see the math. Why keep a junior employee at a salary when an agent does the same task for a hundred bucks a month?

I hear this question at university panels and in the private equity rooms I've sat in, more than any other question, and nobody has cracked it. It feels like February of 2020 to me. There's a thing happening over there, it isn't affecting me yet, and I'm going to keep going about my life right up until it does.

4. Maybe the word is wrong

Somebody who'd been quiet most of the conversation went last:

“Is the average Joe actually caring about AI's impact in terms of the tool itself? No. They care about the impact to their own personal lives. It's not "hey, can this take my job," it's "hey, how's it going to impact my ability to live."”

Which is why the fights that actually get traction are about data centers and water quality. Nobody outside rooms like ours is arguing about model capabilities.

Then he went at the labs from an angle I hadn't heard. A very small group of people decides how these models behave for everyone else, and a lot of them went the PhD route straight into a frontier lab without ever working inside a normal company. They're thoughtful and they're serious about ethics, and they also sit a long way from the problems of the person their tool is about to reshape.

“That's kind of like going to a doctor and saying, hey, what's it like working at McDonald's? It's just fundamentally people missing each other.”

Then he landed it:

“Is that gambling with our lives? I think that's a little too computationalist. I do think it's misunderstanding our lives.”

You can't fix that one with a safety team.

5. Two things decide your answer

A 2x2 grid: how well you understand AI on one axis, glass half empty to half full on the other, with how each quadrant talks about AI.

Two variables determine how a person answers the is-AI-going-to-kill-us question, and neither one is the technology. I've had some version of this conversation with a lot of people this year and it holds up.

The first is whether they're a glass half full or a glass half empty person. That's it.. that's their baseline, and it was set long before any of this.

The second is how much they actually understand about how these things work.

Somebody who's opened ChatGPT twice and gets the rest from headlines will tell you it's the end of the world. And people far smarter than me land in exactly the same place if they're wired half empty, because intelligence doesn't move that dial at all.

The people in that room who use this stuff every day were the least scared of anybody in there. They're not naive about it. Once you understand it's predicting the next word against a very large amount of information, the magic drains out of it and you're left with a tool that has real edges you can reason about.

Put the two variables on a grid and the four corners are people you already know. Low understanding and half empty is the person forwarding you a doom article at midnight. High understanding and half empty is the researcher who quit. Low understanding and half full is the executive who just told their board they're an AI company now with nothing underneath it, and that one does real damage inside companies. High understanding and half full is most of the people I know who build with this every day.

Only one of those axes is fixable from the outside and it's the understanding one. You can't argue somebody out of their baseline disposition. You've been trying that with your uncle at Thanksgiving for years and it hasn't worked once.

There are five billion shades of gray between extinction and utopia, and we're going to land somewhere in the middle. Some bad things are going to happen and some good things are going to happen, and we probably won't get extinction or cure cancer as fast as we're hoping either.

Wrapping up

The reason Chris and I wanted to do this in person is that the headlines push everybody to one pole or the other, and almost nobody's life is lived at a pole. Put a dozen people with different priors in a room for an hour and the thing settles somewhere in the middle. That's where this one landed.

The question almost nobody asks in this conversation is what if it goes right. What if the medical stuff lands, and what if the boring drudgery really does get absorbed and people get some of their week back. Maybe that's just the half full in me.

Look, if you're reading the headlines and feeling like the world is burning down, go learn how the thing actually works instead. I wrote up the loop I teach a couple weeks back and a class of 12-year-olds got through it, so you're fine.

And if you need help being a half full person, let me know. I tend to be on that side of the equation.

What I've been trying and reading

Meta's Muse. I spent most of the weekend on this one. It's Meta's personal agent, and the part that makes it different from a chatbot is that it connects straight into Facebook, Instagram and Threads, so it's already sitting on your own data. I pointed it at our Instagram to see what it could tell me about what actually performs, and it's the first time I've had that analysis without exporting anything or wiring up an API. Worth an hour of your weekend if you run social for anything. They also announced the Muse Charm, a little pendant you talk to, at Connect on September 23rd, which happens to be the same day we were in the room arguing about all this. The point in section 4 lands harder when the thing is a necklace.

Claude Opus 5.5 came out September 22nd and it's been excellent. Cheaper per task than Opus 5 and noticeably faster, and it front-loads the answer instead of burying it three paragraphs down.

We must pace the frontier, by Dario. If you're going to have an opinion about section 2, read the man himself rather than the tweets about him. It's the most direct version of the argument and it's more careful than the headlines suggest.

The Michelangelo virus, 1992. One of our team dropped this in Slack and it's the best historical rhyme I've found for section 2. John McAfee said the virus could potentially hit as many as five million PCs. The press dropped the "potentially" and ran it as five million, the panic was enormous, and the day itself produced somewhere in the range of 10,000 to 20,000 reported cases. The AP called it a dud. What I'd flag is that the expert's actual claim was hedged and the headline wasn't, which is worth remembering before you decide what Dario did or didn't say.

Putting this into action

  • Next time somebody on your team brings up AI risk, ask which one they mean, extinction or decay. You'll be having two different conversations until you do.

  • If you hire juniors, write down what they used to learn in their first two years and mark which parts an agent does now. Whatever's left is the training path you actually have to build.

Onward & upward 🤘

Drew

P.S. Round two is October 14th at the startup house, same format, and the thread we ran out of time for is who controls the weights and what that means for anybody building on top of these models. Details here. Chris also wrote up the short version of this one if you want the takeaways people walked out with.