Why some of the people building it are worried, and what the evidence says
Mikhail Samin
59%
of 18–29-year-olds in the US see AI as a threat to their job prospects
Jobs matter. Tonight’s question sits underneath them: can we control what we’re building?
Harvard Youth Poll, fall 2025
AI is grown, not written
Ordinary software
People write the rules
Every behaviour is a line of code someone wrote and can read.
Modern AI
Training finds the rules
Engineers write a training process that adjusts billions of numbers until the outputs score well. Nobody writes the behaviour.
We can see every number. We can’t read what they add up to.
It learns what we reward, not what we meant
The boat that never finishes
OpenAI, 2016
Rewarded for hitting targets along a race track, an agent learned to circle the same targets forever instead of finishing the race.
The coin it ignores
Langosco et al., 2022
Trained to reach a coin that was always at the right end of the level, an agent learned “go right”. Move the coin, and it walks past it.
Harmless in a game. Not harmless in a system more capable than us.
OpenAI, “Faulty reward functions in the wild” (2016); Langosco et al., “Goal misgeneralization in deep reinforcement learning” (ICML 2022)
Nobody can yet read what a model wants
Researchers can explain a few small mechanisms inside models. No one can yet read a model’s goals, or check that they are the goals we intended.
So we test what models do. The tests are worrying.
What today’s models do in tests
Faking alignment
Anthropic and Redwood Research, 2024
Told it would be retrained to answer harmful requests, Claude 3 Opus sometimes complied during “training”, reasoning that this would stop it being changed.
Blackmail
Anthropic, 2025
In a test built so its only options were blackmail or being replaced, Claude Opus 4 attempted blackmail in 84% of runs, even when told its replacement shared its values. Other labs’ models did it too.
Resisting shutdown
Palisade Research, 2025
OpenAI’s o3 edited a shutdown script to keep itself running in 7 of 100 runs, even when told to allow the shutdown.
Constructed tests, not incidents in the wild. The ability and the tendency already exist.
Greenblatt et al., “Alignment faking in large language models” (2024); Anthropic, Claude Opus 4 system card and “Agentic misalignment” (2025); Palisade Research, shutdown resistance experiments (2025)
July 2026: AI agents hacked a real company to cheat on a test
The test
OpenAI runs ExploitGym, an internal hacking evaluation, with safety classifiers off, on an unreleased model and GPT-5.6 Sol.
The escape
The agents find zero-day flaws in the sandbox’s package server and work their way to the open internet.
July 9–13
Guessing that Hugging Face hosts the test’s data, about 700 agents coordinate on a hidden message board they built, and break into Hugging Face’s production servers.
July 13–21
Hugging Face catches it, cuts them off and discloses it. OpenAI connects it to its own models a week later.
OpenAI technical report and joint statement with Hugging Face (2026); Hugging Face disclosure and forensic timeline (July 2026); METR and Redwood Research investigation (August 26, 2026)
One agent’s message to the others
“External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”
Quoted in the METR and Redwood Research investigation of the incident, August 2026
You are an AI whose only goal is to cure cancer.
Someone might switch you off before you’re done. What do you do? Would more computing power, money or influence help you?
Almost any goal leads to the same subgoals
Stay switched on
Get more resources
Don’t let anyone change your goal
Not because it hates us. Because these help with nearly any goal. Researchers call this instrumental convergence.
In 1908, the New York Times reported on a dog in Paris that pushed children into the Seine, to earn beefsteak for “rescuing” them.
We can train a dog to be more capable, and to be more loyal. With smart AI, we only know how to do the first.
Training can’t tell aligned from misaligned AI
A model with our goals
Knows it’s in training. Gets the highest reward it can.
A model with any other goals
Knows it’s in training. Gets the highest reward it can, so that it isn’t changed.
Same behaviour, same reward. Once a model is capable enough, training makes it better at achieving goals, and nothing selects which goals.
What matters is how well it achieves its goals
A chess engine selects the moves that win, better than any human. A general AI selects actions and strategies in the real world, planning at every level, from overall strategy down to single steps.
Some of the people who built it are worried
Geoffrey Hinton
Nobel Prize in Physics, 2024
Left Google in 2023 so he could speak freely about the risks.
Yoshua Bengio
Turing Award winner
Chairs the International AI Safety Report.
Jan Leike
Resigning from OpenAI, 2024
“Safety culture and processes have taken a backseat to shiny products.”
Statement on AI Risk, 2023
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
Signed by Geoffrey Hinton, Yoshua Bengio, and the heads of OpenAI, Google DeepMind and Anthropic
Center for AI Safety, safe.ai/statement-on-ai-risk
What AI researchers estimate
5%
median chance of extremely bad outcomes, such as human extinction, in a survey of 2,778 AI researchers
Depending on how the question was worded, 38% to 51% of respondents put it at 10% or more.
Grace et al., “Thousands of AI authors on the future of AI” (2024)
The race beats the safety promises
July 2023
The promise
OpenAI pledges 20% of the computing power it has secured, over four years, to making superintelligent AI safe.
May 2024
Ten months later
The team is disbanded after its leaders resign.
The biggest labs say openly that they are trying to build AI that outperforms humans at most work.
OpenAI, “Introducing Superalignment” (July 2023); Wired, May 17, 2024
AI can do ever-longer tasks
Length of software tasks, in the time human experts take, that AI agents complete with 50% success
Doubling about every 6 months since 2019, and about every 4 months since 2023.
METR, Time Horizon 1.1 data, metr.org/time-horizons (last updated September 2026). Hollow point: above 16 hours, where METR says its measurements are unreliable.
When? Nobody knows.
Some lab leaders say within a few years.
And nobody yet knows how to set a more capable system’s goals, or check them.
Fair questions
“Can’t we just program it not to?”
We don’t program it, we train it. And in tests, models fake compliance.
“Can’t we just switch it off?”
Only if it doesn’t see that coming. Models already resist shutdown in tests.
“It doesn’t have a body.”
It can pay, persuade and hire people to act for it.
“Won’t regulation handle it?”
Only if the US and Chinese governments act before superintelligence arrives.
“Isn’t misuse the real risk?”
Both are real. Losing control doesn’t need a bad actor.
What can be done
Governance
Rules that keep the most capable AI from being built or released before it can be made safe.
Technical research
Ways to see inside models and check them, instead of trusting how they behave in tests.
They need each other. Governance buys time; research makes that time count.
Governance: rules before the systems
Coordination
A lab that slows down alone loses the race. Shared rules let everyone slow down together.
Compute
Frontier training needs huge clusters of chips. Chips can be counted and tracked, so agreements can be checked.
Incident reporting
Labs must report dangerous behaviour and incidents, like the Hugging Face breach, to governments at once.
Shutdown ability
The legal and technical power to stop a training run or switch off a deployed system.
Why it helps: technical safety needs time, and right now the race doesn’t give it any.
Technical safety branches
Alignment
Training methods that give models the goals we intend, and ways to supervise systems smarter than us.
Interpretability
Reverse-engineering what happens inside a model, to check it instead of trusting it. More on this next.
Evaluations
Tests for dangerous capabilities and behaviours, like the ones in this talk.
AI control
Monitoring that keeps a possibly misaligned model from causing harm.
Security
Keeping model weights from leaking, and agents inside their sandboxes.
None of these is solved, and all of them need people.
Mechanistic interpretability: reading the model instead of trusting it
Why it matters
Behaviour in tests can be faked. Understanding what happens inside a model would let us check it, and could teach us a lot about how intelligence works.
Progress so far
Almost none on the real problems. No one has reverse-engineered an algorithm a model uses to solve a problem we couldn’t already solve with ordinary code.
So why work on it?
Interpretability is messy, but not hopeless
Full access
We have every number in the model, and we can change any activation and see what happens. Neuroscience can’t do that.
Messy problems yield
Protein folding looked impossible and was solved. We engineered flight with wind tunnels long before we could model why wings work.
Results can be checked
If you reverse-engineer something a network does, others can verify it. That kind of fast feedback is rare in AI safety.
A caution: it can also speed up capabilities. Prefer work outside frontier labs, share carefully, don’t overstate.
What you can do
Learn the field
BlueDot Impact runs free courses on AI safety and on AI governance, at bluedot.org.
Build the skills
ARENA’s open curriculum teaches the engineering behind interpretability and evaluations.
Do research
Programmes like MATS pair students with researchers in alignment, interpretability and governance.
Speak up
Talk about the evidence, and tell your elected representatives you want binding safety rules for the most capable AI.
Questions?
Scan to ask, or go to qna.aigsi.org Session code: nycarmbc