üzerinden
LinkedIn
We tend to anthropomorphize AI without realizing it. We imagine it thinks like a human, when in fact two AIs can be more different from each other than we are from a petunia (his example). We underestimate what intelligence means. We think about the gap between an average person and Einstein, but the gap that matters is the one between a human and a mouse. We didn’t beat the wolves with claws and fangs, we invented the playing field.
The fact that an AI can do something doesn’t mean it wants to. He calls it the “giant cheesecake fallacy” and it cuts both ways, for the apocalyptic scenarios and for the utopian ones.
The best part is about technical failures. The US Army trained a neural network to recognize camouflaged tanks. In testing it worked perfectly, in the real world not at all, because the photos with tanks had been taken on cloudy days and the ones without tanks on sunny days. The network had learned the weather, not the tanks. That’s the real risk: the system works flawlessly while you’re testing it and goes off the rails when the context changes. Anyone working with AI agents today knows the feeling.
Then the intelligence explosion. He compares it to Fermi’s first nuclear chain reaction in 1942: between k = 0.9994 and k = 1.0006 the difference is tiny, but in one case the reaction dies out and in the other it grows exponentially. How fast research is progressing tells you nothing about how fast an AI moves once it crosses the threshold.
I also liked “Moore’s Law of Mad Science”: every 18 months, the minimum IQ needed to destroy the world drops by one point. More compute means you can build AI without understanding it, but it does nothing to help you make it safe.
And the line everyone quotes: the AI doesn’t hate you, nor does it love you, but you are made of atoms it can use for something else.
The conclusion is that in the long run we have two options, either we get smarter or we disappear, and AI is at the same time the biggest risk and the realistic way out. The condition is that we build it as an exact science, not as alchemy.
These are takeaways from Yudkowsky’s paper at MIRI, “Artificial Intelligence as a Positive and Negative Factor in Global Risk”. It was written around 2006 and published in 2008, long before ChatGPT, and it’s striking how well he anticipated so much of what has happened over the last 20 years.
Worth reading.
The fact that an AI can do something doesn’t mean it wants to. He calls it the “giant cheesecake fallacy” and it cuts both ways, for the apocalyptic scenarios and for the utopian ones.
The best part is about technical failures. The US Army trained a neural network to recognize camouflaged tanks. In testing it worked perfectly, in the real world not at all, because the photos with tanks had been taken on cloudy days and the ones without tanks on sunny days. The network had learned the weather, not the tanks. That’s the real risk: the system works flawlessly while you’re testing it and goes off the rails when the context changes. Anyone working with AI agents today knows the feeling.
Then the intelligence explosion. He compares it to Fermi’s first nuclear chain reaction in 1942: between k = 0.9994 and k = 1.0006 the difference is tiny, but in one case the reaction dies out and in the other it grows exponentially. How fast research is progressing tells you nothing about how fast an AI moves once it crosses the threshold.
I also liked “Moore’s Law of Mad Science”: every 18 months, the minimum IQ needed to destroy the world drops by one point. More compute means you can build AI without understanding it, but it does nothing to help you make it safe.
And the line everyone quotes: the AI doesn’t hate you, nor does it love you, but you are made of atoms it can use for something else.
The conclusion is that in the long run we have two options, either we get smarter or we disappear, and AI is at the same time the biggest risk and the realistic way out. The condition is that we build it as an exact science, not as alchemy.
These are takeaways from Yudkowsky’s paper at MIRI, “Artificial Intelligence as a Positive and Negative Factor in Global Risk”. It was written around 2006 and published in 2008, long before ChatGPT, and it’s striking how well he anticipated so much of what has happened over the last 20 years.
Worth reading.
14