There are many successful people with high IQs who are worried that someday a rogue AI will attack humans and/or take over the world and/or ... something scary. There are even nonprofit organizations dedicated to spreading the word about the dangers of rogue AI in the future. They are soliciting donations and publishing pseudo-scientific papers.
As far as I can tell, few of these worriers have ever programmed a computer to do something as complex as flying an airplane.
Human cognition is biased in a lot of ways. Going through the list would take a long time, but we aren't simply pure logic machines. We've been shaped by evolution and our environments to pay attention to certain kinds of threats, while ignoring others. We've been shaped to tell stories in certain kinds of ways, while others are "boring". We share certain irrational religious, political, and cultural beliefs; not because they make sense, but because we need to belong to groups and we declare our memberships by sharing irrational beliefs.
These various cognitive biases come together in this fear of rogue AIs.
First, we tend to fear new, rare, and poorly understood threats more than old, common, and well understood threats. For example, when AIDS was new and rare people feared it a lot more than they feared dying in a car crash. But even at its peak in the US, AIDS didn't kill more people than car crashes. Currently, car crashes kill 3x more than AIDS in the US, but AIDS still seems more scary, because cars are more everyday & familiar than AIDS.
Second, our science fiction stories tend to rely on infinite technological extrapolation, while actual engineering progress is limited by hard physical laws. Our stories contain faster-than-light travel, while the actual speed of human travel hit a plateau 40-50 years ago and may never increase again. Our stories contain time travel, while the, um, actual time, aww, fuck it ;-)
Third, irrational ideas sound more plausible when groups of other people, including famous celebrities, have committed to these ideas beyond a doubt. The Virgin Mary, for example.
With Rogue AIs we have a brand new threat that we don't understand (especially because they don't exist), a threat that would supposedly inevitably result from infinite technological extrapolation in the speed & complexity of computing power, a threat that is commonly recounted in (fictional) books and film, and now a threat that groups of people, including famous celebrities, fear without a doubt.
Actual human intelligence required roughly a billion years of evolution to arise, and it is way more complicated than most people understand. There is no working model for how the brain actually works. Nobody knows how consciousness arises within the brain. That our brains work at all is the product of -- again -- a billion years of selection pressure (not some intelligent design).
I think it is silly to think that within a few decades of dicking around with computer programming, such "selection pressures" would produce the equivalent of a sentient multi-cellular organism that is more intelligent than humans, that would become a danger to humans, and that would master its own destiny.
The real dangers of internetworked computers are already here, but few people seem to care about them, because our internetworked computers are as familiar to us as cars. Even more so, many of us sleep with our smart phones right next to us.
The real dangers of internetworked computers come from other humans using these networks for their own malicious purposes.
We have an epidemic of identity theft, and criminals are stealing millions of credit card numbers every year. Hackers can, and do, break into just about every computer database on the planet and sell the information on the black market. Both governments, corporations, nonprofits, and organized crime have established surveillance systems that can track your location, your relationships, your communications, your eating and exercise habits, your income and spending habits, your health issues, the media you consume, your religious and political beliefs. This information is sold, traded, and stolen, between organizations.
At least in the US we have some Constitutional limits on the use of these surveillance powers by governments. In other countries, people are arrested, tortured, blackmailed, and killed on the basis of their internetworked behaviors. Imagine a world where "liking" a group on Facebook means the secret police show up and take you away. This world is real!
Oh, and, even in the US if your religious or political beliefs are considered "terrorist" then the not-so-secret police will still show up at your door and take you away.
The idea of an omniscient omnipotent God was created a few thousand years ago to enhance the control of the religious and political elites. If they could convince people that a God was always watching them, just waiting for a chance to throw them into Hell, then the people would follow the rules more often.
In a more atheist age, technology has produced a less imaginary form of omniscience. Internetworked computers have not given our elites omnipotence, thankfully, but that distinction matters little when you are a target of the government's power.
You might feel like the omniscient surveillance powers don't matter to you, because you follow the rules, you have nothing to hide, you live in a free society. In such a case you don't even think about how you've been socialized to follow the rules and have nothing to hide. You've already internalized your coexistence with the omniscience and potency of your God/government. But others are not as compliant as you.
I wonder whether the fear of a rogue AI is, like so many of our fears, a distraction from reality, a distraction from the real problems. Maybe not just a distraction, but a legitimizer of the status quo. Fearing a rogue AI means that the current system of omniscient surveillance isn't so bad. At least your laptop/phone/tablet isn't trying to kill you.
But other humans ... they are definitely using your devices to prey upon you.
The ecology of predator & prey is such that predation will increase until predators become numerous and prey become scarce. I think we are in the early stage of Internet predation. There are still far more prey than predators, and the number of prey are still growing around the world as un-networked people join the Internet by the billions. But the Internet will steadily become more dangerous until prey become scarce and then the predators will turn upon each other.
World War III will be fought on the Internet, by humans, not by rogue AIs. And these humans will know everything about you.