Posted on 10/21/2019 12:29:21 PM PDT by LibWhacker
Last March, Chinese researchers announced an ingenious and potentially devastating attack against one of Americas most prized technological assetsa Tesla electric car.
The team, from the security lab of the Chinese tech giant Tencent, demonstrated several ways to fool the AI algorithms on Teslas car. By subtly altering the data fed to the cars sensors, the researchers were able to bamboozle and bewilder the artificial intelligence that runs the vehicle.
In one case, a TV screen contained a hidden pattern that tricked the windshield wipers into activating. In another, lane markings on the road were ever-so-slightly modified to confuse the autonomous driving system so that it drove over them and into the lane for oncoming traffic.
Teslas algorithms are normally brilliant at spotting drops of rain on a windshield or following the lines on the road, but they work in a way thats fundamentally different from human perception. That makes such deep learning algorithms, which are rapidly sweeping through different industries for applications such as facial recognition and cancer diagnosis, surprisingly easy to fool if you find their weak points.
Leading a Tesla astray might not seem like a strategic threat to the United States. But what if similar techniques were used to fool attack drones, or software that analyzes satellite images, into seeing things that arent thereor not seeing things that are? Artificial intelligence-gathering
Around the world, AI is already seen as the next big military advantage.
Early this year, the US announced a grand strategy for harnessing artificial intelligence in many areas of the military, including intelligence analysis, decision-making, vehicle autonomy, logistics, and weaponry. The Department of Defenses proposed $718 billion budget for 2020 allocates $927 million for AI and machine learning. Existing projects include the rather mundane (testing whether AI can predict when tanks and trucks need maintenance) as well as things on the leading edge of weapons technology (swarms of drones).
The Pentagons AI push is partly driven by fear of the way rivals might use the technology. Last year Jim Mattis, then the secretary of defense, sent a memo to President Donald Trump warning that the US is already falling behind when it comes to AI. His worry is understandable.
In July 2017, China articulated its AI strategy, declaring that the worlds major developed countries are taking the development of AI as a major strategy to enhance national competitiveness and protect national security. And a few months later, Vladimir Putin of Russia ominously declared: Whoever becomes the leader in [the AI] sphere will become the ruler of the world.
The ambition to build the smartest, and deadliest, weapons is understandable, but as the Tesla hack shows, an enemy that knows how an AI algorithm works could render it useless or even turn it against its owners. The secret to winning the AI wars might rest not in making the most impressive weapons but in mastering the disquieting treachery of the software.
Battle bots
On a bright and sunny day last summer in Washington, DC, Michael Kanaan was sitting in the Pentagons cafeteria, eating a sandwich and marveling over a powerful new set of machine--learning algorithms.
A few weeks earlier, Kanaan had watched a video game in which five AI algorithms worked together to very nearly outmaneuver, outgun, and outwit five humans in a contest that involved controlling forces, encampments, and resources across a complex, sprawling battlefield. The brow beneath Kanaans cropped blond hair was furrowed as he described the action, though. It was one of the most impressive demonstrations of AI strategy hed ever seen, an unexpected development akin to AI advances in chess, Atari, and other games.
The war game had taken place within Dota 2, a popular sci-fi video game that is incredibly challenging for computers. Teams must defend their territory while attacking their opponents encampments in an environment that is more complex and deceptive than any board game. Players can see only a small part of the whole picture, and it can take about half an hour to determine if a strategy is a winning one.
The AI combatants were developed not by the military but by OpenAI, a company created by Silicon Valley bigwigs including Elon Musk and Sam Altman to do fundamental AI research. The companys algorithmic warriors, known as the OpenAI Five, worked out their own winning strategies through relentless practice, and by responding with moves that proved most advantageous.
It is exactly the type of software that intrigues Kanaan, one of the people tasked with using artificial intelligence to modernize the US military. To him, it shows what the military stands to gain by enlisting the help of the worlds best AI researchers. But whether they are willing is increasingly in question.
Kanaan was the Air Force lead on Project Maven, a military initiative aimed at using AI to automate the identification of objects in aerial imagery. Google was a contractor on Maven, and when other Google employees found that out, in 2018, the company decided to abandon the project. It subsequently devised an AI code of conduct saying Google would not use its AI to develop weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people.
Workers at some other big tech companies followed by demanding that their employers eschew military contracts. Many prominent AI researchers have backed an effort to initiate a global ban on developing fully autonomous weapons.
To Kanaan, however, it would be a big problem if the military couldnt work with researchers like those who developed the OpenAI Five. Even more disturbing is the prospect of an adversary gaining access to such cutting-edge technology. The code is just out there for anyone to use, he said. He added: war is far more complex than some video game.
AI Surge
Kanaan is generally very bullish about AI, partly because he knows firsthand how useful it stands to be for troops. Six years ago, as an Air Force intelligence officer in Afghanistan, he was responsible for deploying a new kind of intelligence-gathering tool: a hyperspectral imager. The instrument can spot objects that are normally hidden from view, like tanks draped in camouflage or emissions from an improvised bomb-making factory. Kanaan says the system helped US troops remove many thousands of pounds of explosives from the battlefield. Even so, it was often impractical for analysts to process the vast amounts of data collected by the imager. We spent too much time looking at the data and not enough time making decisions, he says. Sometimes it took so long that you wondered if you couldve saved more lives.
A solution could lie in a breakthrough in computer vision by a team led by Geoffrey Hinton at the University of Toronto. It showed that an algorithm inspired by a many-layered neural network could recognize objects in images with unprecedented skill when given enough data and computer power.
Training a neural network involves feeding in data, like the pixels in an image, and continuously altering the connections in the network, using mathematical techniques, so that the output gets closer to a particular outcome, like identifying the object in the image. Over time, these deep-learning networks learn to recognize the patterns of pixels that make up houses or people. Advances in deep learning have sparked the current AI boom; the technology underpins Teslas autonomous systems and OpenAIs algorithms.
Kanaan immediately recognized the potential of deep learning for processing the various types of images and sensor data that are essential to military operations. He and others in the Air Force soon began lobbying their superiors to invest in the technology. Their efforts have contributed to the Pentagons big AI push.
But shortly after deep learning burst onto the scene, researchers found that the very properties that make it so powerful are also an Achilles heel.
Just as its possible to calculate how to tweak a networks parameters so that it classifies an object correctly, it is possible to calculate how minimal changes to the input image can cause the network to misclassify it. In such adversarial examples, just a few pixels in the image are altered, leaving it looking just the same to a person but very different to an AI algorithm. The problem can arise anywhere deep learning might be usedfor example, in guiding autonomous vehicles, planning missions, or detecting network intrusions.
Amid the buildup in military uses of AI, these mysterious vulnerabilities in the software have been getting far less attention.
Moving targets
One remarkable object serves to illustrate the power of adversarial machine learning. Its a model turtle.
To you or me it looks normal, but to a drone or a robot running a particular deep-learning vision algorithm, it seems to be a rifle. In fact, at one point the unique pattern of markings on the turtles shell could be recrafted so that an AI vision system made available through Googles cloud would mistake it for just about anything. (Google has since updated the algorithm so that it isnt fooled.)
The turtle was created not by some nation-state adversary, but by four guys at MIT. One of them is Anish Athalye, a lanky and very polite young man who works on computer security in MITs Computer Science and Artificial Intelligence Laboratory (CSAIL). In a video on Athalyes laptop of the turtles being tested (some of the models were stolen at a conference, he says), it is rotated through 360 degrees and flipped upside down. The algorithm detects the same thing over and over: rifle, rifle, rifle.
The earliest adversarial examples were brittle and prone to failure, but Athalye and his friends believed they could design a version robust enough to work on a 3D-printed object. This involved modeling a 3D rendering of objects and developing an algorithm to create the turtle, an adversarial example that would work at different angles and distances. Put more simply, they developed an algorithm to create something that would reliably fool a machine-learning model.
The military applications are obvious. Using adversarial algorithmic camouflage, tanks or planes might hide from AI-equipped satellites and drones. AI-guided missiles could be blinded by adversarial data, and perhaps even steered back toward friendly targets. Information fed into intelligence algorithms might be poisoned to disguise a terrorist threat or set a trap for troops in the real world.
Athalye is surprised by how little concern over adversarial machine learning he has encountered. Ive talked to a bunch of people in industry, and I asked them if they are worried about adversarial examples, he says. The answer is, almost across the board, no.
Fortunately, the Pentagon is starting to take notice. This August, the Defense Advanced Research Projects Agency (DARPA) announced several big AI research projects. Among them is GARD, a program focused on adversarial machine learning. Hava Siegelmann, a professor at the University of Massachusetts, Amherst, and the program manager for GARD, says these attacks could be devastating in military situations because people cannot identify them. Its like were blind, she says. Thats what makes it really very dangerous.
The challenges presented by adversarial machine learning also explain why the Pentagon is so keen to work with companies like Google and Amazon as well as academic institutions like MIT. The technology is evolving fast, and the latest advances are taking hold in labs run by Silicon Valley companies and top universities, not conventional defense contractors.
Crucially, theyre also happening outside the US, particularly in China. I do think that a different world is coming, says Kanaan, the Air Force AI expert. And its one we have to combat with AI.
The backlash against military use of AI is understandable, but it may miss the bigger picture. Even as people worry about intelligent killer robots, perhaps a bigger near-term risk is an algorithmic fog of warone that even the smartest machines cannot peer through.
Two bits are either equal or not. No computer knows what close means, in the end it is true or false.
No computer knows what close means, in the end it is true or false.
But a demon sitting astride a quantum computer might.
kirk: mr. spock always tells the truth.
Mr. Spock: I am lying.
AI: (head explodes)
Keep in mind that this summary is from the same MIT that assured us in the 1980’s that it’d be impossible to create missile defense because, as the thinking went, “you can’t hit a bullet with another bullet”.
I seen this my whole life in Business. IT constantly try to sell management with the notion the machine can replace the person. To an extent that is true. As long as it involves no reasoning at all the computer can be taught to weld a bead or fasten lug nuts on a car wheel.
Where the computer will always fail is anytime it has to reason. As I keep telling my bosses for the last 30 years, “the computer can not be taught when to cheat, people can.”
The computer can absolutely be taught to reason with regard to limited problems and can absolutely be taught when to cheat.
Spoken like someone that’s never played Madden. Computers cheat all the time.
“same MIT that assured us in the 1980s that itd be impossible to create missile defense because, as the thinking went, you cant hit a bullet with another bullet.”
Funny that MIT was so out of the loop. My brother was working on algorithms to do just that (with missiles) in the 70s.
What you are claiming was pretty much a correct assessment in say, the 1960s.
Video game world tie in:
Tencent owns 100% of Riot Games, a bit if Activision/Blizzard, some bits of Discord, some streaming services, and funded Reddit to about 150million...
There’s more, but this is off the top of my head.
Nope. That is a deductive process the computer cannot match
No that not cheating. That operating on a different set of programming. It quite different
No it based on reality that the IT posse simply refuses to learn. Your sort of IT bot have been preaching the same nonsense since the 1970s and it always just around the corner, never here.
You cannot make the computer do deductive reasoning. It always limited to it programming. I work with current generation tech and it far short of the wonder the IT bots sold to Management. “why can we not do this” and it because the computer cannot think outside it programming. The person can.
This article outlines how cutting edge IT tech is being baffled by simply human cheating. Did you READ the article? It proves the point I am making.
Tech is not smart enough to recognize the flaw and adapt. Humans do almost without conscious thought
In Huntsville, AL? Us Alabamians are kinda proud of the geeks there.
To be fair to MIT, at the time they said it was when their main U.S. Senator, Ted Kennedy, was doing anything he could to make then President Regan look bad. And Reagan was pushing SDI, which meant Kennedy had to oppose it. He did that by getting a lot of the deans and chairs within MIT to come testify to Congress and talk on Sunday shows and such and explain to us plebes how it can't be done.
All your comments are well suited to a world of sequential, von Neumann programming constructs, but you’d have to go back in time or to a business programming environment to find that.
No, Pentagon and Crystal City, VA.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.