Al Security Warning from Signal app’s Meredith Whittaker - The Hidden Dangers of Agentic Al
0:00. constructed. We can get into that as well. But um an agent agentic Al this the thing that is the new buzzword that everything that
0:08 used to be an assistant now is um I think there’s a real danger that we’re facing and signal is clocking this
0:15 pretty closely of the introduction of this sort of notion of agentic Al into
0:21 our devices and and lives in part because what we’re doing is giving so much control to these systems that are
0:30 going going to need access to data. So if you you read the kind of you you read the marketing on this, the the value ad
0:37 is something like can look up a concert, book a ticket, schedule it in your calendar and message all your friends
0:45 that it’s booked, right? And so we can just put our brain in a jar because the thing is doing that and we don’t have to touch it, right? And so what would it
0:52 need to do that? Well, it would need access to our browser, an ability to drive that. It would need our credit card information to pay for the tickets.
1:02 It would need access to our calendar, everything we’re doing, everyone we’re meeting. It would need access to
open and send that message to our friends. And it would need to be able to drive that across our entire system with
1:17 something that looks like root permission, accessing every single one of those databases, probably in the clear because there’s no model to do that encrypted.
1:26 And if we’re talking about a sufficiently powerful model Al model that’s powering that, there’s no way that’s happening on device. Even though ondevice isn’t a prof prophylactic isn’t
1:35 going to really solve a lot of those issues, that’s almost certainly being sent to a cloud server where it’s being processed and sent back. So there’s a
1:43 profound issue with security and privacy that is haunting this sort of hype around agents and that is ultimately
1:51 threatening to break the blood brain barrier between the application layer and the
OS layer by conjoining all of
1:59 these separate services, muddying their data and doing things like undermining the privacy of your signal messages
2:06 because hey, the agents got to get in, the agents got to text your friends, the agents text your friends, the agents got to pull the data out of your text and got to summarize that so that again your brain can sit in a jar and you’re not doing any of that yourself.
2:17 You’re doing something else. So I think we need to be really careful, you know, when I think about the immediate concerns, not simply the history of Al and the fact that it’s, you know,
2:26 predicated on this larger surveillance model. There’s a real issue right now of the undermining that Al systems are
2:33 poised to do in these privacy and security guarantees in the name of this sort of, you know, magic genie bot that’s going to take care of the exigencies of life.
AI Security Warning from Signal App’s Meredith Whittaker - The Hidden Dangers of Agentic AI. Constructed. We can get into that as well.
But, , an agent agentic AI this the thing that is the new buzzword that everything that used to be an assistant now is, , I think there’s a real danger that we’re facing and Signal is clocking this pretty closely of the introduction of this sort of notion of agentic AI into our devices and lives in part because what we’re doing is giving so much control to these systems that are going to need access to data.
So if you read the kind of you read the marketing on this, the value add is something like can look up a concert, book a ticket, schedule it in your calendar, and message all your friends that it’s booked, right? And so we can just put our brain in a jar because the thing is doing that and we don’t have to touch it, right?
And so what would it need to do that? Well, it would need access to our browser, an ability to drive that. It would need our credit card information to pay for the tickets.
It would need access to our calendar, everything we’re doing, everyone we’re meeting. It would need access to open and send that message to our friends.
And it would need to be able to drive that across our entire system with something that looks like root permission, accessing every single one of those databases, probably in the clear because there’s no model to do that encrypted.
And if we’re talking about a sufficiently powerful model AI model that’s powering that, there’s no way that’s happening on device. Even though on-device isn’t a prophylactic isn’t going to really solve a lot of those issues, that’s almost certainly being sent to a cloud server where it’s being processed and sent back.
So there’s a profound issue with security and privacy that is haunting this sort of hype around agents and that is ultimately threatening to break the blood-brain barrier between the application layer and the OS layer by conjoining all of these separate services, muddying their data and doing things like undermining the privacy of your Signal messages because, hey, the agents got to get in, the agents got to text your friends, the agents text your friends, the agents got to pull the data out of your text and got to summarize that so that again your brain can sit in a jar and you’re not doing any of that yourself.
You’re doing something else. So I think we need to be really careful, you know, when I think about the immediate concerns, not simply the history of AI and the fact that it’s, you know, predicated on this larger surveillance model.
There’s a real issue right now of the undermining that AI systems are poised to do in these privacy and security guarantees in the name of this sort of, you know, magic genie bot that’s going to take care of the exigencies of life.