The Spy at Harriton High

StrydeHax Blog ^ | 21 February 2010 | Stryde

[ShadowAce]

This investigation into the remote spying allegedly being conducted against students at Lower Merion represents an attempt to find proof of spying and a look into the toolchain used to accomplish spying. Taking a look at the LMSD Staff List, Mike Perbix is listed as a Network Tech at LMSD. Mr. Perbix has a large online web forum footprint as well as a personal blog, and a lot of his posts, attributed to his role at Lower Merion, provide insight into the tools, methods, and capabilities deployed against students at LMSD. Of the three network techs employed at LMSD, Mr. Perbix appears to have been the mastermind behind a massive, highly effective digital panopticon.

PanoMasterMind

The primary piece of evidence, already being reported on by a Fox affiliate, is this amazing promotional webcast for a remote monitoring product named LANRev. In it, Mike Perbix identifies himself as a high school network tech, and then speaks at length about using the track-and-monitor features of LanRev to take surreptitious remote pictures through a high school laptop webcam. A note of particular pride is evident in his voice when he talks about finding a way outside of LANRev to enable "curtain mode", a special remote administration mode that makes remote control of a laptop invisible to the victim. Listen at 35:47, when he says:
"you're controlling someone's machine, you don't want them to know what you're doing"
-Mike Perbix

It isn't until 37 minutes into the video till Perbix begins talking about the Theft Tracking feature, which causes the laptop to go into a mode where it beacons its location and silent webcam screenshots out to an Internet server controlled by the school.

Click to watch an excerpt of Mike Perbix's spycast

The beacon feature appears to have been one of the primary methods for remote spying, however, network footprints abound over the details and architecture of the remote administration effort. In this post, Perbix discusses methods for remotely resetting the firmware lockout used to prevent jailbreaking of student laptops. A jailbreak would have allowed students to monitor their own webcam to determine if administrators were truly taking pictures or if, as the school administration claimed, the blinking webcams were just "a glitch."

Perbix also maintains a prolific blog, where in this blog post he describes using the remote monitoring feature to locate a stolen laptop:

"As a prime example, we initially attempted to recover a stolen laptop that reported back to us it's internet address and DNS name. The police went to the house and were befuddled to find out the people we knew had the laptop was not the family that lived there...well, we eventually found out that they were the neighboring house and were borrowing the unsecured WI-FI."

In a September 2009 post that may come to haunt this investigation, Perbix posted a scripting method for remote enable/disable of the iSight camera in the laptops. This post makes a lot more sense when Perbix puts it in context on an admin newsgroup, in a post which makes it clear that his script allows for the camera to appear shut down to user applications such as Photo Booth but still function via remote administration:
"what this does is prevent internal use of the iSight, but some utilities might still work (for instance an external application using it for Theft tracking"

What's the purpose of shutting down a camera for the user of the laptop but still making it available to network administrators? Ask yourself: if you wanted to convince someone that a webcam blinking was a glitch, would disabling the cameras help make your case?

We Found the Glitch, Mrs. Buttle

The truly amazing part of this story is what's coming out from comments from the students themselves. Some of the interesting points:

• Possession of a monitored Macbook was required for classes
  
• Possession of an unmonitored personal computer was forbidden and would be confiscated

  
  

• Disabling the camera was impossible
  
• Jailbreaking a school laptop in order to secure it or monitor it against intrusion was an offense which merited expulsion

When I spoke at MIT about the wealth of electronic evidence I came across regarding Chinese gymnasts, I used the phrase "compulsory transparency". I never thought I would be using the phrase to describe America, especially so soon, but that appears to be exactly the case. On a familiar note, the authorities are denying everything. As one reads comments on this story, a consistent story begins to emerge:

"My name is Manuel Tebas. I was a student at Harriton High School, in the graduating class of 2009. We were the first year on the one-to-one laptop initiative. [...] I saw your post about removing webcam capability from the Macbook. It is possible - I did it last year. I will preface this by saying that when I did it, I was almost expelled, saved only by the fact that there was, at the time, no rule against doing so."

"I remember that the laptop was a requirement in school for many classes. That may remain so."

" had brought in my own personal computer to work on a project for school one day. I was doing a presentation involving programs not available on the regular computers, only in specific labs. I happened to have a copy of my own. My personal property was confiscated from me in a study hall when I was working on a school assignment because it was against the schools 'code of conduct'."

"Hi, I'm a 2009 Graduate of Harriton Highschool. [...] I and a few of my fellow peers were suspicious of this sort of activity when we first received the laptops. The light next to the web cam would randomly come on, whether we were in class, in study hall or at home minding our own business. We reported it multiple times, each time getting the response: "It's only a malfunction. if you'd like we'll look into it and give you a loaner computer."


"The webcam couldn't be disabled due through tough tough security settings. Occasionally we would notice that the green light was on from time to time but we just figured that it was glitching out as some macbooks do sometimes. Some few covered it up with tape and post its because they thought the IT guys were watching them. I always thought they were crazy and that the district, one of the more respectable ones within the state, would never pull some shit like this. I guess I was wrong."

"I am the father of a 17 y/o Harrington High student. She has had one of these laptops for 2 years. She has noticed the "green light" coming on but was not computer literate enough to know what initiated it"

Browse as many web forums as you like, the comments above are highly representative. Students were told green webcam activation lights going off at home were a glitch, were required to use a jailed computer, were threatened with expulsion if they attempted to jailbreak the computer to find the truth, and were not allowed to use computers they controlled.

Inside LANRev

With some of my colleagues, I began a reverse engineering effort against LANRev in order to determine the nature of the threat and possible countermeasures. Some of the things we found at first left us aghast as security pros: the spyware "client" (they call it an agent) binds to the server permanently without using authentication or key distribution. Find an unbound agent on your network with Bonjour, click on it, you own it. The server software, with an externally facing Internet port... runs as root. I'm not kidding. For those unfamiliar with the principle of least privilege- this is an indicator of a highly unskilled design. Unfortunately, when we got down to basic forensics, LANRev appears to cover its tracks well. Here's a screenshot of the server application monitoring a tracked host:

Tracking intervals available at the top; screenshots and webcam shots in the lower right pane. No webcam shot is visible here as a webcam was not connected during testing

In order to spy on my computer, I had to mark it for spying. The icon for spying is a detective hat and a magnifying glass; very Sherlock Holmes

Once I had the agent installed, I used dtrace to monitor its activity as it hung around and spied on my system. The log below is an edited trace of the agents activity during a spy interval. It uses a fixed dump point, /tmp/Image, as its save file before uploading to the server, sadly this is wiped. Only a full forensics scan which picks up deleted files will have a chance of picking up the history of the spying on a particular computer. On laptops with a webcam, a second fixed save point, /tmp/Image1, is used to save the webcam pic.

For the technically inclined, I've highlighted some of the key points, use of the system screengrabber, the use of RawCamera, the fixed save point, etc. We're still working on our technical writeup of this software and hope to update soon.

During our testing, we infected a laptop with LANRev, then closed the lid, hoping to activate the LANRev feature which takes a webcam picture when the computer wakes. As my colleague Aaron opened the lid of his Mac, the green webcam light flickered, ever so briefly. It wasn't a glitch. It was a highly sophisticated remote spy in his system. And even though he was in control, the effect was still very creepy.

Here's one last capture from the Windows version of the administration console, showing a forced remote webcam snapshot. We've pixellated this, but rest assured the real thing looks very detailed

In other news on the case, subpoenas have been issued, the FBI is on the case, the candy in question has been caught red-fingered, and some enterprising chap is ready to cash in with a t-shirt. Doug Muth's hands on screenshots provide the best first hand encounter with the client end of the spyware in question. What amazes me most is that the family and lawyer filing the suit appear to have done no digital forensics going in, and no enterprising student hacker ever jailbroke a laptop and proved this was going on. The greatest threat to this investigation now is the possibility that the highly trained technical staff at LMSD could issue a LANRev script to wipe digital forensic evidence off all the laptops. This is why it is imperative for affected parents to have the hard drive removed from their children's laptops and digitally imaged before the laptop is connected to a network. With enough persistence, and enough luck, we may eventually learn the truth.

-stryde.hax

---

update 2/23/2010 6:00pm

If you haven't already, you must watch this PBS Documentary - How Google Saved a School. At five minutes in, you can see all these same features in use, in a school setting, by a principal. Remote surreptitious observation. Remote camera use. All used by a principal to observe kids and make sure they're working. There are a lot of school districts, administrators, IT professionals, and security professionals who see nothing wrong with this documentary. They see remote administration software in use in this way and they don't think it's wrong, and they don't think it's spyware. Some of them even believe that the extension of this functionality into the home doesn't make it spyware, or even wrong. But this is my personal blog, and it's my personal opinion that they're wrong. As an expecting parent, I don't ever want my kids on the business end of Remote Desktop Curtain Mode, even at school. I'm a security professional, and a big part of my education and my professional development was tinkering and tearing apart computer systems to gain understanding, learn how they work, and change their use. I believe that computer security is knowledge in practice; it's using your knowledge to protect yourself. These kids are learning that security is something that happens to you. That's backwards. DARPA thinks we're not raising a generation with applicable security skills. I think they're right; I think this is a recipe for the next generation of phishing victims. I'd like to see a school system where a kid can bring in x64 Ubuntu or Haiku OS that he secured him/herself. I'd like to see a school system where kids teach each other how to defend against remote webcam use. Instead, we've got kids who can't run Terminal. Not my kids.
update 2/23/2010 4:12pm

A note for anyone wishing to contact me privately: if you'd like me to write back, please leave a return email. My email is still stryde dot blog at gmail dot com.
It's Not Spyware!!!
I've received a lot of positive feedback about this entry; however, if there's one consistent complaint amongst my detractors, it's my classification of LANRev as spyware. So here is my response. Confusing remote admin software with spyware has a long history stretching back to Cult of the Dead Cow's first Bo2k release. I'm not as funny as them so I don't even try. It's true however that remote administration tools and spyware exist on the same spectrum, just ask the guys at Spectresoft. Spyware authors and remote admin authors often have to solve the same problems, like bypassing OS protections and getting around antivirus. It's a transition that's easily made. So where's the dividing line? The line is basically in how its used. Remote admin usually solves constructive tasks, like remote patch management, inventory location tracking, remote software installation. And sometimes it means screensharing in order to solve problems. I personally have sat at home as a network tech worked on my corporate laptop over a VPN. No problem. My personal opinion is this: when you see a piece of software with dedicated functionality for taking webcam screenshots surreptitiously and removing the evidence on disk, to me that's crossed the line into spyware. I'm certain that others in the industry will disagree with me. That's fine; let's have the debate. I don't mind losing a technical argument, as long as it's on merit.

update 2/23/2010 11:28am

My colleague Aaron pointed out to me today that the reason LANRev is using the raw camera device is that Apple implemented security measures to prevent remote activation of the webcam in OSX. LANRev was designed to bypass this security measure. Those who disagree with my spyware assessment, ask yourself, "what kind of software bypasses OS security measures?"
On the topic of whether or not we yet have proof of illegal use, I would ask you to listen carefully to the webcast, and listen for the word "house" at 1:28. Listen for "yes we have used it."update 2/23/2010 10:00am

I've removed Mr. Perbix's picture from my blog. I try very hard to stick to verifiable facts when I write here; this blog post is made up references to primary documents that show a verifiable pattern of action. But I feel that some readers are getting carried away. Myself and Aaron Rhodes spent hours reading forum posts, messages, and communications from Mike Perbix, his "digital shadow". The impression we both got was of a man who was charged with enormous responsibility, worked very hard, was very adept, and was fanatical about protecting kids and the assets he was charged with managing. I don't have all the facts yet, but the impression I got was of someone who was trying to build a state of the art capability and revelled in the promise of technology. If I had to put my finger on what when wrong here, I would say that someone cared too much. Personally I'm much more interested in who this capability was distributed to, and its persistent pattern of access, than I am in the person who built it. If you're reading this, please, let us not participate in a rush to judgement especially against a guy who worked this hard. Yes, he built the capability. Yes it was used. But if it was abused or simply misguided, that remains to be proven. I for one reserve judgement. For now, what bothers me most is this: When an organ of the State (in this case, a school) builds a system to conduct a search by activating webcams off of school grounds, the only way to determine if the ensuing search will be unreasonable or illegal is to conduct the search. The thought process behind that is unfathomable to me, no matter how much I read about it.
update 2/22/2010 8:30pm

I've created a network footprinting capability for parents, students, anyone who may be concerned that they are infected with the LANRev agent. The capability is documented in my next blog entry. One piece of feedback I continue to get is speculation on what can be seen in a packet sniffer. The answer for now is: not much. A block cipher and compression are in use in serial. It's a tough problem; we're working on it.
update 2/22/2010 5:30pm

In a strange twist, the makers of LANRev have come out with a statement saying that school network techs should never have used their software to engage in theft recovery:"We discourage any customer from taking theft recovery into their own hands," said Stephen Midgley, the company's head of marketing, in an interview Monday. "That's best left in the hands of professionals."

I've watched the 50 minute screencast repeatedly, where Perbix describes his use of this feature outside of school grounds repeatedly during a conversation with Absolute Software employees. They were enthusiastic... now they're throwing LMSD under the bus? I believe this can best be described as intense PR spin. It also completely confirms what I've asserted here, that LANRev was the implant of choice for this school.

[Cyber Liberty]

What about the microphone?

[ShadowAce]

But first you have to suspect that the webcam is being used inappropriately/illegally.

Well, I was talking specifically about this situation, and what the other kids can do with their laptops.

What about the microphone?

A drop of superglue would solve it quite nicely. :)

[goldi]

The principal is denying it now:

http://www.philly.com/philly/news/breaking/85207407.html?cmpid=15585797

This is getting to sound more like “Balloon Boy.” Someone’s lying.

[goldi]

The principal is denying it now:

http://www.philly.com/philly/news/breaking/85207407.html?cmpid=15585797

This is getting to sound more like “Balloon Boy.” Someone’s lying.

[Vinnie]

The school district could save millions by NOT supplying each student w/ a laptop.

IMHO that is waaay over the top.

The kids likely spend most of their online time looking at porn and nude girlfriends.

[LibertyRocks]

In a way I feel for the kids because the adults should have taken the lead on making sure the computer was safe. But, then I would argue the kids and their parents had reasonable expectations to privacy which were callously disregarded by the school and the tech. I’m just glad we homeschool. I don’t need another reason to be paranoid about the safety of my kids...

[metmom]

Well, that’s what the public schools have been grooming us for for generations.

Looks like they’ve succeeded in a lot of cases.

[metmom]

She later added that, in more than a decade as assistant vice principal, she had "never disciplined a student" for actions beyond school property that had no connection to a school-related event, apparently in response to the Robbins lawsuit's allegation the student learned of the webcam surveillance from a school disciplinary action.

I wondered how long it would take for that to happen.

Since it's he said/she said, she can safely deny it and then demand proof that she's lying, knowing that no hard copies exist, or have been destroyed.

However, if nothing else, this did bring to light the fact that these computers had capabilities not honestly disclosed to the students and their families.

[metmom]

The adults should have put their foot down and refused to have their kids saddled with such a onerous burden.

[goldi]

“However, if nothing else, this did bring to light the fact that these computers had capabilities not honestly disclosed to the students and their families.”

That’s where the school district made their big mistake, hopefully out of ignorance, not malice. Perhaps the principal was unknowingly duped, made a scapegoat by someone (the kid and his parents, the IT Department, someone else?). What a career killer. No matter. The cat’s out of the bag. The computers are compromised, the school’s reputation is in tatters.

That IT department or whatever they call themselves — they’re something else. The FBI will straighten it all out.

[grey_whiskers]

I am corroded with jealousy that I failed to come up with that pun first!

Cheers!

[Tired of Taxes]

•Possession of a monitored Macbook was required for classes

•Possession of an unmonitored personal computer was forbidden and would be confiscated

Wow. So, I guess saying the students weren't "allowed" to use other computers was the right way to put it.

[Tired of Taxes]

I was wondering how long it was going to take before they slandered the plaintiff and set up the IT guy. Less than a week...that's got to be a record.

I'm just seeing this article tonight. But, I was talking to my DH (a programmer) earlier today. He was incredulous that anyone working in IT would go along with a plan involving spying via webcam on anyone at home. He said most people working in IT aren't shy about refusing to do something that's wrong because they KNOW the higher-ups will turn around and blame them later.

Yet, here's the school's IT guy admitting to it openly and proudly. :-0 Obviously, he was asked by the school to do it. But, it seems the IT department should've known better.

[Bikkuri]

I read an article about a year or so ago that mentioned something about the cam actually being in the center of the laptop monitor.. so that when you were online chatting with someone, it would appear that you were looking directly at them...

I’m not sure if this technology is out there yet.. but THAT is a scary thought.. and I don’t think that tape would work >.<

Have you heard anything about that, and how far along they are with that kind of technology?

Bikk

[ShadowAce]

I read the same information, but have not heard anything more on the subject.

[pray4liberty]

But, it seems the IT department should've known better.

True, and most in their right minds would refuse, but it wouldn't surprise me in the least if the two-faced school administrators told him (1) their lawyers said it was OK and perfectly legal to do or (2) don't ask, don't tell, and don't get caught.

Oops.

[pray4liberty]

They’ll claim ignorance of the law. That will be their defense, and they’ll think that’s sufficient.

Probably, but it won't be good enough. I am really surprised the U.S. Department of Education's Office of Civil Rights in Philly hasn't joined this investigation.

Whoever got the big idea to do this needs to go where some prison pervert will be watching them 24/7.

[pray4liberty]

Re: your post #19... sounds to me you ran up against the zero tolerance brigade, as have I and scores of other parents battling the monsters abusing the system and using our taxpayers money doing it. That is exactly what they do to parents. It has accelerated under No Child Left Behind.

If they likewise get run over by the zero tolerance, no-excuses, everybody-gets-punished-the-same-regardless system, well...maybe that's a good lesson they need to learn.

[pray4liberty]

Especially notice the bit in the story about the admin bragging about sending cops to hunt down a stolen laptop, and how they went to the wrong house because the real perps were stealing wireless.

I worked in network security for a time. Your warnings are what we told our customers: it was imperative to secure the router with a passkey and install a firewall on every computer in the house. No excuses. This was the worst-case scenario that we outlined.

[Tired of Taxes]

it wouldn't surprise me in the least if the two-faced school administrators told him (1) their lawyers said it was OK and perfectly legal to do or (2) don't ask, don't tell, and don't get caught.

Yeah, that wouldn't surprise me, either.