The Spy at Harriton High

StrydeHax Blog ^ | 21 February 2010 | Stryde

[ShadowAce]

This investigation into the remote spying allegedly being conducted against students at Lower Merion represents an attempt to find proof of spying and a look into the toolchain used to accomplish spying. Taking a look at the LMSD Staff List, Mike Perbix is listed as a Network Tech at LMSD. Mr. Perbix has a large online web forum footprint as well as a personal blog, and a lot of his posts, attributed to his role at Lower Merion, provide insight into the tools, methods, and capabilities deployed against students at LMSD. Of the three network techs employed at LMSD, Mr. Perbix appears to have been the mastermind behind a massive, highly effective digital panopticon.

PanoMasterMind

The primary piece of evidence, already being reported on by a Fox affiliate, is this amazing promotional webcast for a remote monitoring product named LANRev. In it, Mike Perbix identifies himself as a high school network tech, and then speaks at length about using the track-and-monitor features of LanRev to take surreptitious remote pictures through a high school laptop webcam. A note of particular pride is evident in his voice when he talks about finding a way outside of LANRev to enable "curtain mode", a special remote administration mode that makes remote control of a laptop invisible to the victim. Listen at 35:47, when he says:
"you're controlling someone's machine, you don't want them to know what you're doing"
-Mike Perbix

It isn't until 37 minutes into the video till Perbix begins talking about the Theft Tracking feature, which causes the laptop to go into a mode where it beacons its location and silent webcam screenshots out to an Internet server controlled by the school.

Click to watch an excerpt of Mike Perbix's spycast

The beacon feature appears to have been one of the primary methods for remote spying, however, network footprints abound over the details and architecture of the remote administration effort. In this post, Perbix discusses methods for remotely resetting the firmware lockout used to prevent jailbreaking of student laptops. A jailbreak would have allowed students to monitor their own webcam to determine if administrators were truly taking pictures or if, as the school administration claimed, the blinking webcams were just "a glitch."

Perbix also maintains a prolific blog, where in this blog post he describes using the remote monitoring feature to locate a stolen laptop:

"As a prime example, we initially attempted to recover a stolen laptop that reported back to us it's internet address and DNS name. The police went to the house and were befuddled to find out the people we knew had the laptop was not the family that lived there...well, we eventually found out that they were the neighboring house and were borrowing the unsecured WI-FI."

In a September 2009 post that may come to haunt this investigation, Perbix posted a scripting method for remote enable/disable of the iSight camera in the laptops. This post makes a lot more sense when Perbix puts it in context on an admin newsgroup, in a post which makes it clear that his script allows for the camera to appear shut down to user applications such as Photo Booth but still function via remote administration:
"what this does is prevent internal use of the iSight, but some utilities might still work (for instance an external application using it for Theft tracking"

What's the purpose of shutting down a camera for the user of the laptop but still making it available to network administrators? Ask yourself: if you wanted to convince someone that a webcam blinking was a glitch, would disabling the cameras help make your case?

We Found the Glitch, Mrs. Buttle

The truly amazing part of this story is what's coming out from comments from the students themselves. Some of the interesting points:

• Possession of a monitored Macbook was required for classes
  
• Possession of an unmonitored personal computer was forbidden and would be confiscated

  
  

• Disabling the camera was impossible
  
• Jailbreaking a school laptop in order to secure it or monitor it against intrusion was an offense which merited expulsion

When I spoke at MIT about the wealth of electronic evidence I came across regarding Chinese gymnasts, I used the phrase "compulsory transparency". I never thought I would be using the phrase to describe America, especially so soon, but that appears to be exactly the case. On a familiar note, the authorities are denying everything. As one reads comments on this story, a consistent story begins to emerge:

"My name is Manuel Tebas. I was a student at Harriton High School, in the graduating class of 2009. We were the first year on the one-to-one laptop initiative. [...] I saw your post about removing webcam capability from the Macbook. It is possible - I did it last year. I will preface this by saying that when I did it, I was almost expelled, saved only by the fact that there was, at the time, no rule against doing so."

"I remember that the laptop was a requirement in school for many classes. That may remain so."

" had brought in my own personal computer to work on a project for school one day. I was doing a presentation involving programs not available on the regular computers, only in specific labs. I happened to have a copy of my own. My personal property was confiscated from me in a study hall when I was working on a school assignment because it was against the schools 'code of conduct'."

"Hi, I'm a 2009 Graduate of Harriton Highschool. [...] I and a few of my fellow peers were suspicious of this sort of activity when we first received the laptops. The light next to the web cam would randomly come on, whether we were in class, in study hall or at home minding our own business. We reported it multiple times, each time getting the response: "It's only a malfunction. if you'd like we'll look into it and give you a loaner computer."


"The webcam couldn't be disabled due through tough tough security settings. Occasionally we would notice that the green light was on from time to time but we just figured that it was glitching out as some macbooks do sometimes. Some few covered it up with tape and post its because they thought the IT guys were watching them. I always thought they were crazy and that the district, one of the more respectable ones within the state, would never pull some shit like this. I guess I was wrong."

"I am the father of a 17 y/o Harrington High student. She has had one of these laptops for 2 years. She has noticed the "green light" coming on but was not computer literate enough to know what initiated it"

Browse as many web forums as you like, the comments above are highly representative. Students were told green webcam activation lights going off at home were a glitch, were required to use a jailed computer, were threatened with expulsion if they attempted to jailbreak the computer to find the truth, and were not allowed to use computers they controlled.

Inside LANRev

With some of my colleagues, I began a reverse engineering effort against LANRev in order to determine the nature of the threat and possible countermeasures. Some of the things we found at first left us aghast as security pros: the spyware "client" (they call it an agent) binds to the server permanently without using authentication or key distribution. Find an unbound agent on your network with Bonjour, click on it, you own it. The server software, with an externally facing Internet port... runs as root. I'm not kidding. For those unfamiliar with the principle of least privilege- this is an indicator of a highly unskilled design. Unfortunately, when we got down to basic forensics, LANRev appears to cover its tracks well. Here's a screenshot of the server application monitoring a tracked host:

Tracking intervals available at the top; screenshots and webcam shots in the lower right pane. No webcam shot is visible here as a webcam was not connected during testing

In order to spy on my computer, I had to mark it for spying. The icon for spying is a detective hat and a magnifying glass; very Sherlock Holmes

Once I had the agent installed, I used dtrace to monitor its activity as it hung around and spied on my system. The log below is an edited trace of the agents activity during a spy interval. It uses a fixed dump point, /tmp/Image, as its save file before uploading to the server, sadly this is wiped. Only a full forensics scan which picks up deleted files will have a chance of picking up the history of the spying on a particular computer. On laptops with a webcam, a second fixed save point, /tmp/Image1, is used to save the webcam pic.

For the technically inclined, I've highlighted some of the key points, use of the system screengrabber, the use of RawCamera, the fixed save point, etc. We're still working on our technical writeup of this software and hope to update soon.

During our testing, we infected a laptop with LANRev, then closed the lid, hoping to activate the LANRev feature which takes a webcam picture when the computer wakes. As my colleague Aaron opened the lid of his Mac, the green webcam light flickered, ever so briefly. It wasn't a glitch. It was a highly sophisticated remote spy in his system. And even though he was in control, the effect was still very creepy.

Here's one last capture from the Windows version of the administration console, showing a forced remote webcam snapshot. We've pixellated this, but rest assured the real thing looks very detailed

In other news on the case, subpoenas have been issued, the FBI is on the case, the candy in question has been caught red-fingered, and some enterprising chap is ready to cash in with a t-shirt. Doug Muth's hands on screenshots provide the best first hand encounter with the client end of the spyware in question. What amazes me most is that the family and lawyer filing the suit appear to have done no digital forensics going in, and no enterprising student hacker ever jailbroke a laptop and proved this was going on. The greatest threat to this investigation now is the possibility that the highly trained technical staff at LMSD could issue a LANRev script to wipe digital forensic evidence off all the laptops. This is why it is imperative for affected parents to have the hard drive removed from their children's laptops and digitally imaged before the laptop is connected to a network. With enough persistence, and enough luck, we may eventually learn the truth.

-stryde.hax

---

update 2/23/2010 6:00pm

If you haven't already, you must watch this PBS Documentary - How Google Saved a School. At five minutes in, you can see all these same features in use, in a school setting, by a principal. Remote surreptitious observation. Remote camera use. All used by a principal to observe kids and make sure they're working. There are a lot of school districts, administrators, IT professionals, and security professionals who see nothing wrong with this documentary. They see remote administration software in use in this way and they don't think it's wrong, and they don't think it's spyware. Some of them even believe that the extension of this functionality into the home doesn't make it spyware, or even wrong. But this is my personal blog, and it's my personal opinion that they're wrong. As an expecting parent, I don't ever want my kids on the business end of Remote Desktop Curtain Mode, even at school. I'm a security professional, and a big part of my education and my professional development was tinkering and tearing apart computer systems to gain understanding, learn how they work, and change their use. I believe that computer security is knowledge in practice; it's using your knowledge to protect yourself. These kids are learning that security is something that happens to you. That's backwards. DARPA thinks we're not raising a generation with applicable security skills. I think they're right; I think this is a recipe for the next generation of phishing victims. I'd like to see a school system where a kid can bring in x64 Ubuntu or Haiku OS that he secured him/herself. I'd like to see a school system where kids teach each other how to defend against remote webcam use. Instead, we've got kids who can't run Terminal. Not my kids.
update 2/23/2010 4:12pm

A note for anyone wishing to contact me privately: if you'd like me to write back, please leave a return email. My email is still stryde dot blog at gmail dot com.
It's Not Spyware!!!
I've received a lot of positive feedback about this entry; however, if there's one consistent complaint amongst my detractors, it's my classification of LANRev as spyware. So here is my response. Confusing remote admin software with spyware has a long history stretching back to Cult of the Dead Cow's first Bo2k release. I'm not as funny as them so I don't even try. It's true however that remote administration tools and spyware exist on the same spectrum, just ask the guys at Spectresoft. Spyware authors and remote admin authors often have to solve the same problems, like bypassing OS protections and getting around antivirus. It's a transition that's easily made. So where's the dividing line? The line is basically in how its used. Remote admin usually solves constructive tasks, like remote patch management, inventory location tracking, remote software installation. And sometimes it means screensharing in order to solve problems. I personally have sat at home as a network tech worked on my corporate laptop over a VPN. No problem. My personal opinion is this: when you see a piece of software with dedicated functionality for taking webcam screenshots surreptitiously and removing the evidence on disk, to me that's crossed the line into spyware. I'm certain that others in the industry will disagree with me. That's fine; let's have the debate. I don't mind losing a technical argument, as long as it's on merit.

update 2/23/2010 11:28am

My colleague Aaron pointed out to me today that the reason LANRev is using the raw camera device is that Apple implemented security measures to prevent remote activation of the webcam in OSX. LANRev was designed to bypass this security measure. Those who disagree with my spyware assessment, ask yourself, "what kind of software bypasses OS security measures?"
On the topic of whether or not we yet have proof of illegal use, I would ask you to listen carefully to the webcast, and listen for the word "house" at 1:28. Listen for "yes we have used it."update 2/23/2010 10:00am

I've removed Mr. Perbix's picture from my blog. I try very hard to stick to verifiable facts when I write here; this blog post is made up references to primary documents that show a verifiable pattern of action. But I feel that some readers are getting carried away. Myself and Aaron Rhodes spent hours reading forum posts, messages, and communications from Mike Perbix, his "digital shadow". The impression we both got was of a man who was charged with enormous responsibility, worked very hard, was very adept, and was fanatical about protecting kids and the assets he was charged with managing. I don't have all the facts yet, but the impression I got was of someone who was trying to build a state of the art capability and revelled in the promise of technology. If I had to put my finger on what when wrong here, I would say that someone cared too much. Personally I'm much more interested in who this capability was distributed to, and its persistent pattern of access, than I am in the person who built it. If you're reading this, please, let us not participate in a rush to judgement especially against a guy who worked this hard. Yes, he built the capability. Yes it was used. But if it was abused or simply misguided, that remains to be proven. I for one reserve judgement. For now, what bothers me most is this: When an organ of the State (in this case, a school) builds a system to conduct a search by activating webcams off of school grounds, the only way to determine if the ensuing search will be unreasonable or illegal is to conduct the search. The thought process behind that is unfathomable to me, no matter how much I read about it.
update 2/22/2010 8:30pm

I've created a network footprinting capability for parents, students, anyone who may be concerned that they are infected with the LANRev agent. The capability is documented in my next blog entry. One piece of feedback I continue to get is speculation on what can be seen in a packet sniffer. The answer for now is: not much. A block cipher and compression are in use in serial. It's a tough problem; we're working on it.
update 2/22/2010 5:30pm

In a strange twist, the makers of LANRev have come out with a statement saying that school network techs should never have used their software to engage in theft recovery:"We discourage any customer from taking theft recovery into their own hands," said Stephen Midgley, the company's head of marketing, in an interview Monday. "That's best left in the hands of professionals."

I've watched the 50 minute screencast repeatedly, where Perbix describes his use of this feature outside of school grounds repeatedly during a conversation with Absolute Software employees. They were enthusiastic... now they're throwing LMSD under the bus? I believe this can best be described as intense PR spin. It also completely confirms what I've asserted here, that LANRev was the implant of choice for this school.

[metmom]

I wonder where the students are who are refusing to comply with this?

If enough of them rebel, the school will have to back down.

[tubebender]

One Ringey Pingey

[metmom]

Great graphic. I may grab that one and post it on any more threads on this topic.

[metmom]

The eye of my Apple?

[meadsjn]

a temp fix while the legalities are being sorted out.

For 2300+ PA state felonies, and 2300+ federal felonies, for all the members of the school administration who ordered, monitored, and used the spy capabilities, and for the technical people who installed the hardware/software, ...

... the temp fixes for such as this is usually called incarceration pending trial and sentencing.

[Antoninus]

There’s an easy way to disable a webcam. It’s called a sticky note.

[trisham]

They know what's good for us. If you've ever heard teachers excuse poor results, it's usually because of:

1. Not enough funding for education.

2. Uncooperative, uninvolved, uncaring parents.

The public school system is out of control.

[brityank]

It seems to be this system can be defeated quite easily—a piece of electrical tape over the webcam.

That only shuts off the use of the camera. It does nothing to prevent the package shipping back any and all pic/vid data stored on the laptop.

Good explanation of the monitoring software, and the work-arounds it had to do to bypass Apple's OS restrictions and lack of security in its operation.

As I've noted previously - It is dangerous to be right, when your government is wrong!

Ping

[little jeremiah]

Isn’t it crazy? When will the hot frogs realize that the plan is to boil them (us) alive?

[JenB]

There have been a lot of people on these threads trying to say “it probably didn’t happen the way the lawsuit says” or “these kids have no rights anyway”. Based on this article and my own knowledge of the field (M.S. in Computer Science) I will say right now that the charges in the lawsuit are correct, the school district has been spying on students in and outside of school. They may or may not have done this only for theft prevention purposes but this is NOT power that a school administrator should have.

Especially notice the bit in the story about the admin bragging about sending cops to hunt down a stolen laptop, and how they went to the wrong house because the real perps were stealing wireless. First off, how would you like to be the innocent people? Second, if the school sent the cops to the wrong address it means they were using more than just the camera to track the laptop. Just the camera would have sent them to the right place. They were actively figuring out what network the computer was on and WHO OWNED THAT NETWORK. Anyone see issues with that? It’s a good thing in that case that they did use the camera too or I bet the cops would have arrested the innocent homeowners. After all, the school apparently had enough evidence to get cops out there (and we’ll assume a warrant but who knows...)

This is VERY VERY disturbing, people!

[JenB]

Yeah? How about disabling the rest of it - like the software that let them figure out the address of the home with the wireless network that a missing computer was connected to? Might need more than a sticky note there.

[metmom]

The kids weren’t allowed to do it.

[Tribune7]

ping

[TruthConquers]

The students themselves are claiming the following:

•Possession of a monitored Macbook was required for classes

•Possession of an unmonitored personal computer was forbidden and would be confiscated

•Disabling the camera was impossible

•Jailbreaking a school laptop in order to secure it or monitor it against intrusion was an offense which merited expulsion

The government schools really do believe that parents and kids are nothing but obstacles in their way, on the road to total power.

They lie. They steal. They demand taxes be paid to them.

Crush the school system, and homeschool.

/rant

[Tribune7]

but this little POS “School” distinct could do this...

It's can vs may and I think the POS administration is going to be taught the difference :-)

[metmom]

What should be the real wake up call is the arrogance of the school district in dictating this whole scenario, down to what the kids can and cannot do off school premises, and the fact that the parents and students just accepted it.

[pray4liberty]

I’ve seen some interesting accusations about the family who brought this suit against the district.

When the law is against you, argue the facts. When the facts are against you, argue the law. When both are against you, attack the plaintiff. - R.Rinkle

I was wondering how long it was going to take before they slandered the plaintiff and set up the IT guy. Less than a week...that's got to be a record. /sarc

[pray4liberty]

What should be the real wake up call is the arrogance of the school district in dictating this whole scenario, down to what the kids can and cannot do off school premises, and the fact that the parents and students just accepted it.

Howard vs. Colonial School District is the Delaware court ruling that started it all.

[TruthConquers]

That is the amazing part, and the part that scares me the most.

The peaceful acceptance of totalitarian power into peoples children’s lives, and not a peep.

[Born Conservative]

It seems to be this system can be defeated quite easily—a piece of electrical tape over the webcam.

But first you have to suspect that the webcam is being used inappropriately/illegally.