Largo [FL.] Loves Linux More Than Ever

We're back in Largo, Florida, checking on advances in the Linux-based network they use to run the city's computers that we wrote about last year. True to Largo's "City of Progress" motto, these guys have not been standing still. Now they're talking about Linux-based terminals in all the city's police cars. Microsoft has tried -- and failed -- to bring them into the proprietary fold. And, possibly most important, we have an amazing cost figure that ought to make you ask your local politicians why their IT operations aren't as efficient as Largo's.

Last time we visited, we spoke mostly with hands-on sysadmins Dave Richards and Mike Pearlman. This time we're also chatting with their boss, IT Manager/CIO Harold A. Schomaker, who is more than a little proud of their latest ultra-cool deal: finding a whole bunch of the NCD thin clients they prefer -- which sell new for around $750 -- on eBay for prices ranging from 50 cents to $5. No, they aren't the latest model, but who cares? These things have no moving parts; the super-cheap used ones are more than adequate to run a KDE desktop and all the apps a typical city employee needs; and with a 10 year expected life it doesn't matter if they're a few years old.

I watch Harold close a winning bid session -- for $5 per unit. Does he gloat a little? Sure. I promise not to reveal the URL of the auctions he's in, but even if I do he's not terribly worried; he says eBay is a great sales medium for things like cameras and laptops that everyone uses, but isn't so great (for sellers) when it comes to highly specialized gear like thin clients, especially in lots of 10 to 50 units each.

This is the kind of thinking that pervades Largo's IT department: provide great functionality, but get the best bang for the taxpayers' buck in every possible way -- including shopping eBay regularly for cheap, used equipment.

Sysadmins who can think instead of react

Instead of going into great detail here about the equipment at the heart of the Largo system, let's take a fast look at this detailed ZDNet article, which does an excellent job of describing its basic hardware and software ingredients.

Now let's talk about the most important feature of this entire system from a sysadmin's or IT manager's point of view: A silent beeper.

An absence of emergencies

We noted on our last Largo visit, and note once again, that these are the least harassed, least worried, calmest sysadmins we have ever met. They have one of the smallest and least-worked help desks we have ever seen -- five people who support 450+ client units and over 800 users, and it is all done without any fuss, muss or hurry. The desktop units, remember, have no moving parts or applicaitons software on them. They rarely break, and if they do it is only a moment's work to swap one out. Monitors eventually get old and dim, but they have a stack of $150 Compaq 17" monitors ready to go, plenty of spare keyboards and mice, and lots of CAT-5 cable. Everything in the server room is backed up and redundant (and neat, with all cables marked) so maintenance there is as under control and worry-free as it is on the client side of things.

Harold, Mike, and Dave all note that when they go to technical conferences and other sysadmin get-togethers, they are usually the only ones in the place who are not getting a steady stream of frantic interruptions.

So what do they do with their time? They research, plan, and think of yet more ways to save Largo taxpayers money on IT while making the city's IT services more efficient and useful.

Linux terminals for every police car

Ruggedized laptops for police cars usually cost at least $4000 each, plus software, and Dave Richards says they really don't last any longer than un-ruggedized ones. So he's been messing with tablet PCs, the kind Microsoft has been pushing like mad this year, except Dave and his guys are running them as Linux thin clients instead of trying to make them be full-tilt computers. This is only now becoming practical because of advances in wireless data communications, but applying the thin client model to mobile computing -- and using tablets instead of laptops -- makes all kinds of sense once Dave explains the reasoning behind the system, which is currently at the prototype and vendor-selection stage, and will probably be fully deployed, after extensive hands-on testing., in 2004 or 2005.

Let's start with the advantage of the thin client setup in mobile computing, using Officer Bustem as an example. He is sitting in his car, writing up a report of a burglary arrest he made earlier, when his shift ends. He needs to turn in the car so Officer Cuffem can use it on his shift.

With a traditional laptop installed in the car, Bustem would need to either transfer his report to a floppy and take it with him or send it over the network to another computer somewhere. But with the thin client system he doesn't need to do anything. The terminal in the car is just one of many interchangable terminals, and his work is stored on the server at police headquarters. He can log on to a terminal in the police station and keep working on his report, no problem. And if tomorrow, when he comes to work, he is assigned a different car, still no problem. His work is on the server, so all his case notes are still accesible to him -- and to his supervisor, Lieutenant Lockemup, who can access them from his car or office and pass word about the burglary suspect's confederate, who is still at large, on to Cuffem and other members of the next shift.

So far, all of this could be done with a low-end laptop, but Dave says the basic durability defect of the laptop form factor is that "it puts the keyboard above the electronics."

So the tablet PC gets mounted permanently in the car, with a keyboard on a cord. Yes, the keyboard is eventually going to wear out or break, but replacing it is a simple plug/unplug operation, much easier than unmounting a laptop or other piece of gear. The keyboards they're looking at are not cheap -- they run around $275 each -- but are super-rugged and have backlit keys for typing in the car at night, which cops often do. Not only that, having the keyboard as a separate unit gives the poor cop a fighting chance of finding a half-comfortable typing position, which is almost impossible to do in the driver's seat of a car no matter what, but is made slightly easier by disconnecting the keyboard from the rest of the unit.

Plus there's handwriting recognition. Dave says it works "fairly well" now on the tablet PCs running Linux, but that the keyboard is likely to be the primary data entry device for a good number of years yet.

So there you are: a computer system for cops in their cars that is better, more flexible, more durable, and a lot less expensive than traditional ones -- all based on a bit of imagination from a couple of sysadmins who are not overwhelmed with reboots and software problems, so they have time to research what the police really need from their in-car data terminals, and figure out how best to give it to them while spending the least possible amount of the taxpayers' money.

These are not brilliant people

We are not knocking Harold, Mike and Dave by saying they are not brilliant. They are smart and they care about their jobs, but none of what they've done is truly original or even new. They know this, and they keep saying it. The real strangeness is not that these guys have managed to build such a wonderful, cost-effective system, but that so few others have done the same thing.

Everything in the Largo IT ecosystem is off-the-shelf standard goods, from hardware to software to the wires that hook everything together. The innovation here comes in making maximum use of everything, and not necessarily in obvious ways -- and in coming up with solutions that are as much social engineering as technical, like the "cybercafe" in the Largo City Hall's employee break room.

A common complaint among employers -- especially government employers -- is that workers spend too much paid time surfing the Net for personal purposes or exchanging personal email. And yet, many people may only have Internet access -- or at least broadband access -- at work. The Largo solution is to put four terminals in the break room, and load them only with Mozilla and Evolution, and encourage workers to surf, chat, and play online all they want during their lunch hours and other breaks.

Many IT shops might hesitate to put in something like this; those that run PCs would need to supply and maintain four complete computers, including (no doubt) Windows, so they'd need to have virus software kept up to date and take care of all the other chores that go along with running a standalone computer. But none of this applies in the Largo IT scheme. The four thin-client units in the break room were purchased for $2 each on eBay and take no maintenance, and besides the client pieces all you have is keyboards, mice, and monitors, and these are not costly items. The biggest thing that makes this sort of niceness possible, though, big enough that it's worth saying over again, is no maintenance!

When you run a servercentric network, you only need to maintain and update one copy of each application you run. As long as your servers are sized correctly for the load you anticipate, plus a substantial margin for error, adding another few terminals is literally no work at all beyond phyiscally plugging things in.

Microsoft tried to bring them into the fold

Dave says, "About a year ago we had two gentlemen from Microsoft come in who spent two or three hours with us." The Microsoft reps asked the Largo people to be frank with them and explain their needs as clearly as possible, which is what happened.

"Mostly it was an issue of scalability," Dave says. This, not money, is what they told the Microsoft people their biggest barrier was. At any given moment, Largo's network may have over 200 people actively logged in and working, often more, and they are all running from a single main server, plus several servers that run specific applications. Even the Microsoft people couldn't refute the fact that Largo's current setup uses far less hardware and is far easier to administer and physically maintain than an equivalent Windows-based system.

This is not to say there is no Microsoft anywhere in Largo. There are between 90 and 100 standalone computers owned by the city, mostly used as Internet terminals in libraries, that run Windows. Harold says, "We're not anti-Microsoft. We have Microsoft here in places where it does a good job."

This is a pragmatic IT shop, not one run by zealots. They don't make decisions quickly, and when they roll out something new it gets a careful beta test before it goes citywide. When these people roll out OpenOffice, with a few StarOffice licenses for workers who need the commercial version's better import filters, it is not a snap judgement, but one that has taken well over a year to make. Ditto their choice to go with SuSE's email server and pay $1100 for a license instead of choosing a free solution (although they have not chosen a groupware package yet; this is a frustrating search, worth a separate article).

Rocking with the BSA

Since Largo still has proprietary software around, the city is still subject to BSA audits. Harold says they have been extra-careful to follow all license terms, to the point of purchasing "probably more licenses than we need."

Other cities in the Tampa Bay area have been fined over missing proprietary software licenses. Largo has not been fined, and Harold wants to keep it that way. "I could lose my job over something like that," he says.

Harold, Mike, and Dave all agree that Largo has not been singled out by either Microsoft or the BSA for any special license enforcement action. "I send in my figures and they seem to accept them without asking any questions," Harold says.

1.3% versus 3%

This is the most important pair of numbers -- and indeed the most important piece of information -- in this entire article: 1.3% and 3%.

Harold A. Schomaker, IT Manager/CIO for the City of Largo, says Largo spends a total of 1.3% of its gross budget on IT. This includes hardware, software, salaries, and incidental expenses. He says the typical small city spends over 3% of its budget on IT, with some approaching 4%. "Between 3% and 4% is about right," he says, "with most closer to 3%."

He is adamant that these are true, quotable figures. We ask several times, since this sort of disparity is far from what we are seeing in recent (Microsoft-sponsored) Linux vs Windows TCO studies.

Don't forget, Harold isn't getting paid by anyone except Largo taxpayers, and his job is to keep their IT expenses as low as he can while providing ever-better IT services to the city employees who use them to do their jobs. In light of this, Harold's comparative cost figures are probably at least as trustworthy as anyone's -- and lots more trustworthy than some.

The "viral" problem

This is a bit of a rant from Dave, with "amens" from his coworkers: He is upset with other local governments that use Visual Basic or ActiveX to make Intranet and Internet applications with which Largo people must interact, in effect making their supposedly "browser-accesible" shared applications and data libraries accessible only for Microsoft Internet Explorer users. And, says Dave, few of the IT people in other governments seem to understand Largo's resistance to Explorer.

"Why don't you just download Explorer? It's free!" is what Dave says many of his counterparts in other governments often tell him. He says most of them don't understand that even though Explorer is "free" it needs Windows licenses to run, and that buying those can add up in a hurry -- and (although he's certainly too polite to say it) -- not having all those Windows licenses is one reason Largo spends less than half as much on IT as many other local governments.

More formally, it is called the network effect, and Microsoft is one of the world's most effective wielders of it as a marketing weapon.

It takes great strength of will and a huge sense of responsibility to one's employers to stand up to the Microsoft network effect, and these characteristics -- more than raw intelligence -- may be what set the Largo IT people apart from their counterparts elsewhere, and make it possibe for them to proudly parade that 1.3% figure not only in front of the City Manager and City Commission to whom they report, but in front of the entire world as an example of how a little skill, a little gumption, a little intelligence, a little imagination, and a lot of resistance to "following the herd" can lead to amazing IT cost savings -- and, at the same time, to better and more reliable IT services.

"Microsoft laughing all the way to the bank. Their market share of server software is actually increasing:

You forgot one little thing, while Microsofts share may be increasing (taking business away from Netware and the like), Linux's share of the market is far outpacing Microsofts gains percentage wise.

Quoted from your link::
In addition to Microsoft's commanding share of the 5.7 million new server-operating-system licenses shipped in 2001, Linux achieved an impressive 25.7 percent of shipments, NetWare accounted for 11.7 percent, and the Unix platforms combined represented 11.6 percent of shipments. Other server operating systems fought over a scant 2 percent of the remaining shipments. OS/400, by our estimate, accounted for about half of that 2 percent share, by the way.

Just give it a few more years and a few more peeved Microsoft customers that get tired of the downtime and freezes, and Linux will be cutting into Microsofts lead like a hot knife through butter. What I really find funny, is that with all of MS's money and Ad saturation, lil bitty ole Linux that is hardy ever advertized through media outlets, gets a better percent gain then big ole MS. Guess what they say is true, "Word of Mouth" far outweighs glitzy ads.

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Largo love Linux more than ever