Pet vs Pet: .NET 'trounces' Java

The Register ^ | 10/29/2002 at 22:57 EST | Andrew Orlowski

[Bush2000]

Pet vs Pet: .NET 'trounces' Java

By Andrew Orlowski in San Francisco
Posted: 10/29/2002 at 22:57 EST

Java expert The Middleware Company has optimized Sun's showcase J2EEPet Store application, and reckon it still runs like a dog. Or in some cases, like a dog with a kennel tied to its hind legs: by refusing to function at all. Sun slammed the tests, which conceal important information, although the testers acknowledge that app servers from different vendors are being used, without naming them - but conceded that on low end hardware Wintel will perform faster. Both platforms were tested on Xeon-based Compaqs.

The original Pet Store wasn't designed to be a benchmark, but a demonstration of a range of programming techniques. Version 2.0 of each application added XML-based web services, and distributed database access with rollback, each of which the Middleware Company measured in a separate suite of tests. They also threw in a price/performance metric.

.NET significantly outperformed the J2EE version on 2,4 and 8-way machines on all three suites: the web benchmarks, TP and web services. In one case, the J2EE Pet Store couldn't handle the transactions at all.

"There are different app servers in each case," Sun's David Harrah told us. "Why that's even been published, I don't even know."

"It's no surprise to us or our engineers that Windows on Intel is faster: it's their home ground. The first Pet Store comparison, that was widely repudiated, showed a 10x advantage. This one shows a 2x and they've got home field advantage."

"That's not our value proposition - Java runs across a spectrum of devices from cellphones to mainframes."

But wasn't the 4 and 8 way the sweet spot of the market?

"I'd say the larger part of the market is developing client side software for devices from cell phones to smartcards," was the reply.

The price performance calculation was based on the system cost of a J2EE app server price totaling $84,990 (to .NET's $36,990). They could just as easily have used Sun's app server which is now bundled for free with Solaris, he pointed out.

The testers explain that the web services part of the application couldn't use the 1.4 version of the Java run time, which they note was 50 per cent faster than the 1.3 they used, thanks to better garbage collection. That alone explains why so many connections were refused in the web services test, they say.

But the benchmark throws up the remarkable statistic that the Java version required 14,004 lines of code, while the .NET version featured just 2,096 (and not the other way round, as we originally stated). The benefit of hindsight, or is one of our class libraries missing?

[Bush2000]

Bwahahahahahahahahaha!!! Poor Dominic. Tied to a kennel.

[TomServo]

Nice find!! ;-)

[Bush2000]

"I'd say the larger part of the market is developing client side software for devices from cell phones to smartcards," was the reply.

This guy is full of crap. Most work is happening on servers.

[TomServo]

Most work is happening on servers.

Yup!

[general_re]

In the interests of equal time - and I ain't got a dog in this fight anyway, so let's mix it up ;) - some folks have taken issue with how the benches were conducted...

[BuddhaBoy]

Come on, dude. I love .NET, but you gotta admit, the Java coding is brain dead verbose.

This wasnt even fair. J2EE doesnt suck THAT bad. Microsoft isnt helped by this comparison.

[Flashman_at_the_charge]

The Sun Pet Store example is not meant to be used as a performance benchmark. It was built as a tutorial. This is an 'apples to oranges' argument.

[BuddhaBoy]

The Sun Pet Store example is not meant to be used as a performance benchmark. It was built as a tutorial. This is an 'apples to oranges' argument.

Actually, its worse than that.

Sun has touted the Pet Store as an BENCHMARK EXAMPLE of how to build Enterprise E-Commerce Applications on J2EE. This thing is supposed to be a standard bearer for all to follow.

The architecture is stupid, verbose, clumsy and without any form of optimization to speak off. If had coded something like that in school, I would have been expelled.

Sun calls that abortion THE STANDARD.

They should hide their heads in shame. (or maybe hire some decent coders)

[toupsie]

Portable ANSI C is faster than both. :)

[Bush2000]

SURRENDER

DOROTHY

[TechJunkYard]

The report states that a Microsoft employee was allowed to tune the .NET app. Were vendors of appserver A or appserver B involved, to tune their own appservers?

No they were not. We are currently working to conduct a different experiment where all the vendors do participate.

[Flashman_at_the_charge]

I agree it's poorly written. The whole project seems to have been given to, let's just say, "less than cutting edge" developers to support. Nevertheless I don't see where performance benchmarking is explicitly mentioned as being one of the purposes of the Pet Shop example. Like they say "If you torture the figures long enough they'll admit to anything".

Regards,
Flashy.

PS: STOP SHOUTING.

[lelio]

That web page is one excuse after another. It boils down to that in the Java world you have a lot of choices and they all suck in their own way. Is that a benefit? Perhaps in that you get to choose the sword you fall upon.
Another point that was brought up is that projects like this take many many man months to plan out. Sun and MSFT are guilty in trying to make you think that you can roll these things out in days.

[Bush2000]

Perhaps you forgot to read this part:

"at various points, we also hired independent consultants experienced in appservers A and B to tune them or provide recommendations, at our own expense."

Nice try.

[FormerLurker]

Sun calls that abortion THE STANDARD.
They should hide their heads in shame. (or maybe hire some decent coders)

That's what they get for laying off most of their American engineers and replacing them with those who either can't or won't speak English...

[BuddhaBoy]

Hey dude, I drank the .NET coolaid, so I am a believer.

I just think the J2EE app could have been better written.

I suppose you dudes are partying in Redmond over the settlement?

[Dominic Harr]

Actually, this is becoming a great joke in the java world.

MS funded the test, set the conditions for the test and controlled the testing environment.

And the MS-run test showed MS was lightning fast, while Java Beans were dog-slow.

Of course, they had to specifically screw the app up to get such a result, (Entity Beans!?) but they didn't mind, I am sure.

They specifically selected 2-3 year old Java technologies and practices known for slow performance (entity beans!?!). They specifically selected app servers that didn't support the current 1.4 jvm, and used the 3 year old 1.3 vm, while using the newest .NET beta.

These selected "experts" were even unable to get a simple pet-store running on one of the selected app servers!!!

They didn't write the app for performance, in fact de-tuned the Sun one (which was *not* a real-world app but a PATTERNS TEACHING TOOL), but allowed MS to specifically write theirs for performance and even make changes to the .NET runtime specifically for this app. They didn't have the vendors tune the appservers or database, but allowed MS to tune theirs.

They didn't use the app servers known to be fastest, and didn't use a single Open-Source server solution (gotta jack up that cost somehow, or else MS would look bad).

I got burned believing this report. I'm a big critic of beans, and I was trying to use this report to slam the EJB devotees, and then I got hit with the fact that this was an MS-paid-for-and-fixed. (Entity Beans!? Why not just say, "fixed test" in the title?). Even when I try to be on their side, I find out they lied and cheated.

Apparently, the company that did the test, The MiddleWare Company, hit some financial trouble recently and was bought by an MS "strategic partner" for $10,000.

And now they falsify a benchmark.

One review by a *real* Java expert.

Ibatis Java Petstore that *is* fast, which they chose not to test for obvious reasons.

Unfortunately, 99% of the folks here lack even the most basic technical knowledge required to discuss this, so if ya'll don't mind, I'll keep my comments restricted to The Server Side's chat board on the topic.

If anyone wishes to discuss this, join me there.

[JonH]

Just one more example of the reasons Sun Micro has almost arrived as a penny stock.

[BuddhaBoy]

From the Server Side:

"In May 2001 Sun presented the Pet Shop application the first time as an example implementation. The source code of Pet Store should act as an architecture template for developers own systems; many vendors of J2EE servers package their products with Pet Store as a sample application.

Now, maybe you should not assume that this is the only place some of us frequent. Sun put that ugly-baby of an application out there as a benchmark, and Sun got burned.