Free Republic
Browse · Search
General/Chat
Topics · Post Article

Skip to comments.

Roadrunner: 130,536 cores break the Petaflop barrier ( the world’s first hybrid supercomputer)
TgDaily ^ | Monday, June 09, 2008 12:13 | Wolfgang Gruener

Posted on 06/09/2008 11:50:57 AM PDT by Ernest_at_the_Beach

Armonk (NY) – IBM is prepared to deliver the Roadrunner supercomputer to the Department of Energy’s National Nuclear Security Administration (NNSA). The system was development over the past 18 months and is not only the first hybrid supercomputer using Cell processors, but also the first commercial system to exceed a performance of 1 PFlops.

Roadrunner is the NNSA’s third IBM-built supercomputer, adding to two IBM-built supercomputers at the Lawrence Livermore National Laboratory: BlueGene/L, completed in 2005, has 131,072 p5 processors and delivers a sustained performance of 280.6 TFlops. The 12,208 processor ASC Purple is estimated to provide a sustained performance of 75.8 teraflops.

The new Roadrunner, which is currently disassembled and will be shipped to the NNSA within 30 days, according to IBM, was originally described to deliver about 2.8x (3.5x peak: 1.6 TFlops ) the combined processing power of BlueGene/L and ASC Purple. Back in 2006, IBM said that the system would bet more than 16,000 Opteron processor cores and more than 16,000 Cell BE CPUs to deliver on the Petaflop goal.

The final system ended up with less processors – 13,896 Opteron cores (6948 dual-core processors) as well as 12,960 Cell BE chips with a total of 116,640 PPE and SPE cores for a combined 130,536 cores. The original design called for about 176,000 cores.   

According to IBM, Roadrunner has the floating point performance of about 100,000 notebooks. To describe the horsepower of the system, IBM said that Roadrunner can do in one day what would take the entire population of the world 46 years on handheld calculators, assuming a pace of one second per calculation.

Other interesting facts of the supercomputer, which IBM said cost about $100 million, include a power consumption of about 3.9 Mwatts, as well as a structure that includes 80 TB of memory, 576 miles of fiber optic cable as well as 3456 tri-blades which can deliver 400 GFlops each.

Also noteworthy is that Roadrunner was supposed to become the flagship product of AMD’s Torrenza platform, a project which was pitched as an open AMD x86 platform that takes advantage of Direct Connect Architecture and the Hypertransport interface. However, we haven’t heard much about Torrenza anymore and we don’t expect AMD to make a big deal of it.


TOPICS: Business/Economy; Computers/Internet; Science
KEYWORDS: hiperfcomputing; hitech; supercomputing
Navigation: use the links below to view more comments.
first 1-5051-56 next last

1 posted on 06/09/2008 11:50:58 AM PDT by Ernest_at_the_Beach
[ Post Reply | Private Reply | View Replies]

To: ShadowAce

AMD Opterons amd Cell chips.


2 posted on 06/09/2008 11:51:54 AM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: rdb3; Calvinist_Dark_Lord; GodGunsandGuts; CyberCowboy777; Salo; Bobsat; JosephW; ...

3 posted on 06/09/2008 11:55:22 AM PDT by ShadowAce (Linux -- The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 1 | View Replies]

To: All
Fact Sheet & Background: Roadrunner Smashes the Petaflop Barrier

***********************EXCERPS*********************

Armonk, NY - 09 Jun 2008: In 2006, the Department of Energy’s National Nuclear Security Administration selected Los Alamos National Laboratory as the development site for Roadrunner and IBM as the computer’s designer and builder. Roadrunner, named after the New Mexico state bird, cost about $100 million, and was a three-phase project to deliver the world’s first “hybrid” supercomputer – one powerful enough to operate at one petaflop (one thousand trillion calculations per second). That’s twice as fast as the current No.1 rated IBM Blue Gene system at Lawrence Livermore National Lab – itself nearly three times faster than the leading contenders on the current TOP 500 list of worldwide supercomputers.

  • Custom Configuration. Two IBM QS22 blade servers and one IBM LS21 blade server are combined into a specialized “tri-blade” configuration for Roadrunner. The machine is composed of a total of 3,456 tri-blades built in IBM’s Rochester, Minn. plant. Standard processing (e.g., file system I/O) is handled by the Opteron processors. Mathematically and CPU-intensive elements are directed to the Cell processors. Each tri-blade unit can run at 400 billion operations per second (400 Gigaflops).
  • The machine was built, tested and benchmarked in IBM’s Poughkeepsie, N.Y. plant, home of the ASCI series of supercomputers the company built for the US government in the late 1990s. IBM’s site in Rochester, Minn. constructed the specialized tri-blade servers. Software development was led by IBM engineers in Austin, Texas and by researchers in IBM’s Yorktown Heights, N.Y. research lab. Roadrunner will be loaded onto 21 tractor trailer trucks later this summer when it is delivered to Los Alamos National Lab in New Mexico.
  • Roadrunner operates on open-source Linux software from Red Hat.
  • Energy Miser. Compared to most traditional supercomputer designs, Roadrunner’s hybrid format sips power (3.9 megawatts) and delivers world-leading efficiency – 376 million calculations per watt. IBM expects Roadrunner to place among the top energy-efficient systems later in June when the official “Green 500” list of supercomputers is issued.
  • IBM is developing new software to make Cell-powered hybrid computing broadly accessible. Roadrunner’s massive software effort targets commercial applications for hybrid supercomputing. With corporate and academic partners, IBM is developing an open-source ecosystem that will bring hybrid supercomputing to financial services, energy exploration and medical imaging industries among others.

Applications for Cell-based hybrid supercomputing include: calculating cause and effect in capital markets in real-time, supercomputers in financial services can instantly predict the ripple effect of a stock market change throughout the markets. In medicine, complex 3-D renderings of tissues and bone structures will happen in real-time, as patients are being examined.

How fast is a petaflop?
Roadrunner operates at speeds exceeding one petaflop -- one thousand trillion calculations per second -- or one million billion calculations per second; or one quadrillion calculations per second.
  • Lots of laptops. That’s roughly equivalent to the combined computing power of 100,000 of today’s fastest laptop computers. You would need a stack of laptops 1.5 miles high to equal Roadrunner’s performance.
  • It would take the entire population of the earth, -- about six billion – each of us working a handheld calculator at the rate of one second per calculation, more than 46 years to do what Roadrunner can do in one day.
  • In the past 10 years, supercomputer power has increased about 1,000 times. Today, just three of Roadrunner’s 3,456 Tri-blade units have the same power as the 1998 fastest computer. A complex physics calculation that will take Roadrunner one week to complete, would have taken the 1998 machine 20 years to finish – it would be half done today! If it were possible for cars to improve their gas mileage over the past decade at the same rate that supercomputers have improved their cost and efficiency, we'd be getting 200,000 miles to the gallon today.

4 posted on 06/09/2008 11:55:34 AM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 2 | View Replies]

To: ShadowAce
* Roadrunner operates on open-source Linux software from Red Hat.
5 posted on 06/09/2008 11:57:37 AM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 3 | View Replies]

To: Ernest_at_the_Beach

When the Top 500 list comes out this month, there should be several PF machines on the list.


6 posted on 06/09/2008 11:57:58 AM PDT by ShadowAce (Linux -- The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Ernest_at_the_Beach

IB4GE


7 posted on 06/09/2008 11:59:41 AM PDT by twntaipan (NOBAMA!)
[ Post Reply | Private Reply | To 5 | View Replies]

To: Ernest_at_the_Beach
Is this the one Joshua used?

... and it's no longer a movie?

8 posted on 06/09/2008 12:03:25 PM PDT by knarf (I say things that are true ... I have no proof ... but they're true.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: All
Supercomputer breaks petaflop barrier
Military machine smashes speed record

**********************

A US military supercomputer has broken the petaflop speed record by performing one thousand trillion calculations per second.

The 'Roadrunner' machine was developed at the Los Alamos Laboratory and uses AMD Opteron chips and the Cell processors found in PlayStation games consoles.

It will be used to model climate systems before being classified by the Army to work on nuclear explosion modelling.

"We replace our high-performance supercomputers every four or five years," said Andy White, leader of supercomputer development at Los Alamos.

"They become outdated in terms of speed, and the maintenance costs and failure rates get too high."

White explained that the use of two kinds of chips represents a revolution in supercomputing.

'Roadrunner' uses fewer than 20,000 processors while the next fastest computer, IBM's Blue Gene/L, uses over 200,000 and runs at half the speed.

The Los Alamos machine uses 7,000 multi-core Opteron processors and parses the data to 12,960 enhanced Cell processors using 57 miles of fibre-optic cabling. The device consumes three megawatts of power, enough to run 1,000 homes.

9 posted on 06/09/2008 12:03:48 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 5 | View Replies]

To: ShadowAce
Guess we are still using Linpack to sort out the speeds...?

How does a Hybrid get measured with a standard benchmark...?

10 posted on 06/09/2008 12:05:46 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 6 | View Replies]

To: Ernest_at_the_Beach

And the sad thing is Vista STILL takes forever to boot up on it...


11 posted on 06/09/2008 12:07:56 PM PDT by apillar
[ Post Reply | Private Reply | To 1 | View Replies]

To: All
Frequently Asked Questions on the Linpack Benchmark and Top500
12 posted on 06/09/2008 12:09:40 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 10 | View Replies]

To: Ernest_at_the_Beach

So 7,000 Opterons...that’s, what, about half of the Opterons AMD sold last year?

I’m not happy with AMD. I went Intel (Core 2 Duo E8400) on my new box because Intel has just blown AMD out of the water on mid- to high-end consumer CPUs. I was an AMD guy since the days of the K6/225...guess I’m part of the Evil Empire now.

Besides, from what I’ve heard, it’d take a petaflop machine to get more than 20 frames per second in Microsoft Flight Simulator X...

}:-)4


13 posted on 06/09/2008 12:11:38 PM PDT by Moose4 (http://moosedroppings.wordpress.com -- Because 20 million self-important blogs just aren't enough.)
[ Post Reply | Private Reply | To 9 | View Replies]

To: Ernest_at_the_Beach

It would be interesting to know how long it would take it to render a movie like ‘Toy Story’.


14 posted on 06/09/2008 12:13:37 PM PDT by Walmartian
[ Post Reply | Private Reply | To 1 | View Replies]

To: Ernest_at_the_Beach
Guess we are still using Linpack to sort out the speeds...?

Yup. That's the standard that they accept.

On a hybrid, I think what I'd do is run the HPL on the Opterons, then run one on the Cells. Not sure how I'd combine the results, though. They don't really go together linearly.

15 posted on 06/09/2008 12:15:17 PM PDT by ShadowAce (Linux -- The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 10 | View Replies]

To: Ernest_at_the_Beach

cost about $100 million, include a power consumption of about 3.9 Mwatts, as well as a structure that includes 80 TB of memory, 576 miles of fiber optic cable as well as 3456 tri-blades which can deliver 400 GFlops each

..

wwowww


16 posted on 06/09/2008 12:23:25 PM PDT by NormsRevenge (Semper Fi ... Godspeed ... ICE toll-free tip hotline 1-866-DHS-2-ICE ... 9/11 .. Never FoRget!!!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Moose4

The K6 sucked. AMD rocked the GHz world (I own all AMDs currently) until Intel figured out what hit them and released the awesomely cool Core2. I plan on having a Core2 Quad on my next machine


17 posted on 06/09/2008 12:23:59 PM PDT by Crazieman (Vote Juan McAmnesty in 2008! Because freedom abroad is more important than freedom at home!)
[ Post Reply | Private Reply | To 13 | View Replies]

To: Ernest_at_the_Beach
Other interesting facts of the supercomputer, which IBM said cost about $100 million, include a power consumption of about 3.9 Mwatts ...

I see I'm going to have to upgrade my AC.

18 posted on 06/09/2008 12:27:15 PM PDT by AZLiberty (Wipe the national hard drive and reinstall the Constitution.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: ShadowAce
Standard processing (e.g., file system I/O) is handled by the Opteron processors.

Mathematically and CPU-intensive elements are directed to the Cell processors. Each tri-blade unit can run at 400 billion operations per second (400 Gigaflops).

19 posted on 06/09/2008 12:31:31 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 15 | View Replies]

To: Ernest_at_the_Beach
Hybrid? Analog and digital? Really?

My, how times have changed.

20 posted on 06/09/2008 12:35:34 PM PDT by LTCJ (God Save the Constitution - Tar/Feathers '08)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Ernest_at_the_Beach

They’re probably getting the PF number through their extrapolation program. They run one unit through the linpack, the use a program to extrapolate the results so they don’t have to bother running the long HPLs.


21 posted on 06/09/2008 12:35:46 PM PDT by ShadowAce (Linux -- The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 19 | View Replies]

To: ShadowAce
It's going to take some hefty programming to make this a usable machine for any thing beyond the specialized stuff ...
22 posted on 06/09/2008 12:39:38 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 21 | View Replies]

To: Ernest_at_the_Beach

Consider that it’s going to the NNSA, I doubt if that will be a problem.


23 posted on 06/09/2008 12:41:49 PM PDT by ShadowAce (Linux -- The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 22 | View Replies]

To: All
This will take some manpower to move to the Cell,...I would think:

SLATEC A Mathematical Library

24 posted on 06/09/2008 12:43:47 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 22 | View Replies]

To: NormsRevenge

It’s less than that crashed B2....


25 posted on 06/09/2008 12:45:03 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 16 | View Replies]

To: Ernest_at_the_Beach
eh---it's just FORTRAN. If the Cell processor has a FORTRAN compiler, it should just compile.

I don't don't about the availability of the compiler, though.

26 posted on 06/09/2008 12:46:52 PM PDT by ShadowAce (Linux -- The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 24 | View Replies]

To: Moose4

If you look at the architecture of the new Intel processors they are more akin to the Socket A AMD and the Mobile P3 processors...

C2D had to step backwards to where AMD and Intel had been there and done that...

Proving that Instructions per clock trumps final speed.

Intel rode that pony until it died...


27 posted on 06/09/2008 12:49:22 PM PDT by MD_Willington_1976
[ Post Reply | Private Reply | To 13 | View Replies]

To: ShadowAce
had to look up NNSA:

National Nuclear Security Administration
From Wikipedia, the free encyclopedia

********************EXCERPT******************

The United States National Nuclear Security Administration (NNSA) is part of the United States Department of Energy. It works to improve national security through the military application of nuclear energy. The NNSA also maintains and improves the safety, reliability, and performance of the United States nuclear weapons stockpile, including the ability to design, produce, and test, in order to meet national security requirements.

NNSA has four missions with regard to National Security:

The NNSA maintains a database containing personal information on 37,000 persons who design and maintain nuclear weapons for the U. S. government.

The NNSA's Office of Secure Transportation (OST) provides safe and secure transportation of nuclear weapons and components and special nuclear materials, and conducts other missions supporting the national security of the United States of America. Since 1974, OST has been assigned responsibility to develop, operate, and manage a system for the safe and secure transportation of all government-owned, DOE or NNSA controlled special nuclear materials in "strategic" or "significant" quantities. Shipments are transported in specially designed equipment and are escorted by armed federal agents.

28 posted on 06/09/2008 12:50:09 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 23 | View Replies]

To: Ernest_at_the_Beach

Damn it all man, what about the Flux Capacitor??!!


29 posted on 06/09/2008 12:53:33 PM PDT by starlifter
[ Post Reply | Private Reply | To 1 | View Replies]

To: MD_Willington_1976; Moose4
Intel's marketing Hype would have you believe that did miracles with the Core2....

AMD is still whopping them in the heavy duty heavy data moving department...know as the Server marketplace.

30 posted on 06/09/2008 12:53:55 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 27 | View Replies]

To: starlifter

Do we still need that?


31 posted on 06/09/2008 12:54:42 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 29 | View Replies]

To: All
I'll bet this machine would be a big help to the FReeper folders.
32 posted on 06/09/2008 12:57:34 PM PDT by Straight Vermonter (Posting from deep behind the Maple Curtain)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Ernest_at_the_Beach

Well duh!

How else to we keep the dust out of the core memory?


33 posted on 06/09/2008 12:58:15 PM PDT by starlifter
[ Post Reply | Private Reply | To 31 | View Replies]

To: starlifter
core memory?

ROFL!

I thing that has been gone for awhile now....

34 posted on 06/09/2008 1:00:05 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 33 | View Replies]

To: Straight Vermonter

The PS3s being used to help achieve petaflop processing with Folding@Home kind of proved a system like this would be useful.


35 posted on 06/09/2008 1:02:59 PM PDT by antiRepublicrat
[ Post Reply | Private Reply | To 32 | View Replies]

To: ShadowAce; Straight Vermonter
IBM XL Fortran for Multicore Acceleration for Linux on System p

A high-performance IBM XL Fortran compiler for the Cell Broadband Engine Processor.

So there is a compiler....for the Cell.

36 posted on 06/09/2008 1:04:55 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 26 | View Replies]

To: Ernest_at_the_Beach

I understood “of,” “and,” and “the.”


37 posted on 06/09/2008 1:11:53 PM PDT by pabianice
[ Post Reply | Private Reply | To 1 | View Replies]

To: All
The Cell architecture

**********************EXCERPT********************

Single precision floating point computation is geared for throughput of media and three-dimensional graphics objects. In this vein, the decision to support only a subset of IEEE floating point arithmetic and sacrifice full IEEE compliance was driven by the target applications. Thus, multiple rounding modes and IEEE-compliant exceptions are typically unimportant for these workloads, and are not supported. This design decision is based the real time nature of game workloads and other media applications: most often, saturation is mathematically the right solution. Also, occasional small display glitches caused by saturation in a display frame is tolerable. On the other hand, incomplete rendering of a display frame, missing objects or tearing video due to long exception handling is objectionable.

38 posted on 06/09/2008 1:13:18 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 36 | View Replies]

To: pabianice
Just remember that Four letter word:

FAST

39 posted on 06/09/2008 1:15:41 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 37 | View Replies]

To: All; ShadowAce; Straight Vermonter
most often, saturation is mathematically the right solution.

I don't know what that means...any idea?

40 posted on 06/09/2008 1:21:31 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 38 | View Replies]

To: All
Old Bit twiddlers might find this of interest...from Gamespot:

The Cell Project at IBM Research, details inside.

41 posted on 06/09/2008 1:51:25 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 40 | View Replies]

To: Ernest_at_the_Beach
I can't tell if they're talking about processor saturation or network saturation.

Either way, I have no idea. :)

42 posted on 06/09/2008 2:04:54 PM PDT by ShadowAce (Linux -- The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 40 | View Replies]

To: Ernest_at_the_Beach

and the super computer market, latest one from IBM has a whack of Opterons and Cells...

http://forums.amd.com/forum/index.cfm?forumid=1

Drop on by the forums...


43 posted on 06/09/2008 2:22:21 PM PDT by MD_Willington_1976
[ Post Reply | Private Reply | To 30 | View Replies]

To: ShadowAce
May be n Numerical Analysis term....found this:

A Theoretical Introduction to Numerical Analysis (Hardcover)

***********************EXCERPT*****************

presents the general methodology and principles of numerical analysis, illustrating the key concepts using numerical methods from real analysis, linear algebra, and differential equations.

***************************

Explains the most fundamental and universal concepts, including error, efficiency, complexity, stability, and convergence. Addresses advance topics, such as intrinsic accuracy limits, saturation of numerical methods by smoothness, and......

Never studied this area of Math,...stayed with Foundations...like Topology...

44 posted on 06/09/2008 2:23:44 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 42 | View Replies]

To: MD_Willington_1976

Thanks...I have some AMD’s/...


45 posted on 06/09/2008 2:25:47 PM PDT by Ernest_at_the_Beach (No Burkas for my Grandaughters!)
[ Post Reply | Private Reply | To 43 | View Replies]

To: Ernest_at_the_Beach

Wow....they probably need 350,000 watts just to power the CPUs.


46 posted on 06/09/2008 3:26:29 PM PDT by Psycho_Bunny (Islam: Imagine a clown car.........with guns.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Ernest_at_the_Beach
According to IBM, Roadrunner has the floating point performance of about 100,000 notebooks.

Now the question is: Will Steve Jobs ever build a machine as powerful as this and sell it for $2500?

Per Moore's Law, the answer is probably not but possibly.

Moore's Law tells us that processing power doubles every 18 months. That's a lot of "times twos" to get to 100,000. In fact it takes about 16 or 17 of them so Jobs would still have to be on the job 25 years from now in 2033. That's not likely but possible.

I can't wait. It will probably be running OS XX

47 posted on 06/09/2008 4:37:40 PM PDT by InterceptPoint
[ Post Reply | Private Reply | To 1 | View Replies]

To: Ernest_at_the_Beach

No idea.


48 posted on 06/09/2008 5:20:03 PM PDT by Straight Vermonter (Posting from deep behind the Maple Curtain)
[ Post Reply | Private Reply | To 40 | View Replies]

To: Ernest_at_the_Beach

I turned down a job offer about 6 years ago to work on, if not this very project, a sister project in the same group in Poughkeepsie. Oh, well.


49 posted on 06/09/2008 11:46:30 PM PDT by TiberiusClaudius
[ Post Reply | Private Reply | To 1 | View Replies]

To: Ernest_at_the_Beach; texas booster; neverdem; Swordmaker

> not only the first hybrid supercomputer using Cell processors, but also the first commercial system to exceed a performance of 1 PFlops

Thanks Ernest.


50 posted on 06/11/2008 10:16:32 AM PDT by SunkenCiv (https://secure.freerepublic.com/donate/_________________________Profile updated Friday, May 30, 2008)
[ Post Reply | Private Reply | To 1 | View Replies]


Navigation: use the links below to view more comments.
first 1-5051-56 next last

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
General/Chat
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson