The iMac ran pretty well with a medium load of software (iTunes/Photo, browser, some Finder windows, etc.) with just 1GB RAM. I guess it's all in personal perception of speed. But with 3GB it barely pauses for anything, no matter how much I run (although I haven't run multiple VMs yet -- that'll slow down any system).
I’ve used this rev B iMac since 1999, and wish I’d paid for more RAM (this one has 96 MB, and uses 97 MB for virtual memory). Ah well.