This applies only to the old-style ads that were just GIF image banners. "Modern" ads that use active content (JavaScript, Flash, etc.) are a common vector for malware and it is important to block these. Blockers that focus on active content (like NoScript or strong browser security settings) would not block ads that are simple images.
An increasing trend is that some websites are now restricting access to content if you are running pop-up blockers. At that point, it's a judgment call for me. If it's a well-known website that I trust, I'll allow pop-ups for that domain only. If I don't know or trust the site, I'll just move on. It all depends on your comfort level.