The main obstacle is that existing e-mail encryption programs are too awkward for the average non-technical user. Fortunately, the NSA scandals have sparked interest in developing user-friendly solutions (such as MiniLock, currently in limited beta release with full release planned for August -- technoneepery details).
For any recipient that you have a public key, you encrypt the message to them. Granted, having one of the messages go out plaintext defeats the crypto, as time goes on less and less would be sent that way. You could also have an option where instead of sending the recipient the message you could just send an email to them requesting their public key and providing yours. Anyone not interested in sending your their key would just not get any email from you.
I've almost gotten to the point that I'm more interested in secure communications than I am with talking to casual acquaintances, or even family members for that matter.