I hear ya but if you dress like Lance Armstrong when you ride and take up the whole lane dont be surprised to be lumped in with them.
As for the attire, bike shorts are the way they are for good reason. The tight-fitting shirt is more optional, and really only needed if you are riding competitively.