No, can't say I've been kissed by a man with a mustache... but I have had a goatie or full beard for many years, and never had any complaints from any ladies over it.
When its stubble or short, definately.
Me, I keep my trimmer generally set to "2", long enough so that pressure against it causes it to give, and not so short that it effectilely won't give and just pricks ya like a bazillion needles. I would say anything 2 and over you are golden... less than that, you run into the poking/irritation problem...
Stubble feels THE worst!