Yes... when a middle-aged man sleeps with young boys some may tend to get that idea.
You can't have it both ways.