Geez.. you sound SO much like me!! I feel the EXACT same way.. NO problem making spepeches.. One on one, ... TAHT'S a tuffy!!
Well, when you're making a speech or whatever, everyone is looking at you, and you have their attention, and you're in control of the situation. One on one, people tend to get distracted, look away, not listen to you...
Sometimes for me, I'd be talking to someone and while I was in mid-sentence, they'd turn around and start talking to someone else.