I can't really speak for others being irritated - I guess because he didn't speak out more strongly in favor of marriage and against same sex marriage. I am not displeased with what he said. He's a politician, not a preacher.
No offense, but the President can't be any more forceful... (unless you want him to call down fire from Heaven and consume all gays in America) He has repeatedly stated that he is against gay marriag and that marriage is between a man and a woman. He has stated that he supports a constitutional amendment.
But, he cannot tell the states what to do. He can only act on the federal level.
Yet... all this is not enough for you.