You make an interesting argument. Gay marriage, as you say, does not affect me as an individual. That being the case, I have no dog in the fight and approach the matter for the perspective of ‘you are free to do anything that causes another no harm.’ I am neither bound by nor a party to the marriage contract of others, gay or straight.
As for marriage and religion, the alternative is civil unions to set the legal aspects of the ‘partnership.’
As an individual, I really don’t see what all the fuss is about. Let people live by their own lights wherever possible.
But that's not what this marriage issue is about. It has never been about "doing your own thing," but about subjugating an enduring culture under a pernicious Fascism.