The constitution doesn't exist to list the rights we're allowed to have. It exists to list the rights and powers WE the people have granted to the government. If it aint in there than WE didn't give them authority to regulate it, legislate it, or do anything else about it.
In fact an argument was made that there should be no Bill of Rights (which came shortly after the constition was orginally signed) because it would leave open an argument that government did indeed enumerate our rights and as an ultimate perversion of the intent they would go on to say if it wasn't in the Bill of Rights than it wasn't a right of the people. Well, that fear came true, didn't it? Hell, even if it is in the Bill of Rights they just keep chipping away at certain things.
So yeah, marriage isn't in there. And as a result the FedGov has no right or authority to even have an opinion on it. But SCOTUS just doesn't follow the constitution any more than the other branches do. They just do what they want. Which is very very sad to see. Sometimes we like an arbitrary ruling they make, other times we hate it. But there is absolutely nothing we can do when they seize power to amend the constitution with the votes of just five citizens.
We do know that the Continental Congress/Congress was issuing some pensions to war widows in 1780,1794,1798, and 1802 which means the feds knew about marriage and what made a legal one.