Of course they do. The same rights as any of the couples who are living together without licensure.
The state began mandating licensure in the beginning of the 20th century. Prior to that people posted marriage bands, committing themselves to the marriage and providing for children as they came along.
Are you completely ignorant of the history of marriage?
What a lot of you Libertarian types never fully grasp is that you cannot have a morality free government. You are going to get some sort of morality imposed on you, the only question is who's?