“If either her parents or my parents had objected it would not have mattered because it was just us against the world. “
Not germane to this story, unless your “it would not have mattered”, means you would have repeatably gone into her house, even after the parents had told you not to.
It does mean that.