You are quite right. Those who refuse to accept it at all, perhaps with violent opposition, would be a subset of those who voted against it.
But the question still exists. It is very difficult for a government based on the consent of the people to have legitimacy if a large minority of the people disapprove.
Or at least more so than for governments basing their legitimacy on some other principle.
Odd. Obama makes it look effortless, between rounds of golf and vacations...