There's no question the court has gained latitude over time. That's the nature of the government under this Constitution. Each precedent creates the foundation for the next, and so on. None of this is a surprise:
When the courts will have a precedent before them of a court which extended its jurisdiction in opposition to an act of the legislature, is it not to be expected that they will extend theirs, especially when there is nothing in the constitution expressly against it? and they are authorised to construe its meaning, and are not under any controul?Brutus, 31 January 1788
"Gained latitude"?
That's a really soft and fuzzy way of saying "seized power". Trouble is, those who seize power aren't particularly soft and fuzzy.
When society rejects the government, the government has lost legitimacy. The battle becomes, as you point out, one of brute force and wills.