As for Scalia... as an American citizen-voter, I have authority over all those people in Washington. All I have to do is convince, or join, a large enough number of fellow-citizens who share my opinion.
“But SC justices aren't elected!”
I bet if enough Americans got together to demand it, they'd find a way to change all kinds of stuff about the SC, including the people on it, and without running afoul of the Constitution.
The fact that you have to join with other like minded citizens is proof positive that YOU do not have authority over anyone but you. On the other hand We the people....have much authority if we chose to exercise it.