You seem to forget that the SC can do the same thing. What's to stop 5 of 9 legal elites from passing a law canceling elections?
Putting that kind of power in the hands of unelected elites is madness.
Someone would have to bring a lawsuit. What would that lawsuit be? That elections are unconstitutional? Please. A court can only rule for this side or the other. A legislature can pass any law they fancy and we have to have a judicial body monitoring them.