“The Chief Justice of the United States and the associate justices of the Supreme Court shall from time to time be allotted as circuit justices among the circuits by order of the Supreme Court. The Chief Justice may make such allotments in vacation.”
So in short, SCOTUS decides.