It only takes five of those nine.
On the present court it only takes one, namely Kennedy. He holds more power on the Court than Chief Justice Roberts because of his position as the swing vote on almost every case that comes before the Court.
If Kennedy dies or is removed from the Court for some reason while Obama is in the Oval Office every constitutional right that we now enjoy will be in very serious jeopardy.