Justice Scalia has said exactly that-- he refuses to consider the 9th Amendment in any case because he has no basis, other than his own preferences, to decide which rights it protects. Chief Justice Burger once tried, in a concurring opinion, to list the rights he thought the 9th Amendment protects-- IIRC, he mentioned the right to travel from state to state, the right of a criminal defendant to be presumed innocent, and a few others.
That's interesting - do you recall the case where Burger said that? That would make two cases where the 9'th is mentioned, since the only one I can think of off the top of my head is Goldberg's concurrence in Griswold...