Yes, that is true; however, the big problem for everyone is that we may know or recognize up to 2500 people, but we, at most, interact in a meaningful way with no more than 20. Research has supported this and the thought is that we evolved or survived in small groups of less than 20.
In any case, this means SCOTUS members are increasingly reflectors of those small number of elites they interact with and nothing else. If one assumes government is society than, perhaps, is what we should expect with cognitive elites. If one sees the culture from a majoritarian perspective as something beyond, and many ways superior to, government, then this isolation is dangerous and doomed to long term failure. I for one see the culture or American civilization as being a greater force in daily life than the judicial elites who consistently try to abort or change it because of their almost missionary religious zeal to improve it.
Your point about the "law clerks" is proof positive the SCOTUS members need something else to influence them except bright law clerks. Indeed, scary as it seems now SCOTUS is talking about using international law as precedent--never mind our Constitution.