That is not the point. In fact, I am not sure that you're right on that, either. However, the point I tried to make is, the "hundreds of years of judicial wisdom" you refer to, have given us such idiocy as the Dred Scott decision, Roe v Wade, and the unbelievable "emanations and penumbras" argument. Just because some Supreme Court Justice from 1850 wrote a decision that, in effect, re-wrote the Constitution, doesn't make it right.
Dred Scott held that:
1. Congress had no authority to regulate slavery in the territories and thus could not grant the territorial legislatures the authority to do that.
2. Blacks are not citizens of the United States and therefore have no standing to sue in federal courts.
Both holdings were overturned by the specified amendments.
You say some precedents are wrong and should be overturned. I agree. Some decisions, like Dred Scott, Plessy, Lochner and Bowers were wrong and have been rightfully overruled. But that's different from saying that every precedent you believe is wrong should be overruled. And that's were conservative activism comes kicking in, I don't think someone like Roberts would do that. He is more likely to limit the scope of decisions he disagrees with. Compare that to Thomas: Scalia says that he homas does not believe in stare decisis, something that is essential to stability in the legal system.