I don't think he went as far as to endorse Griswold.
Roberts got lucky and was able to duck directly addressing Griswold under questioning by Schumer. Here's how Byron York described it:
Schumer ... said, "And on the Griswold case and the right to privacy there, you said, in reference to Sen. Kohl's question, 'I agree with the Griswold Court's conclusion that marital privacy extends to contraception and the availability of that. The Court, since Griswold, has grounded the privacy right in that case in the liberty interests protected under the due process clause.' That is your accurate view?""Yes, sir," Roberts said.
Well, yes, that was what the Court did. And that was what Roberts said the Court did. But did Roberts agree with what the Court did? That would have been a natural Schumer question, but the senator didn't ask.
But my memory of commentary after that session (which was Day 2 or Roberts's confirmation hearings) was that everybody was saying "Roberts supports Griswold."
I suspect something similar happened with Specter: That she made a response to an inept question from Specter re Griswold; and Specter either misunderstood her answer (the way many commentators did with Roberts); or, he's spinning it.