Souter’s “judicial restraint” was to prevent restoration of the Constitutional order which had been devastated by the Judicial activism of Progressive judges.
Souters actions were to preserve the Progressive revolution in the courts.
I frequently disagreed with Souter opinions. Perhaps more than Kagan opinions or Breyer opinions. I think Souter was just not quite SCOTUS material. This is without regard to where he was ideologically.