I am pretty sure that is the point
Souter does not want people that knew him commenting on his thinking and he does not want people who knew him harmed by what is in his papers.
he does not want people who knew him harmed by what is in his papers.”
If Souter was exemplary in his personal life and in how he conducted himself as a officer of the courts, who could be hurt by it?