Others were strictly war fighters but could be assigned to the camps at any time. When the U.S. troops liberated the horror show that was Dachau they lined up the SS men they had at hand against a wall and shot the lot of them without a trial, a rather understandable reaction given the boxcars of corpses they had just seen. Unfortunately those men weren't the guards responsible. Those guards had fled the scene a day before leaving the new arrivals there to hold the bag.
The real problem was that relatively few of the ones truly responsible were punished. Certain high-profile culprits such as Rudolf Hoess and Hans Frank were, but Nikolaus Wachsmann estimates that only about 15% of the commandants and guards who actually committed the atrocities were punished.
Through Wachsmann's KL and Arad's Belzec, Sobibor, Treblinka I have come to harden my attitude quite a bit. Hound these men into their old age? Yes. Because so many of them earned it.
Hound them on the basis of what evidence? As you said, many were avenged upon had little to do with the brunt of the evil.
Credibility suffers as time passes.