To repeat: the annulment wouldn't mean they were never married in any sense at all. They were probably joined in the eyes of the law. But not in the eyes of the Church, and that because of some defect at the beginning that would have annulled the formation of lifelong bond.
Yes, I'd attend the "second marriage" if the Church had determined it was actually the first and only marriage with a lifelong sacramental bond.
I find the annulment out a convenient "after-the-fact" determination which can be applied to any marriage. The mere fact that there might be adultery involved is simply deemed the "evidence" of some non-binding intent.
In other words, "Catholic divorce."