There is the possibility that the shooter discovered the affair, lured the guy to the property and then shot an unarmed man as part of a premeditated plot to eliminate his rival.
If law enforcement cannot demonstrate that such a setup took place, it will be very hard to charge him and castle doctrine applies.
If the wife invited the man to her home, then it should not apply. He was a guest, no matter how unwelcome he was to one resident.
The husband was killed going to his rivals home after he found out about the affair, not the other way around.