I live near train tracks, about 500 yards, and the loudness does vary, depending on the wind direction.
You are right, the wind does influence how sound can be heard, tho I have no idea why.
As someone pointed out, the speed of sound is far faster than any wind but apparently that is not the whole story.