I have always been unaffiliated in NC and occasionally I have had to explain to poll workers that I can vote in any primary. They do record on the public record which primary you voted in.
I know what you mean. I had a poll worker tell me one time that whatever party I voted for in the primary, I had to vote the same in the general election. I knew better but another poll worker piped up to correct it.