How about NO.
If someone goes to a race, pays WAY too much money for tickets, and he decides to fly a confederate flag at his tailgating set up, or wear it as a hat or a shirt, you have NO right to tell him he cannot.
Because that is where the confederate flag is shown, in the spectators area, the tracks have removed them already.
So focus on your driving, as mediocre as it is, and leave the few people that even bother to watch your sport anymore alone.