If they are not field dressed immediately they can be gamey, also a gut shot deer tastes strong. Additionally, they need to be hung to drain (the same day they're shot), something that can only be done if it's cold out. To make them taste best, quite a bit of care is required.
But give me deer meat any day, I far prefer it over the best sirloin. We had a steak and venison fondue recently, even my wife and daughter agreed the venison was better than the steak.
OK, you can have my share then. :)
Where I grew up, the first day of deer hunting season was a school holiday. I got to taste venison killed by almost every male relative I had over the age of 14, prepared in almost every way imaginable. It didn't matter; I just never liked the taste. My Dad and Mom did like it; my Dad especially liked venison jerky. Not me though; go figure.