I think the issue isn't the species themselves as much as the fact that they're not slaughtered and butchered at licensed and inspected facilities. Depending on the source, it might be a tricky proposition to do so -- in towns where hunting is popular, butchers have to maintain separate tables and sets of tools for livestock and game.
Yep, after reading more of the comments, agree that the issue with most of the freepers is that the meat wasn't inspected. But not all of them.