It isn't about whaling. It's about freedom. Can screaming-meemies tell a free person what they can and cannot do with their free time? Your argument can easily be applied to my firearms hobby, my neighbor's animal heart-and-eye collection (yes, I'm not kidding... don't ask), or my friend's ultra-weird (and inefficient, and probably not road-worthy) motor-vehicle "creations". Non-endangered whales have no more protections than any other wild mammal, fish, or bird. Should your misplaced emotional attachment to an entire class/genus/species supercede someone else's emotional enjoyment of hunting/killing/eating them?
The difference between prey animals like deer and creatures like whales and other primates is simply not that hard to grasp. Deer, pigs, goats and the like were PUT here to provide food for humans and other predators; whales weren't.