We have holly, too, but only two, and I cut them back when they get in my way. My husband would never prune anything, if it were left to him, and the lugustrum would be blocking the windows up to the second floor!
We get deer in the subdivision, since we have woods and fields, but not at our house.
Holly doesn’t seem to mind being pruned. I like all of our plants except the arborvitae, because the deer just walk right up to the house and eat it back to the trunk. We lost two japanese maples over the winter, probably due to voles.