This article says that the trees should be taken inside if there is a danger of frost. That’s really not practical or doable for some people. These can be heavy. And who wants an orchard in their house.
I wonder if you only have to be concerned about this when they are young and have been newly potted.