Yes, I would think it very difficult to define the "operational" size of the Roman Empire at any given time since so much of it was conquered territory in which the inhabitants were never really "Romans".
Contrast that with the huge number of Celts, who as The Lost Tribes of Israel may not have know who they really were, at least retained some semblance of tribal identity even to this day. (As in birds of a feather flock together.)