Would you apply that same label to Lee's men during their campaigns in the Maryland and Pennsylvania during 1862 and 1863?
Obviously, not because Lee paid most of them in Confederate money for what they obtained.
Furthermore, the citizens in Maryland and Pennsylvania were much better off materially than those in Virginia during the end of the war. In several instances, the bummers were stealing from people in dire poverty.