Your analogy would apply if it were actually swapping; but it's not. For example, if you buy George Strait's latest CD, and I buy James Taylor's latest CD, and we swap them, then I will have George Strait's cd and you will have James Taylor's cd, and both cd's were legally purchased (i.e. paid for). That would be a true swap.
But if I make a copy of James Taylor's cd and you make a copy of George Strait's cd and we swap those copies, now you have George Strait's and James Taylor's cd's and I also have George Strait's and James Taylor's cd's, but each of us only paid for one cd. James and George each have been denied their rightful royalties for their work. The fact that each of them are wealthy and won't miss the royalty from the measley one CD that we stole from them is completely irrelevant.