This well may be, but (i) one has to differentiate the existing wealth and wealth creation, and (ii) the model is incomplete: one has to model explicitly choices other than abstention from the fraudulent market (the victim typically goes to some other market when suffering a setback in a given one --- witness the real estate bubble after the burst of the DotCom bubble --- in which case even creation of new wealth is unchanged).
I agree with you when you say that this point fascinates many. My personal take is that it is because of the "common sense" perpetrated by media and uneducated folks: fraud is equated with robbery, hence loss. People do not understand, of course, that the loss is NOT macroeconomic: there is a countervailing gain to the robber. But, if a robbed one feels a loss, the folks reason, there must be a loss.
The problem is that markets are essentially an amoral tool and we are trying to overlay morality on top of this. Markets are a collective tool, morality is a quality that individuals bring with them when they participate.