I don't doubt that point, as swinging is mostly about hedonism whereas pedophilia is a deep pathology involving power and control issues.
But the real question is this: couldn't a crafty pedophile use swinging as a means to access children?