Chapter 10 of Hebrews tells us who it hurts:
29 How much worse punishment, do you think, will be deserved by the one who has trampled underfoot the Son of God, and has profaned the blood of the covenant by which he was sanctified, and has outraged the Spirit of grace? 30 For we know him who said, “Vengeance is mine; I will repay.” And again, “The Lord will judge his people.” 31 It is a fearful thing to fall into the hands of the living God.
My point exactly: they’re not hurting the church, they’re just hurting themselves.