I believe that any person involved in a divorce who makes allegations of abuse, unfit parenting, etc they cannot prove (or are later proven false) should experience whatever they were trying to foist on their spouse, e.g. loss of visitation/custody, payment of child support and/or alimoney.
So let's say that you suspect your ex-spouse of using drugs, but you can't prove it. You should lose all visitation because you couldn't prove something to be true? I think that if it can be proven that you made the allegations knowing them to be false, in order to gave custody, then you should lose automatically and further sanctions (i.e. financial, or criminal) should be used. But many things that are true may or may not be able to be proven to the standard that the court requires.
Keep in mind that denial of all visitation don't just punish the parent, but the child as well.