I agree with your opinion. Let the first one with no sin cast the first stone.
True, Jesus said, “Let he who is without sin, throw the first stone” but THEN he said, “Go, and sin no more”. People forget to quote the second sentence.