He is a federal judge. Congress can impeach him, not the “people of California.”
I know he is not elected, but there must be some mechanism to remove a negligent judge from the bench. If a judge literally loses his mind, can’t they force him to step down?