Forget all the legalistic nonsense, it will be utterly ignored by the Census and ignored by the courts, and will result only in this guy wasting his own time and money.
Everything this Census worker has done is specifically legally authorized by the US code and Census workers are specifically legally authorized to enter private property so that they can knock on a door or leave a note.
He's never produced any evidence of "harassment" other than a note on the door; ignore the notes long enough, and don't attempt to call, and they will give up (eventually.)
you sure don't understand the law!