When I worked for the Census Bureau decades ago we were given specific instructions for the ACS survey.
The “visit” list was only people who had not responded to the mailed survey.
Probably 90%+ were friendly and cooperative.
Another 5% refused to cooperate.
Another 5% tried to avoid us in various ways.
The way we handled the last two groups was that we contacted neighbors and asked them basic questions that neighbors would be likely to know (not about income, though).
So—if you don’t want to answer the questions you can refuse and the census taker should be ok with that. If they give you a hard time just ask for the phone number of their supervisor and tell them you plan to complain about harassment.
My bosses always told stories about "the quivering curtain".