I have had a similar experience, although not identical.
Refused to answer and was similarly threatened by the census taker. I laughed at her, and said you can wright down anything that you observe that is relevant to the census questions. BUt I will not answer them directly. I pointed out that the IRS and Socialist Stupidity Admin had other info that she was asking for and other information was in the county clerks office and the registrar’s office and that the agency should look there.
She similarly stated that federal regulations required that I answer her questions to which I laughed and said I was not going to assist her in stealing more of my money. This went on for close to 40 minutes to which I was polite but asserted my rights, quoted the constitution, and similarly refused to answer questions or advised that they already had that information.
What ended the conversation was when I stated that I would not answer any questions without the presence of my lawyer.
The law does state that you must answer any and all questions
“Your response to this survey is required by law (Title 13, U.S. Code, Sections 141 and 193). Title 13, as changed by Title 18, imposes a penalty for not responding. We estimate this survey will take about 40 minutes to complete.”
That verbiage doesn’t lead to any mandate, but a bureau web page singles out Section 121 of Title 13, which says that any adult refusing to answer a bureau survey shall be fined.
I had a very similar experience, multiple drawn out phone arguments and one in person.