All the ones who refuse to answer (and I wouldn’t either), probably contribute to the “skewed results” that the Census is getting from only those immigrants and low-income people and illegals answering it — instead.
So, they are getting their answers — which actually show, instead, that you and me don’t exist in this society... :-)
I am really torn about this. I value my privacy highly but I also value history highly. I suppose I should admit I am somewhat of a genealogy nut too. I use the old censuses to find my family and where they came from, where their parents came from and what they did for a living. I can even go to the old address, which in some cases are still there, and see the actual house they lived in over 100 years ago. It is a precious part of my history. I hate to see people not answer the census, but I also understand. Hopefully people will fill out the important parts with the correct history and I don't care what you put in the other parts.
And as you said, not filling it out indicates you don't exist. That most certainly won't opt you out of the present, too many trails will still lead to you, but it might deny your descendants the fact of your existence in the future.