I suspect federal law can and does provide “necessary and proper” power for census workers to enter gated communities.
You forget the white privilege clause of the US Constitution.