You did fine...
The law however is exactly as he stated. You are by law, supposed to comply with the survey. They usually try to do it by mail, I have a few of them that I never replied to and would have handled it in exactly the same way as you did.
However, the law has no teeth. It has no enforcement mechanism for a penalty, unless they decide to do it by executive order...which would add another constitutional violation to Obama’s growing list.
I think the last time they actual brought someone to trial to fine them was back in the 1970s and the Census specifically went after them because they were publicly announcing on a radio program that they weren't going to fill out their census forms.