Although the law authorizes the Census Bureau to fine you $100-$5000, it has never happened.
Of course, you always run the risk of being the test case.