Robberies would increase if folks kept their money at home. That’s the one thing I’d be worried about. That and a house fire.
I’ll bet there would be more dead burglers/robbers. And you can get pretty well fireproofed safes.
I’m with you, though. However, even if you DON’T do it, if a lot of people do, you’d still be more likely to be robbed.
I would only worry about my funds in the bank if I was going to participate in something that might put it at risk regarding the government. That’s why I don’t do long road trips with a lot of cash. I just deposit locally and pull the money out at my destination. Civil asset forfeiture makes the cops the modern highwaymen.