As things stand right now, "We" means taxpayers. And of course, with the current welfare state, they wouldn't go hungry or homeless. There'd be no need for "we" (meaning you and me as individuals) to help them.
That is a problem associated with socialism, not a problem associated with liberty.