No one goes to Hell for ignoring the poor. They go to Hell because of their sin.
The parable doesn’t mention any sin of Dives other than ignoring Lazarus.