Well some homes are no longer there. Others are uninhabitable. I don't advocate sending anyone back, but I don't advocate forcing people to leave at gunpoint either.
The people who left have no idea of what happened to their houses and in that respect they might no worse off than the people who stayed. Wouldn't you say that those who left (and obeyed authorities) have an equal right to their homes?