Why should that be Israel's problem?
The stupid ones listened to their fellow Arabs who told them to leave, that they would squash Israel, kill all the Jews and they could move back in and get all the loot.
Didn't quite work out as planed.
So there they were, out of their homes, across the border and Israel was not going to let them back in. They had openly declared themselves enemies.
The nations that attacked Israel suddenly found these undesirables parked on their doorstep permanently. They didn't like it much.
So they passed laws that "Palestinians" could not own property or open businesses or do a bunch of other things to encourage them to "push the Jews into the sea".
And here we are.
It was a bit more complicated then that but not much.