I call that racism.
The trapped spirits had followed a cult leader, predicting the end of the world, to a cave in a desert to "wait out the end times". When the time and day came and nothing happened, the people (then alive) begged Kane (their leader) to let them flee the cave. But he wouldn't let them. And they died.
Flash forward, the little girl is asked why the spirits remain and keep troubling her and she (and Tangina, the psychic) effectively reply -- the lost spirits can see the light of the little girl's innocence through the darkness, and so they keep following her: They have nowhere else to go.
Ipso, Victoria Peterson. She, obviously, can only see the light of the Duke Three's innocence, and has only that to follow in her pursuit of whatever it is she thinks will give her "relief".