We can't go there, (well, maybe one time only) but they can travel back and forth.
Same with benevolent beings (angels) - they can travel back and forth, but we can only go there once (forever).
Humans have a soul and a physical body which is our armor,created by god,to protect us from them.
Our physical bodies ordinarily prevent us from crossing over into their realm.