That's a very anthropocentric way of looking at it.
There may be entities in our universe–or in dimensions parallel to ours–that we've never even imagined. Beings that occupy a reality layer so orthogonal to our own that we couldn't perceive them any more than a bacterium in your gut could grasp your inner monologue, your mortgage worries, or the concept of a symphony.
They might not be made of anything physical at all–no ships, no bodies, no “little gray men.” Just pure consciousness or information.
Physics (String theory and M-theory) already hints at possibilities like this, suggesting up to 11 dimensions, most of them curled up so tiny we can't detect them. A being that moves freely through those extra dimensions could appear and disappear, pass through solid matter, or slip between realities as easily as we walk through a door. To us, it would seem like pure magic.
I'm not saying anything like this is visiting us, but it does show how limited our assumptions might be when we insist that any visitor must behave like we would or obey the laws of our physics.
Whatever, but please do not make me taxes for UFO exploration.