Out there is a big place. The distances are actually beyond the imagination of most humans.
"They" may not even know our galaxy exists, let alone us.
Your explanation supposes "their" intelligence is equal to or less than ours.
No it doesn't. It talks of unimaginable distances, not intelligence. The most intelligent life simply may not be able to travel through interstellar space for a variety of reasons, lack of resources, biological etc.
Given the size of the universe, I'd say the probability that we're the smartest ones in that universe is - zero.
Again due to the size of the universe, which you and others, including myself, don't even understand, even the most intelligent life in the universe might have no means of leaving their planet.
On the other hand, spacecraft no larger then a flea, carrying thousands could have made the trip many times, and come and go completely undetected. HA!
We just simply don't know.