Visible light lasers will easily burn your retina. It all comes down to power level and how long you are exposed.
It is much more complicated than you imply. The aversion reflex prevents most damage. The determining factor is retinal illumination.