If he’s outside a library there shouldn’t be any reason to stop him — unless he’s blocking the door.
Being annoying sounds like a good reason.