So what? Let them stand there if they want to.
Get some shade there. Don’t deny them the shade of the fence/wall!