Crosswinds are one of the main problems. Look at the early rocket launches where winds aloft sent the rocket off in a random direction.
I’ve heard that the Space X Falcon 9 is more sensitive to adverse winds because of its long and narrow profile or “fineness ratio”.