A drop in U.S. demand is still a drop in total demand. If the U.S. was only a small part of world demand, it might not be noticeable. But the U.S. is a large part of world demand. And when our economy is off, everyone else's tends to be too.
If the rest of the world increased demand as the U.S. decreased so that total demand stayed the same, you wouldn't see a price change. Usually the rest of the world economies are affected by ours, and their demands drops along with ours.
OPEC is the one spoiler in the whole equation because they can influence supply to manipulate price. But even they typically want the economies running well, because they have other investments now too.