But this cannot be a noticable impact as we were transporting most of the crude oil to refine in the first place.
It's noticeable in the short run. We do have significant refining capacity in the US, even if it's not enough. If the capacity goes down, then there will be a shortage until it either comes back up, or they are able to increase our imports of refined gasoline.
Another thing to think about is that if you've got to import refined products instead of crude oil, then you are going to be putting greater pressure on the balance of trade. A lot of the value is added at the refinery level. Instead of paying $40 a barrel, you're paying $2 a gallon, and that means a bigger trade deficit. A bigger trade deficit means that the dollar declines, and you pay even more for oil. But that only happens over the long term, ie. several years.