Jayna thought it might have been a photo vest, but I'm sure other vests have loads of pockets and are similar. If it was a fisherman's vest, he could have answered Ellison that he wanted the ammonium nitrate to blow up fish. That is easier than a rod and reel.
I think he was probably wearing something that he thought looked pretty normal. The guy was not dumb, and would not have come in with a vest that screamed "nutcase suicide bomber." However, he obviously didn't look like a normal farmer or fertilizer customer.
Incidentally, I believe ammonium nitrate is used in manufacturing meth, and this might have been what the clerk was looking for. Here in Florida the places where people manufacture meth can usually be detected at some distance because of the smell, which is supposedly due to the fertilizer component.