More likely it was creative differences between Alex Chilton and Chris Bell.
The first album tanking was probably what caused the rift about band direction. Obviously both of them thought they had produced a pretty decent intial release. To see it sell a mere 10k copies must have been hard to accept. Much easier to accept if the product is crappy.