Not really why they are called toxins. There are a host of toxins that cause neither fever or inflammation. They simply put an end to metabolism, you die.
For instance the chemo patients we see are far more likely to succumb to a toxin than you or I might be.
See my post above for the definition of a toxin. When someone gets sick from an infection, it is the toxin produced that is making them sick. When you get a vaccine, you are getting ( most often) the toxin along with parts of the cell membrane.