I think he will only vote for removing one that shouldn't be there in the first place, or as you said in all other cases.
Precisely; Mr. Paul's is a literal interpretation of the Second Amendment. I suspect he would say, 'If the Constitution needs to be amended in the pursuit of liberty, then amend it. Do not pass unconstitutional legislation or it dilutes respect for the Constitution in the construction of ALL legislation.' In that respect, he would be right.
Unfortunately, there really are places where one must draw a limit line on "keep and bear arms," because technology has so enormously changed what constitutes "arms." Assuming a logic of literal interpretation, Mr. Paul would consequently vote against laws limiting private ownership of nuclear weapons or weaponized smallpox and sinkspur would say that it was proof that he was crazy. Perhaps, but then he might also demand a Constitutional Amendment to articulate what should be the limits on private arms ownership.
If so, IMO, he would be right to do so. Technology has changed many things, and not all for the better. It is our challenge in the pursuit of liberty to ascertain the operant principles of self-defense, and an armed militia, necessary (AND SUFFICIENT) to insure a free people, and craft legislation that accurately articulates the key distinctions in light of those technical changes, a task so far beyond most legislators that it gives one pause to even hope for such a resolution.