This is not true. There was no travel ban to Pakistan in 1981.
My understanding is that Reagan lifted travel restrictions to Pakistan in 1981, however the State department issued a travel advisory:
http://dosfan.lib.uic.edu/ERC/travel/cis/southasia/TA_Pakistan1981.pdf
The reality was, however, that Pakistan was only issuing 30-day visas to the press, diplomats and those willing/able to provide large bribes. They did not want Americans there.