You have it wrong. Pakistan did not allow entry to anyone with a US passport at that time (1981). So the question is NOT whether it was OK to go to Pakistan on a US passport, but whether Pakistan allowed US passports entry at that time.
Moreover, during that same time period (mid-1981), New York Times assistant news editor Barbara Crossette visited Pakistan and wrote a column about her experiences, a piece in which she offered tips to other Americans who might wish to visit that country. Additionally, her column prompted a(emphasis added)follow-up letter fromJohn S. Brims, then the U.S. Consul General in Lahore, Pakistan, in which he "welcome[d] an influx of Americans" who might have been inspired to come to Pakistan by her article and also offered helpful tips for tourists traveling to that country by rail from India....Many other news articles from 1981 reference either Americans' traveling to Pakistan or the Pakistan government's making efforts to encourage visits from U.S. tourists.