I don't know how much of this "refusal" story is true. PM Orban went to Boston to accept a prize; I didn't hear anything about him wanting to go to DC until he came back to Hungary and the leftist papers here began whining about the trip.
We were in the middle of an election campaign, and the commies threw all sorts of lies and distortions at Fidesz, with the obedient assistance of the media. My impression was that, annoyed by the PM receiving a prize, the leftists tried to change the subject and it was simply an expedient for them that he hadn't been to DC. "He is now a persona non grata in the White House!" they began trumpeting.
Spin like this was very typical of the Socialists' campaign; they did it day by day, month after month. The reason that this lie became so famous and others didn't is that it was later picked up by the Washington ComPost.
It only adds to the hypocrisy that the same leftist papers routinely denigrate Bush as an evil capitalist unilateralist racist yada yada yada yada ... You know, the same stuff you hear in the States. This sort of forked tongue is also very typical of our leftists: In 2000, they vilified the Austrian government for restricting immigration and employment of foreigners, while in 2002, they castigated Orban for letting Romanians be employed in Hungary, wringing their hands about "the effects on the labor market." They accused the Fidesz government of being racist and persecuting Gypsies; then later, they started rumors about Orban himself being a Gypsy. (With the obvious intention of getting racists to vote against him!) They painted Fidesz as anti-Semitic, while at the same time they servilely echoed the lies about Islam being "a religion of peace."