There is significant cultural overlap between Yemen and the poorer rural provinces of southwestern Saudia Arabia (Asir, Najran, Jizan), with no clearly defined border between the two countries. The tribes go back and forth with little regard for the border, in the same way that the Pashtuns ignore the border between Pakistan and Afghanistan.
This area of Saudi Arabia is where Bin Laden and many of his henchmen came from. The more urbanized inhabitants of the north consider people from that area to be ignorant, violent rubes of questionable loyalty, and it is difficult for them to advance in the Saudi military or other national institutions. I suppose this feeds their aggrievement with the Saudi government and contributes to the radicalism associated with that area.
With so many of them, including Bin Laden, having Yemeni ancestry, it is no wonder they can claim citizenship in either country.
-ccm
As a cultural aside; when Saudis tell jokes, the Yemeni is usually the butt of them a la "Blonde" "Polish" and "Irish" jokes here.