I’m pretty sure Baker’s family has Lebanese roots although I’m having trouble with google finding anything. It would partly explain his animus towards all things Israeli.
You might be thinking of someone else. Spencer Abraham?
Baker’s family goes back at least to the pre-Civil War South, and were not known for being friendly to non-similar ethnic groups.