Agreed, Caan's parents were Jewish immigrants from Germany, he didn't look the part and he overcompensated.
I think Duvall looked the part (German-Irish) but he didn't fit a character who had been raised/adopted by that family. That could have been a Director's choice.
___________________________________________
I grew up around the corner from the Gambino family (the source for the Godfather). They lived on Club drive in Massapequa. The real consigliore was a waspy guy who looked a lot like Duvall. I won't tell you his name but it was the same as a particularly feisty CW General.
http://classic.mapquest.com/maps?city=Massapequa&state=NY&address=%5B1-7%5D+Club+Dr&zipcode=11758&country=US&latitude=40.652415&longitude=-73.457455&geocode=BLOCK
FWIW...Jerry Sienfeld lived just west on Riviera Drive and the Baldwins lived just east on Clearwater.