The biggest factor in driver distraction when using a cell phone is that the driver is attempting to have a conversation while driving -- NOT that the driver is holding a phone to his ear while driving.
Specifically, having a conversation with someone who doesn't share your environment. Conversations with passengers rarely cause problems because they know your situation; they'll usually shut up when you need to focus on the road, and if they don't you won't hesitate to tune them out.
But yeah, hands-free devices are a feel-good non-solution, and thus very popular with politicians.