I have a hands-free system in my car and all the controls are right on the steering wheel. Talking on it is no more distracting than to a passenger or listening to the radio. So if they are going to ban hands-free, they might as well ban speaking to passengers and listening to radios as well!
If there are passengers in the car, they should be the only ones allowed to talk and listen to the radio. The driver should not be able to talk to passengers or listen to the radio.
I think they should every time I get behind someone engaged in conversation with their passenger while looking at them, gesticulating, and ignoring the traffic.