I am able to talk on the phone and drive at the same time and it is far less of a distraction to have the phone to my ear than fumbling for hands free buttons.
...but, there are people out there that simply don’t have a clue. I saw one guy stopped at a right turn that had no stop. He was talking on the phone. Others drive gradually slow down and they get more into their phone call.
If you can’t drive and talk on the phone at the same time, don’t! You are ruining it for the rest of us that can.
Hands' free is much less distracting for me. It leaves me with both hands on the wheel and you are not distracted with 'talking into the phone' which tends to isolate you from external events.