It IS different than talking on the phone. I can drink a milkshake or eat a candy bar while driving a car, and not feel impaired in the least.
But, to paraphrase Chris Rock in this video How Not To Get Your Ass Kicked by The Police, if you are driving down the road, eating a Taco Bell taco, spilling it on your lap, and trying to keep yourself clean, maybe you SHOULD snuff yourself out in a car accident...
The reason I think there is something to the concept your brain processes interaction differently when the other person isn't there, I think back to watching people carrying on a phone conversation on a land line. Ever watch that? People doodling, drawing, fiddling with stuff, while their disembodied eyes look the corners in the ceiling as they wrap the phone cord around themselves. You ever see someone do stuff like that while they talk with someone in the same room with them? If you have, it is pretty rare.
Now, put that same person in a car talking on the phone, doing 40 mph with traffic entering and exiting, pedestrians, people on bikes, etc. I really don't think the analogy is to people eating or talking to someone sitting next to them.
I guess that my experience is different from yours.
I almost always drive myself, with nobody else in the car. Occasionally (rarely) I have another person in the car with me, in the front passenger seat.
I have found that talking to her while driving (it is almost invariably a ‘her’) is quite similar in its distraction potential to talking to someone else on the phone while driving.