I'm glad I live in europe then! Here both males and females can drive. In general males drive more aggressively and females slightly less concentrated, but it adds up to fairly reasonable driving on average.
Anyhow, the problem with talking on the phone as opposed to listening to music or movie events etc is, that you actively have to concentrate on chatting. Especially if it's work related, or otherwise complicated stuff.
Concentrating on chatter means you're very much less responsive to change. You'll do just fine so long you only have to follow the road, but if a tire blows, another car breaks down, or there's roadwork where there wasn't any in the morning, you're much much slower at responding to the new situation, and may find yourself breaking a full second later (or whatever task is required).
If you ain't using handsfree stuff it gets even worse, as you'll automaticly try to place those items safely before you can do anything else with that hand. Even though just dropping the stuff might be the obvious best-response.