As someone who has performed driving distraction research in past years, I have to agree that these laws mandating hands-free phones are silly. The primary distractor in using a phone in vehicle is cognitive - you are paying attention to the conversation instead of the driving situation. We found the impact was as strong as cataloging your CD collection while driving - equivalent to physical distractors.
To "Annoyed": many folks who self-judge themselves as being able to handle distraction actually perform as poorly as anyone else in controlled experiments. It's of course possible that you are the one exception in 10,000 individuals, but public policy should not be exception based.
Better methods to improve driver control behavior would not be ones that many would accept - a separate sealed compartment for the driver (to avoid distraction by passengers), induction of stimulants (to improve attention and reaction time), and cognitive workload monitoring (to maintain appropriate levels of performance over time, fatigue detection, etc.).
Simpler and more appealing, perhaps, would simply disable use of a phone in a moving vehicle (you could use it while stopped); we found in general passenger distraction to be less than that of a phone participant because they shared contextual cues. In addition, food, loose materials that move while driving, etc. would also have to be banned (and at least according to a somewhat discredited AAA study were a larger influence on physical distraction of the driver than, e.g., holding the phone).
And even better would be just to make the road as well as the cockpit more predictable. Sealing roads in vacuum tunnels should do the trick ;-).
Frankly, once autonomous vehicle technology gets to the point of mass reliability, I can easily see that being mandated, with manual control eliminated. In general, driving as it exists today is too difficult a task for humans to perform reliably.