What I see as a problem with the iPhone is something that many users like; it has an immersive user interface that you have to pay attention to. On a phone with a physical button to answer a call, that button is always in the same place and for many people, muscle memory will put your finger in the right place. With virtual buttons, you certainly can look at the screen to see what to do - but having to look does take more of your attention. For some apps that's good, for some it's not.
Putting a button with a different function in a familiar place and changing what built-in gestures do, or making you click OK before you correct a password instead of highlighting the field are small things; if they're small things that irritate you frequently, they're bad things. I view a phone as something you use on the move, with patience for navigating interfaces, without the fine control of a mouse and keyboard, often when you're in a hurry. Getting the experience right for the phone means being simpler and more accomplished than on a desktop/notebook; there's less margin for error.
Nearly all of the interface 'glitches' here (including the moving answer button) break Apple's Human Interface Guidelines for the iPhone, so it's developers who could give users an extra second of delight rather than frustration each time. We included examples of apps that get the experience right for contrast, because it's the difference between a seamless user experience and something that breaks you out of flow.