I've been out of the ATM business for a few years but,
when i left non of the machines had USB ports.
Yes they all do come from the factory keyed with a "standard" key unless you specify otherwise and pay more. Then you have to have unimpeded access to get into the top of the machine. Where it was proprietary PCBs.
Sure you could pull jumpers and use a laptop to load software but you have to have pretty intimate knowledge of the machines. and there were many security checks in place between the communications between the individual components that if one was not right the rest refused to work with it and the security was only getting stronger when i left.
Older machines were even worse when the main boards were old 386s and they were located inside the safe. Sure they keys to get into the top of the machine were the same but to do a software change to them you had to get into the safe.
So while on one hand I think it is possible that this guy could hack one, I find it unlikely that the mfgs got that sloppy on their security as to put a USB port with enough access to the rest of the systems in to their machines.
Unless it was one of the smaller lesser know mfgs , i remember one that just put a whole PC inside (windows 2k I think) but it was a bit player in the market.
The others I dealt with Tidel, Triton, Tranax and a small but growing at the time company out of Ohio whose name escapes me at the moment all were decent as far as security went.
And then there was Diebold, yeah the voting machine diebold, but there weren't too many of them in the Mom&Pop stores/nightclubs/bars where the smaller standalone machines were. I don't remember much of their inner workings but, given the weakness in voting machine security , I wouldn't put it past them.