Possibly this, if you are very, very careful:
Cut off a toothpick to get a flat end
A teeny bit of gel superglue on the flat tip
glue the end of the toothpick to the ball
See if it pulls out.
Either the toothpick and ball will come out (yay! Fixed!), or the toothpick breaks off and it's no worse than it is now.