It's not as if the Settings app couldn't replace the Control Panel and do a good job of it. It's just that Microsoft hasn't demonstrated that it can. For something as critical as system configuration, that's worrisome.
If the Settings app had already presented itself as a mature and thorough option, I wouldn't care about the transition. The problem is that Microsoft has created a dual reliance on the Settings app and the Control Panel.
Building the Settings app piecemeal and moving options over a little at a time has created a bit of a mess. It's only natural for people to be worried about what comes next. On top of that, Windows 10 was once solid, but continues to get more buggy and unstable. That also isn't increasing anyone's confidence in Microsoft to implement a new system critical feature without a safety net.