My Files does allow you to delete, rename and remove files. I can't remember any version that haven't been able to do that. At one point, right after Google dropped native SD card support in Android (4-5 years ago?), file management on SD cards became really messy for about a year. During that time it was a nuisance to manage files on SD cards. Google was thankfully bullied into adding native SD card support on their next major update.
Third party apps like Solid Explorer and ES does add nifty features, like network share support and other things, but like the BFG said, beware of ad funded apps. They're often buggy and annoying.