I actually find the fact that the Xbox controllers use AA batteries and a replaceable battery pack more of an advantage than a disadvantage. I hate the PS4 controllers that have internal batteries as they are such a pain in the backside. I know you mention that AA batteries aren't exactly environmentally friendly as you use them then throw them - but have you heard of rechargeable AA batteries. In fact using AA batteries over internal batteries is far more environmentally friendly.
Think about this - years ago some motorola phones used to have an option to either run off AA batteries or it's own lithium ion cell. When the lithium ion cell died - what did you do? You just went down the local high street, bought a pack of AA rechargeable batteries, plugged them in and off you go - phone fixed. Now with modern phones (and most other electronic devices) with internal batteries - battery fails and stops accepting charging, what do you do? You could go to some guy in the market and see if they'll swap the battery over for you, although as I discovered with one of my phones the NFC antenna is stuck to the battery - so replacing the battery kills the NFC in the phone if the guy changing the battery does it wrong, or you go buy another phone and chuck the entire phone (and in some cases the entire phone is superglued together in a way that changing the battery means the phone will never be the same again.
This example obviously is about phones - but this is one annoyance of many people with phones now, since Apple decided it was a good idea to seal the battery into the device and prevent replacing the battery when the battery does refuse to accept charge that leaves you with an entirely useless device. Batteries have a lifetime that is far shorter than the lifetime of the actual electronic device and will need replacing at some point. Having AA batteries in a games controller is a far more better idea than having internal batteries that are sealed in the controller as when the AA batteries fail if they are rechargeable you just stick them in the charger - we have a rapid 1,000mAh charger so in about 2.5 hours a 2,500mA battery is ready to go again (and if you can't wait that long most rechargeable AAs come in packs of 4 and the controller only takes two so you can just rotate them). When the battery finally reaches its end of rechargeable life then you just replace the batteries rather than the entire controller. I know you mention that the rechargeable batteries in your other Xbox controllers have lasted long enough, but not all batteries last that long, plus the Xbox rechargeable batteries tend to be NiMH because they're based on AA batteries, and NiMH can have a lot longer charge time than LiOn batteries (like the ones that are used in PS4 controllers). For a controller there is absolutely no need to use a LiOn battery as they don't really pull enough power to warrant it.
The other advantage that using AAs has is it allows many alternative options for charging. You could go for the standard Microsoft charging set, but as it's just using AA batteries it does give many other manufacturers the change to produce their own versions. I have bought a charging kit for my Series X controller, it's not the official Microsoft version but it's a really good charging kit. It has two small metal contacts on the back and a docking station so when I'm done with the controller I can just drop it on the docking station and leave it charging. The docking station itself has a red light on it which changes to green when it's charge and even has a pass through USB port on the back - it has to be one of the best charging stations I've ever found - it was a Venom Twin Charging Dock for Series X - they do a single version but I got the double version for when I invest in an extra controller. On the PS4 I've got a similar docking station for the PS4 controller, but the annoying thing is because the PS4 controller has to be charged by it's stupid USB port and the battery is internal to make it dockable I had to add this tiny plastic connector to the USB port which now sticks out of the top of the controller, because the batteries on the Xbox controller are replaceable it's replaced the entire back of the controller and the controller is still flat with no extra bits sticking out of USB ports (plus the USB port on the controller can still be used without having to remove the adapter.
There are far more advantages for having removable batteries and I much prefer that Microsoft went down this route and not the Sony route of making them internal.