Thanks everyone, I just came back from erasmus a few days ago, I tried the software robert600 linked, it's really good! The only drawback is that in a 100" screen you can see the diference between that Staxrip (CQP) and DVDFab (CRF), no really a big issue since it's barely noticable, not even on my 65" HDR TV. I also tested CPU encoding...I could not tell the diference whatsoever from GPU to CPU, same file size, same quality on my friends 100" Proyector and my 65" TV.
I upgraded to a GTX 1070, now everything works fine, it seems it was also a bug from DVD Fab that is now fixed. For my surprise, Staxrip can enable 10 bit HEVC encoding using NVENC GPU Acceleration, so it works with my GTX 1070, I tried testing and I got around 240 fps encode time, the color depth was from an 8 bit source, it improves things, as it looks a bit sharper, but it's no big deal, I will continue with 8 bit HEVC.
I did more testing and finally got it working. I took Angels and Demons Mastered in 4K Disc and I made 2 copies at CRF 20 using CUDA on DVD Fab.
-H.264 File: 12,3 GB Encoding average speed: 515 fps took about 5.5-6 Minutes
-H.265 File: 9,85 GB Encoding average speed: 400 fps (That's a beast compared even to the i7-7700K which does it in 16 fps) took about 7-8 minutes.
As I mentioned before, I did a 10 bit H.265 at 240 fps, but it didn't look much better, just a little bit (size was also 9,85 GB). The 8 bit H.265 file obviously looked better than the H.264, here the difference was much more instant than 8 vs 10 bit HEVC. I also tried to compress even more the H.265 file to see when it would start to look similar to the H.264 and it was at CRF 23-24 somewhere around there the quality was the same as H.264 or at least very similar, but size came down to 5,44 GB, and with a DTS-HD MA 5.1 track included...that's seriously unbelievable.
Conclusion: Only encode in HEVC 10 bit when the source is 10 bit, as the quality is barely noticable from 8 bit and the encode speed is much slower (a bit less than half).
Here H.265 leaves you with 2 choices depending on your needs:
-1: Achieve the exact same quality of an AVC file by saving half the file size (50% savings)
-2: Whatever preset is used in AVC, will have much better Quality in HEVC and will also save you a bit of space (about 20%)