Aye .. looking at your settings for HandBrake, they are a little too "extreme".
As PA2 said... at 4GB per BluRay, you can use x264 instead.
for x265 settings, use:
Preset: Faster
Constant Quality: 23 to 25 (i can barely tell the difference, and only in fast scenes)
Sound: DTS is great, but...