There aren't any good ones, stick with x264.
I did x265 in DIVX but I could get better quality in Handbrake at the same file size with x264. Since the only real advantage to x265 is file size it was ridiculous that DIVX had a minimum file size that didn't make sense.
Decoding x265 is also problematic since most devices don't have a hardware decoder so it's mainly a PC likely that could decode it in software.
I've tried basically ALL the good video software out there, and I've come back to Handbrake and optimized my profile for the best quality/size.
Here's my basic profile for Handbrake:
Profile-> High profile (MP4)
- anamorphic-> None
- Keep Aspect Ratio
- Cropping (as applicable)
- FILTER-> usually only Decomb for anything interlaced like DVD-Video, or some deblocking for really bad quality anime/cartoon otherwise not used
- Variable framerate (almost always)
- 2-pass encoding
- Bitrate kbps (typical 800 to 2000 for anime/cartoon; 5000 to 10,000 for HD video)
- Optimize Video-> SLOW or MEDIUM (slow for final BluRay compression)
*(the smaller the file size, the more important the Optimize setting but in general more than Medium is overkill)
- audio (as applicable. I tend to use AAC 128kbps stereo or 192kbps Surround, or uncompressed DTS for larger BluRay file)
I also use the following software:
- mkvtoolnix (mux in or remove audio/subs; or join video)
- DVD Decrypter (remove copyright on DVD)
- DVD Fab (only use for DVD or BluRay copyright removal)
- Imgburn (create ISO image of DVD folder for playback on my Western Digital media player)