The GTX 770 M 3GB GDDR5 edges out the 860 when it comes to overall performance.
The GTX 860M has a 218 MHz higher core clock speed than the GeForce GTX 770M, but the GeForce GTX 770M has 40 more Texture Mapping Units than the GTX 860M. As a result, the GeForce GTX 770M exhibits a 23.7 GTexel/s better Texture Fill Rate than the GTX 860M. This still holds weight but shader performance is generally more relevant, particularly since both of these GPUs support at least DirectX 10.
The GTX 860M has a 218 MHz higher core clock speed than the GeForce GTX 770M, but the GeForce GTX 770M has 8 more Render Output Units than the GTX 860M. As a result, the GeForce GTX 770M exhibits a 3 GPixel/s better Pixel Fill Rate than the GTX 860M. However, both GPUs support DirectX 9 or above, and pixeling performance is only really relevant when comparing older cards.
The GTX 860M was released less than a year after the GeForce GTX 770M, and so they are likely to have similar driver support for optimizing performance when running the latest games.
The GeForce GTX 770M has 1024 MB more video memory than the GTX 860M, so is likely to be much better at displaying game textures at higher resolutions. This is supported by the fact that the GeForce GTX 770M also has superior memory performance overall.
The GeForce GTX 770M has 16 GB/sec greater memory bandwidth than the GTX 860M, which means that the memory performance of the GeForce GTX 770M is slightly better than the GTX 860M.
The GeForce GTX 860M 2GB has 640 Shader Processing Units and the GeForce GTX 770M has 960. However, the actual shader performance of the GTX 860M is 864. The GeForce GTX 770M having 96 better shader performance and an altogether better performance when taking into account other relevant data means that the GeForce GTX 770M delivers a noticeably smoother and more efficient experience when processing graphical data than the GTX 860M.