Typically only effects and filters are rendered using the GPU, and even then this kind of task shouldn't result in heavy GPU load. This might be why you are seeing GPU usage increase only intermittently and only to low levels.
You should still see performance gain over CPU-only rendering, though. Try previewing a clip with complex effects applied to it with GPU acceleration on vs. off. You should see higher preview framerates when GPU acceleration is on.