Low GPU utilization

I am trying to use Lucent to visualize deep neurons, but whatever I do it seems like GPU is under-utilized:
Examining utilization via `nvidia-smi`  I see low utilization (~10%) with occasional peaks at ~50%, but never above that.
This happens both for cppn prior as well as fourier image representation.

Any suggestions?