yeh that'll probably be it tbf... the cuda drivers are specifically for scientific computing and are pretty rubbish for anything else unfortunately... even amd ones are like that :(
however a way i found around it is to just push my gpu compute envs to docker and voila (also avoids the pain of installing the drivers cos nvidia actually provides a cuda docker image) :D