CUBLAS_STATUS_NOT_INITIALIZED

[isabellapu]

Wondering what this issue means/if it is a known issue:

File "/u/ipu/miniconda3/envs/styledrop2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/u/ipu/jiboji-styledrop/StyleDrop-PyTorch/open_clip/transformer.py", line 332, in forward
x = r(x, attn_mask=attn_mask)
File "/u/ipu/miniconda3/envs/styledrop2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/u/ipu/miniconda3/envs/styledrop2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/u/ipu/jiboji-styledrop/StyleDrop-PyTorch/open_clip/transformer.py", line 252, in forward
x = x + self.ls_1(self.attention(self.ln_1(x), attn_mask=attn_mask))
File "/u/ipu/jiboji-styledrop/StyleDrop-PyTorch/open_clip/transformer.py", line 247, in attention
return self.attn(x, attn_mask=attn_mask)
File "/u/ipu/miniconda3/envs/styledrop2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/u/ipu/miniconda3/envs/styledrop2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/u/ipu/jiboji-styledrop/StyleDrop-PyTorch/open_clip/transformer.py", line 140, in forward
q, k, v = F.linear(x, self.in_proj_weight, self.in_proj_bias).chunk(3, dim=-1)
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)