[Issue]: Memory access fault by GPU node-1: amdgpu: [gfxhub] page fault #204
Description
Problem Description
On Linux kernel 6.18.3 & 6.18.4 ComfyUI crashes after a little while when doing stuff, e.g. wan VAE encoding.
No failures occur after rolling back to kernel 6.17.9.
Memory access fault by GPU node-1 (Agent handle: 0x55f8b54eec30) on address 0x7f2a53193000. Reason: Page not present or supervisor privilege.
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Related: ROCm/ROCm#5051 (comment)
Operating System
Arch Linux
CPU
AMD Ryzen 7 5800X
GPU
RX 7900 GRE
ROCm Version
ROCm 7.1
ROCm Component
No response
Steps to Reproduce
Run several wan workflows
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
journal logs
Jan 09 18:55:45 alexpc kernel: gmc_v11_0_process_interrupt: 632 callbacks suppressed
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc systemd-coredump[134090]: Process 91762 (python) of user 60105 terminated abnormally with signal 6/ABRT, processing...
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc systemd[1]: Started Process Core Dump (PID 134090/UID 0).
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00801A31
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Faulty UTCL2 client ID: SDMA0 (0xd)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: MORE_FAULTS: 0x1
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: WALKER_ERROR: 0x0
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: PERMISSION_FAULTS: 0x3
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: MAPPING_ERROR: 0x0
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: RW: 0x0
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318c000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318c000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318b000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318c000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318c000 from client 10
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:8 pasid:32792)
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: Process python pid 91762 thread python pid 91762
Jan 09 18:55:45 alexpc kernel: amdgpu 0000:0b:00.0: amdgpu: in page starting at address 0x00007f2a5318d000 from client 10
Jan 09 18:57:12 alexpc systemd-coredump[134091]: [🡕] Process 91762 (python) of user 60105 dumped core.
Module /dev/zero (deleted) without build-id.
...
Module /dev/zero (deleted) without build-id.
Module libsharpyuv.817afaaa.so.0 without build-id.
Module libz.db8a765a.so.1 without build-id.
Module libwebp.efd97cc6.so.7 without build-id.
Module libjpeg.2394f5ee.so.8 without build-id.
Module libpng16.c2d532b4.so.16 without build-id.
Module librocroller.so without build-id.
Module libmagma.so without build-id.
Stack trace of thread 91764:
#0 0x00007f2ed709890c n/a (libc.so.6 + 0x9890c)
#1 0x00007f2ed703e3a0 raise (libc.so.6 + 0x3e3a0)
#2 0x00007f2ed702557a abort (libc.so.6 + 0x2557a)
#3 0x00007f2db948d33f _ZN4rocr4core7Runtime14VMFaultHandlerElPv (libhsa-runtime64.so + 0x8d33f)
#4 0x00007f2db948c0b3 _ZN4rocr4core7Runtime15AsyncEventsLoopEPv (libhsa-runtime64.so + 0x8c0b3)
#5 0x00007f2db94332cd _ZN4rocr2os16ThreadTrampolineEPv (libhsa-runtime64.so + 0x332cd)
#6 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#7 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91762:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70931b4 n/a (libc.so.6 + 0x931b4)
#3 0x00007f2ed711acb5 epoll_wait (libc.so.6 + 0x11acb5)
#4 0x00007f2ed6621981 n/a (select.cpython-313-x86_64-linux-gnu.so + 0x2981)
#5 0x00007f2ed757483d _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17483d)
#6 0x00007f2ed7648b59 PyEval_EvalCode (libpython3.13.so.1.0 + 0x248b59)
#7 0x00007f2ed768956c n/a (libpython3.13.so.1.0 + 0x28956c)
#8 0x00007f2ed768618d n/a (libpython3.13.so.1.0 + 0x28618d)
#9 0x00007f2ed76828f8 n/a (libpython3.13.so.1.0 + 0x2828f8)
#10 0x00007f2ed7682522 n/a (libpython3.13.so.1.0 + 0x282522)
#11 0x00007f2ed7682323 n/a (libpython3.13.so.1.0 + 0x282323)
#12 0x00007f2ed7680991 Py_RunMain (libpython3.13.so.1.0 + 0x280991)
#13 0x00007f2ed7635beb Py_BytesMain (libpython3.13.so.1.0 + 0x235beb)
#14 0x00007f2ed7027635 n/a (libc.so.6 + 0x27635)
#15 0x00007f2ed70276e9 __libc_start_main (libc.so.6 + 0x276e9)
#16 0x000055f8a72ba045 _start (/usr/bin/python3.13 + 0x1045)
Stack trace of thread 91779:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
#3 0x00007f2ed7095e9e pthread_cond_wait (libc.so.6 + 0x95e9e)
#4 0x00007f2c63637c7b blas_thread_server (libscipy_openblas-b75cc656.so + 0x437c7b)
#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91812:
#0 0x00007f2db94620f6 _ZN4rocr4core14BusyWaitSignal11WaitRelaxedE22hsa_signal_condition_tlm16hsa_wait_state_t (libhsa-runtime64.so + 0x620f6)
#1 0x00007f2db9461dea _ZN4rocr4core14BusyWaitSignal11WaitAcquireE22hsa_signal_condition_tlm16hsa_wait_state_t (libhsa-runtime64.so + 0x61dea)
#2 0x00007f2db9465931 _ZN4rocr3HSA25hsa_signal_wait_scacquireE12hsa_signal_s22hsa_signal_condition_tlm16hsa_wait_state_t (libhsa-runtime64.so + 0x65931)
#3 0x00007f2ed177c61c _ZN9roctracer11hsa_support6detailL34hsa_signal_wait_scacquire_callbackE12hsa_signal_s22hsa_signal_condition_tlm16hsa_wait_state_t (libroctracer64.so + 0x1161c)
#4 0x00007f2e367d73ea _ZN3amd3roc13WaitForSignalE12hsa_signal_sbb (libamdhip64.so + 0x3d73ea)
#5 0x00007f2e367e177a _ZN3amd3roc10VirtualGPU14HwQueueTracker16CpuWaitForSignalEPNS0_15ProfilingSignalE (libamdhip64.so + 0x3e177a)
#6 0x00007f2e367e44de _ZN3amd3roc10VirtualGPU21releaseGpuMemoryFenceEb (libamdhip64.so + 0x3e44de)
#7 0x00007f2e367eaa0c _ZN3amd3roc10VirtualGPU12submitMarkerERNS_6MarkerE (libamdhip64.so + 0x3eaa0c)
#8 0x00007f2e367b4946 _ZN3amd7Command7enqueueEv (libamdhip64.so + 0x3b4946)
#9 0x00007f2e367b7640 _ZN3amd9HostQueue6finishEb (libamdhip64.so + 0x3b7640)
#10 0x00007f2e36482cbe _ZN3hip6Device14SyncAllStreamsEbb (libamdhip64.so + 0x82cbe)
#11 0x00007f2e3658afbc _ZN3hip8ihipFreeEPv (libamdhip64.so + 0x18afbc)
#12 0x00007f2e36592940 _ZN3hip7hipFreeEPv (libamdhip64.so + 0x192940)
#13 0x00007f2ed34b2a24 _ZN3c103hip19HIPCachingAllocator6NativeL15uncached_deleteEPv (libc10_hip.so + 0x19a24)
#14 0x00007f2ea3bb3e76 _ZN2at6native40raw_miopen_convolution_forward_out_32bitERKNS_6TensorES3_S3_N3c108ArrayRefIlEES6_S6_lbbb (libtorch_hip.so + 0x2fb3e76)
#15 0x00007f2ea3baeef3 _ZN2at6native30miopen_convolution_forward_outERNS_9TensorArgEPKcRKS1_S6_N3c108ArrayRefIlEES9_S9_lbbb (libtorch_hip.so + 0x2faeef3)
#16 0x00007f2ea3baf287 _ZN2at6native18miopen_convolutionERKNS_6TensorES3_RKSt8optionalIS1_EN3c108ArrayRefIlEESA_SA_lbb (libtorch_hip.so + 0x2faf287)
#17 0x00007f2ea391690a _ZN2at12_GLOBAL__N_112_GLOBAL__N_132wrapper_CUDA__miopen_convolutionERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_SB_bb (libtorch_hip.so + 0x2d1690a)
#18 0x00007f2ea394077e _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_S8_RKSt8optionalIS6_ENS_8ArrayRefINS_6SymIntEEESF_SF_SE_bbEXadL_ZNS5_12_GLOBAL__N_112_GLOBAL__N_132wrapper_CUDA__miopen_convolutionES8_S8_>
#19 0x00007f2ebe8c85c7 _ZN2at4_ops18miopen_convolution4callERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_SB_bb (libtorch_cpu.so + 0x2cc85c7)
#20 0x00007f2ebdb074cf _ZN2at6native12_convolutionERKNS_6TensorES3_RKSt8optionalIS1_EN3c108ArrayRefIlEESA_SA_bSA_lbbbb (libtorch_cpu.so + 0x1f074cf)
#21 0x00007f2ebeec0d60 _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_S8_RKSt8optionalIS6_ENS_8ArrayRefINS_6SymIntEEESF_SF_bSF_SE_bbbbEXadL_ZNS5_12_GLOBAL__N_112_GLOBAL__N_147wrapper_CompositeExplicitAutograd>
#22 0x00007f2ebe516c64 _ZN2at4_ops12_convolution4callERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_bSC_SB_bbbb (libtorch_cpu.so + 0x2916c64)
#23 0x00007f2ebdb00846 _ZN2at6native11convolutionERKNS_6TensorES3_RKSt8optionalIS1_EN3c108ArrayRefIlEESA_SA_bSA_l (libtorch_cpu.so + 0x1f00846)
#24 0x00007f2ebeec08c5 _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_S8_RKSt8optionalIS6_ENS_8ArrayRefINS_6SymIntEEESF_SF_bSF_SE_EXadL_ZNS5_12_GLOBAL__N_112_GLOBAL__N_146wrapper_CompositeExplicitAutograd__co>
#25 0x00007f2ebe514049 _ZN2at4_ops11convolution4callERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_bSC_SB_ (libtorch_cpu.so + 0x2914049)
#26 0x00007f2ebdb0bb60 _ZN2at6native13conv3d_symintERKNS_6TensorES3_RKSt8optionalIS1_EN3c108ArrayRefINS8_6SymIntEEESB_SB_SA_ (libtorch_cpu.so + 0x1f0bb60)
#27 0x00007f2ebeff659e _ZN3c104impl28wrap_kernel_functor_unboxed_INS0_6detail24WrapFunctionIntoFunctor_INS_26CompileTimeFunctionPointerIFN2at6TensorERKS6_S8_RKSt8optionalIS6_ENS_8ArrayRefINS_6SymIntEEESF_SF_SE_EXadL_ZNS5_12_GLOBAL__N_112_GLOBAL__N_141wrapper_CompositeImplicitAutograd__conv3d>
#28 0x00007f2ebea63d4d _ZN2at4_ops6conv3d4callERKNS_6TensorES4_RKSt8optionalIS2_EN3c108ArrayRefINS9_6SymIntEEESC_SC_SB_ (libtorch_cpu.so + 0x2e63d4d)
#29 0x00007f2ed1e80f9d _ZN5torch8autogradL18THPVariable_conv3dEP7_objectS2_S2_ (libtorch_python.so + 0x680f9d)
#30 0x00007f2ed7594c9d n/a (libpython3.13.so.1.0 + 0x194c9d)
#31 0x00007f2ed75611db _PyObject_MakeTpCall (libpython3.13.so.1.0 + 0x1611db)
#32 0x00007f2ed7572676 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x172676)
#33 0x00007f2ed75ce575 n/a (libpython3.13.so.1.0 + 0x1ce575)
#34 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
#35 0x00007f2ed75ce575 n/a (libpython3.13.so.1.0 + 0x1ce575)
#36 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
#37 0x00007f2ed75ce575 n/a (libpython3.13.so.1.0 + 0x1ce575)
#38 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
#39 0x00007f2ed75ce575 n/a (libpython3.13.so.1.0 + 0x1ce575)
#40 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
#41 0x00007f2ed7646b01 n/a (libpython3.13.so.1.0 + 0x246b01)
#42 0x00007f2ed769920f n/a (libpython3.13.so.1.0 + 0x29920f)
#43 0x00007f2ed75611db _PyObject_MakeTpCall (libpython3.13.so.1.0 + 0x1611db)
#44 0x00007f2ed7572676 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x172676)
#45 0x00007f2ed75ce6c3 n/a (libpython3.13.so.1.0 + 0x1ce6c3)
#46 0x00007f2ed76569d6 n/a (libpython3.13.so.1.0 + 0x2569d6)
#47 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
#48 0x00007f2ed75ce6c3 n/a (libpython3.13.so.1.0 + 0x1ce6c3)
#49 0x00007f2ed76569d6 n/a (libpython3.13.so.1.0 + 0x2569d6)
#50 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
#51 0x00007f2ed7646b79 n/a (libpython3.13.so.1.0 + 0x246b79)
#52 0x00007f2ed769920f n/a (libpython3.13.so.1.0 + 0x29920f)
#53 0x00007f2ed75611db _PyObject_MakeTpCall (libpython3.13.so.1.0 + 0x1611db)
#54 0x00007f2ed757b769 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17b769)
#55 0x00007f2ed75ce6c3 n/a (libpython3.13.so.1.0 + 0x1ce6c3)
#56 0x00007f2ed76569d6 n/a (libpython3.13.so.1.0 + 0x2569d6)
#57 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
#58 0x00007f2ed75ce6c3 n/a (libpython3.13.so.1.0 + 0x1ce6c3)
#59 0x00007f2ed76569d6 n/a (libpython3.13.so.1.0 + 0x2569d6)
#60 0x00007f2ed757582f _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17582f)
#61 0x00007f2ed765d80d n/a (libpython3.13.so.1.0 + 0x25d80d)
#62 0x00007f2ed65c77d5 n/a (_asyncio.cpython-313-x86_64-linux-gnu.so + 0x57d5)
#63 0x00007f2ed65c8eb5 n/a (_asyncio.cpython-313-x86_64-linux-gnu.so + 0x6eb5)
Stack trace of thread 91962:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91964:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91965:
#0 0x00007f2ed321de6d gomp_barrier_wait_end (libgomp.so + 0x1de6d)
#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91778:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
#3 0x00007f2ed7095e9e pthread_cond_wait (libc.so.6 + 0x95e9e)
#4 0x00007f2c63637c7b blas_thread_server (libscipy_openblas-b75cc656.so + 0x437c7b)
#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91968:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91963:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91780:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
#3 0x00007f2ed7095e9e pthread_cond_wait (libc.so.6 + 0x95e9e)
#4 0x00007f2c63637c7b blas_thread_server (libscipy_openblas-b75cc656.so + 0x437c7b)
#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91967:
#0 0x00007f2ed321de66 gomp_barrier_wait_end (libgomp.so + 0x1de66)
#1 0x00007f2ed321b498 gomp_thread_start (libgomp.so + 0x1b498)
#2 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#3 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91777:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
#3 0x00007f2ed7095e9e pthread_cond_wait (libc.so.6 + 0x95e9e)
#4 0x00007f2c63637c7b blas_thread_server (libscipy_openblas-b75cc656.so + 0x437c7b)
#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91977:
#0 0x00007f2ed711670d ioctl (libc.so.6 + 0x11670d)
#1 0x00007f2db9528730 hsakmt_ioctl (libhsa-runtime64.so + 0x128730)
#2 0x00007f2db951f4e3 hsaKmtWaitOnMultipleEvents_Ext (libhsa-runtime64.so + 0x11f4e3)
#3 0x00007f2db948bc73 _ZN4rocr4core7Runtime15AsyncEventsLoopEPv (libhsa-runtime64.so + 0x8bc73)
#4 0x00007f2db94332cd _ZN4rocr2os16ThreadTrampolineEPv (libhsa-runtime64.so + 0x332cd)
#5 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#6 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91994:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
#3 0x00007f2ed709e1f6 n/a (libc.so.6 + 0x9e1f6)
#4 0x00007f2ed76cb115 _PySemaphore_Wait (libpython3.13.so.1.0 + 0x2cb115)
#5 0x00007f2ed76cafad _PyParkingLot_Park (libpython3.13.so.1.0 + 0x2cafad)
#6 0x00007f2ed76a8ec7 n/a (libpython3.13.so.1.0 + 0x2a8ec7)
#7 0x00007f2ed76a8d77 n/a (libpython3.13.so.1.0 + 0x2a8d77)
#8 0x00007f2ed759a3f2 n/a (libpython3.13.so.1.0 + 0x19a3f2)
#9 0x00007f2ed756364d PyObject_Vectorcall (libpython3.13.so.1.0 + 0x16364d)
#10 0x00007f2ed7572676 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x172676)
#11 0x00007f2ed75ce3bc n/a (libpython3.13.so.1.0 + 0x1ce3bc)
#12 0x00007f2ed76cb1d8 n/a (libpython3.13.so.1.0 + 0x2cb1d8)
#13 0x00007f2ed76cb15c n/a (libpython3.13.so.1.0 + 0x2cb15c)
#14 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#15 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 91996:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
#3 0x00007f2ed709ef08 n/a (libc.so.6 + 0x9ef08)
#4 0x00007f2ed76cb0ad _PySemaphore_Wait (libpython3.13.so.1.0 + 0x2cb0ad)
#5 0x00007f2ed76cafad _PyParkingLot_Park (libpython3.13.so.1.0 + 0x2cafad)
#6 0x00007f2ed38084b8 n/a (_queue.cpython-313-x86_64-linux-gnu.so + 0x14b8)
#7 0x00007f2ed380861a n/a (_queue.cpython-313-x86_64-linux-gnu.so + 0x161a)
#8 0x00007f2ed75e6573 n/a (libpython3.13.so.1.0 + 0x1e6573)
#9 0x00007f2ed756364d PyObject_Vectorcall (libpython3.13.so.1.0 + 0x16364d)
#10 0x00007f2ed757b769 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17b769)
#11 0x00007f2ed75ce3bc n/a (libpython3.13.so.1.0 + 0x1ce3bc)
#12 0x00007f2ed76cb1d8 n/a (libpython3.13.so.1.0 + 0x2cb1d8)
#13 0x00007f2ed76cb15c n/a (libpython3.13.so.1.0 + 0x2cb15c)
#14 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#15 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 92160:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
#3 0x00007f2ed709e1f6 n/a (libc.so.6 + 0x9e1f6)
#4 0x00007f2ed76cb115 _PySemaphore_Wait (libpython3.13.so.1.0 + 0x2cb115)
#5 0x00007f2ed76cafad _PyParkingLot_Park (libpython3.13.so.1.0 + 0x2cafad)
#6 0x00007f2ed76a8ec7 n/a (libpython3.13.so.1.0 + 0x2a8ec7)
#7 0x00007f2ed76a8d77 n/a (libpython3.13.so.1.0 + 0x2a8d77)
#8 0x00007f2ed759a3f2 n/a (libpython3.13.so.1.0 + 0x19a3f2)
#9 0x00007f2ed756364d PyObject_Vectorcall (libpython3.13.so.1.0 + 0x16364d)
#10 0x00007f2ed7572676 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x172676)
#11 0x00007f2ed75ce3bc n/a (libpython3.13.so.1.0 + 0x1ce3bc)
#12 0x00007f2ed76cb1d8 n/a (libpython3.13.so.1.0 + 0x2cb1d8)
#13 0x00007f2ed76cb15c n/a (libpython3.13.so.1.0 + 0x2cb15c)
#14 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#15 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
Stack trace of thread 130827:
#0 0x00007f2ed709f002 n/a (libc.so.6 + 0x9f002)
#1 0x00007f2ed709316c n/a (libc.so.6 + 0x9316c)
#2 0x00007f2ed70937dc n/a (libc.so.6 + 0x937dc)
#3 0x00007f2ed709ef08 n/a (libc.so.6 + 0x9ef08)
#4 0x00007f2ed76cb0ad _PySemaphore_Wait (libpython3.13.so.1.0 + 0x2cb0ad)
#5 0x00007f2ed76cafad _PyParkingLot_Park (libpython3.13.so.1.0 + 0x2cafad)
#6 0x00007f2ed38084b8 n/a (_queue.cpython-313-x86_64-linux-gnu.so + 0x14b8)
#7 0x00007f2ed380861a n/a (_queue.cpython-313-x86_64-linux-gnu.so + 0x161a)
#8 0x00007f2ed75e6573 n/a (libpython3.13.so.1.0 + 0x1e6573)
#9 0x00007f2ed756364d PyObject_Vectorcall (libpython3.13.so.1.0 + 0x16364d)
#10 0x00007f2ed757b769 _PyEval_EvalFrameDefault (libpython3.13.so.1.0 + 0x17b769)
#11 0x00007f2ed75ce3bc n/a (libpython3.13.so.1.0 + 0x1ce3bc)
#12 0x00007f2ed76cb1d8 n/a (libpython3.13.so.1.0 + 0x2cb1d8)
#13 0x00007f2ed76cb15c n/a (libpython3.13.so.1.0 + 0x2cb15c)
#14 0x00007f2ed709698b n/a (libc.so.6 + 0x9698b)
#15 0x00007f2ed711a9cc n/a (libc.so.6 + 0x11a9cc)
ELF object binary architecture: AMD x86-64