Can't run the train phoenix-script

[Maydaytyh]

I run the train phoenix-script and got the following error. How can I fix it?

RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

My server has 2 80G-A100.

[zhjohnchan]

Hi @Maydaytyh,

Thanks for your attention! Set --nproc_per_node to the number of GPUs, i.e., 2.

Best,
Zhihong

[Maydaytyh]

Hi @Maydaytyh,

Thanks for your attention! Set --nproc_per_node to the number of GPUs, i.e., 2.

Best, Zhihong

Thank you! It works. And when I run it on single 80G-A100, the 80G memort seems to be insufficient. How much Cuda memory does Phoenix need?

[zhangzhuoran997997]

Can you share the package version such as peft? I can't run the train phoenix-script

[Maydaytyh]

Can you share the package version such as peft? I can't run the train phoenix-script

The version of peft in my environment is 0.3.0.