TensorFlow on Multiple Vector Engines

[jklinkenberg]

Description

The regular tensorflow has the opertunity to use multiple GPUs via data parallelism techniques such as MirroredStratgey.
For NVIDIA GPUs it is sufficient to set e.g. export CUDA_VISIBLE_DEVICES=0,1 to tell TensorFlow which Device IDs to use.

After looking in the source it is possible to set VE_NODE_NUMBER but this only refers to a single Vector Engine correct?
Is there a possibility to run TensorFlow on multiple Vector Engines?

[k-ishizaka]

Yes, you are correct. Current tensorflow-ve does not support multiple VEs.

Just FYI. I know someone tried horobod to use multiple VEs with tensorflow-ve, but I have never tried it.

[keichi]

I found the tf-samples repository contains scripts for (seemingly) multi VE training. Is multi VE training now supported?

[k-ishizaka]

Thank you for positing it. It is supported in the sense that the sample worked a year ago. Current tf-ve supports one VE at once, but you can select VE number by setting VE_NODE_NUMBER. It means that you can use multiple VEs by using multiple tensorflow prcesses. The sample uses two VEs in this way with MultiWorkerMirroredStrategy. But we have tested only the sample.

Our local version of tf-ve supports multiple VEs in a single process. With this version, you can use MirroredStrategy which might be better than multi worker. If you are interested in it, I will try to make it public. But it is also not well tested.

[keichi]

Thanks for the information. Your local version supporting MirroredStrategy definitely seems easier to use, but let me try out the public version w/ MultiWorkerMirroredStrategy first and see if it works.