VCK integration proposal

[Tomcli]

Integration Proposal

1. Implement a new module that handles creating the volumemanage for VCK
2. Insert logic to provision volumemanage resource and monitor it for completion before executing the training job workload.
3. To make it more elastic, we need to come up with some algorithm on how much data replicas we need for each job. Then create some labels/tags to allow users to reuse the same dataset volume.
4. Need to figure out a shared file storage for all the learner pods (required for many distributed learning methods) and a way to store the model results for our users.

For more details, please refer to https://github.com/IBM/FfDL/blob/vck-patch/etc/examples/vck-integration.md

[animeshsingh]

Need to figure out a shared file storage for all the learner pods (required for many distributed learning methods) and a way to store the model results for our users.

So this shared file storage, will be satisfied the PVC work? Or you explicitly need NFS under the covers?

[Tomcli]

Many distributed learning methods required shared file storage to sync with the other workers. Currently all our workers are mounted on the same input and result bucket, so we have that satisfied. However, with VCK that pulls the data to HostPath, each K8s node will have their own path for the input and result directory. So we need to figure out a shared place where we should store the result files and other files that are required to shared among all the workers.

With the PVC work, this definitely could be solved for the NFS use case because it is mounted with PV. However, for S3 or Pachyderm using VCK we still have the same issue since VCK technically create replicas in the HostPath for the files (can be from multiple sources) that you want to cache.