Add RayCluster support for DGX Cloud Lepton #389
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add support for launching a RayCluster on DGX Cloud Lepton and submitting RayJobs on the clusters using the lepton SDK. This uses the new RayCluster feature on DGX Cloud Lepton to dynamically spawn clusters up and down via the Python SDK and jobs can be submitted to deployed clusters directly. Signed-Off-By: Robert Clark <roclark@nvidia.com>
Signed-off-by: Zoey Zhang <zozhang@nvidia.com>
… RayCluster and linting Signed-off-by: Zoey Zhang <zozhang@nvidia.com>
Removed the placeholder Slurm packager handling comments from the Lepton RayCluster code. For now, the "workdir" parameter should be used for transferring local data to the remote Ray cluster. Signed-Off-By: Robert Clark <roclark@nvidia.com>
Fix issue to ensure the proper head node resource shape is used if it isn't explicitly given by the user. Signed-Off-By: Robert Clark <roclark@nvidia.com>
Updated the comments in the LeptonRayCluster and LeptonRayJob classes to accurately reflect the code. Signed-Off-By: Robert Clark <roclark@nvidia.com>
The RayJob logs stream would sometimes timeout and reset, causing a very long output of logs in the terminal as it continually resets. Signed-Off-By: Robert Clark <roclark@nvidia.com>
The head node resource shape for a LeptonRayCluster should be optional. If it isn't specified by the user, it should default to the same shape used for the worker nodes. Signed-Off-By: Robert Clark <roclark@nvidia.com>
Added an example to the Ray quick-start guide on how to use RayClusters and RayJobs with NeMo-Run on DGX Cloud Lepton. Signed-Off-By: Robert Clark <roclark@nvidia.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-Off-By: Robert Clark <roclark@nvidia.com>
zoeyz101
requested changes
Nov 14, 2025
Contributor
zoeyz101
left a comment
zoeyz101
left a comment
There was a problem hiding this comment.
A couple small changes requested and some minor questions
| | `run.ray.job.RayJob` | Lifecycle of a Ray **job** (submit ⇒ monitor ⇢ logs ⇢ cancel). | same | | ||
|
|
||
| The two helpers share a uniform API; the chosen *Executor* decides whether we talk to the **KubeRay** operator (K8s) or a **Slurm** job under the hood. | ||
| The two helpers share a uniform API; the chosen *Executor* decides whether we talk to the **KubeRay** operator (K8s), **DGX Cloud Lepton's RayCluster**, or a **Slurm** job under the hood. |
Contributor
There was a problem hiding this comment.
three instead of two helper and I would say three helper executors
Contributor
Author
There was a problem hiding this comment.
I interpreted the helpers being the RayCluster and RayJob objects with the different executors being the backends.
Contributor
There was a problem hiding this comment.
I interpreted this as the helpers being the executors. @hemildesai can you confirm? RayJob and RayCluster do not have the same API
Signed-Off-By: Robert Clark <roclark@nvidia.com>
Allows users to specify how long to wait for a RayCluster to be created on DGX Cloud Lepton. Signed-Off-By: Robert Clark <roclark@nvidia.com>
Signed-Off-By: Robert Clark <roclark@nvidia.com>
Signed-Off-By: Robert Clark <roclark@nvidia.com>
Signed-Off-By: Robert Clark <roclark@nvidia.com>
Signed-Off-By: Robert Clark <roclark@nvidia.com>
roclark
force-pushed
the
roclark/leptonraycluster
branch
from
65a6cbe to
c45e6b0
Compare
roclark
force-pushed
the
roclark/leptonraycluster
branch
from
c45e6b0 to
b0a22a8
Compare
roclark
force-pushed
the
roclark/leptonraycluster
branch
from
b0a22a8 to
9991173
Compare
roclark
force-pushed
the
roclark/leptonraycluster
branch
from
9991173 to
7252cb1
Compare
roclark
force-pushed
the
roclark/leptonraycluster
branch
from
7252cb1 to
34b93cc
Compare
roclark
force-pushed
the
roclark/leptonraycluster
branch
from
34b93cc to
b04d893
Compare
roclark
force-pushed
the
roclark/leptonraycluster
branch
from
b04d893 to
f637645
Compare
Signed-Off-By: Robert Clark <roclark@nvidia.com>
roclark
force-pushed
the
roclark/leptonraycluster
branch
from
f637645 to
5cec02e
Compare
roclark
commented
Nov 25, 2025
Contributor
Author
There was a problem hiding this comment.
@hemildesai - I updated the pyproject.toml file as this update requires a newer minimum version of leptonai and it looked like the ray packages were missing from extras. Do you prefer that I update the uv.lock file with this update, or is there a mechanism to update it automatically when pyproject.toml is updated?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added
LeptonRayClusterandLeptonRayJobclasses backed by theRayClusterandRayJobhelpers, respectively, to launch Ray jobs and clusters using the RayCluster feature on DGX Cloud Lepton. Developers can now launch a RayCluster and RayJobs via the SDK remotely on their DGX Cloud Lepton node groups.