Delayed worker registration #2122
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Workers can now delay registration with the scheduler until they are fully ready to accept tasks. This prevents the scheduler from assigning tasks to workers that are still initializing (e.g., cold pods that need time to warm CAS connections, download container images, etc.). New LocalWorkerConfig options: - warmup_delay_secs: Delay in seconds before registering (default: 0) - verify_cas_before_register: Verify CAS health before registering - readiness_check_script: Custom script to run as readiness check - readiness_timeout_secs: Timeout for readiness checks (default: 600s) The warmup phase executes after connecting to the scheduler endpoint but before calling register_worker(). If a readiness script is configured, it will retry until success or timeout. This addresses the root cause of stalled tasks when cold workers register before being ready to execute actions.
…d-worker-registration * 'main' of github.com:TraceMachina/nativelink: Support ignorable platform properties (#2120)
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
Add example files demonstrating the new worker warmup feature: - worker_readiness_check.sh: Example readiness check script that verifies Docker daemon, disk space, CAS connectivity, and GPU availability before worker registration. - worker_with_warmup.json5: Example worker configuration showing how to use warmup_delay_secs, readiness_check_script, and readiness_timeout_secs options. These examples help users configure workers to delay registration until they are fully ready to accept tasks, preventing stalled jobs during cold pod startups.
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
Adds three new tests to verify warmup delay functionality: - warmup_delay_is_applied_before_registration: Verifies that the configured warmup delay is applied before worker registers with scheduler - readiness_check_script_success: Verifies readiness check script runs and allows registration when it returns success - readiness_check_script_with_warmup_delay: Verifies both features work together (readiness check runs first, then warmup delay) Also adds setup_local_worker_with_real_sleep() test utility that uses actual tokio::time::sleep instead of the no-op mock, enabling tests to verify timing behavior. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
Apply formatjson5 and shfmt formatting to: - worker_with_warmup.json5 - Use JSON5 style (unquoted keys, trailing commas) - worker_readiness_check.sh - Add spaces around redirects (2> /dev/null) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
MarcusSorealheis
temporarily deployed
to
production
GitHub Actions
Inactive
Collaborator
|
/build-image |
|
Image built and pushed! |
palfrey
approved these changes
Jan 28, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change allows the worker to complete initialization (warming CAS connections,
verifying disk space, etc.) before accepting tasks. This is particularly
useful for cold pod startup where initialization can take several minutes.
Fixes # (issue)
Type of change
Please delete options that aren't relevant.
not work as expected)
How Has This Been Tested?
Please also list any relevant details for your test configuration
Checklist
bazel test //...passes locallygit amendsee some docsThis change is