Add visited nodes tracking and parallelism bypass for subworkflow nodes

[sshardool]

Tracking issue

Fixes #6873

Why are the changes needed?

Problem: In deeply nested subworkflow executions, workflow nodes can get stuck or experience significant slowdowns due to:

Redundant evaluations: The same workflow node being evaluated multiple times within a single execution round when it's already running
Parallelism bottlenecks: Already-running workflow nodes (subworkflows/launchplans) being blocked by max parallelism limits, preventing progress on child nodes even though the parent workflow has already consumed its parallelism slot.

Impact:

• Prevents workflows from getting stuck in nested subworkflow and launchplans scenarios
• Improves execution throughput for workflows with many nested reference launchplans

What changes were proposed in this pull request?

1. Visited Nodes Tracking:

• Added VisitedNodes interface to ControlFlow in ExecutionContext
• Tracks workflow nodes using fully-qualified unique IDs (generated via common.GenerateUniqueID)
• Shared across entire execution tree (all nested subworkflows share the same ControlFlow)
• Only workflow nodes (NodeKindWorkflow) are tracked; task nodes are excluded
• Nodes are added to visited set after successful handling
• Already-visited nodes return NodeStatusRunning without re-evaluation

2. Configurable Parallelism Bypass:

• New config: node-config.bypass-parallelism-check-for-workflow-nodes (default: false)
• When enabled and max parallelism is reached:
  • Workflow nodes in Running, Queued, Succeeding, or Failing phases bypass the check
  • Nodes in NotYetStarted phase are still blocked (must wait for parallelism slot)
  • Task and array nodes remain subject to parallelism limits
• Config is logged once at executor initialization for visibility

3. Additional Fix:

• Fixed error swallowing in RecursiveNodeHandler where return status, nil should have been return status, err

How was this patch tested?

Multiple reproduction scenarios are included in the draft PR, all of which resulted in stuck workflow node. These test workflows also expose the issues with the max-parallelism handling overall.

Stuck workflows recovered immediately after deployment and newly launched flows are not stuck any more since the fix has been deployed.
Several unit tests were added as well.

Labels

Please add one or more of the following labels to categorize your PR:

• added: Flag for allowing already running subworkflow nodes to be evaluated even after max-parallel nodes have been evaluated in the current cycle - bypass-parallelism-check-for-workflow-nodes
• changed: Explicit tracking added for nodes with have been visited in the current cycle.
• fixed: Fixes the stuck subworkflow behavior in [BUG] Flyte subworkflow nodes can be stuck even after referenced launchplans executions are complete #6873, can be generalized to all node types where this issue is possible.

This is important to improve the readability of release notes.

Setup process

Example workflows that be be registered and executed with pyflyte:
https://github.com/sshardool/flyte/blob/4feb41fd0036dde78009bc9962ef269546e54f08/workflows/max-parallel/max_parallelism_workflows.py

Screenshots

Screenshots included in #6873

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

[codecov]

Codecov Report

❌ Patch coverage is 82.69231% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.99%. Comparing base (575c0af) to head (249f0f8).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
flytepropeller/pkg/controller/nodes/executor.go	78.04%	4 Missing and 5 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6875      +/-   ##
==========================================
+ Coverage   56.96%   56.99%   +0.03%     
==========================================
  Files         929      929              
  Lines       58152    58195      +43     
==========================================
+ Hits        33125    33170      +45     
+ Misses      21985    21981       -4     
- Partials     3042     3044       +2     

Flag	Coverage Δ
unittests-datacatalog	53.51% <ø> (ø)
unittests-flyteadmin	53.10% <ø> (-0.04%)	⬇️
unittests-flytecopilot	43.06% <ø> (ø)
unittests-flytectl	64.02% <ø> (+0.01%)	⬆️
unittests-flyteidl	75.71% <ø> (ø)
unittests-flyteplugins	60.13% <ø> (ø)
unittests-flytepropeller	53.79% <82.69%> (+0.16%)	⬆️
unittests-flytestdlib	63.29% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.