[BUG][2.5] Aborting jobs that use flower

[SlamanigG]

Description
When aborting a NVFlare job that uses Flower for the training logic, the abort does not interrupt the training loop (it only prevents executing the next round or evaluation).

To Reproduce
Steps to reproduce the behavior:

1. Clone the hello-fower example (branch 2.5).
2. Start POC mode.
3. Inspect what python processes are running before submitting the job.
4. Submit the flower_pt job.
5. Optional: Inspect what python processes are running during submitting the job.
6. Abort the job using abort_job.
7. Inspect what python processes are running after submitting the job. See that the CPU/GPU load does not drop after aborting the job but the training continues in the background.

Expected behavior
Training loop should be stopped by abort signal.

Environment:

• OS: macOS 13.4.1, RedHat Enterprise Linux 9.6 (Plow)
• Python Version: 3.11
• NVFlare Version: 2.5.2 ([this commit])
• Flower Version: 1.11.0rc0
• Tested on CPU

Is there some kind of interrupt signal that i can listen to in client.py and pass to my train() function to stop it when the job is aborted?

[YuanTingHsieh]

Hi @SlamanigG

I observed the same behavior using NVFLARE 2.5 with Flower 1.11.0.

The newer NVFLARE versions (2.6 and 2.7) are designed to work with Flower 1.23.0.

For NVFLARE 2.5, we needed to add the following lines:

diff --git a/nvflare/app_common/tie/executor.py b/nvflare/app_common/tie/executor.py
index f40bca989..d86ebc4f9 100644
--- a/nvflare/app_common/tie/executor.py
+++ b/nvflare/app_common/tie/executor.py
@@ -138,6 +138,9 @@ class TieExecutor(Executor):
             self._notify_client_done(Constant.EXIT_CODE_FATAL_ERROR, fl_ctx)
         elif event_type == EventType.END_RUN:
             self.abort_signal.trigger(True)
+            if self.connector:
+                self.logger.info(f"stopping connector {type(self.connector)}")
+                self.connector.stop(fl_ctx)
 
     def execute(self, task_name: str, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) -> Shareable:
         if task_name == self.configure_task_name:

This fix is already included in NVFLARE 2.6 and later.

We recommend trying the newer versions if possible. Otherwise, you can manually apply this change to 2.5.

[holgerroth]

Hi @SlamanigG, the flower team worked on some updates that might help with this issue adap/flower#6151

Would you mind checking if this issue persists with the latest version of nvflare and the flwr-nightly build?