Exiting python process leaves Nextflow running

[bmyers-ozette]

We have a workflow that allows a user to cancel their running Nextflow job from our Python application which is using the nextflowpy library. However, we have noticed that despite the Python process exiting, Nextflow continues to run.

I had a couple thoughts for possible solutions:

1. Return the PID of the nextflow process in the Execution object, allowing us to handle the process termination.
2. Handle process termination within nextflowpy so that catches the exit signal and ensure the Nextflow process is also terminated, keeping all process management encapsulated within the library.

Would love to hear your thoughts on this or if there's an existing approach we might have missed. Thanks!

[samirelanduk]

We had this issue also for a web application which uses nextflow.py. We came to a similar solution, but ultimately I decided to add this logic to the web application, rather than here in the nextflow.py layer, as there are too wide a variety of ways to cancel depending on the executor used - local, slurm, AWS etc. This could be revisited one day, especially as we already support custom ways of submitting executions with custom runners.