Output files only saved from first task

[gcapes]

Considering the advanced_workflow.yaml workflow, it runs with the latest matflow, but the point of the workflow was to show the output is the same from the input file generator, as that from the second task.

However despite using save_files for every task, I only get a set of files saved, in artifacts/submissions/tasks/t_0. There's nothing for the second task i.e. artifacts/submissions/tasks/t_1 doesn't exist, but artifacts/submissions/0/tmp contains input_file.json and processed_file.json that I was expecting in artifacts/submissions/tasks/t_1.

[gcapes]

@aplowman Would you take a look?

[gcapes]

Check whether this is still the case using the latest matflow

[gcapes]

@aplowman It is still the case.

I can get the intended result for the second task using requires_dir: true, but don't really understand why this should be necessary given I've used e.g. save_files: [my_input_file]. It seems that without requires_dir: true, the files only get written to the submissions/tmp directory, which I guess is fine, but there is something extra going on for the input_file_generator and output_file_parser shortcuts. Files created using an input file generator or output file parser are written to theexecute directory (and presumably the submissions/tmp dir too), and using save_files copies them into the tasks directory too.

It looks like what is happening for a standard action is that save_files copies a file (if it exists) from execute to tasks but fails silently if the file doesn't exist. This means that requires_dir: true is a prerequisite for copying the file from tmp to execute(? )

From a user perspective this is counter-intuitive, and it would be useful if either the action would fail in this scenario, or better, that save_files had an implicit requires_dir: true baked into it.