New RunTime Error: Failing to Build Model 2 Output File (which is, in fact, present) #1325
Description
Attempting to train with input samples (pixel ZIPs) which have two layers, and so requiring equivalent models. Go to run, and in the "created resources" tab inside of "Workflow runs" I can see the output files for each model get made, at which point the job fails with this message:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "/code/Rodan/rodan/jobs/base.py", line 876, in run
).format(opt_name, outputs)
RuntimeError: The job did not produce the output file for Model 2.
{'Model 2': [{'resource_type': 'keras/model+hdf5', 'uuid': UUID('d9cad397-d397-4c4b-a1c2-d4e25cc10e6b'), 'is_list': False, 'resource_temp_path': '/tmp/tmpwa3apxba/72283b46-cb0c-4ae8-ae26-b006752b9362'}], 'Background Model': [{'resource_type': 'keras/model+hdf5', 'uuid': UUID('cf79c4fd-2354-451d-9566-2a9a0c9cc05e'), 'is_list': False, 'resource_temp_path': '/tmp/tmpwa3apxba/bd6501c6-37ab-40d9-af32-3b5bb4e5b449'}], 'Model 1': [{'resource_type': 'keras/model+hdf5', 'uuid': UUID('cb01b188-9cbf-404b-ac64-e740f106f99c'), 'is_list': False, 'resource_temp_path': '/tmp/tmpwa3apxba/107f9037-6e63-42b8-98ca-8378cbd9fd8e'}], 'Log File': [{'resource_type': 'text/plain', 'uuid': UUID('bf2b50af-d20a-4e17-a884-bbbc9631a12a'), 'is_list': False, 'resource_temp_path': '/tmp/tmpwa3apxba/a572096d-339c-48f2-8f67-882f3281f091'}]}
Even though as you can see in the image below, the resource has been successfully created.
I'm going to keep experimenting with this, and will update this thread as I learn more.