Implement `generate_volume_element_from_statistics_dual_phase_orientations_Dream3D` task schema

[gcapes]

Old task schema
Old Dream3D script

Files from Aiden's workflow are here: #427 (comment)

[gcapes]

I'm not clear how to do this. The old schema has

name: orientations
  context: phase_1
  alias: orientations_phase_1

so given there are two orientations, I'm not sure how to handle it - basically #428 again.

I also don't know whether I can use script:dream_3D/generate_volume_element_statistics.py, given there are two orientations.

[gcapes]

Demo of one way to handle this is given here: 9d70a28

[gcapes]

What is an input_map and an output_map? These look like input_file_generators and output_file_parsers, so that's how I'm going to attempt this :)

[gcapes]

I think I have the structure of the task schema now, with placeholders for three processing scripts.

[gcapes]

Hi @aplowman,
I've given this a good go here: 3bfbfda, however I'm getting an error I've not managed to figure out:

(.venv) cipher-workflows $ matflow demo-workflow go cipher-aiden
Traceback (most recent call last):
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/bin/matflow", line 8, in <module>
    sys.exit(cli())
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/demo/cli.py", line 192, in make_and_submit_demo_workflow
    out = app.make_and_submit_demo_workflow(
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/app.py", line 1641, in <lambda>
    return lambda *args, **kwargs: func(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/app.py", line 3116, in _make_and_submit_demo_workflow
    wk = self._make_demo_workflow(
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/app.py", line 3011, in _make_demo_workflow
    wk = self.Workflow.from_file(
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/workflow.py", line 1304, in from_file
    template = cls._app.WorkflowTemplate.from_file(
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/workflow.py", line 702, in from_file
    return cls.from_YAML_file(path_, variables=variables)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/workflow.py", line 628, in from_YAML_file
    return cls._from_data(data)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/workflow.py", line 552, in _from_data
    wkt = cls.from_json_like(data, shared_data=cls._app._shared_data)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/app.py", line 1694, in _shared_data
    return cast("Mapping[str, Any]", self.template_components)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/app.py", line 1688, in template_components
    self._load_template_components()
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/app.py", line 1778, in _load_template_components
    ts_list = self.TaskSchemasList.from_json_like(schemas, shared_data=self_tc)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/object_list.py", line 474, in from_json_like
    return super().from_json_like(json_like, shared_data=shared_data)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/json_like.py", line 431, in from_json_like
    return cls._from_json_like(copy.deepcopy(json_like), shared_data)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/json_like.py", line 616, in _from_json_like
    from_json_like_item(
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/json_like.py", line 570, in from_json_like_item
    out.extend(
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/json_like.py", line 574, in <genexpr>
    else chd_cls.from_json_like(
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/json_like.py", line 431, in from_json_like
    return cls._from_json_like(copy.deepcopy(json_like), shared_data)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcfhttps://github.com/hpcflow/matflow-new/blob/develop/matflow/data/workflows/fit_yield_funcs.yaml#L83C9-L90C39low/sdk/core/json_like.py", line 629, in _from_json_like
    obj = cls(**json_like_copy)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/task_schema.py", line 177, in __init__
    self._validate()
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/hpcflow/sdk/core/task_schema.py", line 746, in _validate
    raise ValueError(
ValueError: Action 0 input 'orientations' of schema 'generate_volume_element_from_statistics_dual_phase_orientations_Dream3D' is not a schema input, but nor is it an action output from a preceding action.

I thought it might have been the different sytax for referencing the inputs from specific tasks, but I tried it like this (syntax copied from https://github.com/hpcflow/matflow-new/blob/develop/matflow/data/workflows/fit_yield_funcs.yaml#L83C9-L90C39) and got the same error 😆

  - schema: generate_volume_element_from_statistics_dual_phase_orientations_Dream3D
    input_sources:
      orientations[phase_1]:
      - source_type: task
        task_ref: task.sample_texture_from_model_ODF_mtex_1
      orientations[phase_2]:
      - source_type: task
        task_ref: task.sample_texture_from_model_ODF_mtex_2

Any pointers?

[gcapes]

This looks to have been fixed in hpcflow-new#881

[gcapes]

Now fixing bugs in my implementations of the task schemas.

[gcapes]

@aplowman
I've just pushed my latest code to software/cipher, and have fixed some errors, but I'm getting this one I can't figure out in /artifacts/submissions/0/js_std/2/js_2_std.log:

: 81%
: [9/10] Generate IPF Colors 
: 90%
: [10/10] Write DREAM.3D Data File 
: Pipeline Complete
Traceback (most recent call last):
  File "/home/mbexegc2/projects/lightform/cipher-workflows/cipher-aiden_2025-10-28_151040/artifacts/submissions/0/scripts/generate_volume_element_from_statistics_dual_phase_orientations_Dream3D_act_3.py", line 141, in <module>
    output = parse_dream_3D_volume_element_from_stats(**func_kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/cipher-aiden_2025-10-28_151040/artifacts/submissions/0/scripts/generate_volume_element_from_statistics_dual_phase_orientations_Dream3D_act_3.py", line 14, in parse_dream_3D_volume_element_from_stats
    orientations_phase_1 = orientations["phase_1"]
TypeError: 'NoneType' object is not subscriptable

I guess it's something to do with the syntax around the multiple orientations?
I'm working with the demo-workflow cipher-aiden, which clearly needs renaming to something more meaningful at some point :)

[gcapes]

The problem above was fixed by hpcflow/hpcflow#884

Next problem :)

(.venv) cipher-workflows $ tail cipher-aiden_2025-10-29_102522/artifacts/submissions/0/js_std/2/js_2_std.log 
       [ 0.11248086, -0.27806147,  0.37088035, -0.87890707],
       [ 0.05853553,  0.54365831,  0.70436877,  0.45262995]]), unit_cell_alignment=UnitCellAlignment(x=<LatticeDirection.A: 0>, y=<LatticeDirection.B: 1>, z=<LatticeDirection.C: 2>), representation=OrientationRepresentation(type=<OrientationRepresentationType.QUATERNION: 0>, euler_definition=None, euler_is_degrees=None, quat_order=<QuatOrder.SCALAR_VECTOR: 0>))
Traceback (most recent call last):
  File "/home/mbexegc2/projects/lightform/cipher-workflows/cipher-aiden_2025-10-29_102522/artifacts/submissions/0/scripts/generate_volume_element_from_statistics_dual_phase_orientations_Dream3D_act_3.py", line 141, in <module>
    output = parse_dream_3D_volume_element_from_stats(**func_kwargs)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/cipher-aiden_2025-10-29_102522/artifacts/submissions/0/scripts/generate_volume_element_from_statistics_dual_phase_orientations_Dream3D_act_3.py", line 40, in parse_dream_3D_volume_element_from_stats
    ori_1 = validate_orientations(orientations_phase_1)
  File "/home/mbexegc2/projects/lightform/cipher-workflows/.venv/lib/python3.10/site-packages/damask_parse/utils.py", line 983, in validate_orientations
    ori_type = orientations.get('type')
AttributeError: 'Orientations' object has no attribute 'get'

The validate_orientations function looks to still work ok in other task schemas, so is there a problem with how the orientations are created/passed to this function?

[aplowman]

You'll probably have to use this function: https://github.com/hpcflow/matflow-new/blob/131f691aa238232dccbdec53bb41a27974fa52c4/matflow/data/scripts/dream_3D/generate_volume_element_statistics.py#L1033