Computational path resolution

[thorwhalen]

The original idea of meshed is to offer tools to combine python objects to create new ones. Of course, pipelines (function composition) and DAGs (generalization thereof) are examples of this, and meshed offers not much more than this at the time of writing this. It's intention though, is to offer much more: Including making classes (such as bind_func_object_attrs in i2), caching, etc.

An underlying problem to enable the destiny of meshed is "path finding" or more precisely "computational path resolution".
The problem is related to DAG slicing. Given a dag and a tuple of input and output identifiers, dag[inputs:outputs] is a dag that gets us from the desired inputs to the desired outputs -- more precisely, it's a callable that let's us compute outputs from the given inputs.

This is a special case of computational path resolution but for a DAG. We want the same but for any collection of objects (along with connectivity information). One instance of this would be if all these objects are DAGs themselves, sharing some identifiers...

For this particular instance, one might think of providing an interface such as get_result(src, src_type) or get_result(src) where src would act as wf, filepath or mic_id and get_result would do what is necessary to get the result.
This interface is a good one, but that's not the question here: The question is what the architecture of get_result itself is. How can we cleanly "create" it from the pieces we have (i.e. the function that gets results from wf, the one from filepath to wf, and the one from mic_id to wf).

Verify architecture ideas against the following considerations:

• Is it open-closed? How do you extend get_result to handle a different source, or enable the user to extend?
• We have a simple pipeline here, but how do we do the same when we if we had several kinds of inputs and targeted outputs and options to link both sides?
• What if a different kind of interface of get_result is warranted. For example, a Model class that might have three methods: wf_to_result, filepath_to_result and mic_id_to_result? Can we reuse the same combinator tools we used for the get_result(src, src_type) function?

Note the relationship with

• routing
• functools.singledispatch (or more like multipledispatch

Proposal

In a nutshell, we want a mk_func so that, given a collection of objs (that somehow contains information on how these can be combined to create computational paths) and a specification of what the inputs and outputs should be,

f = mk_func(objs, inputs, outputs)

is a function that uses objs internally to computationally connect the said inputs and outputs.

These input and output specifications can be expressed as identifiers, or types, or whatever.

An instance of this is where the objs are all DAGs that share some identifiers. We use these identifiers to identifier what combination of these DAGs (or possibly sub-dag of these) we need to connect the desired inputs to the outputs.

Appendix: DAGs used above

from meshed import code_to_dag
@code_to_dag
def model_pipe():
    result = model(wf)
    
@code_to_dag
def filepath_to_wf():
    wf = get_wf_from_filepath(filepath)
    
@code_to_dag
def mic_to_wf():
    wf = get_wf_from_mic(mic_id)

from meshed.util import incremental_suffixes as suff

(
    filepath_to_wf.copy(next(suff)) + 
    mic_to_wf.copy(next(suff)) + 
    model_pipe.copy(next(suff))
).dot_digraph()

(model_pipe + from_filepath).dot_digraph()

(model_pipe + from_mic).dot_digraph()

[thorwhalen]

This is another thing I wrote about it, in a section of the Ideas wiki, having forgotten that I already had devoted a whole wiki to it.

Path finders

This digraph doesn't represent a DAG computation. In fact, it's not a DAG since it has a directed cycle!

Instead here each node represents a variable kind and an edge represents a function that can transform/compute one kind from another.

The general object illustrated here is one that relates sets of variable kinds to other ones.
(Note often functions require more than one variable, but we don't represent this case here, for simplicity.)

The intended use of such an object is to allow a user to get one set of variables from another set of variables without having to manually specify how to do so. For example, a user can specify that they need to have the wfs corresponding to a specific folder_path, and the object will be able to find a computational path to do so. It may be that there's several ways (such as is the case here) to do so, in which case, some rule will decide which path to offer. This rule could for example assume that the path with the least edges must be more efficient and use that.

Note as well that this object can have cycles: Here, we may want to save some wfs as files of a folder_path folder. Our current object enables us to do that too.

One typical application of such path finder object arises when we want to overload a function to "Postelize" it, enabling it to handle several types of inputs. For example, our function could have a wfs input, waveforms which the function expects to be given as a list of lists.

• If we wanted to enable the user to specify a folder containing .wav files that should be used to create these waveforms, we could include some if... then... transform code in the function to handle this.
• Better would be to bring this code outside the main function, and use it inside that main function to do our bidding.
• Even more elegant, and reusable, would be to write a decorator that can be applied to any function using wfs (recognized by name and/or type, say).
• Our path finder object brings it yet to another level, allowing us to encapsulate not only this logic from folder_path to wfs, but from many different intertwined variable kinds.

[thorwhalen]

Known applications

• Postelizing function arguments. For example, in ZipReader's init, but there are many more of these.
• StringTemplate -- the new StrTupleDict. You can clearly see the attempt to codify the combinations of the four different types in str_template_key_trans which uses the _method_names_for_path_type