How to represent a for loop

@liamhuber

It's closely related to #24, but since it's getting pretty long, I open a new issue.

Let's talk about the following workflow:

def my_workflow(n):
    for ii in my_list(n):
        n = my_function(ii)
    return n

Now, following the logic from #24, there would be an edge from inputs.n to my_list, then my_list to my_function, finally my_function to output. Now, however, there are two significant differences:

The output of my_function is not the input of my_list, as my_list is evaluated only once when n is inserted
ii is not the output of my_list, but it's each element of the output of my_list.

So I'm feeling that maybe our edge representation is not good enough to cover the for loop. What do you think? @liamhuber

@liamhuber

The output of my_function is not the input of my_list, as my_list is evaluated only once when n is inserted

Agreed, of course. The cyclicity against which you're contrasting was information explicitly carried in 'type': 'while', I wouldn't generally expect my_list to get called again or for my_function to be able to impact it.

2. ii is not the output of my_list, but it's each element of the output of my_list.

Analogously, I think this is what is carried by 'type': 'for'.

So I'm feeling that maybe our edge representation is not good enough to cover the for loop. What do you think? @liamhuber

I can't say the edge representation is necessarily optimal, but I do think in light of the above, it's sufficient: 'while' carries information to modify the edges to add cyclicity, 'for' carries information to modify the edges to access elements (and be directed to copies/re-invocations of the body node(s)).

I do rather start to worry that while all of this has the pro that we can write natural-ish python, it is only natural-ish.
We insist that all non-declarative actions be wrapped in functions, we support some but not all operations (here I think of zip, later unpacking **kwargs terrifies me, and we haven't handled operators (even simple + or ==) at all.

I'm not saying we should stop now or abandon this line of attack, I'm just saying my philosophical concern of whether it's easier to learn something new from scratch or unlearn parts of something you already know is starting to rear its ugly head again...

I'm working on a data model for the for nodes (#85), and IMO it's not too bad. However, I'm really struggling to figure out how we're going to parse python into it and have the workflow function still produce the same result when you just run it as a function. This last constraint is a key part of the idea that the universal decorators shouldn't actually modify the function.

Take your initial example (fleshed out with the functions it calls, and not reusing the input variable):

def my_list(n):
    return list(range(n))

def my_function(x):
    return x**2

def my_workflow(n):
    for ii in my_list(n):
        z = my_function(ii)
    return z

My understanding is that from a workflow perspective, we want an n=3 invocation to give us back the list [0, 1, 4]. But of course if we directly run my_workflow(3) we just get 4.

I can see how we use ast to infer that z is listlike because of the scope in which it's declared -- i.e. from that python definition we can build the "right" recipe model -- but I don't understand how to keep the raw function and recipe synchronized without expanding the parsing tool to understand list appending in some way such that we can directly write something like

@flowrep.workflow
def my_workflow(n):
    z = []
    for ii in my_list(n):
        z.append(my_function(ii))
    return z

You mean something like this?

def create_list():
    return []

def append(new_value, my_lst):
    return my_lst + [new_value]

@flowrep.workflow
def my_workflow(n):
    z = create_list()
    for ii in my_list(n):
        x = my_function(ii)
        z = append(x, z)
    return z

I think that part of the reason I'm struggling with this is that I've been thinking about this as a workflow system first, and python only second. In my perspective, for a WfMS it should be perfectly possible to have a graph where the root-most node is a for-node. E.g.,

from flowrep import model

def my_function(x):
    return x**2

atomic_recipe = {
    "type": "atomic",
    "inputs": ["x"],
    "outputs": ["squared"],
    "fully_qualified_name": "__main__.my_function",
    "unpack_mode": "tuple",
}

for_recipe = model.ForNode.model_validate(
    {
        "type": "for",
        "inputs": ["xs"],
        "outputs": ["squared_xs"],
        "body_node": {
            "label": "square_node",
            "node": atomic_recipe,
        },
        "input_edges": {"square_node.x": "xs"},
        "output_edges": {"squared_xs": "square_node.squared"},
        "nested_ports": ["x"],
    }
)

I should be able to pass that for_recipe to my WfMS, instantiate a "living" node out of it, and if I pass that node the xs input [0, 1, 2], then I expect to get back from that the squared_xs output [0, 1, 4].

In fact, I think I'm actually more extreme than this. In principle, the WfMS itself can be non-python. As long as it has the capability to execute and exchange data with the python "atomic" nodes, the WfMS can be written in what ever it wants. Even more extremely, one could consider that our AtomicNode is actually a python AtomicNode, and in principle we could have atomic nodes in any language with which the WfMS is capable of liaising, and the flowrep recipes should and will all continue to be consistent and valid.

Now, to recant a bit of my heresy, I'm not actually clear that this flexibility persists -- or at least not in any remotely easy way -- when we get over to semantikon and start using python objects to validate both data type and ontological type connections. And further, I am still fully on-board with the goal that we want to let users write new workflows in a python(-like) syntax. So I'm not in any way suggesting a shift away from python, just pointing out that (outside the atomic nodes) the recipes are actually python-agnostic and just need to fulfill procedural programming robustly.

Ok, but let's be practical and come back to the python. If we actually want to parse a for-loop from python, we're going to need to have it inside a function definition, and thus the for-loop won't be the top-level node, but some static macro will (a WorkflowNode).

From the perspective of defining a recipe, we can accomplish this straightforwardly:

from flowrep import parser

@parser.atomic("ns")
def my_list(n):
    return list(range(n))

workflow_recipe = model.WorkflowNode.model_validate(
    {
        "type": "workflow",
        "inputs": ["n"],
        "outputs": ["squared_values"],
        "nodes": {
            "n_items": my_list.flowrep_recipe,
            "square_loop": for_recipe,
        },
        "input_edges": {"n_items.n": "n"},
        "edges": {"square_loop.xs": "n_items.ns"},
        "output_edges": {"squared_values": "square_loop.xs"},
    }
)

This is then a perfectly unambiguous for-loop specification, and we could easily turn it into a python execution. Indeed, it's not dissimilar to how for-loops are specified in pyiron_workflow, so I know for a fact it's doable and pretty easy.

What's difficult is then the converse problem; how do we write a python function that gets parsed back into such a simple recipe format? I.e. what do we put in the function body here, and how do we put constraints on what the parser-valid pseudo-python can be to guarantee a cleanly parsed result -- Such that my_loop_workflow.flowrep_recipe: WorkflowNode from below is the same as what is written above (or at least functionally equivalent -- I don't care if node labels change or whatever).

@parser.workflow("squared_values")
def my_loop_workflow(n):
    ...

I can easily write python that behaves the same way, such that the raw function call gives back the same thing I expect from the workflow:

# @parser.workflow("squared_values")
def my_loop_workflow(n):
    list_data = my_list(n)

    loop_output_collector = []
    for x in list_data:
        xsq = my_function(x)
        loop_output_collector.append(xsq)

    return loop_output_collector

my_loop_workflow(3)
>>> [0, 1, 4]

But I don't know if it's reasonable to ast-parse this into a recipe like above. I'm not worried about name changes, e.g. node labels being auto-generated like "n_items" --> "my_list_0", "square_loop" --> "for_0", "square_node" --> "my_function_0", nor internal edge specifications getting different names, e.g. the difference in names for the output_edges of the for-recipe "squared_xs": "square_node.squared" --> "loop_output_collector": "my_function_0.squared". What I hope to get here is just a recipe with matching topology and terminal IO. I am worried about whether it will be hard/ugly/impossible to identify new symbols assigned to an empty list, and to track those symbols being appended to inside a given loop. Will we be able to fail cleanly if someone re-uses the same declared variable in separate (non-nested!) loop? Even if all this is possible, is there some much clearer and more clever way to constrain our pseudo-python?

Basically, I find the recipe model to be succinct, unambiguous, robust, and sufficiently readable, but I don't want to shoot us in the foot by making a lovely recipe format that is all but impossible to parse from a python function.

Sorry I still don't really understand where the problem is. I presume you are saying my representation does not suffice, but I don't see what makes you think so.

Indeed, that is my claim. I'm saying that if I used the workflow you linked as a node in a workflow in a WfMS, I would want the output of that macro to be a list of data; if I just run that function as a function, I get back a single data value. I think it would be really helpful if the functions-decorated-as-recipes could always just be run directly as python functions¹ and that they should return the same result as if we use the recipe in a WfMS

admittedly losing all the provenance, retrospection, ability to easily ship components off to executors, etc. ↩

Hm I guess I know what you mean but I still don't really see the problem. It's true that the body consists of two steps: execution of a single node and appending of the results (at least in this case). This single node is defined in the way you described, so there's nothing wrong about it?

Ah, that's because apparently I didn't actually read the linked function. Yes, it appends and I would expect the workflow and raw function call outputs to be identical.

I still see an asymmetry that in the WfMS we are free to write the loop directly, but if we want to parse a loop node from a python definition, that loop node always exists inside a static macro. But this is actually the case parsing all the flow controls. When parsing via a decorated python function, it is trivially(?) true that these flow controls all need to occur inside the context of a static workflow.

So looking at it again, I think this is fine:

@flowrep.workflow
def my_workflow(n):
    z = create_list()
    for ii in my_list(n):
        x = my_function(ii)
        z = append(x, z)
    return z

It creates a ForNode inside a WorkflowNode, where a WfMS would in principle be able to do the for-loop in isolation, but my_workflow(5) and wfms.WfMS(my_workflow.flowrep_recipe).run(n=5) (or whatever) should indeed be producing identical output.

For very practical reasons of making the parsing easier to implement and more robust, I still think it's possible that z = create_list() will be tough, and that we'll want to follow the pattern of search for ...; my_var = []; ...; for ...; my_var.append(...); .... But that's just a practical consideration that the function quoted above might be hard to parse, not a philosophical comment that parsing it would be bad.

Navigation Menu

Repository navigation

How to represent a for loop #23

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

How to represent a for loop #23

Description

Activity

liamhuber commented on May 23, 2025

liamhuber commented on May 23, 2025

liamhuber commented on Jan 20, 2026

samwaseda commented on Jan 21, 2026

liamhuber commented on Jan 21, 2026

samwaseda commented on Jan 26, 2026

liamhuber commented on Jan 27, 2026

Footnotes

samwaseda commented on Jan 27, 2026

liamhuber commented on Jan 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions