Repository navigation
New release please? #213
Description
Activity
I fear we have cyclic dependencies now, as I am just waiting for
flowrep/pyiron_workflowto support the latest version ofpydantic: #199Reacted by Liam HuberEither we need to migrate the flowrep<->PWD converter here, or here needs to stop demoing the conversion. I'm fine with either, or another solution, but that's what immediately presents itself to me.
Either we need to migrate the flowrep<->PWD converter here, or here needs to stop demoing the conversion. I
I let this percolate, and I think the crux of the problem is that this repo is both the repo that provides the data format and the repo that demonstrates the integration of the format with other tools. This setup fundamentally interferes with those tools depending on this repo themselves.
It hasn't come into focus as a problem until now because (afaik) none of the other tools go out of their way to interact with the data format. Now that
flowrepdoes, we wind up with the circular dependency.My recommended solution would be to break up the repo that holds the data format to which we expect others to adhere, from demonstrations of that adherence/compatibility/usage. Barring that, we don't actually need to move the flowrep<->PWD converter here, but we do need to move it out of
flowrep.So your suggestion would be to split the package, having the
models.pydistributed as standalone package with minimal dependencies and the rest would be a shared toolbox, with the interfaces for the individual packages ideally being merged into the individual workflow managers? I had a similar discussion with Sebastian before: #162 and we discussed this in #203I am generally open to this idea, we just have to make sure changes remain simple in the future. When the protocol is defined in a separate package, then we always have to update both when we implement changes to the protocol.
Yes, that's my suggestion. Except you can additionally keep the shared toolbox alongside the models if you want. The core of the problem is demoing each of the workflow managers and hoping those WfMS hold their own interfaces to the data model.
we just have to make sure changes remain simple in the future. When the protocol is defined in a separate package, then we always have to update both when we implement changes to the protocol.
I'm not sure I agree with your claim that changes must remain simple. You have a semantically versioned data model; if you make minor or patch the updates to the workflow interfaces should be straightforward; if you make big major version changes, it may be more involved to get the individual WfMS to talk to the data model again. But this is true whether those interfaces are housed here alongside the model, or separately.
If you allow/encourage individual WfMS to create their own interfaces to the data model, you'll have to accept that different tools will update to newer versions of the data model at different rates (and if you additionally have a separate, single repo demoing all the possible interfaces, those packages may delay updating the demos to the new version). On the other hand, if you keep control of the interfaces -- either here or as helper code in a demoing repo -- then you preclude those WfMS from explicitly declaring any knowledge of the data model.
I don't think there's a free lunch here: PWD can control all WfMS interfaces to itself and guarantee that it can demo the latest PWD protocol at all times, or the WfMS can be responsible for their own PWD-compliance but then PWD can't guarantee the timing of when those WfMS demonstrate the latest PWD protocol.
My downstream CI is blocked:
But
main:HEADis ready up to 2.14.0:python-workflow-definition/pyproject.toml
Line 19 in 97a244c