[New WfMS] Pipefunc #117

Activity

1. jan-janssen commented on May 25, 2025

   

   jan-janssen

   on May 25, 2025

   Member

   More actions

   Yes, I feel there is a raise of more light-weight workflow solutions. As part of the Exascale Computing Project (ECP) we also played around with a similar idea, which lead to the development of https://github.com/pyiron/executorlib . pipefunc as well as executorlib do not provide any specific data storage functionality, but instead primarily focus on distributing Python functions and executing those. The limitation I see with pipefunc is that they do not support MPI parallel Python functions, which was one of the features we needed in the context of the ECP. The other focus for executorlib was staying as close to the Python Standard Library as possible, following the Executor pattern https://docs.python.org/3/library/concurrent.futures.html#executor-objects which pipefunc and parsl also use as their backend. Another tool in this direction is https://github.com/facebookincubator/submitit but again it does not provide advanced features like MPI support and so on.
2. jan-janssen

   changed the title [-]Pipefunc[/-] [+][New WfMS] Pipefunc[/+] on May 25, 2025