On a linear chain of 1,200 nodes, FunctionGraph.get_upstream_nodes now works since the traversal was made iterative, but running the same graph with the default synchronous Driver.execute still fails with RecursionError. The default recursion limit (1000) is enough to break it. The task-based executor runs the same graph fine.

Current behavior

With a linear chain node_0, node_1, up to node_1199, where each node depends on the previous one, and the default recursion limit:

• fg.get_upstream_nodes(["node_1199"]) returns 1,200 nodes.
• dr.execute(["node_1199"]) raises RecursionError, with 991 frames in dfs_traverse at hamilton/execution/graph_functions.py line 208.
• fg.nodes_between("node_0", "node_1199") raises RecursionError too, in a separate recursive dfs_traverse at line 393.
• At 900 nodes all three succeed.
• With enable_dynamic_execution and SynchronousLocalTaskExecutor, execute on the 1,200-node chain returns 1200.0.

Stack Traces

  File "hamilton/execution/graph_functions.py", line 208, in dfs_traverse
    dfs_traverse(n, node_dependency_type)
  File "hamilton/execution/graph_functions.py", line 208, in dfs_traverse
    dfs_traverse(n, node_dependency_type)
  (the same frame repeats about 990 times)
RecursionError: maximum recursion depth exceeded

Steps to replicate behavior

1. pip install "apache-hamilton==1.90.0"
2. Run this script. It generates the chain with exec into a module object:

import platform
import sys
import traceback
import types
from importlib.metadata import version

from hamilton import driver
from hamilton.execution import executors

print(f"python: {platform.python_version()}")
print(f"apache-hamilton: {version('apache-hamilton')}")
print(f"recursion limit: {sys.getrecursionlimit()}")


def make_chain(n: int) -> types.ModuleType:
    """node_0() -> float, node_i(node_{i-1}: float) -> float = node_{i-1} + 1."""
    mod = types.ModuleType(f"chain_{n}")
    src = ["def node_0() -> float:\n    return 1.0\n"]
    for i in range(1, n):
        src.append(f"def node_{i}(node_{i - 1}: float) -> float:\n    return node_{i - 1} + 1\n")
    exec("\n".join(src), mod.__dict__)
    sys.modules[mod.__name__] = mod
    return mod


def deepest_frame(exc: BaseException) -> str:
    frames = traceback.extract_tb(exc.__traceback__)
    counts = {}
    for f in frames:
        key = (f.filename.rsplit("/", 1)[-1], f.lineno, f.name)
        counts[key] = counts.get(key, 0) + 1
    (fname, lineno, func), count = max(counts.items(), key=lambda kv: kv[1])
    return f"{count} frames at {fname}:{lineno} in {func}"


for n in (900, 1200):
    last = f"node_{n - 1}"
    dr = driver.Builder().with_modules(make_chain(n)).build()
    fg = dr.graph

    up, _ = fg.get_upstream_nodes([last])
    print(f"[{n}] fg.get_upstream_nodes: OK, {len(up)} nodes")

    for label, call in (
        ("dr.execute", lambda: dr.execute([last])[last]),
        ("fg.nodes_between", lambda: len(fg.nodes_between("node_0", last))),
    ):
        try:
            print(f"[{n}] {label}: OK, {call()}")
        except RecursionError as e:
            print(f"[{n}] {label}: RecursionError ({deepest_frame(e)})")

mod = make_chain(1200)
dr_tasks = (
    driver.Builder()
    .with_modules(mod)
    .enable_dynamic_execution(allow_experimental_mode=True)
    .with_local_executor(executors.SynchronousLocalTaskExecutor())
    .build()
)
print("[1200] task-based dr.execute: OK,", dr_tasks.execute(["node_1199"])["node_1199"])

print("expected: dr.execute returns 1200.0 for the 1,200-node chain")

Output (the Hamilton slack banner printed on errors is removed):

python: 3.12.3
apache-hamilton: 1.90.0
recursion limit: 1000

[900] fg.get_upstream_nodes: OK, 900 nodes
[900] dr.execute: OK, 900.0
[900] fg.nodes_between: OK, 900
[1200] fg.get_upstream_nodes: OK, 1200 nodes
[1200] dr.execute: RecursionError (991 frames at graph_functions.py:208 in dfs_traverse)
[1200] fg.nodes_between: RecursionError (993 frames at graph_functions.py:393 in dfs_traverse)
[1200] task-based dr.execute: OK, 1200.0

expected: dr.execute returns 1200.0 for the 1,200-node chain

Library & System Information

• python 3.12.3
• apache-hamilton 1.90.0
• Linux (x86_64)

graph_functions.py is the same on main at fa786ea.

Expected behavior

dr.execute(["node_1199"]) on the sync Driver returns 1200.0, the same as the task-based executor, without needing sys.setrecursionlimit.

Additional context

Cause, with lines at main fa786ea: the sync Driver reaches graph_functions.execute_subdag through FunctionGraph.execute (hamilton/graph.py line 1156). execute_subdag computes nodes with a nested recursive dfs_traverse (hamilton/execution/graph_functions.py lines 197-208) that calls itself once per dependency level, so a chain deeper than the recursion limit fails. The iterative rewrite in #1483 changed FunctionGraph.directional_dfs_traverse, which backs get_upstream_nodes / get_downstream_nodes, but not this function, and its regression test calls get_upstream_nodes directly, so it passes while execute on the same graph fails. graph_functions.nodes_between (lines 353-393, used by FunctionGraph.nodes_between and execution/grouping.py line 216) has the same recursive shape.

Proposed fix: rewrite execute_subdag's dfs_traverse as an explicit-stack post-order walk that keeps the per-edge dependency_type and the existing computed and overrides short-circuits, and do the same for nodes_between. Then add a test that runs Driver.execute on a chain longer than the recursion limit.

I'm happy to take this one and will open a PR with the fix and the test.