.NET: Python: [Bug]: Declarative custom-function regexes swallow the formula tail, silently emitting formula source or failing compilation #9173

Description

DeclarativeWorkflowState._eval_custom_function matches its custom-function dialect with regexes anchored to the end of the whole formula:

# python/packages/declarative/agent_framework_declarative/_workflows/_declarative_base.py
match = re.match(r"(?:Concat|Concatenate)\((.+)\)$", formula.strip())   # L729
match = re.match(r"UserMessage\((.+)\)$", formula.strip())              # L750
match = re.match(r"AgentMessage\((.+)\)$", formula.strip())             # L758
match = re.match(r"MessageText\((.+)\)$", formula.strip())              # L765

(.+) is greedy and \)$ only requires the formula to end with some ) — not that the paren closes the call being matched. So any formula that starts with one of these names and ends with ) is captured whole, and everything after the call's own closing paren is swallowed into the argument list.

eval() consults _eval_custom_function before PowerFx and returns on any non-None result, so the mis-parse wins and the formula never reaches the engine that would have handled it correctly.

Two symptoms follow, depending on what the swallowed tail looks like:

For Concat/Concatenate the swallowed tail still starts and ends with ", so the literal branch strips the outer quotes and emits the formula's own source text as data — silently wrong output, no error. For MessageText the tail is re-evaluated as a PowerFx fragment (="hi") & Upper("there"), producing a spurious compilation error for a formula that is valid PowerFx.

A related consequence: because the alias only applies when Concat( is at position 0, Concat resolves to two different functions depending on where it sits in the formula. =Concat("a") is intercepted as the Copilot Studio string alias, while =Lower("B") & Concat("a") reaches real PowerFx and fails with Invalid number of arguments: received 1, expected 2-3 — the table-aggregator signature.

What I expected: a custom-function regex to match only when the call spans the entire formula, so that a formula merely beginning with Concat(/Concatenate(/UserMessage(/AgentMessage(/MessageText( falls through to PowerFx instead of being mis-parsed.

What happens instead: the tail after the call's closing paren is absorbed into the arguments, yielding either formula source text as output or a compilation error.

Steps to reproduce

1. Install agent-framework-declarative with a working PowerFx engine (dotnet + powerfx).
2. Define a declarative workflow whose SendActivity.activity.text is a formula that starts with one of the five custom-function names and continues with any operator, e.g. =Concat(Local.First, " ") & Upper(Local.Last).
3. Run the workflow and inspect the output.

Code Sample

import asyncio

from agent_framework_declarative import WorkflowFactory

CASES = [
    ('=Concat("a") & Lower("B")', "ab"),
    ('=Concatenate("x") & Upper("y")', "xY"),
    ('=MessageText("hi") & Upper("there")', "hiTHERE"),
    ('=Concat("a", "b")', "ab"),  # control: no trailing operator, works
]


async def main():
    for formula, expected in CASES:
        workflow = WorkflowFactory().create_workflow_from_definition({
            "name": "concat_tail_repro",
            "actions": [{"kind": "SendActivity", "id": "show", "activity": {"text": formula}}],
        })
        try:
            actual = (await workflow.run({})).get_outputs()
        except Exception as e:
            actual = f"{type(e).__name__}: {e}"
        print(f"{formula}\n    expected: {expected!r}\n    actual:   {actual}\n")


asyncio.run(main())

Output:

=Concat("a") & Lower("B")
    expected: 'ab'
    actual:   ['a") & Lower("B']
=Concatenate("x") & Upper("y")
    expected: 'xY'
    actual:   ['x") & Upper("y']
=MessageText("hi") & Upper("there")
    expected: 'hiTHERE'
    actual:   ValueError: Power Fx failed compilation: Error 4 - 5 : Unexpected characters. Characters are used in the formula in an unexpected way.
=Concat("a", "b")
    expected: 'ab'
    actual:   ['ab']

Error Messages / Stack Traces

The Concat/Concatenate cases raise nothing — the workflow emits incorrect text. The MessageText case raises:

ValueError: Power Fx failed compilation: Error 4 - 5 : Unexpected characters. Characters are used in the formula in an unexpected way.

Package Versions

Source checkout 40763bed1a5e94a1698ba9ebf009e5622eb099aa: agent-framework-core 1.20.0, agent-framework-declarative 1.2.0, powerfx 0.0.34, dotnet 10.0.303.

Python Version

Python 3.13.2 on Windows x64.

Additional Context

• Distinct from Python: Declarative concatenation treats quoted expressions as literal text #9072, which concerns the argument literal heuristic (arg.startswith('"') and arg.endswith('"')) inside an otherwise correctly-delimited call. This report is about the outer regex selecting the wrong extent of the formula in the first place; the control row Concat("a", "b") evaluates correctly, so argument handling is not what fails here. The two interact: once the extent is fixed, Python: Declarative concatenation treats quoted expressions as literal text #9072's fix governs the arguments; while the extent is wrong, a Python: Declarative concatenation treats quoted expressions as literal text #9072 fix would change these cases from literal garbage to evaluating a malformed fragment rather than making them correct.
• A matching-paren scan (or requiring the capture to be balanced before accepting the match) would confine each handler to formulas that are entirely one custom call, which is what the surrounding _preprocess_custom_functions path already assumes for the nested case (Upper(MessageText(...)) works today).
• Separately, _powerfx_functions.concat_text implements the standard PowerFx table Concat(table, expression, separator) but is registered nowhere — CUSTOM_FUNCTIONS["Concat"] points at the varargs string concat_strings, and concat_text is referenced only from tests. Worth deciding whether the table form should be reachable at all, given the positional split described above.
• No model or network access is needed for this reproduction.