UN-2900 [FIX] Flag variables that cannot resolve under single-pass ex…

…traction Single pass builds ONE combined prompt -- every field declared up front in a single JSON schema, answered in one LLM call -- so no prompt's output exists to feed another prompt's variable. The runtime reflects that: the enterprise single_pass_extraction plugin calls the shared replacement service with structured_output={}, and both replace_static_variable and replace_dynamic_variable return the prompt UNCHANGED when their lookup misses. The literal {{...}} is then sent to the LLM, silently degrading the answer. This is not specific to custom_data, despite the ticket title. CUSTOM_DATA is in fact the ONLY variable type that survives single pass, because it resolves from the tool's own custom_data and never consults the variable map. STATIC and DYNAMIC variables both fail on their own, with no custom_data involved: static : "check {{invoice_number}}" -> "check {{invoice_number}}" dynamic : "via {{https://.../x[cust_id]}}" -> unchanged custom : "{{custom_data.client.name}}" -> "Acme GmbH" (works) Adds find_unresolvable_single_pass_variables() and surfaces the result per prompt as single_pass_unresolvable_variables when the tool has single-pass enabled. Warning only -- deliberately NOT a save-time block, because existing projects may already carry this combination and users toggle single pass on and off; a hard refusal would break them retroactively and be order-dependent. Classification pairs with VariableReplacementService in the worker, which keeps its own copy of the variable regexes; noted in the docstring since drift would make validation and runtime disagree. Known gap: this covers Prompt Studio authoring, not an already-exported tool running single pass via API deployment, which keeps failing silently until the tool is re-saved. That argues for pairing this with placeholder-stripping at runtime later, not for widening this change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sXFBEu2GHatXq2CV2ShPF
hari-kuriakose · Aug 28, 2026 · Aug 28, 2026 · Aug 28, 2026 · Aug 28, 2026 · Aug 28, 2026
commit f45d4e0f9e505d21c0822265119d0a4e114a9431
diff --git a/backend/prompt_studio/prompt_studio_core_v2/prompt_variable_service.py b/backend/prompt_studio/prompt_studio_core_v2/prompt_variable_service.py
@@ -64,6 +64,41 @@ def identify_variable_type(variable: str) -> VariableType:
                 variable_type = VariableType.STATIC
         return variable_type
 
+    @staticmethod
+    def find_unresolvable_single_pass_variables(prompt: str) -> list[str]:
+        """Variables in ``prompt`` that cannot resolve under single-pass extraction.
+
+        UN-2900. Single pass builds ONE combined prompt — every field is declared
+        up front in a single JSON schema and answered in one LLM call — so no
+        prompt's output exists to feed another prompt's variable. The runtime
+        reflects this: the enterprise ``single_pass_extraction`` plugin calls the
+        shared replacement service with ``structured_output={}``, and both
+        ``replace_static_variable`` and ``replace_dynamic_variable`` return the
+        prompt UNCHANGED when their lookup misses. The literal ``{{...}}`` is then
+        sent to the LLM, silently degrading the answer.
+
+        CUSTOM_DATA is the one exception: it is resolved from the tool's own
+        ``custom_data`` and never consults the variable map, so it works
+        identically in both modes and is not reported here.
+
+        Returns the offending variable strings, in prompt order, or an empty list.
+
+        NOTE: classification pairs with ``VariableReplacementService`` in the
+        worker (``executor/executors/variable_replacement.py``), which keeps its
+        own copy of these regexes. If one side's patterns change, this validation
+        and the runtime behaviour will disagree.
+        """
+        unresolvable: list[str] = []
+        for variable in PromptStudioVariableService.extract_variables_from_prompt(
+            prompt=prompt
+        ):
+            variable_type = PromptStudioVariableService.identify_variable_type(
+                variable=variable
+            )
+            if variable_type != VariableType.CUSTOM_DATA:
+                unresolvable.append(variable)
+        return unresolvable
+
     @staticmethod
     def extract_variables_from_prompt(prompt: str) -> list[str]:
         variable: list[str] = []

diff --git a/backend/prompt_studio/prompt_studio_core_v2/serializers.py b/backend/prompt_studio/prompt_studio_core_v2/serializers.py
@@ -18,6 +18,9 @@
 from prompt_studio.prompt_profile_manager_v2.models import ProfileManager
 from prompt_studio.prompt_studio_core_v2.constants import ToolStudioKeys as TSKeys
 from prompt_studio.prompt_studio_core_v2.exceptions import DefaultProfileError
+from prompt_studio.prompt_studio_core_v2.prompt_variable_service import (
+    PromptStudioVariableService,
+)
 from prompt_studio.prompt_studio_output_manager_v2.output_manager_util import (
     OutputManagerUtils,
 )
@@ -226,6 +229,19 @@ def to_representation(self, instance):  # type: ignore
 
             # Add coverage to serialized data
             serialized_data["coverage"] = coverage
+
+            # UN-2900: warn (do not block) when a prompt uses variables that
+            # cannot resolve under single pass. Surfaced per prompt so the user
+            # sees it while authoring instead of discovering it as a degraded
+            # answer after paying for the run.
+            serialized_data["single_pass_unresolvable_variables"] = (
+                PromptStudioVariableService.find_unresolvable_single_pass_variables(
+                    prompt=prompt.prompt
+                )
+                if instance.single_pass_extraction_mode and prompt.prompt
+                else []
+            )
+
             output.append(serialized_data)
 
         data[TSKeys.PROMPTS] = output