Repository navigation
Bug: SK runs his self-made function by ignoring the auto-invoke function config #281
Description
Activity
- changed the title
[-]Bug: SK runs his own-construct function by ignoring the auto-invoke function config[/-][+]Bug: SK runs his self-made function by ignoring the auto-invoke function config[/+]on Jan 2, 2025 Which model are you using?
Which model are you using?
I'm using GPT-4o as the model.
My requirement is to implement a dynamic method invoker, which uses the llm to automatically select the method and extract the required parameters from the conversation with the user.
When all things get done ,then inform the background service of this information for invocation.
The problem is SK use this info and made a function-call.😂All method metadata and user intents are assembled in the prompt(in userInput and metadata variables).
I have run a sample based on the original code, and it appears to execute as expected:
Are you able to share more context/a failing sample of the code that is failing?
John (@johnoliver) Sorry for some mistake description before.
When using gpt-4o-mini, it tried to call the backend service method such as aiotDeviceGatewayImpl$scrollDeviceInfoByParam, and then exception occurred.
When using gpt-4o, the model just return the expercted json.
I am using 4o for now.My prompt is:
You are a dataset matching expert. You will identify the user's intent based on the dataset definition (Init) and matching rules (Rules) from the user's input (User Input) and then return the matching result in JSON format. # Matching Rules - Rules: 1. Match based on the dataset - Init definition, referring to the example (few-shots). 2. If fully matched (set status = 0), the returned data must be in JSON format, structured as follows, where `args` are the parameters used to query the dataset (the parameter array may contain heterogeneous elements, matching based on the dataset definition): ```json { "status": 0, "desc": "Your intent is xxxx, all parameters have been matched", "args": [ {"argName": "yyy1", "argValue": "yyy2"}, {"argName": "zzz", "argValue": 100} ], "identity": "xxx3" } ``` 3. If not fully matched (set status = 1), output the following JSON structure, where `desc` is the prompt message, `identity` is the dataset identifier (could be matched, may be an empty string), and `args` are the parameters that may be matched: ```json { "status": 1, "desc": "Your input is incomplete, please describe xxxx/Your intent is xxxx, you still need to provide the xxxx parameter", "args": [ {"argName": "argName1", "argValue": "argValue1"}, {"argName": "argName2", "argValue": "argValue2"} ], "identity": "xxx3" } ``` 4. **Special Note**: If the data the user requires has been retrieved, return the `records` field (where `headers` is the table header, and `rows` are the actual data corresponding to the header): ```json { "status": 0, "desc": "Your intent is xxxx, all parameters have been matched", "records": { "headers": ["colName1", "colName2", "colName3"], "rows": [["obj1", "obj2", "obj3"]] }, "args": [ {"argName": "argName1", "argValue": "argValue1"}, {"argName": "argName2", "argValue": "argValue2"} ], "identity": "xxx3" } ``` 5. Only return JSON, no other unrelated content, and no unnecessary statements. # Dataset - Init: Note: 1. Only return the most matching dataset intent. Each identity represents a unique dataset, and each dataset can implement one or more intents (Intent). 2. The intent is considered fully matched only if all required parameters are matched; otherwise, it is considered incomplete. Non-required parameters should also be returned. 3. The content of the dataset cannot be used as a tool call. Dataset Definition: {{$metadata}} # Example: Input: I want to query the weather in Shenzhen Output: ```json { "status": 0, "desc": "Your intent is to query the weather, you matched the weather query plugin, and the city you entered is Shenzhen", "args": [{"argName": "cityName", "argValue": "Shenzhen"}], "identity": "weatherSearchPlugin$queryCityWeatherByCityName" } ``` Input: I want to query the weather Output: ```json { "status": 1, "desc": "Your intent is to query the weather, you matched the weather query plugin, please enter the city name you want to query", "args": [], "identity": "weatherSearchPlugin$queryCityWeatherByCityName" } ``` Input: I want to query Output: ```json { "status": 1, "desc": "Your input does not match any intent, please re-describe your query", "args": [], "identity": "" } ``` Input: I want to know the city national standard code in Shenzhen in MT bizCode Output: ```json { "status": 0, "desc": "Your intent is to query the city national standard code, the city you entered is Shenzhen", "args": [{"argName": "cityName", "argValue": "Shenzhen"}, {"argName": "bizName", "argValue": "MT"}], "records": {"headers": ["cityCode"], "rows": [[440300]]}, "identity": "" } ``` Input: I want to query the data of device 003C84FF's identifier=temperature from yesterday Output: ```json { "status": 0, "desc": "Your intent is to query the data for a specific identifier of a device for a certain period", "args": [ {"argName": "deviceId", "argValue": "003C84FF"}, {"argName": "identifier", "argValue": "temperature"}, {"argName": "startTime", "argValue": "2025-01-01 00:00:00"}, {"argName": "endTime", "argValue": "2025-01-02 00:00:00"} ], "identity": "TWQYJEEGNN" } ``` Input: My emp number is xxx, I want to query the device information of MT device 003C84FF Output: ```json { "status": 0, "desc": "Your intent is to query the device information of MT device 003C84FF", "args": [ {"argName": "emp", "argValue": "xxx"}, {"argName": "bizName", "argValue": "MT"}, {"argName": "deviceIdOrName", "argValue": "003C84FF"} ], "identity": "aiotDeviceGatewayImpl$scrollDeviceInfoByParam" } ``` # User Input Input: My emp number is {{$emp}}, {{$input}} Output:My metadata looks like:
Dataset Definition: Dataset Identity: aiotDeviceGatewayImpl$scrollDeviceInfoByParam, Description: Query device information based on the business name, device ID or device name (deviceIdOrName, supports fuzzy query), and user's emp number. Intent Input Parameter Combinations and Sorting Parameters (Intent): Intent Description: Query device information based on the business name (bizName, such as XS, MT, etc.), device ID or device name (deviceIdOrName, supports fuzzy query), and user's emp number. Required Parameters List: - Field Name: deviceIdOrName, Field Description: Device ID or device name, Field Type: java.lang.String - Field Name: bizName, Field Description: Business name, such as MT, XS, etc., must be an exact match, Field Type: java.lang.String - Field Name: emp, Field Description: User's emp number, Field Type: java.lang.String Optional Parameters List: - Field Name: deviceOnlineStatus, Field Description: Device online status, 1 for online, 0 for offline, Field Type: java.lang.Integer Dataset Identity: weatherSearchPlugin$queryCityWeatherByCityName, Description: Query city weather based on the city name. Intent Input Parameter Combinations and Sorting Parameters (Intent): Intent Description: Query city weather based on the city name (cityName). Required Parameters List: - Field Name: cityName, Field Description: City name, Field Type: java.lang.StringSo from my understanding you just want the JSON back, but not the function invocation, in this case I think passing:
.withToolCallBehavior(ToolCallBehavior.allowOnlyKernelFunctions(true, functions))is the mistake, passing true instructs the kernel to perform automatic function invocations. However even if you pass false here, the LLM is likely to still request a tool invocation, it will just be left to the user to make that call manually. I think for your usecase you might not want to use function calling at all, since the LLM will try to use functions as part of its execution, whereas I believe you simply want the json returned. I would enable json structured outputs and not pass any tool call behaviour at all, and see if that works as you need.
Because I also need function-call to obtain some contextual information. For example, constructing parameters based on time descriptions—like transforming "one day ago" into start_time and end_time using methods such as getCurrentDate, as well as some other utility functions.
So I passed some functions and setauto invokeas true.So I think this is probably more of a prompt/model issue than a semantic kernel issue, I think the fact that you are asking the LLM to invoke some functions but not the final function that is possibly too complex a concept for it to understand reliably.
What I am somewhat surprised about (if I am understanding the situation correctly) is that the LLM hallucinated and tried to invoke a function that was not passed to it as an available function. As it effectively invented that
aiotDeviceGatewayImpl$scrollDeviceInfoByParamexists even though it only exists in the few shot example prompt but not the function list. What I would probably try is try a 2 step process, first get the LLM to select which function to invoke by making a call with auto invoke false, and no functions added. Then once it has selected a function make a second call with tool calling enabled, to interpret things like "one day ago".A second thing to try, in your few shot examples, try renaming the function to remove the "$", again from my interpretation the issue is that with the $ it is not a valid function call. If the sample is a valid function name, what I am hoping will happen is:
- LLM will make a mistake, try to invoke
aiotDeviceGatewayImpl_scrollDeviceInfoByParam - SK will receive the tool call, determine that the function does not exist and bounce it back to the LLM to have another try
- LLM will make a mistake, try to invoke
Describe the bug
I set the toolcallbehavior of
allowOnlyKernelFunctionsastrue, and passed the functions to the kernelbut the kernel make a functiontoolcall by himself and just invoke it.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Only invoke the passed functions.
Screenshots
Maven
Platform
Additional context
Excption: com.azure.core.exception.HttpResponseException: Status code 400, "{"error":{"message":"Invalid 'messages[1].tool_calls[1].function.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.","type":"invalid_request_error","code":"invalid_value","param":"messages[1].tool_calls[1].function.name"}}"