Skip to main content

Response Validation Framework

Overview​

The Response Validation Framework is introduced to determine valid responses because nowadays LLMs exhibit nondeterministic behavior while creating responses. A prompt may succeed 9 out of 10 times but can fail on occasion.

User Experience​

When a user runs a GPTfy prompt from the GPTfy User Console and receives an invalid response, a warning message will inform them that the response is invalid.

RVF 1

Sustainable Solution for Complex and Simple Responses​

  • Declarative Handling: Simple includes/excludes conditions can be easily specified
  • Complex Handling: For more complex cases, customers can use Apex

Characteristics of Response Validation Framework​

Add Filter (Plus) Icon​

When clicked, it will allow the user to add filters which will be based on three actions as below:

  1. Sequence - What sequence the action should be given, so that we can add/use them in the condition

  2. Action - A picklist that will show three values:

    • Includes - The value given with this will be considered as values that are acceptable while being a part of response (e.g., positive)

      RVF 2

    • Excludes - The value given with this will be considered as values that are unacceptable while being a part of response (e.g., Error)

      RVF 3

    • Apex - This action will allow an apex class that can have its own custom logic. The return method has to be boolean. Sample class is added, this will look if the response has error or exception in response, then it will fail.

      RVF 4

  3. Value - The value that needs to be included or excluded or apex class name

Note: All these three values are required.

Apex Class Example​

Apex class for response validation framework that will verify whether the response doesn't have the terms 'Error' or 'Exception' in it. If it has, it will fail.

global class AIResponseValidationSample implements ccai.AIResponseValidationInterface {
    global static Boolean invokeApex(String response, ccai__AI_Prompt_Response_Validation__c responseValidation){
        if(response.containsIgnoreCase('Error') || response.containsIgnoreCase('Exception')){
            return false;    
        }        
        return true; 
    }
}

Additional Controls​

  • Refresh Icon - This will refresh the actions and its values. When the condition and error message is saved, this will disable them from editing, replacing the save icon with pencil icon.

  • Modify Condition (Pencil) Icon - When clicked, the condition text box gets enabled and also the 'Save' icon is enabled. This will enable the user to give in their own condition by using the Filter sequence number.

    RVF 5

  • Error Message Text Box - The error message that needs to be shown when the values in the exclude took part in the response

    RVF 6

Security Audit Integration​

When the error message is popping out, the corresponding security audit will be marked as errored.

RVF 7

Important Notes​

This response framework feature is introduced to all the four types of the prompts.

This feature is supported in all the following places:

  • Console
  • AI Mass processing
  • Test console
  • Canvas
  • REST API
  • Flow