Generate Chat Completion

Description

Given a user prompt, an optional instruction, and other optional inputs such as the Model ID, etc., generates a chat completion response. This functionality mimics the popular ChatGPT capabilities. This activity enables users to interact with GPT-3.5 and GPT-4 models and simulate a dialogue experience with the model via context and user roles.

Project compatibility

Windows | Cross-platform

Configuration

• Connection ID - The connection established in Integration Service. Access the dropdown menu to choose, add, or manage connections.

• Model - The large language model (LLM) to use for the chat completion. You can choose between gpt-35-turbo and gpt-4. You can read more on the differences between these models in the Microsoft documentation.
  Note: This activity enables users to select any GPT-3.5 or GPT-4 model that has been deployed into your Azure instance. If you don’t see a model in the dropdown, please reach out to your Azure Administrator and ensure that they have deployed the model to production.
• Prompt - The input to use for the chat completion request. Read more about Prompt engineering. This field supports String type input.
• Knowledge base - Indicates whether or not the chat completion will be executed with unique context stored in a vector database. If select, other fields will be required. Boolean value. Default value is False.
• Vector DB - The vector DB to utilize as an internal knowledge base. Select an option from the drop-down menu. For now, only Pinecone is available. This field is displayed if you set Knowledge base to True.
• Pinecone API key - If using your own instance of a Pinecone vector database to get a chat completion with retrieval augmented generatione, you must provide the API Key. This can be stored as an asset if security is a concern. This field supports String type input. This field is displayed if you select Pinecone for the Vector DB field.
• Pinecone environment - Environment where the Pinecone database is hosted. Please consult your instance. This is a required field if retrieval augmented generation is desired. Select an option from the available drop-down list.
• Pinecone index - The Pinecone index to search for pulling in relevant content. Select an option from the available drop-down list. This field is optional.
• Pinecone namespace - The specific Pinecone namespace to search for pulling in relevant context. Select an option from the available drop-down list. This field is optional.

Manage Properties

Use the Manage Properties wizard to configure or use any of the object's standard or custom fields. You can select fields to add them to the activity canvas. The added standard or custom fields are available in the Properties panel (in Studio Desktop) or under Show additional properties (in Studio Web).

Additional properties

• API version - The API version to use for this operation. this follows the YYYY-MM-DD format. This field supports String type input.
• Instruction - Instructions which guide the AI to respond in a way that matches your intent. Defaults to null. This field supports String type input.
• Max Tokens - The maximum number of tokens allowed for the prompt and generated answer. Fewer tokens are less expensive. Most models support a maximum of 4096 tokens, however, some models support only 2048. Default value is 3840. Read more information about tokens.
• Temperature - A number between 0 and 2. Higher values like 0.8 make the output more random, while lower values like 0.2 make it more focused and deterministic. Default value is 1.
• N - The number of completion choices you want the AI to return. The minimum value of this field is 1. Default value is 1.
• Top P - A number between 0 and 1. The lower the number, the fewer tokens are considered. Defaults to 1.
• Stream - Specifies whether to stream back partial progress. If set, tokens are sent as data-only server-sent events as they become available, with the stream terminated by a data:[DONE] message. Defaults to False.
• Presence Penalty - A number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics. Defaults to 0.
• Frequency Penalty - Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. Defaults to 0.

Output

• Top Generated Text - The top generated chat completion string. If you are interested in only returning a single response (i.e., N=1, which is the default), this will be the output you most likely will want to use. It is a Text data type which is easy to assign a variable to and use directly in other activities/text inputs.
• Chat Completion - The full ChatCompletion object that includes:
  • Choices - or responses from the model. You can set the total number of responses that you want back from the model using the ‘N’ field in the additional options of the activity which defaults to 1. This will return as an array with the first object starting at 0. Ex. Referencing a single object in an array within a ‘For Each’ control: Response.Choices(0).Text. For more information, see the Chat Completion object documentation.