Custom Model

You can use a custom LLM that isn't built into Monacopilot by setting up a model when you create a new CompletionCopilot. This feature lets you connect to LLMs from other services or your own custom-built models.

Example

const copilot = new CompletionCopilot(process.env.HUGGINGFACE_API_KEY, {
    // You don't need to set the provider if you are using a custom model.
    // provider: 'huggingface',
    model: {
        config: (apiKey, prompt) => ({
            endpoint:
                'https://api-inference.huggingface.co/models/openai-community/gpt2',
            headers: {
                Authorization: `Bearer ${apiKey}`,
                'Content-Type': 'application/json',
            },
            body: {
                inputs: prompt.user,
                parameters: {
                    max_length: 100,
                    num_return_sequences: 1,
                    temperature: 0.7,
                },
            },
        }),
        transformResponse: response => ({text: response[0].generated_text}),
    },
});

Configuration

The model option accepts an object with two functions:

Function	Description	Type
config	A function that receives the API key and prompt data, and returns the configuration for the custom model API request.	(apiKey: string, prompt: { system: string; user: string }) => { endpoint: string; body?: object; headers?: object }
transformResponse	A function that takes the raw/parsed response from the custom model API and returns an object with the text property.	(response: unknown) => { text: string | null; }

The config function must return an object with the following properties:

Property	Type	Description
endpoint	string	The URL of the custom model API endpoint.
body	object or undefined	The body of the custom model API request.
headers	object or undefined	The headers of the custom model API request.

The transformResponse function must return an object with the text property. This text property should contain the text generated by the custom model. If no valid text can be extracted, the function should return null for the text property.

NOTE

Please ensure you are using a high-quality model, especially for coding tasks, to get the best and most accurate completions. Also, use a model with very low response latency (preferably under 1.5 seconds) to enjoy a great experience and utilize the full power of Monacopilot.