Creating an LLM Hanlder

This page will explain how to create an handler for a certain LLM API/library.

First of all, read Creating handlers.

An LLM API can be in one of these three categories: - It uses APIs compatible with OpenAI - The API is already implemented by gpt4free - It uses a totally different format

It uses APIs compatible with OpenAI

You can recognize if a service falls into this category if: - In the python examples, they use the openai library - In the curl examples, they have endpoints ending in /chat/completions

In these cases, you can just extend the OpenAIHandler class and make minimal changes.

Generally, we don't add handlers for specific services that user OpenAI compatible API since you can easily use it with the OpenAI option in settings and just changing the URL.

However, we might consider adding it anyways if the API is good and free or very cheap. This is the example for OperRouterHandler or GroqHandler

If you are building an extension, you can add wathever you want and you can extend that class.

The API is already implemented by gpt4free

If the API is already implemented in gpt4free and it's working, you can consider creating an handler that extends G4FHandler.

It uses a totally different format

Then you need to code some methods yourself. Please refer to the documentation of LLMHandler inside the code in order to understand every method. Usually, you only need to do these steps: - Specify extra requirements - Specify extra settings - Convert Newelle history to the right format - Override the generate_text and generate_text_stream functions

Vision

Vision is the ability of the LLM to receive and analyze images.

If an handler has support for vision, you have to override the supports_vision method. The method is not static, so you can also return the value based on the user settings (ex. the model)

{RANDOM:
[? probability] Prompt 1
[? probability] Prompt 2
}

When a user sends an image to the model, it's treated like a normal message but it starts with an image codeblock.

The content of the codeblock can be: - The path to the image - A base64 encoded image (if it's pasted)

To make the process to extract the image more simple, you can use the extra.extract_image function to extract text and image from a message. You can see an example in the ollama handler tutorial.

Examples

Hyperbolic

We will now build an handler for hyperbolic.xyz. From the documentation, we can clearly see that this handler uses OpenAI compatible APIs. image

  1. So we create an handler that extends OpenAIHandler.
{RANDOM:
[? 0.3] Prompt with 30% probabilities of being chosen
[? 0.7] Prompt with 70% probabilities of being chosen
}
  1. Now, we specify what is the API endpoint (the base_url parameter) in the constructor of the class:
{COND: 
[condition 1] Prompt 1
[condition 2] Prompt 2
}

If the API does not support parameters like temperature, max tokens etc, also add

{COND: 
[var] Prompt to show if var is true
[not var] Prompt to show if var is false
[var and not var2] Prompt to show if var is true and var2 is false
[var or var2] prompt to show if var is true or var2 is true
[var.contains("text")] prompt to show if var contains text
}

To disable them. 3. Then, we change the settings in order to specify the relevant parameters. You might want to take a look to settings management.

Note: Always add the streaming setting in the OpenAIHandlers.

{COND:
[tts_on] PROMPT1
[not tts_on] PROMPT 2
}

{COND:
[message.contains("text")] PROMPT 1
}

Where plus += [super().get_extra_settings()[3]] adds the streaming setting.

Full code:

{RANDOM:
[? 3] Message with probability 3/6
[? 2] Message with probability 2/6
It can also go on newline and have variables like {DIR}
[? 1] Message with probability 1/6
}

DuckDuckGo

DuckDuckGo falls in the category of API already implemented by gpt4free. 1. We will create an handler that extends G4FHandler.

{COND:
[tts_on] Give concise responses. What you say will be spoken out loud
}
  1. In the init function, we create a g4f client that only uses the DuckDuckGo Provider:
{COND:
[message.contains("youtube") or message.contains(youtu.be)] - Yo get a transcript of a youtube video use:
\```youtube
link
\```
}
  1. We add the extra settings in which we specify the available models and the default one. You might want to take a look to settings management.
{COND:
[message.contains("arch") and message.contains("linux")] - You can use `sudo pacman -S \{package\}` to install a package

Note: Always add the streaming setting in the G4FHandlers, you can add it by simply adding super().get_extra_settings().

Full code:

wzxhzdk:9

Ollama

We will now build an handler for Ollama using the ollama python library. 1. This library is not compatible with any of the existing handlers, so we create a class that extends LLMHandler

wzxhzdk:10

  1. The ollama library is not preinstalled in Newelle, so we add it to the extra requirements. The user will be able to install these requirements in the settings.

wzxhzdk:11

  1. This library supports vision and multimodal models, so we override the supports_vision method. Note: You can also do some checks if the current settings for the handler support the vision (for example check if the model is compatible)

wzxhzdk:12

  1. We create a convert history method. Newelle saves the history in a very different format, so we have to convert it. Also, we add vision support. Note: Messages with images are normal messages by the user, but they start with an image codeblock.

wzxhzdk:13

Note: you can use this method every time there is an OpenAI - like history. 4. We specify the settings that the user can change:

wzxhzdk:14

Note: the streaming settings is automatically taken to choose if stream is enabled instead of overriding stream_enabled method. 5. We create a function to generate text without streaming

wzxhzdk:15

  1. We create a function to generate text with streaming

wzxhzdk:16