# LLM - Large Language Model

In this app, LLM is used for several purposes:
1. Extracting knowledge from docs;
2. Generating responses to user queries.

## Configure LLM

After logging in with an admin account, you can configure the LLM in the admin panel.

1. Click on the `Models > LLMs` tab;
2. Click on the `New LLM` button to add a new LLM;

    ![llm-config](https://github.com/user-attachments/assets/993eec34-a99a-4acf-b4b7-a4ee8e28e3d5 "LLM Config")

3. Input your LLM information and click `Create LLM` button;
4. Done!

import { Callout } from 'nextra/components'

<Callout>
If you want to use the new LLM while answering user queries, you need switch to `Chat Engines` tab and set the new LLM as LLM.
</Callout>

## Supported LLM providers

Currently Autoflow supports the following LLM providers:

### OpenAI

To learn more about OpenAI, please visit [OpenAI](https://platform.openai.com/).

### Google Gemini

To learn more about Google Gemini, please visit [Google Gemini](https://gemini.google.com/).

### Vertex AI

To learn more about Vertex AI, please visit [Vertex AI](https://cloud.google.com/vertex-ai).

### Amazon Bedrock

To use Amazon Bedrock, you'll need to provide a JSON Object of your AWS Credentials, as described in the [AWS CLI config global settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-global):

```json
{
    "aws_access_key_id": "****",
    "aws_secret_access_key": "****",
    "aws_region_name": "us-west-2"
}
```

To learn more about Amazon Bedrock, please visit [Amazon Bedrock](https://aws.amazon.com/bedrock/).

### Gitee AI

Follow the UI to configure the Gitee AI provider. To learn more about Gitee AI, please visit [Gitee AI](https://ai.gitee.com/serverless-api).

### OpenAI-Like

Autoflow also support the providers that conform to the OpenAI API specification.

To use OpenAI-Like LLM providers, you need to provide the **api_base** of the LLM API as the following JSON format in **Advanced Settings**:

```json
{
    "api_base": "{api_base_url}"
}
```

#### OpenRouter

Default config:

```json
{
   "api_base": "https://openrouter.ai/api/v1/"
}
```

To learn more about OpenRouter, please visit [OpenRouter](https://openrouter.ai/).

{/*
#### ZhipuAI BigModel

Default config:

```json
{
    "api_base": "https://open.bigmodel.cn/api/paas/v4/",
    "is_chat_model": true
}
```

To learn more about BigModel, please visit [BigModel](https://open.bigmodel.cn/). 
*/}

#### Ollama

Default config:

```json
{
    "api_base": "http://localhost:11434"
}
```

To learn more about Ollama, please visit [Ollama](https://ollama.com/).

#### vLLM

Default config:

```json
{
    "api_base": "http://localhost:8000/v1/"
}
```

To learn more about vLLM, please visit [vLLM](https://docs.vllm.ai/en/stable/).

#### Xinference

If you assigned a model uid different from the model name, you need to fill in **model uid** in the box `model`.

Default config:

```json
{
    "api_base": "http://localhost:9997/v1/"
}
```

To learn more about Xinference, please visit [Xinference](https://inference.readthedocs.io/en/latest/).


#### Azure OpenAI

To learn more about Azure OpenAI, please visit:

- [Azure OpenAI documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference)
- [Create and deploy an Azure OpenAI Service resource](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal)

After creating the Azure OpenAI Service resource, you can configure the API base URL in the **Advanced Settings**:

```json
{
  "azure_endpoint": "https://<your-resource-name>.openai.azure.com/",
  "api_version": "<your-api-version>",
  "engine": "<your-deployment-name>"
}
```

You can find those parameters in the [Deployment Tab](https://ai.azure.com/resource/deployments) of your Azure OpenAI Service resource.

<Callout>
Do not mix `Model version` and `API version` up, they are different.
</Callout>

![Azure OpenAI Service Deployment Tab - LLM](https://github.com/user-attachments/assets/158f845c-5f38-40d7-b66a-8528d7df178e)

#### Novita AI

Default config:

```json
{
    "api_base": "https://api.novita.ai/v3/openai"
}
```

To learn more about Novita AI, please visit [Novita AI](https://novita.ai/).

#### DeepSeek

DeepSeek provides chat model `deepseek-chat`.

Default config:

```json
{
    "api_base": "https://api.deepseek.com/v1",
    "is_chat_model": true
}
```

To learn more about DeepSeek, please visit [DeepSeek](https://www.deepseek.com/).