Platform For AI:Deploy a RAG-based LLM chatbot

Step 1: Deploy a RAG service

1. Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Enter Elastic Algorithm Service (EAS).

2. On the Model Online Service (EAS) page, click Deploy Service. In the Scenario-based Model Deployment area, click RAG-based Smart Dialogue Deployment.

   

3. On the RAG-based LLM Chatbot Deployment page, configure the parameters and then click Deploy. When the Service Status changes to Running, the service is successfully deployed. Service deployment usually takes about 5 minutes. The actual duration may differ based on the number of model parameters or other factors. The following table describes the key parameters.

   • Basic Information

     Parameter	Description
     Version	The following two versions are supported for deployment: LLM-Integrated Deployment: Deploy the LLM service and RAG service within the same service. LLM-Separated Deployment: Deploy only the RAG service. However, you can freely switch and connect the LLM service for greater flexibility.
     Model Type	When you select LLM-Integrated Deployment, you need to choose the LLM to deploy. You can select an open-source model based on your specific use case.

   • Resource Information

     Parameter	Description
     Deployment Resources	If you select LLM-Integrated Deployment, the system automatically matches an appropriate instance type. If you change the instance type to other instance types, the model service may fail to be started. If you select LLM-Separated Deployment, we recommend that you select the ecs.g6.2xlarge and ecs.g6.4xlarge instance types that include more than 8 vCPUs and 16 GB of memory.

   • Vector Database Settings

     RAG supports building vector databases using Faiss (Facebook AI Similarity Search), Elasticsearch, Hologres, OpenSearch, or ApsaraDB RDS for PostgreSQL. Select a version type based on your scenario to serve as the vector database.

     Faiss

     Faiss enables the creation of a local vector database efficiently, eliminating the necessity to acquire or activate online vector databases.

     Parameter	Description
     Version Type	Select FAISS.
     OSS Address	Select the OSS storage path created in the current region to store the uploaded knowledge base files. If no storage path is available, you can refer to Quick Start for Console to create one. Note If you choose to use a custom fine-tuned model deployment service, ensure that the selected OSS storage path does not overlap with the path of the custom fine-tuned model to avoid conflicts.

     Elasticsearch

     Specify the connection information of an Alibaba Cloud Elasticsearch instance. For information on how to create an Elasticsearch instance and prepare configuration items, see Prepare the vector database Elasticsearch.

     Parameter	Description
     Version Type	Select Elasticsearch.
     Private Network Address/port	Configure the private network address and port of the Elasticsearch instance in the format http://<private network address>:<private network port>. For information on how to obtain the private network address and port number of an Elasticsearch instance, see View the basic information of an instance.
     Index Name	Enter a new index name or an existing index name. For an existing index name, the index schema must meet the requirements of the RAG-based chatbot. For example, you can enter the name of the index that is automatically created when you deploy the RAG-based chatbot by using EAS.
     Account	Specify the logon name that you configured when you created the Elasticsearch instance. The default logon name is elastic.
     Password	Specify the password that you configured when you created the Elasticsearch instance. If you forget the logon password, you can reset the instance access password.

     Hologres

     Specify the connection information of a Hologres instance. If you have not activated a Hologres instance, you can refer to Purchase Hologres for more information.

     Parameter	Description
     Version Type	Select Hologres.
     Invocation Information	Specify the host information of the designated VPC. Go to the instance details page of the Hologres Management Console. In the Network Information area, click Copy after Designated VPC to obtain the host information before the domain name :80.
     Database Name	Specify the database name of the Hologres instance. For more information about how to create a database, see Create a Database.
     Account	Specify the custom account that you created. For more information, see Create a Custom User, where you select Select Member Role and choose Superuser.
     Password	Specify the password of the custom account that you created.
     Table Name	Enter a new table name or an existing table name. For an existing table name, the table schema must meet the requirements of the RAG-based chatbot. For example, you can enter the name of the Hologres table that is automatically created when you deploy the RAG-based chatbot by using EAS.

     OpenSearch

     Specify the connection information of an OpenSearch instance of Vector Search Edition. For information on how to create an OpenSearch instance and prepare configuration items, see Prepare the vector database OpenSearch.

     Parameter	Description
     Version Type	Select OpenSearch.
     Access Address	Specify the public endpoint of the OpenSearch instance of Vector Search Edition. You need to enable the public access feature for the OpenSearch instance of Vector Search Edition. For more information, see Prepare the vector database OpenSearch.
     Instance ID	Obtain the instance ID from the OpenSearch Vector Search Edition instance list.
     Username	Enter the username and password that you specified when you created the OpenSearch instance of Vector Search Edition.
     Password
     Table Name	Enter the name of the index table of the OpenSearch instance of Vector Search Edition that you created. For more information about how to prepare the index table, see Prepare the vector database OpenSearch.

     ApsaraDB RDS for PostgreSQL

     Specify the connection information of the ApsaraDB RDS for PostgreSQL instance. For information on how to create an ApsaraDB RDS for PostgreSQL instance and prepare configuration items, see Prepare a vector database by using ApsaraDB RDS for PostgreSQL.

     Parameter	Description
     Version Type	Select RDS PostgreSQL.
     Host Address	Specify the internal network address of the ApsaraDB RDS for PostgreSQL instance. You can go to the ApsaraDB RDS for PostgreSQL console page and view the Database Connection page of the ApsaraDB RDS for PostgreSQL instance.
     Port	The default value is 5432. Enter the value based on your actual situation.
     Database	Specify the name of the database that you created. For more information about how to create a database and an account, see Create an Account and a Database, where: When you create an account, select Account Type and choose Privileged Account. When you create a database, select Authorized Account and choose the privileged account that you created.
     Table Name	Specify the name of the database table.
     Account	Specify the privileged account and password that you created. For more information about how to create a privileged account, see Create an Account and a Database, where you select Account Type and choose Privileged Account.
     Password

   • VPC

     Parameter	Description
     VPC (VPC)	If you selected Separate LLM Deployment when deploying the RAG service, you must make sure it is connected to the LLM service: Public access: Select a virtual private cloud (VPC) that can be accessed over the Internet. For more information, see configure Internet access. Private access: Associate the same VPC with both the RAG service and the LLM service. If you need to use a model from Model Studio or perform Q&A with web search, you must configure a VPC that can be accessed over the Internet. For more information, see Configure Internet access. Network requirements for the vector database: The Faiss vector database does not require network access. Hologres, Elasticsearch, or ApsaraDB RDS for PostgreSQL can be accessed by EAS through the public network or private network. Private network access is recommended. Private network access requires that the VPC configured in EAS is consistent with the VPC of the vector database. For more information about how to create a VPC, a switch, and a security group, see Create and Manage a VPC and Create a Security Group. EAS can only access OpenSearch through the public network. For more information about how to configure the access method, see Step 2: Prepare Configuration Items.
     vSwitch
     Security Group Name

Step 2: Debug on the web UI

After the RAG service is successfully deployed, click View Web Application in the Service Method column to launch the web UI.

Follow the steps below to upload your knowledge base file on the web UI and test the Q&A chatbot.

Step 3: API calling

The following content describes the API calling methods of common RAG features. For more information about the API calling methods of other features, such as managing knowledge base indexes and updating the configurations of a RAG service, see Applicable to images with the versions earlier than v0.3.0.

Obtain invocation information

1. Click the RAG service name to go to the Service Details page.

2. In the Basic Information area, click View Invocation Information.

3. In the Invocation Information dialog box, obtain the service access address and token.

   Note

   You can use public network access or private network access.

   • To use public network access, the client must support Internet access.

   • To use private network access, the client must be in the same VPC as the RAG service.

Upload knowledge base files

You can upload local knowledge base files through the API. You can query the status of the file upload task based on the task_id returned by the upload interface.

In the following example, replace <EAS_SERVICE_URL> with the access address of the RAG service and <EAS_TOKEN> with the token of the RAG service. For more information about how to obtain the access address and token, see Obtain Invocation Information.

• Upload a single file

   # Replace <EAS_TOKEN> and <EAS_SERVICE_URL> with the service token and the service access address, respectively.
   # Replace the path after "-F 'files=@" with the path to your file.
   # Configure index_name as the name of your knowledge base index.
     curl -X 'POST' /api/v1/upload_data \
    -H 'Authorization: ' \
    -H 'Content-Type: multipart/form-data' \
    -F 'files=@example_data/paul_graham/paul_graham_essay.txt' \
    -F 'index_name=default_index'

• Upload multiple files. Use multiple -F 'files=@path' parameters with each parameter corresponding to a file to be uploaded, as shown in the example:

    # Replace <EAS_TOKEN> and <EAS_SERVICE_URL> with the service token and the service access address, respectively.
    # Replace the path after "-F 'files=@" with the path to your file.
    # Configure index_name as the name of your knowledge base index.
    curl -X 'POST' /api/v1/upload_data \
    -H 'Authorization: ' \
    -H 'Content-Type: multipart/form-data' \
    -F 'files=@example_data/paul_graham/paul_graham_essay.txt' \
    -F 'files=@example_data/another_file1.md' \
    -F 'files=@example_data/another_file2.pdf' \
    -F 'index_name=default_index'

• Query the task upload status

  # Replace <EAS_TOKEN> and <EAS_SERVICE_URL> with the service token and access address, respectively.
  # task_id indicates the task ID returned when you upload knowledge base files. 
  curl -X 'GET' '<EAS_SERVICE_URL>/api/v1/get_upload_state?task_id=2c1e557733764fdb9fefa0635389****' -H 'Authorization: <EAS_TOKEN>'

Single-round conversation request

import requests

SERVICE_URL = 'http://xxxx.****.cn-beijing.pai-eas.aliyuncs.com/'
headers = {
    'accept': 'application/json',
    'Content-Type': 'application/json',
    'Authorization': 'MDA5NmJkNzkyMGM1Zj****YzM4M2YwMDUzZTdiZmI5YzljYjZmNA==',
}

def test_post_api_query(url):
    data = {
       "question":"What is PAI?" 
    }
    response = requests.post(url, headers=headers, json=data)

    if response.status_code != 200:
        raise ValueError(f'Error post to {url}, code: {response.status_code}')
    ans = dict(response.json())

    print(f"======= Question =======\n {data['question']}")
    if 'answer' in ans.keys():
        print(f"======= Answer =======\n {ans['answer']}")
    if 'docs' in ans.keys():
        print(f"======= Retrieved Docs =======\n {ans['docs']}\n\n")
 
# LLM 
test_post_api_query(SERVICE_URL + 'api/v1/query/llm')
# Retrieval
test_post_api_query(SERVICE_URL + 'api/v1/query/retrieval')
# Chat (Knowledge Base)
test_post_api_query(SERVICE_URL + 'api/v1/query')

Multi-round conversation request

LLM and Chat (Knowledge Base) support sending multi-round conversation requests. The following code example shows how to do this:

# Send a request. 
curl -X 'POST'  '<service_url>api/v1/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question": "What is PAI?"}'

# Provide the session ID returned for the request. This ID uniquely identifies a conversation in the conversation history. After the session ID is provided, the corresponding conversation is stored and is automatically included in subsequent requests to call an LLM.
curl -X 'POST'  '<service_url>api/v1/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question": "What are the benefits of PAI?","session_id": "ed7a80e2e20442eab****"}'

# Provide the chat_history parameter, which contains the conversation history between you and the chatbot. The parameter value is a list in which each element indicates a single round of conversation in the {"user":"Inputs","bot":"Outputs"} format. Multiple conversations are sorted in chronological order.
curl -X 'POST'  '<service_url>api/v1/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question":"What are its features?", "chat_history": [{"user":"What is PAI?", "bot":"PAI is Alibaba Cloud's platform for AI..."}]}'

# If you provide both the session_id and chat_history parameters, the conversation history is appended to the conversation that corresponds to the specified session ID. 
curl -X 'POST'  '<service_url>api/v1/query' -H 'Authorization: <service_token>' -H 'accept: application/json'  -H 'Content-Type: application/json'  -d '{"question":"What are its features?", "chat_history": [{"user":"What is PAI?", "bot":"PAI is Alibaba Cloud's platform for AI..."}], "session_id": "1702ffxxad3xxx6fxxx97daf7c"}'

import requests

SERVICE_URL = 'http://xxxx.****.cn-beijing.pai-eas.aliyuncs.com'
headers = {
    'accept': 'application/json',
    'Content-Type': 'application/json',
    'Authorization': 'MDA5NmJkN****jNlMDgzYzM4M2YwMDUzZTdiZmI5YzljYjZmNA==',
}

def test_post_api_query_with_chat_history(url):
    # Round 1 query
    data = {
       "question": "What is PAI?"
    }
    response = requests.post(url, headers=headers, json=data)

    if response.status_code != 200:
        raise ValueError(f'Error post to {url}, code: {response.status_code}')
    ans = dict(response.json())
    print(f"=======Round 1: Question =======\n {data['question']}")
    if 'answer' in ans.keys():
        print(f"=======Round 1: Answer =======\n {ans['answer']} session_id: {ans['session_id']}")
    if 'docs' in ans.keys():
        print(f"=======Round 1: Retrieved Docs =======\n {ans['docs']}")
   
    # Round 2 query
    data_2 = {
       "question": "What are the benefits of PAI?",
       "session_id": ans['session_id']
    }
    response_2 = requests.post(url, headers=headers, json=data_2)

    if response.status_code != 200:
        raise ValueError(f'Error post to {url}, code: {response.status_code}')
    ans_2 = dict(response_2.json())
    print(f"=======Round 2: Question =======\n {data_2['question']}")
    if 'answer' in ans.keys():
        print(f"=======Round 2: Answer =======\n {ans_2['answer']} session_id: {ans_2['session_id']}")
    if 'docs' in ans.keys():
        print(f"=======Round 2: Retrieved Docs =======\n {ans['docs']}")
    print("\n")

# LLM
test_post_api_query_with_chat_history(SERVICE_URL + "api/v1/query/llm")
# Chat (Knowledge Base)
test_post_api_query_with_chat_history(SERVICE_URL + "api/v1/query")

Precautions

This practice is subject to the maximum number of tokens of an LLM service and is designed to help you understand the basic retrieval feature of a RAG-based LLM chatbot:

• The chatbot is limited by the server resource size of the LLM service and the default number of tokens. The conversation length supported by the chatbot is also limited.

• If multi-round conversation is not required, it is recommended to disable the with chat history feature to effectively reduce the possibility of reaching the limit.

  Web UI operation method: On the Chat tab of the RAG service web UI, deselect the Chat history check box.

  

FAQ

References

• You can also use EAS to complete the following scenario-based deployment:

  • Quickly deploy LLMs in EAS

  • Use ComfyUI to deploy an AI video generation model service

  • Use Stable Diffusion web UI to deploy an AI painting service

• EAS provides stress testing methods for LLM and common services. It can help create stress testing tasks to perform testing with a few clicks, achieving the performance of EAS services in a comprehensive manner. For more information, see Automatic service stress testing.