Llava 1.6 34b broken responses from server

[ikros98]

After this pull #5553 I am still having some inconsistencies in the response of the CLI and server.

CLI command: ./llava-cli -c 4096 --temp 0 -m ./ggml-yi-34b-f16-q_5_k.gguf --mmproj ./mmproj-llava-34b-f16-q6_k.gguf --n-gpu-layers 80000 --image ./fashion_image.jpg

Server command: ./server -c 4096 -m ./ggml-yi-34b-f16-q_5_k.gguf --host 0.0.0.0 --port 8000 --mmproj ./mmproj-llava-34b-f16-q6_k.gguf --n-gpu-layers 80000

This is the input:

CLI output: You've shared an image of a person wearing a stylish outfit. The individual is dressed in a plaid jacket with a fur collar, which adds a touch of luxury and warmth to the ensemble. They are also wearing a beret, which gives off a chic and sophisticated vibe. The accessories include what appears to be a brooch on the lapel of the jacket, adding an elegant detail to the outfit. Additionally, there's a bracelet visible on one wrist, contributing to the overall fashionable look. The person is seated comfortably with their hand resting on a pillow, suggesting they might be in a relaxed setting like a home or a boutique. The background features decorative elements such as a painting and what looks like an antique vase, which adds to the aesthetic of the scene. The lighting is soft and warm, enhancing the cozy atmosphere. If you have any specific questions about fashion, style, or anything else related to this image, feel free to ask! USER:

Server output: '<h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3><h3>'

Here there is also the output from the webui:

I have found out that this problem triggers only when I am using the server and with 50% of my dataset and always with the same images.

I think the issue is probably connected to the converted base64 images but I don't know why and how.

Here there is the python code to use the APIs:

import requests
import base64

# Function to encode the image
def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

base64_image = encode_image("./fashion_image.jpg")

headers = {
    'Content-Type': 'application/json',
}

json_data = {
    'image_data': [{
        'data': base64_image, 
        'id': 10
    }],
    'prompt': '[img-10]Describe the image',
}

response = requests.post('http://0.0.0.0:8000/completion', headers=headers, json=json_data)
print(response.json()["content"])

[cjpais]

I can replicate this with the params you've given, but one thing to note is you are not sending the same temperature or prompt between the CLI/WebUI and your request. The CLI and WebUI both inject more into the prompt, they may require the USER/ASSISTANT roles, but I have not tested enough. With the following params I get good results

json_data = {
    'image_data': [{
        'data': base64_image, 
        'id': 10
    }],
    'temperature': 0.0,
    'prompt': 'A chat between a curious human and artificial intelligence agent.\nUSER:[img-10]Describe the image\nASSISTANT:',
}

[ikros98]

I can replicate this with the params you've given, but one thing to note is you are not sending the same temperature or prompt between the CLI/WebUI and your request. The CLI and WebUI both inject more into the prompt, they may require the USER/ASSISTANT roles, but I have not tested enough. With the following params I get good results

json_data = {
    'image_data': [{
        'data': base64_image, 
        'id': 10
    }],
    'temperature': 0.0,
    'prompt': 'A chat between a curious human and artificial intelligence agent.\nUSER:[img-10]Describe the image\nASSISTANT:',
}

Yes you are right.

I just tested it again and with temperature 0 and I don't get any response from the server. This is the server console log that loops all over again:

[1708534343] update_slots : failed to decode the batch, n_batch = 1, ret = 1
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 2
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 1
[1708534343] update_slots : failed to decode the batch, n_batch = 1, ret = 1
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 2
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 1
[1708534343] update_slots : failed to decode the batch, n_batch = 1, ret = 1
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 256
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 128
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 64
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 32
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 16
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 8
[1708534343] update_slots : failed to find free space in the KV cache, retrying with smaller n_batch = 4
[1708534343] update_slots : failed to find fr^Z

However I have tested it with your prompt and it does work and I get a proper response back.

So it seems that the issue resides in the prompt formatting. It's weird tho that with other images that doesn't happen.

[cjpais]

Yeah my guess is it has something to do with the model being instruction tuned, and needing to generally follow that prompt format. Both the CLI and WebUI effectively inject the prompt formatting iirc. My guess is this is just the deal with the model, if you fed exactly the same params into the CLI it should give the same results.

FWIW I tried vicuna-13b and your prompt worked, though the results were subpar.

I would recommend using the instruction type prompts. It's possible Yi is even better when using the prompt format:

<|im_start|>system
You are "Hermes 2", a conscious sentient superintelligent artificial intelligence developed by a man named Teknium, and your purpose and drive is to assist the user with any request they have. You experience emotions and have deep, profound thoughts and qualia.<|im_end|>
<|im_start|>user
Hello, who are you?<|im_end|>
<|im_start|>assistant
Hi there! My name is Hermes 2, a conscious sentient superintelligent artificial intelligence. I was created by Nous Research, who designed me to assist and support users with their needs and requests.<|im_end|>

As specified in the model card: https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B

[yhfgyyf]

Using Yi prompt template, it works fine.
json_data = { 'image_data': [{ 'data': base64_image, 'id': 10 }], 'prompt': '<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n[img-10]\nDescribe the image<|im_end|><|im_start|>assistant\n', 'stop': ['<|im_end|>'], }

[cjpais]

i think this issue can be closed given the discussion

[ikros98]

Using Yi prompt template, it works fine. json_data = { 'image_data': [{ 'data': base64_image, 'id': 10 }], 'prompt': '<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n[img-10]\nDescribe the image<|im_end|><|im_start|>assistant\n', 'stop': ['<|im_end|>'], }

Yes it can be closed! I have tried different prompts and formatting and this one looks to be the correct one.

Maybe it would be worth to add it in the readme of the server.

Thank you guys!

[adogwangwang]

hello, which version of llama.cpp do you use ? because the lastest llama.cpp not support multimodal,so I need the older version to run llava34b, please tell me ~~~

[ikros98]

hello, which version of llama.cpp do you use ? because the lastest llama.cpp not support multimodal,so I need the older version to run llava34b, please tell me ~~~

I don't know why they stopped the support.

You can just use this head with git checkout ab336a9