when will baichuan2 be supported?

[niubi-AI]

i think it is the best model in Chinese. i tried llama.cpp to run baichuan2, but failed

[niubi-AI]

the following message was displayed:

CUDA error 9 at D:\llama.cpp\ggml-cuda.cu:6517: invalid configuration argument

[BarfingLemurs]

If the first model will load, and there are no architectural changes to the second model, then it should load in cpu mode.

[niubi-AI]

sth changed , but not much

[goerch]

sth changed , but not much

What do you see? I'm interested because we are trying to fix Aquila here and I think I have heard of Baichuan being mentioned in the same context (@KerfuffleV2 : any ideas?).

[KerfuffleV2]

I think I have heard of Baichuan being mentioned in the same context (@KerfuffleV2 : any ideas?).

Sorry, I don't know. I've heard of Baichun but never messed with it. There was a recent pull that was supposed to add support for Baichun models in general: #3009

I guess Baichuan 2 is different and wouldn't have been included in that.

@dansinboy Did you try what BarfingLemurs said and try to run it in pure CPU mode?

[gewanbo]

I have tried to use convert-baichuan-hf-to-gguf.py to convert Baichuan2-7B-Chat to gguf format, it is successful, and then I tried to quantize the model, it is successful. But both of these two models can't loaded. The error at the end:

llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type  f16:  226 tensors
llm_load_print_meta: format         = GGUF V2 (latest)
llm_load_print_meta: arch           = baichuan
llm_load_print_meta: vocab type     = SPM
llm_load_print_meta: n_vocab        = 125697
llm_load_print_meta: n_merges       = 0
llm_load_print_meta: n_ctx_train    = 4096
llm_load_print_meta: n_ctx          = 512
llm_load_print_meta: n_embd         = 4096
llm_load_print_meta: n_head         = 32
llm_load_print_meta: n_head_kv      = 32
llm_load_print_meta: n_layer        = 32
llm_load_print_meta: n_rot          = 128
llm_load_print_meta: n_gqa          = 1
llm_load_print_meta: f_norm_eps     = 1.0e-05
llm_load_print_meta: f_norm_rms_eps = 1.0e-06
llm_load_print_meta: n_ff           = 11008
llm_load_print_meta: freq_base      = 10000.0
llm_load_print_meta: freq_scale     = 1
llm_load_print_meta: model type     = 7B
llm_load_print_meta: model ftype    = mostly F16 (guessed)
llm_load_print_meta: model size     = 7.51 B
llm_load_print_meta: general.name   = Baichuan2-7B-Chat
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 0 '<unk>'
llm_load_print_meta: LF token  = 1099 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.09 MB
error loading model: create_tensor: tensor 'token_embd.weight' has wrong shape; expected  4096, 125697, got  4096, 125696,     1,     1
llama_load_model_from_file: failed to load model
Traceback (most recent call last):
  File "llama2longchain.py", line 48, in <module>
    llm = LlamaCpp(
  File "/Users/www/miniconda3/envs/Test/lib/python3.8/site-packages/langchain/load/serializable.py", line 75, in __init__
    super().__init__(**kwargs)
  File "/Users/www/miniconda3/envs/Test/lib/python3.8/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for LlamaCpp

I'm looking forward that this will be supported, thanks.

[akawrykow]

Looks like an issue with the vocab size? We have:
n_vocab = 125697
But it is expecting a tensor with size 125696 so probably some token got omitted somewhere.

[KerfuffleV2]

Please check out #3299 and see if that fixes your issue. Also if anyone can test other Baichuan models like Baichuan1 that would be appreciated.

I converted the 7b base model. It seems to work now:

main: prompt: '从前有一只小狐狸，他'
main: number of tokens in prompt = 8

sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.000000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0

 从前有一只小狐狸，他住在森林里。有一天，一只大灰狼来到森林里，看见了一只小狐狸，就问他：“你叫什么名字？” 小狐狸说：“我叫小狐狸。”“那好，我以后就叫你‘小狐狸’吧！” 大灰狼说完，就把小狐狸给吃掉了。

edit: Also, yikes. What's with that story? Maybe he deserved to die for having such an uncreative name.

[goerch]

@KerfuffleV2 : We are checking at the same time :)

For me the following commands seem to work without any change:

python.exe convert.py models\Baichuan2-7B-Base
.\build\bin\Release\main.exe -m models\baichuan2-7B-Base\ggml-model-f16.gguf -p "The meaning of life is " --temp 0 --color

Same goes for models\Baichuan-7B. The models look to me like having LlaMa architecture with sentencepiece-based tokenizers.

Edit: obvious question: does convert-baichuan-hf-to-gguf.py do something special during conversion?

[KerfuffleV2]

I actually didn't even try with the plain convert.py, I just assumed since there was a special script for it that I needed to use that. Maybe the "fix" is to just delete the Baichuan-specific script? There was a point a while back where a number of other scripts got deleted for being obsolete/non-functional.

Can anyone confirm that Baichuan(2) is exactly the same architecture as LLaMA?

---

edit: It has its own LLM_ARCH_BAICHUAN and there's special handling in llama.cpp for when that architecture is set. The specific conversion script also sets that architecture. I didn't compare the code between that and normal LLaMA carefully.

I'm not sure what the implications are of converting the Baichuan models as if they're LLaMA. Presumably someone at some point thought they needed to be handled differently (Baichuan1 at least). If that's not the case then we can rip out a bunch of special case stuff in llama.cpp too.

edit: Also, apparently you can convert Baichuan2 to Baichuan1 pretty easily: https://github.com/baichuan-inc/Baichuan2/blob/main/README_EN.md#migrating-inference-optimizations-from-baichuan-1-to-baichuan-2

[goerch]

Can anyone confirm that Baichuan(2) is exactly the same architecture as LLaMA?

Here is some documentation I found.

It has its own LLM_ARCH_BAICHUAN

Interesting.

Edit: I diff'd llama.cpp code for LLM_ARCH_LLAMA and LLM_ARCH_BAICHUAN and didn't notice any differences. @ggerganov : we could discuss if command argument support for ARCH would be sensible in convert.py then?

[KerfuffleV2]

I grabbed the 13B base version: https://huggingface.co/baichuan-inc/Baichuan2-13B-Base

convert.py actually can't handle this one:

Exception: failed to guess 'n_ctx'. This model is unknown or unsupported.
Suggestion: provide 'config.json' of the model in the same directory containing model files.

The dedicated conversion script did convert it, however it doesn't actually work properly. With the prompt "从前有一只小狐狸" it just repeats "一只大老虎。" (a big tiger) forever. Don't think it matters what the prompt is, it gets stuck repeating the same thing. It's not just nonsense though, which is interesting.

I thought it might have been because of

llm_load_print_meta: f_norm_eps     = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-06

f_norm_eps = 0.0 looks weird, but hacking the loader to use 1.0e-06 or 1.0e-05 made no difference. I think there's something architecturally different with the 13B model.

I think using the convert to Baichuan1 thing I mentioned above might make it work, but I didn't get a chance to try that yet.

---

Doing:

    for name in model_part.keys():
        data = model_part[name]
        if name == 'lm_head.weight':
            print('>>> Normalizing lm_head.weight')
            data = torch.nn.functional.normalize(data)

does not seem to have any effect. No idea what's wrong at this point.

---

According to: https://github.com/baichuan-inc/Baichuan-13B/blob/main/README_EN.md#model-details

For Baichuan1, the 13B uses ALiBi instead of RoPE. Might be the same in Baichuan2, the result does look like an attention problem so I can believe this is the issue. I guess this would mean it's not really a problem with conversion but with how llama.cpp handles the graph.

It seems like there is code to use ALiBi for 13B though: https://github.com/ggerganov/llama.cpp/blob/324f3403d54ae4499a1d68623161015f7419fb76/llama.cpp#L2975-L2983

edit: Last edit, I promise. I tried hacking llama.cpp to treat it like a 7B (in other words, to use RoPE). Also didn't work: "从前有一只小狐狸,从19870年,在19月2月19月19日19日19日19日19日19日19日1919191919191919191"

[goerch]

@KerfuffleV2 : 7B's config.json contains

  "max_position_embeddings": 4096,
  "model_max_length": 4096,

but 13B's contains only

  "model_max_length": 4096,

So I'm now trying with

        if "max_sequence_length" in config:
            n_ctx = config["max_sequence_length"]
        elif "max_position_embeddings" in config:
            n_ctx = config["max_position_embeddings"]
        elif "model_max_length" in config:
            n_ctx = config["model_max_length"]
        else:
            raise Exception("failed to guess 'n_ctx'. This model is unknown or unsupported.\n"
                            "Suggestion: provide 'config.json' of the model in the same directory containing model files.")

in convert.py. Will report back.

[KerfuffleV2]

So I'm now trying with

I'm pretty sure the context size saved in GGUF is purely cosmetic. You can set -c to whatever you want, the only difference is you'll get a warning if it's above the what was defined in the metadata. (Also I guess it affects where metadata gets dumped.) As far as I know there's no functional effect though.