Converting alpaca-native-GPTQ models into ggml models

[BadisG]

Expected Behavior

Hello,

I wanted to convert the alpaca-native 7b GPTQ file (pt file) into a ggml file with the convert-gptq-to-ggml.py script https://github.com/ggerganov/llama.cpp/blob/master/convert-gptq-to-ggml.py

Current Behavior

The problem is that I have this error

D:\Large Language Models\CONVERTISSEURS\gptq to ggml>python convert-gptq-to-ggml.py alpaca-native-4b
it.pt tokenizer.model out.bin
32000
32001
Traceback (most recent call last):
  File "D:\Large Language Models\CONVERTISSEURS\gptq to ggml\convert-gptq-to-ggml.py", line 35, in <
module>
    assert tokenizer.vocab_size() == n_vocab
AssertionError

32000 is the tokenizer.vocab_size() (Number of tokens on the tokenizer.model)
32001 is the n_vocab (Number of tokens on the model)

The model that is trained with alpaca has 1 more token and it's this one:
"[PAD]": 32000

It looks like that if we want to convert the alpaca native GPTQ models we need to create a new tokenizer.model that has this "PAD" token in it.

The problem is that I have no idea how to do that... if someone can help me on this I'll appreciate!

[Ronsor]

I wrote a tool to add additional tokens to tokenizer.model: https://github.com/Ronsor/llama-tools

The token list:

C [PAD]

would work with the script I wrote.

[BadisG]

@Ronsor I used your script and it looks like it did actually add the token on the tokenizer.model

But now I have a new error... looks like the issue is more complex than I thought 😅

D:\Large Language Models\CONVERTISSEURS\gptq to ggml>python convert-gptq-to-ggml.py alpaca-native-4b
it.pt tokenizer.model out.bin
32001
32001
Processing non-Q4 variable: model.embed_tokens.weight with shape:  torch.Size([32001, 4096])  and ty
pe:  torch.float32
Processing non-Q4 variable: model.norm.weight with shape:  torch.Size([4096])  and type:  torch.floa
t32
  Converting to float32
Processing non-Q4 variable: lm_head.weight with shape:  torch.Size([32001, 4096])  and type:  torch.
float32
Traceback (most recent call last):
  File "D:\Large Language Models\CONVERTISSEURS\gptq to ggml\convert-gptq-to-ggml.py", line 153, in
<module>
    convert_q4(f"model.layers.{i}.self_attn.q_proj", f"layers.{i}.attention.wq.weight", permute=True
)
  File "D:\Large Language Models\CONVERTISSEURS\gptq to ggml\convert-gptq-to-ggml.py", line 94, in c
onvert_q4
    zeros = model[f"{src_name}.zeros"].numpy()
KeyError: 'model.layers.0.self_attn.q_proj.zeros'

[comex]

Looks like the zeros issue corresponds to a recent commit to GPTQ-for-LLaMa (with a very non-descriptive commit message) which changed the format. The change is not actually specific to Alpaca, but the alpaca-native-GPTQ weights published online were apparently produced with a later version of GPTQ-for-LLaMa. The zeros and scales are now separate for every group of 32 weights, but the zeros are now themselves scaled and quantized… I don't really understand how that makes sense. I'll figure it out when I have a chance.

[BadisG]

On the convert_q4(src_name, dst_name, permute=False): function I changed:

    zeros = model[f"{src_name}.zeros"].numpy()
    ...
    qweight = model[f"{src_name}.weight"].numpy().T # transpose

to

    zeros = model[f"{src_name}.qzeros"].numpy()
    ...
    qweight = model[f"{src_name}.qweight"].numpy().T # transpose

That results in those dimensions.

    print(grouped.shape) -> (4096, 128, 4)
    print(scales_rep.shape) -> (32, 524288, 1)
    print(addends_rep.shape) -> (32, 65536, 1)

Which gives an error because we cannot concanetate those objects anymore.

Here's a comparaison with the regular llama-7b-gptq model (that works well with the converter)

    print(grouped.shape) -> (4096, 128, 4)
    print(scales_rep.shape) -> (4096, 128, 1)
    print(addends_rep.shape) -> (4096, 128, 1)

At this point I'm stuck, as I'm uncertain about which elements (groupings, scales, addends) to modify in order to achieve the desired concatenation

[BadisG]

@comex I'm not sure it was a good idea to convert your addends and scales into int32, those tensors have really small numbers and we're loosing all the informations like that:

[comex]

They're not 'really' int32s. Each int32 is actually 8 4-bit weights packed together. And they're not converted directly from float to integer; they have to be interpreted together with the addends and scales.

[daboe01]

maybe you are lucky with this one?
https://huggingface.co/ozcur/alpaca-native-4bit/tree/main
maybe this was generated just before the zeros patch was merged.

[Belluxx]

maybe you are lucky with this one? https://huggingface.co/ozcur/alpaca-native-4bit/tree/main maybe this was generated just before the zeros patch was merged.

Just tried, it fails with KeyError: 'model.layers.0.self_attn.q_proj.zeros'

[comex]

I spent some time today working on this but didn't finish.

[BadisG]

oobabooga merged a PR that makes the alpaca-7b-4bit-GPTQ-native works now
oobabooga/text-generation-webui@49c10c5

That's funny it worked because it uses the exact same tokenizer model (the one with 32000 token) even though this model has one more.

[daboe01]

cool! do you see any significant improvements from GPTQ?

[BadisG]

@daboe01 I have the RTN quantized on llama cpp and the GPTQ quantized on the webui but it would be hard to compare the 2 as they are a bit differents in the way they work.

The best comparaison would be RTN vs GPTQ in llama.cpp with a perplexity test, I'll wait for @comex to do his magic! 👀