Phi-2: changed model schema

[bearn01d]

Currently, the conversion of Phi-2 using convert-hf-to-gguf.py fails, due to the recently changed model format on HuggingFace:
https://huggingface.co/microsoft/phi-2/commit/cb2f4533604d8b67de604e7df03bfe6f3ca22869 and https://huggingface.co/microsoft/phi-2/commit/eb8bbd1d37d258ea74fb082c53346d33056a83d4.
Adaptation would be very desirable, but also needs to include adaptation to the new model schema including renamed layers.

[ggerganov]

Please test #4903

[bearn01d]

Will do so by hopefully Monday. Thanks a lot!

[Nold360]

Tested successful with vanilla latest phi-2! :)

This is probably a different issue, but i tried it on mlabonne/phixtral-4x2_8 and it gives me different error messages. (same with TheBloke/phixtral-4x2_8-GPTQ )

convert using master build:

$ podman run -ti --rm -v/data/llm:/models reg.dc/llama.cpp /bin/bash
root@a26bbf2a1588:/app# ./convert-hf-to-gguf.py /models/phixtral-4x2_8-GPTQ/
Loading model: phixtral-4x2_8-GPTQ
gguf: This GGUF file is for Little Endian only
Set model parameters
Could not find any of ['partial_rotary_factor']

Using #4903 i get this error instead:

$ podman run -ti --rm -v/data/llm:/models reg.dc/llama.cpp:phi /bin/bash
root@8a3f79989c68:/app# ./convert-hf-to-gguf.py /models/phixtral-4x2_8-GPTQ
Loading model: phixtral-4x2_8-GPTQ
gguf: This GGUF file is for Little Endian only
Set model parameters
Set model tokenizer
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
gguf: Adding 50000 merge(s).
gguf: Setting special token type bos to 50256
gguf: Setting special token type eos to 50295
gguf: Setting special token type unk to 50256
gguf: Setting special token type pad to 50256
gguf: Setting chat_template to {% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '
' + message['content'] + '<|im_end|>' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
' }}{% endif %}
Exporting model to '/models/phixtral-4x2_8-GPTQ/ggml-model-f16.gguf'
gguf: loading model part 'model.safetensors'
output.bias, n_dims = 1, torch.float16 --> float32
output.weight, n_dims = 2, torch.float16 --> float16
output_norm.bias, n_dims = 1, torch.float16 --> float32
output_norm.weight, n_dims = 1, torch.float16 --> float32
token_embd.weight, n_dims = 2, torch.float16 --> float16
blk.0.attn_norm.bias, n_dims = 1, torch.float16 --> float32
blk.0.attn_norm.weight, n_dims = 1, torch.float16 --> float32
blk.0.attn_qkv.bias, n_dims = 1, torch.float16 --> float32
Can not map tensor 'transformer.h.0.mixer.Wqkv.g_idx'

Next i tried cognitivecomputations/dolphin-2_6-phi-2 which also results in different error messages..

So it seems this relates to #4615 somehow.

The new error looks familiar to the one of the phixtral model:

$ podman run -ti --rm -v /data/llm/:/models reg.dc/llama.cpp:phi /bin/bash                                          
root@0bc13cd79adf:/app# ./convert-hf-to-gguf.py /models/dolphin-2_6-phi-2/       
Loading model: dolphin-2_6-phi-2                                                 
gguf: This GGUF file is for Little Endian only                                   
Set model parameters                                                             
Could not find any of ['partial_rotary_factor'] 

[lastrosade]

Sorry for resurrecting this, but I'm having the same issue with dolphin-2_6-phi-2 where using convert-hf-to-gguf.py yields partial_rotary_factor

I'm running commit 87c91c0 which is the latest as I am typing this.

Is the model itself using an outdated arch or config?