llama : add Mixtral support

[fakerybakery]

Hi,
Please add support for Mistral's MOE model Mixtral.

[ddh0]

Seconded!

EDIT: There is an early implementation here: https://github.com/dzhulgakov/llama-mistral

[fakerybakery]

Does the hacky implementation support quantization?

[ddh0]

Does the hacky implementation support quantization?

It barely supports inference, lol

[lxe]

What's the general effort in modifying gguf-py / convert.py to support quantizing/converting this mistral MoE architecture?

[fakerybakery]

Has anyone seen this?

https://github.com/stanford-futuredata/Megatron-LM/blob/f385caf934b84e71c946c4342362270edae02173/tools/run_text_generation_server.py

[lxe]

Here's a hacked together conversion script: lxe@2dd8944

[irony]

Here's a hacked together conversion script: 2dd8944

Nice!

[mayfer]

Here's a hacked together conversion script: 2dd8944

conversion worked! inference fails with "error loading model: unknown model architecture: 'moe'", any tips?

[fakerybakery]

Hi @lxe, so your script supports conversion but not inference?

[lxe]

It's a work in progress. It can convert to gguf format (with wrong layer names) but there's no logic for quantizing under 8 bits or loading and running the model.

Yet.

[clemens98]

so how much ram or vram is needed to run it?

[leedrake5]

Here's a hacked together conversion script: 2dd8944

conversion worked! inference fails with "error loading model: unknown model architecture: 'moe'", any tips?

Same here. Mystified by what the problem here is. Mac silicon

Update: I'm dumb, just downloaded the script. With all files it creates a 93 Gb half precision model. I've got a system that in theory can run this (128 Gb M3 Max) but still get an error:

error loading model: unknown model architecture: 'moe'
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '~/GitHub/text-generation-webui/models/ggml-model-f16.gguf'

[mayfer]

It requires changes to llama.cpp. each model type has some custom code, it's not all generic. check out source code and see what kind of custom stuff there is for qwen etc.