llama : add BERT support

[ggerganov]

There is a working bert.cpp implementation.
We should try to implement this in llama.cpp and update the embedding example to use it.

The implementation should follow mostly what we did to integrate Falcon.
Here are the main steps:

• Update gguf.py with BERT arch KV pairs and tensors
• Python convert script using gguf.py to generate F16 model
• add tokenizer implementation in llama.cpp
• add function to build BERT graph
• add any new ops in ggml if needed
• add CUDA offloading
• add tokenizer tests

[dranger003]

@monatis Happy to help with testing or any other tasks you may need help with. Just keep in mind my understanding of LLM architecture is quite limited.

[m0saan]

@ggerganov Is this claimed by somebody? otherwise I can give this a shot!

[ggerganov]

I think @monatis is already working on it

[monatis]

Yes I'm on the work, I'll raise a PR early this week.

[nortekax]

@monatis did this ever happen? I did not see a PR, RAG (retrieval augmented generation) is a major application for which bert models are heavily used. It would be good to add bert model support.

[ggerganov]

Field report: https://bloop.ai/blog/gpu_with_ggml

[monatis]

Ouch, after that I was diving deep into LLaVA. Now that it's shipped, I can go back to this one.

[yunghoy]

Maybe m0saan can take care of this?

[CyrilPeponnet]

While on it maybe worth adding https://jina.ai/news/jina-ai-launches-worlds-first-open-source-8k-text-embedding-rivaling-openai/ :p doesn’t seems to be a regular Bert model though see https://huggingface.co/jinaai/jina-bert-implementation/tree/main