llama : improve sep token handling #14272
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
ggerganov
reviewed
Jun 19, 2025
ggml-ci
ggerganov
approved these changes
Jun 19, 2025
compilade
reviewed
Jun 20, 2025
ggml-ci
ggml-ci
Nexesenex
pushed a commit
to Nexesenex/croco.cpp
that referenced
this pull request
Jun 20, 2025
gabe-l-hart
added a commit
to gabe-l-hart/llama.cpp
that referenced
this pull request
Jun 20, 2025
* mamba2-sync: (24 commits) sync : ggml Add `ggml_roll` (ggml/1274) docs : fix the link to llama.h (ggml-org#14293) CUDA: add conv_2d_transpose (ggml-org#14287) lint : remove trailing whitepace (ggml-org#14304) vocab : prevent tokenizer overflow (ggml-org#14301) sycl: add usage of enqueue_functions extension (ggml-org#14244) Implement GGML_CPU_ALL_VARIANTS for PowerPC (ggml-org#14286) llama : improve sep token handling (ggml-org#14272) cuda : synchronize graph capture and cublas handle destruction (ggml-org#14288) ggml : fix repack work size for mul_mat_id (ggml-org#14292) ggml: Update KleidiAI to v1.9.0 (ggml-org#14277) model : more uniform output id handling (ggml-org#14275) ubatch : new splitting logic (ggml-org#14217) CUDA: add conv_2d_dw (ggml-org#14265) ggml-cpu : remove unnecesary arm feature detection (ggml-org#14281) gguf-py : make sentencepiece optional (ggml-org#14200) server : add server parameters for draft model cache type (ggml-org#13782) build : suppress gcc15 compile warnings (ggml-org#14261) sycl: Cleanup codepaths in Get Rows in sycl backend (ggml-org#14215) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sequence classification models (classifiers/rerankers) require a different pattern of special tokens than other models, so far this has been hard-coded to the most common pattern, but as we are getting support for more models this is starting to fail.
The way
AutoTokenizerhandles this is by using a specialTemplateProcessorintokenizer.json. This PR does not add full support for this as that would require major changes to thellama.cpptokenizer, but instead simply adds a newadd_sep_tokenstate that can be used together with the existingadd_bos_tokenandadd_sep_tokento figure out what tokens to add.SpecialVocabwill now crudely parse thetokenizer.configprocessor to figure out which of these to add to metadata, thenllama-serverandllama-embeddingwill add the appropriate tokens (however, still in the fixed pattern as before, so if we ever add support for a model with an unusual pattern, we will most likely have to rework things).Added a new
--cls-separatorparameter forllama-embeddingto automatically split sequence pairs and insert the correct tokens.