| Copyright | (c) 2025 Tushar Adhatrao |
|---|---|
| License | MIT |
| Maintainer | Tushar Adhatrao <[email protected]> |
| Safe Haskell | None |
| Language | Haskell2010 |
Llama.Tokenize
Description
Synopsis
- tokenize :: Vocab -> String -> Bool -> Bool -> IO ([LlamaToken], Int)
- tokenToPiece :: Vocab -> LlamaToken -> Bool -> IO String
- detokenize :: Vocab -> [LlamaToken] -> Bool -> Bool -> IO String
Documentation
tokenize :: Vocab -> String -> Bool -> Bool -> IO ([LlamaToken], Int) Source #
Tokenize a string into tokens
tokenToPiece :: Vocab -> LlamaToken -> Bool -> IO String Source #
Convert a token to a piece of text
detokenize :: Vocab -> [LlamaToken] -> Bool -> Bool -> IO String Source #
Detokenize tokens into a string