Documentation

Check if the backend supports remote procedure calls (RPC).

Check if the backend supports GPU offloading.

Check if the backend supports locking model memory into RAM (no swapping).

Check if the backend supports memory mapping models.

Get maximum number of devices supported by the backend (e.g., GPUs).

Get current time in microseconds since some unspecified epoch.

Get the maximum context size (n_ctx) of the model in the given context.

Get the batch size (n_batch) used by the context.

Get the unbatched size (n_ubatch).

Get the maximum number of sequences supported.

Get the pooling type used by the context.

Detach the internal threadpool from the context.

Allocate and initialize a new LlamaContextParams with defaults.