ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

[hellostronger]

System Info

a container from FROM nvidia/cuda:11.7.1-cudnn8-devel-ubuntu22.04

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/executor.py line 11
import tensorrt_llm.bindings as tllm I do not understand tensorrt_llm.bindings
how can i fix the problem ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

Expected behavior

ModuleNotFoundError: No module named 'tensorrt_llm.bindings' the problem not allert

actual behavior

ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

additional notes

I con from annother resposity optimum-nvidia which use TensorRT-LLM, then i clone the main branch , then create a new python file just add
'''
from optimum.nvidia.pipelines import pipeline

then  it run and  the error  just like this
![image](https://github.com/NVIDIA/TensorRT-LLM/assets/48884346/0db237cc-d9a4-4cfc-b6d8-a14f951b2b94)

i  want to solve the proplem

[MartinMarciniszyn]

The bindings module will be found if you install the software following the instructions. Please provide the exact steps to reproduce the issue.

[ArlanCooper]

how to solve it?

[ArlanCooper]

i do the steps as instructions,as following:

conda create -n tensorrt python==3.10.12
conda activate tensorrt
conda install mpi4py
pip install mpmath==1.3.0
pip install tensorrt_llm -U --extra-index-url https://pypi.nvidia.com
# Check installation
python -c "import tensorrt_llm"

but it reports the errors as following:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/powerop/work/rwq/TensorRT-LLM/tensorrt_llm/__init__.py", line 46, in <module>
    from .hlapi.llm import LLM, ModelConfig
  File "/home/powerop/work/rwq/TensorRT-LLM/tensorrt_llm/hlapi/__init__.py", line 1, in <module>
    from .llm import LLM, ModelConfig
  File "/home/powerop/work/rwq/TensorRT-LLM/tensorrt_llm/hlapi/llm.py", line 18, in <module>
    from ..executor import (GenerationExecutor, GenerationResult,
  File "/home/powerop/work/rwq/TensorRT-LLM/tensorrt_llm/executor.py", line 11, in <module>
    import tensorrt_llm.bindings as tllm
ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

[MartinMarciniszyn]

We do not support installations in conda environments. This may work but it has not be validated by us. Hence, we cannot provide support for these kinds of installations. Please deploy with docker for full support of the functionality as explained in the installation instructions.

[Ednaordinary]

Renaming .pyi files to .py in the bindings folder seemed to work in a python venv, not sure if it will break anything

Actually! might be related to the next error I got:

File "tensorrt_llm/runtime/model_runner_cpp.py", line 34, in <module>
    DataType.FLOAT: torch.float,
    ^^^^^^^^^^^^^^
AttributeError: type object 'DataType' has no attribute 'FLOAT'

[MarktHart]

The bindings module will be found if you install the software following the instructions. Please provide the exact steps to reproduce the issue.

For future reference:

This is also a problem when following the steps when building a Docker image. You can solve it by setting ENV TORCH_CUDA_ARCH_LIST= .

[peterschmidt85]

I have the same issue even though I'm using the base CUDA 12.1 Docker image (without conda).

[MartinMarciniszyn]

@peterschmidt85 , please post the exact steps to reproduce the issue.

[peterschmidt85]

@peterschmidt85 , please post the exact steps to reproduce the issue.

@MartinMarciniszyn,

1. Run nvidia/cuda:12.1.0-devel-ubuntu22.04

docker run --rm --runtime=nvidia --gpus all --entrypoint /bin/bash -it nvidia/cuda:12.1.0-devel-ubuntu22.04

2. apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev
3. pip3 install tensorrt_llm==0.9.0 -U --extra-index-url https://pypi.nvidia.com
4. python3 -c "import tensorrt_llm"

Fails with the problem that it is not compatible with NumPy 2.

After pip install "numpy<2", I try it again, and it fails with

ModuleNotFoundError: No module named 'tensorrt_llm.bindings'

I was trying to install random versions of tensorrt and tensorrt_llm and once I got it working but later couldn't find how I did it.

[MartinMarciniszyn]

@peterschmidt85, please test this with the latest release 0.10.0.

[peterschmidt85]

5. python3 -c "import tensorrt_llm"

Wow, 0.10.0 works. Thank you!

Is there any reason why the older version stopped working? I supposed many pinned it and depend on it.

[MartinMarciniszyn]

I suspect this might be due to some Python dependencies that were not correctly pinned and evolved in the meanwhile. However, this is just a guess.