Getting Error in installing vllm error on Jetson orin nx jetpack6.2

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
, , ,
1

2025-05-08 10-10-32屏幕截图
2025-05-08 10-10-32屏幕截图1067×666 79.8 KB

jetson orin nx 16G


$ nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver

Copyright (c) 2005-2024 NVIDIA Corporation

Built on Wed_Aug_14_10:14:07_PDT_2024

Cuda compilation tools, release 12.6, V12.6.68

Build cuda_12.6.r12.6/compiler.34714021_0

$ python

Python 3.10.0 | packaged by conda-forge | (default, Nov 20 2021, 02:50:31) [GCC 9.4.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import torch

>>> print(torch.__version__)

2.6.0+cu126

>>> print(torch.cuda.is_available())

True

When I install vllm from https://pypi.jetson-ai-lab.dev/jp6/cu126 I get the following error


pip install vllm-0.8.6+cu126-cp310-cp310-linux_aarch64.whl

Processing ./vllm-0.8.6+cu126-cp310-cp310-linux_aarch64.whl

ERROR: Wheel 'vllm' located at /home/ygsj/dependency/vllm-0.8.6+cu126-cp310-cp310-linux_aarch64.whl is invalid.
3

Hi,

Is docker an option for you?
As we have the vllm container so you don’t need to install it manually.

https://hub.docker.com/r/dustynv/vllm/tags

Thanks.

4

Hello, your suggestion inspired me, but I couldn’t get vllm running with Docker. I encountered a

RuntimeError: operator torchvision::nms does not exist.

But the vllm was successfully run using jetson-containers by jetson-containers run $(autotag vllm)

5

Hi,

jetson-container also uses docker to launch the container but with some jetson-specific settings.
Have you tried to use the same command as jetson-container (which can be found in the console log) to see if it can work?

Thansk.

6

Actually, I don’t really understand Docker, but I can show you the output of my run and the versions of each package.When installing VLLM with Jetson container, I noticed that the VLLM version installed with Jetson container is 0.6.6.post1+cu126.
The versions of other packages are.
python3.10,torch2.5.0,torchvision0.20.0,transformers-4.52.0.dev0,jetpack6.2.
This may be useful for you.

8

jetson-containers run -v /home/ygsj/workspace/Qwen/Qwen2___5-3B-Instruct:/model/Qwen/Qwen2___5-3B-Instruct/ dustynv/vllm:0.6.6.post1-r36.4.0
V4L2_DEVICES:

ARM64 architecture detected

Jetson Detected

SYSTEM_ARCH=tegra-aarch64

  • sudo docker run --runtime nvidia --env NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics -it --rm --network host --shm-size=8g --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /var/run/dbus:/var/run/dbus --volume /var/run/avahi-daemon/socket:/var/run/avahi-daemon/socket --volume /var/run/docker.sock:/var/run/docker.sock --volume /home/ygsj/dependency/jetson-containers/data:/data -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro --device /dev/snd -e PULSE_SERVER=unix:/run/user/1000/pulse/native -v /run/user/1000/pulse:/run/user/1000/pulse --device /dev/bus/usb --device /dev/i2c-0 --device /dev/i2c-1 --device /dev/i2c-2 --device /dev/i2c-4 --device /dev/i2c-5 --device /dev/i2c-7 --device /dev/i2c-9 -v /run/jtop.sock:/run/jtop.sock --name jetson_container_20250515_143735 -v /home/ygsj/workspace/Qwen/Qwen2___5-3B-Instruct:/model/Qwen/Qwen2___5-3B-Instruct/ dustynv/vllm:0.6.6.post1-r36.4.0
9

Hi,

Do you mind testing our latest vLLM container to see if it can work?

jetson-containers run dustynv/vllm:0.8.6-r36.4-cu128-24.04

Thanks.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.