Jetson AGX Orin: LLaVa Tutorial failed in building text-generation-webui-torchvision

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi, I’m trying to follow LLaVa tutorial(LLaVA - NVIDIA Jetson AI Lab) to test LLaVa on Jetson AGX Orin 32GB, and getting some error while starting web UI with multimodal extension.

I’ve succeeded to download model(TheBloke/llava-v1.5-13B-GPTQ) with some other images (dustynv/text-generation-webui:r36.2.0 or text-generation-webui:r36.4-cu126-22.04-{installable one} I guess), but when trying to run

jetson-containers run --workdir=/opt/text-generation-webui $(autotag text-generation-webui) \
  python3 server.py --listen \
    --model-dir /data/models/text-generation-webui \
    --model TheBloke_llava-v1.5-13B-GPTQ \
    --multimodal-pipeline llava-v1.5-13b \
    --loader autogptq \
    --disable_exllama \
    --verbose

or at least

jetson-containers build text-generation-webui

always get error while checking torchvision:

┌────────────────────────────────────────────────────────────────┐
│ > TESTING  text-generation-webui:r36.4-cu126-22.04-torchvision │
└────────────────────────────────────────────────────────────────┘

docker run -t --rm --network=host --runtime=nvidia \
  --volume /home/$USER/jetson-containers/packages/pytorch/torchvision:/test \
  --volume /home/$USER/jetson-containers/data:/data \
  --workdir /test \
  text-generation-webui:r36.4-cu126-22.04-torchvision \
    /bin/bash -c 'python3 test.py


testing torchvision...
Traceback (most recent call last):
  File "/test/test.py", line 12, in <module>
    import torchvision
  File "/usr/local/lib/python3.10/dist-packages/torchvision/__init__.py", line 10, in <module>
    from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils  # usort:skip
  File "/usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py", line 164, in <module>
    def meta_nms(dets, scores, iou_threshold):
  File "/usr/local/lib/python3.10/dist-packages/torch/library.py", line 828, in register
    use_lib._register_fake(op_name, func, _stacklevel=stacklevel + 1)
  File "/usr/local/lib/python3.10/dist-packages/torch/library.py", line 198, in _register_fake
    handle = entry.fake_impl.register(func_to_register, source)
  File "/usr/local/lib/python3.10/dist-packages/torch/_library/fake_impl.py", line 31, in register
    if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist
[13:32:54] Failed building:  text-generation-webui

Traceback (most recent call last):
  File "/home/$USER/jetson-containers/jetson_containers/build.py", line 129, in <module>
    build_container(**vars(args))
  File "/home/$USER/jetson-containers/jetson_containers/container.py", line 244, in build_container
    test_container(container_name, pkg, simulate)
  File "/home/$USER/jetson-containers/jetson_containers/container.py", line 431, in test_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'docker run -t --rm --network=host --runtime=nvidia   --volume /home/$USER/jetson-containers/packages/pytorch/torchvision:/test   --volume /home/$USER/jetson-containers/data:/data   --workdir /test   text-generation-webui:r36.4-cu126-22.04-torchvision     /bin/bash -c 'python3 test.py' 2>&1 | tee /home/$USER/jetson-containers/logs/20250528_133240/test/text-generation-webui_r36.4-cu126-22.04-torchvision_test.py.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.

This is my current versions of sw

$USER@ubuntu:~$ jetson-containers build text-generation-webui

Namespace(packages=['text-generation-webui'], name='', base='', multiple=False, build_flags='', build_args='', use_proxy=False, package_dirs=[''], list_packages=False, show_packages=False, skip_packages=[''], skip_errors=False, skip_tests=[''], test_only=[''], simulate=False, push='', no_github_api=False, log_dir=None, log_level=None, log_colors=None, log_status=None, debug=False, verbose=False, version=False)

┌───────────────────────┬────────────────────────┐
│ L4T_VERSION   36.4.3  │ JETPACK_VERSION  6.2   │
│ CUDA_VERSION  12.6    │ PYTHON_VERSION   3.10  │
│ SYSTEM_ARCH   aarch64 │ LSB_RELEASE      22.04 │
└───────────────────────┴────────────────────────┘

$ jetson-containers text-generation-webui

These are versions of my torch-related packages:

$USER@ubuntu:~$ pip list | grep torch
torch                 2.7.0
torch_tensorrt        2.6.0.dev0+44375f208
torchvision           0.22.0
2

Let me tag @dusty_nv that can help you

Best,
Raffaello

4

Hi @acensia ,
Same issues here. RuntimeError: operator torchvision::nms does not exist
And I modify the Dockerfile under the jetson-containers/packages/pytorch/torchvision/ Make the FORCE_BUILD=off to on [ref]

#---
# name: torchvision
# group: pytorch
# config: config.py
# depends: [pytorch, cmake]
# test: test.py
#---
ARG BASE_IMAGE
FROM ${BASE_IMAGE}

ARG TORCHVISION_VERSION \
    #FORCE_BUILD=off
    FORCE_BUILD=on

COPY build.sh install.sh /tmp/torchvision/

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        libjpeg-dev \
        libpng-dev \
        zlib1g-dev \
	&& rm -rf /var/lib/apt/lists/* \
	&& apt-get clean

RUN /tmp/torchvision/install.sh || /tmp/torchvision/build.sh

Maybe you can give it a try, after that use jetson-containers to build torchvision again.

$ jetson-containers build torchvision

1 Like
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.