Build failed with cuBLAS

[AlexiaChen]

mathxh@mathxh-server:~$ CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python
Defaulting to user installation because normal site-packages is not writeable
Collecting llama-cpp-python
  Downloading https://mirrors.aliyun.com/pypi/packages/82/2c/9614ef76422168fde5326095559f271a22b1926185add8ae739901e113b9/llama_cpp_python-0.1.50.tar.gz (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 4.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in ./.local/lib/python3.10/site-packages (from llama-cpp-python) (4.5.0)
Building wheels for collected packages: llama-cpp-python
  Building wheel for llama-cpp-python (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for llama-cpp-python (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [109 lines of output]


      --------------------------------------------------------------------------------
      -- Trying 'Ninja' generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      Not searching for unused variables given on the command line.
      -- The C compiler identification is GNU 11.3.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is GNU 11.3.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done (0.6s)
      -- Generating done (0.0s)
      -- Build files have been written to: /tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f/_cmake_test_compile/build
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying 'Ninja' generator - success
      --------------------------------------------------------------------------------

      Configuring Project
        Working directory:
          /tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f/_skbuild/linux-x86_64-3.10/cmake-build
        Command:
          /tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/cmake/data/bin/cmake /tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f/_skbuild/linux-x86_64-3.10/cmake-install -DPYTHON_VERSION_STRING:STRING=3.10.6 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.10 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.10.so -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.10 -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.10 -DCMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/ninja/data/bin/ninja -DLLAMA_CUBLAS=on -DCMAKE_BUILD_TYPE:STRING=Release -DLLAMA_CUBLAS=on

      Not searching for unused variables given on the command line.
      -- The C compiler identification is GNU 11.3.0
      -- The CXX compiler identification is GNU 11.3.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      CMake Warning (dev) in CMakeLists.txt:
        A logical block opening on the line

          /tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f/CMakeLists.txt:9 (if)

        closes on the line

          /tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f/CMakeLists.txt:31 (endif)

        with mis-matching arguments.
      This warning is for project developers.  Use -Wno-dev to suppress it.

      -- Found Git: /usr/bin/git (found version "2.34.1")
      fatal: not a git repository (or any of the parent directories): .git
      fatal: not a git repository (or any of the parent directories): .git
      CMake Warning at vendor/llama.cpp/CMakeLists.txt:105 (message):
        Git repository not found; to enable automatic generation of build info,
        make sure Git is installed and the project is a Git repository.


      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
      -- Found Threads: TRUE
      -- Found CUDAToolkit: /usr/local/cuda/include (found version "12.1.105")
      -- cuBLAS found
      -- The CUDA compiler identification is unknown
      CMake Error at /tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/cmake/data/share/cmake-3.26/Modules/CMakeDetermineCUDACompiler.cmake:603 (message):
        Failed to detect a default CUDA architecture.



        Compiler output:

      Call Stack (most recent call first):
        vendor/llama.cpp/CMakeLists.txt:188 (enable_language)


      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/skbuild/setuptools_wrap.py", line 666, in setup
          env = cmkr.configure(
        File "/tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/skbuild/cmaker.py", line 356, in configure
          raise SKBuildError(msg)

      An error occurred while configuring with CMake.
        Command:
          /tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/cmake/data/bin/cmake /tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f/_skbuild/linux-x86_64-3.10/cmake-install -DPYTHON_VERSION_STRING:STRING=3.10.6 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.10 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.10.so -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.10 -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.10 -DCMAKE_MAKE_PROGRAM:FILEPATH=/tmp/pip-build-env-t04dq_3v/overlay/local/lib/python3.10/dist-packages/ninja/data/bin/ninja -DLLAMA_CUBLAS=on -DCMAKE_BUILD_TYPE:STRING=Release -DLLAMA_CUBLAS=on
        Source directory:
          /tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f
        Working directory:
          /tmp/pip-install-qzc9tklk/llama-cpp-python_6fcd1f65f0b54bae869069faa39cdc1f/_skbuild/linux-x86_64-3.10/cmake-build
      Please see CMake's output for more information.

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for llama-cpp-python
Failed to build llama-cpp-python
ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects

[AlexiaChen]

Seems can not detect cuda arch. but I installed cuda as follows and CUDA toolkit already are found from CMAKE above:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda

My Display card:

*-display
       description: VGA compatible controller
       product: GA106M [GeForce RTX 3060 Mobile / Max-Q]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: /dev/fb0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=nouveau latency=0 mode=1920x1080 resolution=1920,1080 visual=truecolor xres=1920 yres=1080
       resources: iomemory:383c0-383bf iomemory:383e0-383df irq:33 memory:fa000000-faffffff memory:383c00000000-383dffffffff memory:383e00000000-383e01ffffff ioport:e000(size=128) memory:fb000000-fb07ffff

[gjmulder]

    Failed to detect a default CUDA architecture.

Did you install CUDA?

[AlexiaChen]

    Failed to detect a default CUDA architecture.

Did you install CUDA?

yes. i already installed cuda i mentioned above. @gjmulder

[gjmulder]

Apologies. So you did.

I saw this error with cmake builds a while back. Can you try building from source as per my attached Dockerfile? This Dockerfile works with the latest code.

Dockerfile.txt

[AlphaAtlas]

I think this is related to a warning/issue I ran into building vanilla llama.cpp with cmake. I fixed it by manually specifying the CUDA architecture: https://cmake.org/cmake/help/latest/variable/CMAKE_CUDA_ARCHITECTURES.html
set(CMAKE_CUDA_ARCHITECTURES 75)

In fact
set(CMAKE_CUDA_ARCHITECTURES native)
Is probably a good thing to put in this repo's cmake file.

[abetlen]

@AlphaAtlas I can add it here temporarily but do you mind reporting this issue in llama.cpp as that's the more appropriate place to set that flag.

[gjmulder]

set(CMAKE_CUDA_ARCHITECTURES native)

This will potentially need a lot of regression testing against different CUDA installs, as I do not have this problem.

EDIT:

Try running:

nvcc --version

I suspect the problem may be the build can't find nvcc:

if (GGML_CUDA_SOURCES)
    message(STATUS "GGML CUDA sources found, configuring CUDA architecture")
    set_property(TARGET ggml PROPERTY CUDA_ARCHITECTURES OFF)
    set_property(TARGET ggml PROPERTY CUDA_SELECT_NVCC_ARCH_FLAGS "Auto")
    set_property(TARGET llama PROPERTY CUDA_ARCHITECTURES OFF)
endif()

[gjmulder]

Closing. Reopen if the issue is not resolved upstream in llama.cpp