Error when trying to write data to GPU DMA memory (using GPU Direct RDMA)

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
, ,
5

I’ve never used dma with a gpu.

I thought gpudirectrdma was the way, so I looked at the gpudirect rdma example and wrote the code.

                        uint8_t *gpuBuf
   user                 cudaHostAlloc(gpuBuf)
                        cuPointerSetAttribute(gpuBuf)
   -------------------------------------------------------------------------------                        
                        // input: gpuBuf virtual address, output: gpuBuf page table 
   kernel               nvidia_p2p_get_pages(gpuBuf_virtAddr, &gpu_page_table) 
  (dirver)              // input: gpu_page_table (out value of nvidia_p2p_get_pages function), output: gpu_dma_mapping
                        nvidia_p2p_dma_map(gpu_page_table, &gpu_dma_mapping)

I think here ‘gpu_dma_mapping_->hw_address’ is the physical address of the gpu dma.
So I gave this address to the fpga.
However, when the fpga tries to write data to this address, an error occurs.

And I have an additional question.

https://github.com/NVIDIA/jetson-rdma-picoevb/blob/master/kernel-module/picoevb-rdma.c#:~:text=ram_offset%2C%20pcie_addr%2C%20len)%3B-,/%20Create%20descriptor%20/,desc%2D%3Enxt_adr_hi%20%3D%200%3B,-return%20pevb_dma(pevb%3B-,/%20Create%20descriptor%20/,desc%2D%3Enxt_adr_hi%20%3D%200%3B,-return%20pevb_dma(pevb)
(Line 709 ~ Line 721)

I don’t understand this part.
Why does dma_alloc_coherent write gpu dma address to dma address allocated and write to hardware register?
Can’t the gpu dma address be directly written to a hardware register?