I am trying to detect device memory leaks using cuda memcheck tool. I intentionally added some cudamalloc() without corresponding cudaFree() and I expected the tool to detect them. But the tool reported no leaks.
Documentation says cudaDeviceReset() should be called at the end of the application for this tool to work.I called this API and still no change.
My application has a loop that has big iteration count based on user inputs and all the cudamalloc() happens inside this loop. My requirement is to catch those cudamalloc() that do not have a corresponding cudaFree() .
We are supposed to call cudaDeviceReset() only when we are about to exit the application, right? So, I cannot use this API in each iteration to avoid leaks.
So, my question is can cuda memcheck be used to detect device memory leaks in this scenario?
If yes, what could be the reason it is not catching the cudamalloc() which does not have a cudaFree()?
As per my understanding cudaDeviceReset() cleans up all the device resources. Does it include the device memory allocated by cudamalloc()? If so, when we call this API , any of the device allocation that was not freed using cudafree() will be freed. Then, how can we expect this tool to catch cudamalloc() that are not freed using cudaFree()? Is this the reason for no leak report by the tool in my usecase?
CUDA-MEMCHECK is no longer supported, can you please try drop-in replacement tool compute-sanitizer? Documentation is available at Compute Sanitizer Documentation. Thanks!
Thanks for the response.
I tried compute sanitizer with Cuda 12.0 on windows. I get below error.
“option is ambiguous and matches ‘binary-patching’, ‘check-api-memory-access’, ‘check-cache-control’,…”
Below is my command:
compute-sanitizer --tool memcheck --leak-check full – coretb --incfg D:\UC_01\test_config.xml --api_mode 0
Could you please point out what is wrong in the above command? Should I do anything explicitly for the tool to work other than usual CUDA tool kit installation? or any specific compiler options?
Could you please also comment on the last part of my original post (repeated below).
“As per my understanding cudaDeviceReset() cleans up all the device resources. Does it include the device memory allocated by cudamalloc()? If so, when we call this API , any of the device allocation that was not freed using cudafree() will be freed. Then, how can we expect this tool to catch cudamalloc() that are not freed using cudaFree()? Is this the reason for no leak report by the tool in my usecase?”
A minor correction in the command. This is the exact one I use.
compute-sanitizer --tool memcheck --leak-check full –- coretb --incfg D:\UC_01\test_config.xml --api_mode 0