Why do I get "CUDA_ERRO​R_ILLEGAL_​ADDRESS" error depending on my input size?

I am using "gpuArray" objects and performing multiple calculations using these "gpuArray" objects. I have a custom CUDA script that was compiled with the "mexcuda" function. My script works perfectly fine and as expected when my input size is small. However, with larger input sizes, I receive  the following error: 

An error occurred during PTX compilation of <image>.
The information log was:
The error log was:
The CUDA error code was: CUDA_ERROR_ILLEGAL_ADDRESS.

The error only seems to occur when indexing the matrix in MATLAB that is outputted from my CUDA script. How can I resolve this error?

Answer 1

[MathWorks Support Team]

"CUDA_ERROR_ILLEGAL_ADDRESS" is a CUDA error that can occur for multiple reasons. Refer to the following five troubleshooting steps which should address the most common reasons for this error:

1) The most likely cause is when accessing a memory address that is not intended to be accessed. For example, trying to access index 10 of an array that only has 5 elements. Check to make sure that all CUDA memory accesses are valid and in the bounds of array sizes. This is the first troubleshooting step that should be taken before trying other steps.

 

2) After checking to see there are no out-of-bounds memory accesses, you should then check if you have the most recent NVIDIA drivers for your GPU. To download the latest NVIDIA drivers, please refer to the following NVIDIA link:

        https://www.nvidia.com/Download/index.aspx?lang=en-us

3) Another potential cause for this CUDA error is when the GPU runs out of memory. GPU memory usage can be monitored by using the "nvidia-smi" command line tool. "nvidia-smi" ships with the CUDA toolkit, for more information regarding "nvidia-smi" please refer to the following NVIDIA documentation link:

        https://developer.nvidia.com/system-management-interface

4) Another useful tool that will help reduce the scope of the issue and give more information besides the simple "CUDA_ERROR_ILLEGAL_ADDRESS" error message, is "cuda-memcheck". This tool is shipped with the CUDA toolkit and is capable of precisely detecting and attributing out of bounds and misaligned memory access errors in CUDA applications. To run "cuda-memcheck" on a CUDA MEX script, you must first compile the CUDA MEX code with the "-G" compiler flag, similar to the following.code:

mexcuda -G myCudaMexFunction.cu

To run "cuda-memcheck" on a MATLAB script that uses a compiled CUDA MEX script, please use the following command:

cuda-memcheck /path/to/matlab -nodisplay -nojvm -batch "cd <to your code>; <name of MATLAB script to run>

For more information on "cuda-memcheck", please refer to the following NVIDIA documentation link:

        https://developer.nvidia.com/compute-sanitizer

5) Another potential cause of "CUDA_ERROR_ILLEGAL_ADDRESS" is when using complex data. If the CUDA MEX function takes complex data as input, it is important to make sure that the complex data is handled correctly on both the MATLAB end and the CUDA receiving end. If the handling of complex data from either side (MATLAB or CUDA) is incorrect, the CUDA error "CUDA_ERROR_ILLEGAL_ADDRESS" can occur randomly or can occur dependent on the input size. For proper complex data handling, for example, in MATLAB, suppose you create the following matrix:

a = complex(ones(4,'gpuArray'),ones(4,'gpuArray'));

If you pass a "gpuArray" to a MEX-function as the first argument (prhs[0]), then you can get a pointer to the complex data by using the following code:

mxGPUArray const * A = mxGPUCreateFromMxArray(prhs[0]);
mwSize numel_complex = mxGPUGetNumberOfElements(A);
double2 * d_A = (double2 const *)(mxGPUGetDataReadOnly(A));

For more information regarding complex data in CUDA MEX functions, please refer to the following MathWorks documentation link:

• Run MEX Functions Containing CUDA Code