"detectHarrisFeatures" much slower when deploying to Jetson Nano GPU (cuda) than CPU

Good evening, i am developing a basic vision tracker with MALTAB/Simulink to deploy on Jetson Nano board.

The code works fine when targeting the CPU: "Generate GPU code" checkbox in the Code Generation Pane of Configuration Parameters is left unchecked. Frame rate is pretty high on my Jetson Nano and corner points are acquired very fast every time. However, if i check the afore mentioned checkbox in order to generate cuda, the tracking stage executes flawlessly but the application hangs when it has to compute new corners and it takes several seconds (almost a minute). Investigating a little bit, it seems like that for some reasons, neither "detectHarrisFeatures" nor "detectMinEigenFeatures" work well for code generation with cuda. The image size is 360 rows x 640 cols.

The parameters in the pane "GPU code" under Code Generation in Configuration Parameters are left at their default values except for Compute capability which is set to 5.3 according to NVIDIA documentation for Jetson Nano.

The code is as follows (and it works perfectly also when executing the simulink model on my laptop running Windows 10 either with or without GPU acceleration).

function pointsout  = KLTtraker(frameGS, nMinPoints, BlockSize, NumPyramidLevels) %#codegen

persistent pointTracker
persistent numPts
persistent oldPoints
persistent newTrack
% persistent bboxPoints

%% Initialize persistent variables
% Create the face detector object.

if isempty(numPts)
    numPts = 0;
end

if isempty(oldPoints)
    oldPoints = single([0 0]);
end

if isempty(newTrack)
    newTrack = true;
end

% Create the point tracker object.
if isempty(pointTracker)
    pointTracker = vision.PointTracker('NumPyramidLevels',NumPyramidLevels,'BlockSize',BlockSize);
    initialize(pointTracker, single([10 10]), frameGS);
end

%% Detection and Tracking

% Find corner points inside the detected region.
if newTrack
    %     assert(all(size(ROI) == [1,4]));
    %     coder.varsize('xyPoints',[100,2]);
    %     pointsy = rand(100,1)*360;
    %     pointsy(pointsy<1) = 1;
    %     pointsx = rand(100,1)*640;
    %     pointsx(pointsx<1) = 1;
    %     xyPoints = single([pointsx pointsy]);
    
    points = detectHarrisFeatures(frameGS);
    points = points.selectStrongest(100);
    % Re-initialize the point tracker.
    xyPoints = points.Location;
    assert(size(xyPoints,1) <500);
    numPts = size(xyPoints,1);
    
    %     coder.varsize('xyPoints',[500,2]);
    
    % extract points from ROI
    if numPts < nMinPoints %insufficient number of points extracted from ROI
        % quit tracking and return do detect phase
        newTrack        = true;
        pointTracker.reset();
        pointsout       = single([0 0]);
        return
    else
        newTrack    = false;
        % initialize Points tracker
        if ~pointTracker.isLocked
            initialize(pointTracker, xyPoints, frameGS);
        end
        step(pointTracker, frameGS);
        setPoints(pointTracker,xyPoints);
        
        %Save a copy of the points
        oldPoints   = xyPoints;
    end
end

% Tracking mode.
[xyPoints, isFound] = step(pointTracker, frameGS);
assert(size(xyPoints, 1) < 101);
visiblePoints = xyPoints(isFound, :);
oldInliers = oldPoints(isFound, :);

numPts = size(visiblePoints, 1);
if numPts >= nMinPoints
    % I don't really care about knowing the transformation here: i just
    % want to reject outliers.
    [~, ~, visiblePoints] = estimateGeometricTransform(...
        oldInliers, visiblePoints, 'similarity', 'MaxDistance', 4);
    
    % there aren't sufficient inliers: quit tracking
    if visiblePoints < nMinPoints
        newTrack    = true;
        pointTracker.reset();
        pointsout   = single([0 0]);
        return
        % all conditions for successful tracking are met: continue and output
        % points
    else
        pointsout       = visiblePoints;
        oldPoints       = visiblePoints;
        setPoints(pointTracker, oldPoints);
    end
    
    % KLT tracker has found an insufficient number of correspondances: quit
    % tracking
else
    newTrack    = true;
    pointTracker.reset();
    pointsout   = single([0 0]);
    return
end

end

Any help is kindly appreciated.

Thanks in advance,

Marco.

Answer 1

[Milan Bansal]

Hi marco fiorio

I understand that you are developing a basic vision tracker using MATLAB/Simulink to deploy on a Jetson Nano board. Your application performs well when targeting the CPU, but you encounter significant delays when generating CUDA code, specifically during the computation of new corners using "detectHarrisFeatures" or "detectMinEigenFeatures".

As per the documentation of "detectHarrisFeatures" and "detectMinEigenFeatures", these fucntions do not support GPU code generation. MATLAB might allow the code generation process to complete, but it will automatically fall back to executing certain operations on the CPU if those operations are not supported on the GPU. This means that the code will not fully benefit from GPU acceleration for those specific operations, leading to performance bottlenecks.

The below solutions may help to resolve this issue.

Try converting the "frameGS" to GPU Array before passing into "detectHarrisFeatures" function.

 frameGS = gpuArray(frameGS);
 points = detectHarrisFeatures(frameGS);

Alternatively, write a custom Harris feature detector function which will support GPU Code generation. Please refer to the below code snippet to implement basic Harris feature detection.

 function corners = customHarrisFeaturesGPU(frameGS)
    % Ensure the input is a gpuArray for GPU computation
    frameGS = gpuArray(frameGS);
    % Parameters
    k = 0.04; % Sensitivity factor 
    threshold = 1e-6; % Threshold 
    sigma = 1; % Standard deviation

    % 1: Gradients of the image
    [Gx, Gy] = imgradientxy(frameGS, 'Sobel');

    % 2: Compute elements of the Harris matrix M for each pixel
    Ix2 = imgaussfilt(Gx.^2, sigma);
    Iy2 = imgaussfilt(Gy.^2, sigma);
    Ixy = imgaussfilt(Gx.*Gy, sigma);

    % 3: Harris response for each pixel
    R = (Ix2 .* Iy2 - Ixy.^2) - k * (Ix2 + Iy2).^2;

    % 4: Thresholding to find potential corner points
    R(R < threshold) = 0;

    % 5: Non-maximum suppression
    corners = imregionalmax(R);

    % Extract corner positions
    [y, x] = find(corners);
    corners = [x, y];
end

Please refer to the following documentation links to learn more about "detectHarrisFeatures" and "detectMinEigenFeatures" functions.

• https://www.mathworks.com/help/vision/ref/detectharrisfeatures.html
• https://www.mathworks.com/help/vision/ref/detectmineigenfeatures.html

Hope it helps!