I want to build up hand gestures detection and recognition algorithm, I need help with my code and sadly it doesn't work

%vid to intialize a variable to save incoming video %videoinput to get video from webcam, video format, adopter number

vid = videoinput('winvideo', 1);

%Frames per trigger specifies the number of frames acquired, inf used %to state that frames will be acquired till an error occurs or stopped

set(vid,'FramesPerTrigger',inf);

%Returned color space is rgb as we are using this format

set(vid,'ReturnedColorSpace','rgb')

%extract images every 5 milli-second

vid.FrameGrabInterval=5;

%start acquiring video

start(vid);

while(vid.FramesAvailable <= 20)

    % Get a snapshot of the current frame
    data = getsnapshot(vid);

    %Skin Color Extraction
    J=rgb2ycbcr(data);
    L=graythresh(J(:,:,2));
    BW=im2bw(J(:,:,2),L);
    BW1=~BW;
    M=graythresh(J(:,:,3));
    BW2=im2bw(J(:,:,3),M);
    o=BW1.*BW2;
    roi=o;

    % Here we do the image blob analysis.
    % We get a set of properties for each labeled region.

    stats = regionprops(roi, 'BoundingBox', 'Centroid');    

    imshow(data);

    hold on

    %This is a loop to bound the skin colour in a rectangular box.

    for object = 1:length(stats)

        bb = stats(object).BoundingBox;
        bc = stats(object).Centroid;
        rectangle('Position',bb,'EdgeColor','y','LineWidth',1)
        plot(bc(1),bc(2), '-m+')

    end

    hold off

end

stop(vid); flushdata(vid);

Answer 1

[Image Analyst]

It looks like your skin detection is all messed up. Whoever told you to use graythresh() for finding skin tone colored blobs in your image is mistaken. You need to customize your threshold to use fairly fixed thresholds. graythresh() tries to automatically find a threshold based on what's there and it can get confused if there is not enough skin tone, or too much, or the presence of other colors. Try the Color Thresholder app on the Apps tab of the tool ribbon to home in on the threshold you should use.

[Ahmed El Sheikh]

No one told me about, I am just trying to build the algorithm. Can you help me?

[Image Analyst]

What did you learn from the Color Thresholder?

I've also attached two demos.

[Ahmed El Sheikh]

Color Thresholder showed me 4 different color spaces each is formed of 3 color bands, these color bands are the values that mix up to form an image that we see.

Skin_Segmentation demo you read an image, converted to RGB then filtering out the noise, then converting to HSV color space to blur out H and S channels but not the V channel (I'd like to know why?) and then you recombined the channels again to have our image with more continous colors (what does continuous color mean) and you got binary image of skin in HSV color space using a threshold of each color band and you anded them all together same for the YCbCr color space.

Skin_Color demo you showed me how does the skin color of human varies in LAB color space and you represented it in C, H, L color bands

Question: Did I fully understand/know what you wanted me to understand/know?

[Image Analyst]

Correct. But with the Color Thresholder, did you pick one of the color spaces and continue on with the app to do the actual thresholding and generate the code? Or did you just exit it there and not proceed?

For skin segmentation we ignore the lightness/value channel because it's not unique to skin. There are lots of stuff in the scene which may have the same lightness as skin and we don't want all of those things. However there are not many things that have the same hue and saturation as skin so we won't get much non-skin clutter. Plus skin can be very dark or very light - it's all over the place so you can't count on lightness as a way to zero in on skin in the image.

The LAB color gamut that you can rotate around is to show you how the skin color gamut is a boomerang-shaped gamut and that there is no way to perfectly get it by "boxing it in" with thresholds in any color space. However you can get close with hue and chroma/saturation segmentation.

[Ahmed El Sheikh]

Yes, I have done that, but it does not work efficiently with all images.And, I have been working on your demo Skin_segmentation and I am using the HSV & YCbCr color spaces and it works very fine in perfect lighting conditions.

Q1 For each and every skin color, I have to re threshold the color bands, is there any way to automatically detect the skin color? Q2 After detecting the skin how can I make sure that what I detected is hand before proceeding to recognize the gesture?

[Image Analyst]

Q1: The hue of the skin, believe it or not, is in a fairly tight range regardless of the skin color or race. That's why I gave you that demo that showed the skin gamut. The gamut was collected over a large variety of skin colors. The better you can "carve out" that gamut, the better. Using hue and saturation only, you can get like a "blade" or "fin" that sticks out from the L or V axis. However using just that will mean that you could get slightly reddish medium grays (located in the middle of the "elbow" of the boomerang as being called skin. So it's not perfect but you can improve your situation by using an uncluttered background with bright lights and using a color calibrated system (use an x-rite Color Checker chart). So it's better to use calibrated CIELAB (see my avatar), but that's a whole other complicated subject to get into. However I frequently use calibrated color (well maybe about half the time depending on the situation).

Q2: There are tons of papers on gesture recognition and I haven't read any of them - haven't needed to. I suggest you look up some of them here: http://www.visionbib.com/bibliography/contents.html

[Ahmed El Sheikh]

regarding question 1 how can I make sure if what I detected of Skin color is of the hand not any other body part? And I will start searching through that link you sent me

[Image Analyst]

You can try filtering based on size or location. For example the hand will be smaller than the face and will probably be lower in the field of view (though maybe not).

[Ahmed El Sheikh]

Can't i use edge detection to compare with an image of a hand, or features extraction?

[Image Analyst]

Why? Do you have flesh colored hands in front of a flesh colored background screen? I'd hope not. Why not attach an image so we can see it? Edge detection is rarely the best solution - I don't know why everyone wants to try it as a first step. I'm pretty sure you're best off looking at the colors of the hand and surrounding scene rather than doing edge detection, because, after all, things like faces and clothing will also have lots of edges in them so how will you tell the difference between hand edges and clothing edges? Maybe if only the hands are moving you can use optical flow, or combine optical flow with color segmentation.