I still don't know what problem is but I found another solution without 'inpolygon':
lat = linspace(66,73,420)'; % latitude
lon = linspace(-5,18,1380)'; % longitude
[LAT,LON] = ndgrid(lat,lon); % 2-D matrix
LAT = LAT(:); LON = LON(:); % 2-D to 1-D
contourf(lon,lat,z) % bathymetry ('z' is depth of the ocean)
[xi,yi] = getpts(gca); % choose mask border with mouse
pgon = polyshape(xi,yi); % 2-D polygons
TFin = isinterior(pgon,LON,LAT); % query points inside polyshape
mask = reshape(TFin,[420,1380]); % from 1-D to 2-D
hold on
plot(xi, yi,'k','LineWidth',2) % polygon (line)
plot(LON(mask),LAT(mask),'r+') % points inside polygon
Maybe it isn't the best solution but it's work.