Filtering points (outliers, errors) in 3D plot

I'm given a 3D graph (x,y,z coordinates are given), and this plane has some outliers both above and under the graph (See error1 attachment). I want to remove these outliers:

Using the following code, I'm able to remove all outliers on the topside of the plane (see error2 attachment). Z is the only vector which contains the errors out of x,y,and z coordinates:

y_peaks = Z;

[a,b] = findpeaks(y_peaks);

for j = 1:length(b)

      c = b(j);

      y_peaks(c) = y_peaks(c-1);
  end

zi2=griddata(X,Y,y_peaks,xi,yi);

figure(6);
surf(zi2)

Using this code, I'm able to remove all outliers under the plane (see error3 attachment):

y_min=Z;
Data=y_min;

[Maxima,MaxIdx] = findpeaks(Data);
DataInv = 1.01*max(Data) - Data;
[c,d] = findpeaks(DataInv);

for j = 1:length(d)

      e = d(j);

      y_min(e) = y_min(e-1);
  end

zimin=griddata(X,Y,y_min,xi,yi);

figure(8);
surf(zimin)

How could I combine the code above, or use other code, to remove the outliers from both sides of my initial 3D plot?

Many thanks and kind regards,

Philip