Excluding data points from optimization process using createOptimProblem function

I have a objective function which produces a 2x13 vector output based on a mathematical model. I'm using 'createOptimProblem' function with 'lsqcurvefit' to compare this output to experimental data of same size, producing a theoretical fit to my experimental data. The problem is that some of my experimental datapoints have huge error, and because of that I would like to exclude them from the optimization process. I would still like to have the fit based on my model to cover the full range of 2x13 points, but I would like the 'lsqcurvefit' function to exclude the experimental datapoints of my choosing when it's calculating the residual error. The attached image shows an example of fit where the experimental datapoints highlighted with green circles are ones I would like to exclude from the residual error calculation. Is there a way to do this using the createOptimProblem function or could you think better ways to do this?

Answer 1

[Matt J]

Instead of using lsqcurvefit's calculation of the residual error, why not do your own cusotmized calculation after the optimization is complete? The thing you want to do has nothing to do with the optimization process that I can see.

[Ville Tiainen]

Maybe I have misunderstood something here, but the way I understand the 'createOptimProblem' function works is that it calculates the objective function value and uses the 'lsqcurvefit' function to calculate the error with respect to the experimental data that I'll give to it. After that it repeats the process untill satisfactory fit is found. The residual error is just a way to judge how good my fit is. It is not minimizing my objective function, but the least squares error of my fit vs the experimental data. To be clear I don't care so much about the value of residual error except that it should be small enough to give me a good fit. What I would like it to do is to do this same thing except that it should ignore the bad data points of my choosing when doing the error calculation.

[Matt J]

You said in your initial post "I would still like to have the fit based on my model to cover the full range of 2x13 points", which sounds like you don't want the optimization process to exclude the defective data.

If you do want the optimization process to exclude defective data, you must simply

(1) remove that data from your ydata input.

(2) modify your model function so that it does not generate predictions of those ydata values.

[Ville Tiainen]

Sorry for the confusion maybe i didn't do a very good job explaining the issue. Indeed I would like to have the fit covering the full range. For each value of x my objective function gives me 2 values of y. For said x value my experimental data could be accurate for only 1 value of y1, while the other y2 value could be rubbish. This is why I would like to exclude only some of experimental values as I have as shown in the figure attached to the first post. The problem is that these excluded values don't come in pairs so I cannot just simply exclude both of them from my ydata like you suggest. As I also would like to have the fit covering the full range I cannot modify my objective function like you suggest next. The problem with excluding only 1 of two experimental y values is then that the vector sizes don't match with the objective function output and the experimental data which throws an error with 'lsqcurvefit' function.

After googling the whole day for a fix it starts to look like this cannot be done using 'createOptimProblem' and 'lsqcurvefit' so if you have any ideas for alternative approach I would very much like to hear it.

[Matt J]

There's no reason your ydata needs to be organized as a 2xN matrix. Discard the undesired values using a logical index keep and arrange ydata as a vector. Here's how you might wrap your current model fuction mdl(x,xdata) to accomplish this:

ydata_new=ydata(keep);  %new ydata

mdl_new=@(x,xdata) wrapper(x,xdata,keep); %new objective function



function [ypred,varargout]=wrapper(x,xdata,keep)

 [ypred,varargout(1:nargout-1)]=mdl(x,xdata);
 ypred=ypred(keep);
end