# How to replace messy data by using linear interpolation of nearby messy data

2 views (last 30 days)
Danupon Subanapong on 13 Nov 2018
Commented: KSSV on 13 Nov 2018
Hello!! I would like to ask for some help. I have been working with the data processing for weeks. I am quite new to this. I have been struggling to find the solution to my problem. My problem is that I have to import data from txt file which 90 % of them are alright, but some contains messy data. I will show an example of input text file in the attached file. I would like to clean this data by replacing it by the linear interpolation of the nearby existing values. My messy data is either in form of ********** or 0.
I have been seeking for an answer for a while, but no solution has been found so far.
Note that the actual input file that I work with is in format of .out not .txt but I cannot attach it here.

#### 1 Comment

KSSV on 13 Nov 2018
YOu can replace the messy data with NaN and use fillmissing

KSSV on 13 Nov 2018
Replace all ****** values in the text file with NaN's and follow the below code:
data = importdata('data.txt') ;
data = data.data ;
t = data(:,1) ;
s1 = data(:,2) ;
s2 = data(:,3) ;
%% Remove outliers in s1
stdDev = nanstd(s1) ;% Compute standard deviation
meanValue = nanmean(s1) ; % Compute mean
zFactor = 1.5; % or whatever you want.
% Create a binary map of where outliers live.
outliers = abs(s1-meanValue) > (zFactor * stdDev);
s1(outliers) = NaN ;
% GEt NaN filled
idx = isnan(s1) ;
s1(idx) = interp1(t(~idx),s1(~idx),t(idx)) ;
%% Remove outliers in s2
stdDev = nanstd(s2) ;% Compute standard deviation
meanValue = nanmean(s2) ; % Compute mean
zFactor = 1.5; % or whatever you want.
% Create a binary map of where outliers live.
outliers = abs(s2-meanValue) > (zFactor * stdDev);
s2(outliers) = NaN ;
% GEt NaN filled
idx = isnan(s2) ;
s2(idx) = interp1(t(~idx),s2(~idx),t(idx)) ;

Show 1 older comment
KSSV on 13 Nov 2018
i jave asked you to replace those stars with nan...did you?
Danupon Subanapong on 13 Nov 2018
Ohh Im sorry it did work.
KSSV on 13 Nov 2018
YOu can also have a look on fillmissing.