MATLAB Answers

Why would column data not get synced when parsing a large csv file?

3 views (last 30 days)
Ronney Lovelace
Ronney Lovelace on 10 Feb 2020
Commented: Ronney Lovelace on 11 Feb 2020
I am trying to parse a csv data file that is quite large - 870,000 rows and 30 columns. When I use the script below, my first column is parsed correctly, but the following column data is off by 8 to 10 rows. When I make smaller csv files from chunks of the original csv file the script works as intended. Is this a bug or am I doing something wrong?
clear all
clc
Traj = open('C:\Pass_1_full_data.csv');
% Output traj file
fileName = 'C:\3rd_test_traj.csv';
for i=1:size(Traj.data,1)
Traj.data(i,1) = i;
end
% Pass 1
startImg1 = 1;
endImg1 = startImg1 + 7200;
Rate = 5; % secs as input data is at 1 Hz
TrajData1 = Traj.data(startImg1:Rate:endImg1,:);
fid = fopen(fileName, 'w') ;
fprintf(fid, '%s, ', Traj.textdata{1,1:end-1}) ;
fprintf(fid, '%s\n', Traj.textdata{1,end}) ;
fclose(fid) ;
dlmwrite(fileName, Data1(1:end,:), 'precision' , 10 , '-append') ;
fclose('all');

  2 Comments

Walter Roberson
Walter Roberson on 10 Feb 2020
You are using a side effect of open() that for csv files it happens to call importdata. importdata processes files in ways that can be difficult to predict unless you know the structure of the entire file.
I would recommend that you switch to readtable() or textscan()

Sign in to comment.

Answers (0)

Sign in to answer this question.

Products


Release

R2019a