Why would column data not get synced when parsing a large csv file?

1 view (last 30 days)
I am trying to parse a csv data file that is quite large - 870,000 rows and 30 columns. When I use the script below, my first column is parsed correctly, but the following column data is off by 8 to 10 rows. When I make smaller csv files from chunks of the original csv file the script works as intended. Is this a bug or am I doing something wrong?
clear all
clc
Traj = open('C:\Pass_1_full_data.csv');
% Output traj file
fileName = 'C:\3rd_test_traj.csv';
for i=1:size(Traj.data,1)
Traj.data(i,1) = i;
end
% Pass 1
startImg1 = 1;
endImg1 = startImg1 + 7200;
Rate = 5; % secs as input data is at 1 Hz
TrajData1 = Traj.data(startImg1:Rate:endImg1,:);
fid = fopen(fileName, 'w') ;
fprintf(fid, '%s, ', Traj.textdata{1,1:end-1}) ;
fprintf(fid, '%s\n', Traj.textdata{1,end}) ;
fclose(fid) ;
dlmwrite(fileName, Data1(1:end,:), 'precision' , 10 , '-append') ;
fclose('all');
  2 Comments
Walter Roberson
Walter Roberson on 10 Feb 2020
You are using a side effect of open() that for csv files it happens to call importdata. importdata processes files in ways that can be difficult to predict unless you know the structure of the entire file.
I would recommend that you switch to readtable() or textscan()

Sign in to comment.

Answers (0)

Products


Release

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!