Why would column data not get synced when parsing a large csv file?

I am trying to parse a csv data file that is quite large - 870,000 rows and 30 columns. When I use the script below, my first column is parsed correctly, but the following column data is off by 8 to 10 rows. When I make smaller csv files from chunks of the original csv file the script works as intended. Is this a bug or am I doing something wrong?
clear all
clc
Traj = open('C:\Pass_1_full_data.csv');
% Output traj file
fileName = 'C:\3rd_test_traj.csv';
for i=1:size(Traj.data,1)
Traj.data(i,1) = i;
end
% Pass 1
startImg1 = 1;
endImg1 = startImg1 + 7200;
Rate = 5; % secs as input data is at 1 Hz
TrajData1 = Traj.data(startImg1:Rate:endImg1,:);
fid = fopen(fileName, 'w') ;
fprintf(fid, '%s, ', Traj.textdata{1,1:end-1}) ;
fprintf(fid, '%s\n', Traj.textdata{1,end}) ;
fclose(fid) ;
dlmwrite(fileName, Data1(1:end,:), 'precision' , 10 , '-append') ;
fclose('all');

2 Comments

You are using a side effect of open() that for csv files it happens to call importdata. importdata processes files in ways that can be difficult to predict unless you know the structure of the entire file.
I would recommend that you switch to readtable() or textscan()

Sign in to comment.

Answers (0)

Products

Release

R2019a

Asked:

on 10 Feb 2020

Commented:

on 11 Feb 2020

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!