How to import and read several large csv in matlab
7 views (last 30 days)
Show older comments
Ricardo Duarte
on 21 Jun 2022
Commented: Sulaymon Eshkabilov
on 22 Jun 2022
Dear all,
I have a huge amont of large csv files (about 170Mb each).
I need to open them and use the information in two of the columns to plot, let's say collumn D and E (see the attachment).
This is what I did so far, however I found that it is not very efficient.
%Load directory
directorio=uigetdir;
x=dir([directorio,'/','*.csv']);
K=length(x);
Atotal=[];
tabletotal=[];
fprintf('Leitura: \n')
tic;
for k=1:K
FILENAME = strcat(x(k).folder,'/', x(k).name);
TotalData=readtable(FILENAME);
table=TotalData(:,[4 7 18 19 27]); %4-vessel name; %7-vessel type; 18-longitude; 19-latitude
fprintf('File_%s: %s \n', num2str(k), FILENAME);
tabletotal=[table; tabletotal];
%Coluna 2= longitude
%Coluna 3= latitude
end
loadAIS=toc;
fprintf('Tempo de carregamento do AIS: %s \n', num2str(loadAIS) );
clear table;
clear TotalData;
A = table2array(tabletotal(:,2:5));
Thank you in advance
1 Comment
Accepted Answer
Sulaymon Eshkabilov
on 21 Jun 2022
If your data file names are sequential, then you may consider using this type of approach here:
FILE = fullfile('C:\Users\...', '*.csv'); % Directory where the files are residing.
LIST = dir(FILE);
N = length(LIST); % N = number of files
D = zeros(???, ???, N); % Memory allocation to speed up the process of data storing
for ii = 1 : N
FFName = fullfile(LIST(ii).folder, LIST(ii).name);
DATA = readmatrix(FFName);
D(:, :, ii) = DATA; % NOW D contains all data from 10000 files
end
... % Select and process the part of D that is necessary
An alternative way might be this one:
P=pwd; % If your files are located in the current directory
P = fullfile('C:\...\...', '*.csv'); % Specify: directory where the files are residing.
S = dir(fullfile(P, '*.csv')); % Select the file extension to suit your data files
M = 0;
for k = 1:numel(S)
F = fullfile(P, S(k).name);
D = D + readmatrix(F);
end
...
2 Comments
Sulaymon Eshkabilov
on 22 Jun 2022
readmatrix() and readtable() are recommended ones due to their efficiency instead of xlsread() or csvread(), etc.
More Answers (0)
See Also
Categories
Find more on Dialog Boxes in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!