How to import and read several large csv in matlab
20 views (last 30 days)
Show older comments
Ricardo Duarte on 21 Jun 2022
Commented: Sulaymon Eshkabilov on 22 Jun 2022
I have a huge amont of large csv files (about 170Mb each).
I need to open them and use the information in two of the columns to plot, let's say collumn D and E (see the attachment).
This is what I did so far, however I found that it is not very efficient.
FILENAME = strcat(x(k).folder,'/', x(k).name);
table=TotalData(:,[4 7 18 19 27]); %4-vessel name; %7-vessel type; 18-longitude; 19-latitude
fprintf('File_%s: %s \n', num2str(k), FILENAME);
%Coluna 2= longitude
%Coluna 3= latitude
fprintf('Tempo de carregamento do AIS: %s \n', num2str(loadAIS) );
A = table2array(tabletotal(:,2:5));
Thank you in advance
Sulaymon Eshkabilov on 21 Jun 2022
If your data file names are sequential, then you may consider using this type of approach here:
FILE = fullfile('C:\Users\...', '*.csv'); % Directory where the files are residing.
LIST = dir(FILE);
N = length(LIST); % N = number of files
D = zeros(???, ???, N); % Memory allocation to speed up the process of data storing
for ii = 1 : N
FFName = fullfile(LIST(ii).folder, LIST(ii).name);
DATA = readmatrix(FFName);
D(:, :, ii) = DATA; % NOW D contains all data from 10000 files
... % Select and process the part of D that is necessary
An alternative way might be this one:
P=pwd; % If your files are located in the current directory
P = fullfile('C:\...\...', '*.csv'); % Specify: directory where the files are residing.
S = dir(fullfile(P, '*.csv')); % Select the file extension to suit your data files
M = 0;
for k = 1:numel(S)
F = fullfile(P, S(k).name);
D = D + readmatrix(F);
Sulaymon Eshkabilov on 22 Jun 2022
readmatrix() and readtable() are recommended ones due to their efficiency instead of xlsread() or csvread(), etc.
More Answers (0)
Find more on Mathematics in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!