How to run a program on an csv file data in batches

Hello

I am trying to import data from csv file with size (1046520 1). Using the importdata function.

K = load('caamel.csv'); 

It is taking over an hour to compute my program.

How do I run my code on the excel file in batches instead of one whole data which is taking forever to see result

I will appreciate your help

thanks

Tino

[Jan]

You mention the importdata function, and show code with the load function, and explain, that the data is a CSV file, and that it is an Excel file. The confusion is perfect.

Please post the code you use and attach a small example file such that the readers can be sure, which format it has.

[Tino]

Sorry it is a csv file and the function load

[Jan]

load cannot import CSV files, so the confusion remains perfect.

[Tino]

I mean to use the word load instead of import Jan. Thanks for your help in advance

[KSSV]

YOu want entire data to be loaded or a part of it to process ?

[Tino]

Hi KSSV

Just some part to be loaded and analysed at a time.

Tino

[Walter Roberson]

Jan, load() can read numeric csv files.

[Walter Roberson]

Tino, see the recent discussion at https://www.mathworks.com/matlabcentral/answers/467449-how-to-increase-reading-speed-from-a-gigabyte-large-file#comment_715761

[M Carpenter]

I think to answer this properly we would still need to see the file Tino, can you attach it?

[Walter Roberson]

A (1046520 1) csv file would not even contain any commas: it would just be one number per line with only whitespace as the delimiter.

[Tino]

Hi Walter, How do I then implement the computation

Thanks in advance

Tino

[Walter Roberson]

You implied that the reason that your code is taking so long is that import (but really load) is slow. I showed timing on a file 50 times larger than yours could take about 20 minutes to load() so your file would take about half a minute to load(). If you were to switch to textscan then it should take roughly 250/50 = 5 seconds. At that point you might as well read in all the data and loop over portions of it.

textscan also permits you to to specify a count to read, if you find that easier than loading everything and looping indexing.