Memory issue to create 4D array

I have 35000 funds image data-set i want to create 4D array of this data-set but when i ran the code and read images one by one than error occurred memory limitation . plz help me to resolve issue.

my code is here.

close all
clear all
Directory= 'C:\Users\Shehzad\Desktop\sample\*.jpeg';
Imgs = dir(Directory);
for j=1:length(Imgs)
    thisname = Imgs(j).name;
    D=Imgs(j).folder;
    thisfile = fullfile(D, thisname);
    Img = imread(thisfile);  % try to read image
    data(:,:,:,j)=Img;
    clear Img;
end
save Data data
save Imgs Imgs

Output is:

Requested 2500x2700x3x630 (11.9GB) array exceeds maximum array size preference. Creation of arrays greater than this limit

may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more

information.

Error in Project (line 10)

data(:,:,:,j)=Img;

Answer 1

[Rik]

The amount of data is probably huge, is not pre-allocated and will probably not fit in memory. Have you calculated an estimated size of the needed array?

If you can't add fysical RAM to your system, there are some tools that you can use to access data still on your disk and only load them to RAM when you need it, like datastore and imageDatastore.

[Shehzad Ali]

First of all thank you.

yes I have done it. its 300GB approximately.

but I divide my dataset into 3 parts, 11706 images in each part. We need appx. 115GB memory required for each part. How can I use datastore and imageDatastore plz tell me about it.

[Walter Roberson]

What would you do with a 115 gigabytes dataset in memory if you were able to construct it?

How much memory do you have?

[Shehzad Ali]

I want to make 4D array of that dataset and then save as data.mat file.

I have 12GB RAM.

[Walter Roberson]

If you use matfile and a -v7.3 .mat then you might be able to extend it one slice at a time without storing the entire array in memory.

What are you going to do with the 140 or so gigabyte .mat file that you get as output? You will never be able to load it on your computer.

[Shehzad Ali]

I am working on Dibatic retinopathy dectection using convolutional neural network. once i made it then I load one by one image through imageDatastore.

[Walter Roberson]

Here it sounds like the "one by one image" that you intend to load with imageDatastore would be each of the three 4D arrays that are 115 gigabytes each ??

I thought Diabetic Retinopathy involved individual RGB images (so 3D), rather than volume scans or time series ?