loading large structure for data analysis

I have developed many tools (scripts) which use the load command to load data structures which contain .data,.textdata, .file, .... In many instances when the tool is called to load data, the structure definition is loaded but the .data is empty. I have tried creating a try / catch pair to identify the issue but it happens way to often with large datasets. Is there another way to read in large data structures that would prevent this issue?

[Matt Tearle]

Can you clarify what kind of file you're loading? It sounds like you're using load to read some kind of formatted text file. If so, what does the file look like, and is there a reason you're using load, rather than, say, csvread, dlmread, or textscan? Also, how large is "large"?

[Dean Weems]

The data being loaded is stored as *.mat. The mat file contains a structure with the following fields: .data, .textdata, .index

They can be rather large files but the failure is not consistent with the size of the .mat file.

[Matt Tearle]

Hmm. Weird. Can you maybe post the relevant lines of code?

I assume, then, you're also creating the mat-files somewhere along the line? ie x.data = ...; x.textdata = ...; save('foo.mat','x'); [somewhere else] foo = load('foo.mat'); x = foo.x; isempty(x.data) % returns true :(

So one thing I'm wondering is if you can narrow down whether the problem is at the save or load stage. First, can you load the mat-file another way (ie interface, command line, command syntax vs fn syntax, etc). Second, maybe you can look at the file sizes of the mat-files (if the data field is the main memory footprint, the size of the mat-file should correlate closely to the size of .data).

I'm assuming, here, that this is happening consistently *for a given file*(?).

[Dean Weems]

I use the gui interface provided in Matlab to select the input file and then simply use the load(file) command. Nothing very difficult. But you are correct the .data is the majority of the data. But it is not consistent with a particular file but fairly consistent with the larger files. I use my code on both 32 and 64 bit systems and the issue exists in both. If I place a break in my code at the load line and then step thru the code using F10 then it sometimes works but I often cut and paste the load command into the Matlab window and continue to execute there until it loads properly.

[Matt Tearle]

Whut? If you repeat the exact command several times it eventually loads ok?

Woah. That doesn't sound good. Sorry, I think i'm out of ideas. Can you give customer support a call? Maybe with the actual data and code, they could work out what's going on. Sorry I couldn't be more help. Anyone else out there have any ideas?

[Matt Tearle]

Actually, just to be absolutely sure (although I doubt it will help, given the behavior you've described)... if you're using the syntax

load('file')

foo = x.data; % 'file' contains a variable called x

can you maybe just check that there isn't some weird name conflict issue by using

tmp = load('file');

x = tmp.x;

foo = x.data;

instead.

[Dean Weems]

tmp is an empty structure with field names but no data. Same result as before.

Answer 1

[Ovidiu Lasca]

I have the same problem on a Win 64 bit system (both XP and Win 7):

I have example.csv file (2.5 GB) which I import into Matlab using the load command. Then I save the array (size 13mil X 22) as example.mat file using the option -v7.3. It comes up to 600 MB.

Next I try to upload it into Matlab again using: EXAM = load('example.mat') If I request size(EXAM), it reports a 1X1 array even though it is supposed to be 13mil by 22. In the Workspace window it is described as "1X1 struc".

Using the command: load example.mat gives me the correct data. Could somebody from Matlab look into this and get back to us? Thanks!

[Matt Tearle]

This is a different issue, and is the expected behavior. Calling |load| with an output argument returns a structure (that contains the data); the command syntax |load foo| loads the variables in foo.mat directly into the workspace