CSV Import doesn't show up properly?

Hey guys,

So I'm trying to import a timeseries from a csv file (dim roughly 210 x 175,000; corresponds to timepoints x voxels, file size roughly 572 MiB). When I load the file generated by my program into excel, it looks good and exactly how it should. However, when I load it into matlab, it imports as a (175000x210)x1 (ie, a 1 dimensional array whose dimensions are totally screwed up). Note that the file has a header column with the coordinates of each voxel, so I am loading the file as:

    M = csvread('file/path/file.csv', 1, 0)

which should read starting at row index 1 and column index 0 (basically, just chop off the top row). why is it loading improperly into matlab? Note that in a smaller matrix that is created by the same program (200 MiB) the file loads properly and is, in that case, 210 x 50000 roughly (note that this matrix is about 3 times smaller). Is this a matlab error, or an error in creating the files? I did not actually write the code and the program is technically in a beta version, but I do not think they would screw up such a simple error, and it does load into excel just fine, it is only loading into matlab that gives it the dimensionality error. Any thoughts/suggestions?

Answer 1

[Star Strider]

The headers may be the glitch. According to the csvread documentation:

• M = csvread(filename) reads a comma-separated value (CSV) formatted file into array M. The file must contain only numeric values. (Emphasis added.)

Perhaps textscan would be preferable in your application.

Your choice.

[Star Strider]

I suspect that it is reading the header as numeric variables, and could be confusing some of the header information as commas, or there could be additional commas in the header.

Consider using textscan if you cannot remove the header manually, and if you know the number of columns of data you have. (If there are ‘hundreds of thousands’ of columns, use repmat to create the format descriptor string.)

Answer 2

[Eric]

Is there a reason why the file would appropriately remove the header (which has a couple hundred thousand columns) perfectly, but does not read the rest of it properly? By definition, if the header is a couple hundred thousand elements, the rest of the matrix would also need to be a few thousand elements in column width. I am very unfamiliar with comma separated values, and only really need to use it in this scope for the forseeable future, so as much as I would love to read about ways of parsing textscan I feel like it would be easier to figure out why csv is not working for me. Here is a snapshot of a small section of the cells in excel. Why would excel be able to perfectly parse the file, but matlab would not? This is very confusing to me.

as you can see, the top is the header with the 3D volume position, the left column is the time, and the other elements are the data points.