Import CSV file to 2012b make Windows 8 Entire UI hang!

I just try to import CSV file, which has 1054419 rows and 16 cols. However, during import, windows 8 has no response at all, the entire UI hang.

I try many times but fail. 2012b preformance is too bad.

I already try to use csvread(), however, it also return error.

Error using dlmread (line 139) Mismatch between file and format string. Trouble reading number from file (row 1u, field 3u) ==> x0000,0,0x0012,1,0,0,3,66,20,2012,6,21,10,44,4\n

Error in csvread (line 48) m=dlmread(filename, ',', r, c);

here is the sample data: 2nd and 4th cols data type is HEX:

18251,0x0000,0,0x0012,1,0,0,3,66,20,2012,6,21,10,44,4 18251,0x0000,10,0x0010,1,1,38,0,226,2432,2012,6,21,10,44,4 18251,0x0000,12,0x000e,0,0,0,20,28,84,2012,6,21,10,44,4 18251,0x0000,13,0x0000,0,1,85,0,91,5440,2012,6,21,10,44,4 18251,0x0000,15,0x000e,0,1,23,0,25,1472,2012,6,21,10,44,4 18251,0x0000,16,0x0000,0,1,44,0,51,2816,2012,6,21,10,44,4 18251,0x0000,18,0x000e,0,0,0,15,17,72,2012,6,21,10,44,4 18251,0x0000,23,0x000e,0,0,0,15,21,68,2012,6,21,10,44,4 18251,0x0000,23,0x000e,0,1,1,0,1,64,2012,6,21,10,44,4 18251,0x0000,25,0x000e,0,1,55,0,225,3520,2012,6,21,10,44,4 18251,0x0000,30,0x000e,0,1,121,0,244,7744,2012,6,21,10,44,4 18251,0x0000,36,0x000e,0,0,1,20,30,92,2012,6,21,10,44,4 18251,0x0000,37,0x0000,0,1,47,0,56,3008,2012,6,21,10,44,4 18251,0x0000,38,0x000e,0,1,78,0,104,4992,2012,6,21,10,44,4 18251,0x0000,42,0x0000,0,0,86,0,87,2680,2012,6,21,10,44,4 18251,0x0000,42,0x0000,0,1,747,0,1490,47608,2012,6,21,10,44,4 18251,0x0000,43,0x0000,0,0,0,15,55,64,2012,6,21,10,44,4

[Andreas Goser]

Only a comment as not necessarily related to your issue. Win 8 is not a supported OS for R2012b: http://www.mathworks.com/support/sysreq/current_release/

[mingming wang]

However, Before I upgrade to 2012b, I use 2012a on windows 8 and import this csv with no issue. Though Matlab itself may has no response during import, it does not lead to the whole windows 8 UI hang, other programs run normally. For matlab, I always care performance first, since I only process large amount data. I also do not understand why Matlab use Jave to develop UI. Java == Slow.

Answer 1

[mingming wang]

I wish data import wizard could provide more powerful data convert rules. Not only replace Blanks with NaN, but also like Hex data convert to Decimal; replace specific string to other value...

I just try to import data by wizard, exclude 2nd and 4th Hex data cols only. Windows 8 UI still Hang. So this is still a performance issue.

Answer 2

[Lisa Justin]

you can try fopen, feof, fgets,regexprep,sscanf and fclose

or

fid=fopen(filename.csv')

a = fread(fid,1,'uint32','l');

 fprintf('%X',a)  b = hex2dec(a); 
a=dec2hex(fread(fid))

[mingming wang]

I try your code, but meet error:

>> fid=fopen('csvnew.csv');
>> a = fread(fid,1,'uint32','l');
>> fprintf('%X',a)
35323831>> b = hex2dec(a); 
Error using hex2dec (line 38)
Input string found with characters other than 0-9, a-f, or A-F.

I open that hex2dec.m file and see the source code:

% Check for out of range values
if any(any(~((h>='0' & h<='9') | (h>='A'&h<='F'))))
   error(message('MATLAB:hex2dec:IllegalHexadecimal'));
end

You can see that if statement. It only judge 0-9, A-F, but not include a-f. However your error message show: a-f. I update my CSV file chars: a-f to upper case: A-F, then re-run the code. But I meet the same error :(

Answer 3

[Walter Roberson]

fopen(), textscan(), fclose(). If you use textscan(), read the hex fields as strings and convert them after using sscanf() with a %i format item.

Alternately, fopen(), fscanf(), fclose(). Use %i for the format items: it will convert the 0x fields but leave the others as decimal.