Binary format that Matlab can read, and how to convert with C/c++?

My current procedure is not optimal. It is:

1. extract terabytes of data with c++.
2. Naivly writing that to disk in ascii format
3. analysing this data using Matlab. I am reading in tons of moderately sized files (half a milliion lines of doubles for each file), rather than a few huge files. Hence tall arrays aren't necessary, just a good format.

So my question is:

1. Instead of writing to disk in ascii format, is there an easy to use c++ library (or matlab library that Matlab coder can convert to C) that converts the data to some binary format that matlab can read?
2. Or alternatively, can matlab read unformatted Fortran data? Probably this is not 'so' ideal since the data switches machines

Also,

1. The reason for preferring binary formats to ascii are file size and speed savings. Also, I was reading about HDF5, and it seemed really hard to use, whereas I would prefer something that will not cost me several days to get up and working.
2. I realize this question would be perhaps better asked on stackexchange on a C forum, but I was afraid I would get responses which Matlab would not be able to work with, hence I am asking this question from the Matlab side.

Answer 1

[Gayatri Rathod]

Hi,

For a more efficient and easy-to-use method to store and read data from C++ in a format that MATLAB can handle. Here are a few options you can consider:

1. Binary File Format: Instead of writing the data to disk in ASCII format, you can write it in a binary format using C++ and then read it in MATLAB. C++ provides functions like “fwrite” and “fread” that allow you to write and read binary data. MATLAB also has corresponding functions (“fwrite”, “fread”) that can read binary data. By using a binary format, you can achieve smaller file sizes and faster read/write operations compared to ASCII. fwrite function, fread function.

If you consider saving data into MATLAB, here are a few options you can consider:

• MATLAB Data Format: MATLAB has its own binary data format called MATLAB MAT-file (.mat) that can store various types of MATLAB data structures, including numeric arrays, matrices, and structs. You can use the MATLAB Engine API for C++ to write data from C++ directly to MAT-files. Saving and loading mat files.
• MATLAB's “save” function with -v6 option: If you want to stick with MATLAB, you can use the “save” function with the -v6 option to save your data in a binary format. You can save your data in MATLAB and then read it back using the “load” function. Save function, load function.

2. Unformatted Fortran Data: MATLAB has limited support for reading unformatted Fortran data using the “fread” function. However, it may not be an ideal solution due to potential issues with different machine architectures and byte order.

Hope it helps!