Variables created in different formats

Hello all,

I have huge datasheet in .CSV format. I load it into the Matalb with readtable command.

Here you can see an example of some data I am using EDIT: Attached: https://drive.google.com/file/d/1ufomb934NVoFwRQDZ6s3upJQFVICsqLL/view?usp=sharing

Then i want to create a variable from the data set.

rate_qpsk  = src_files{ii}{:,validatestring('LTE_Adv__Q_PCellDLQPSKRate',src_files{ii}.Properties.VariableNames)};
rate_16    = src_files{ii}{:,validatestring('LTE_Adv__Q_PCellDL16QAMRate',src_files{ii}.Properties.VariableNames)};
rate_64    = src_files{ii}{:,validatestring('LTE_Adv__Q_PCellDL64QAMRate',src_files{ii}.Properties.VariableNames)};
rate_128   = src_files{ii}{:,validatestring('LTE_Adv__Q_PCellDL256QAMRate',src_files{ii}.Properties.VariableNames);

Now, for some reason holf of teh variable are created as double and other half is created as cell array.

I woul just change the format but ...

Please advise why Matlab works for some of the columns differently than for others and how to avoid this situation.

Thanks for all the comments/help.

[Koula Lysi]

I am new to Matlab. I went though a similar issue and solved the problem using the command table2array(cell2table(rate_64)). I am sure that they do exist better methods though.

[Karel Starý]

Thanks for hint.

Unfortunately this approach givs me same error as changing the format for the variable only -> Conversion to double from cell is not possible.

Still good try.

Answer 1

[Stephen23]

Your file is not really a Comma-Separated Values (CSV) file, it actually uses a tab as the values delimiter and comma as the decimal radix character. So you need to tell READTABLE this when you import it. Because of confusion with rather unfortunate date formats you will also need to specify the date format to correctly import those dates as DATETIME. For example:

fnm = 'data.csv';
opt = detectImportOptions(fnm, 'Delimiter','\t', 'DecimalSeparator',',', 'VariableNamingRule','preserve');
opt = setvaropts(opt, 'Time', 'InputFormat','d/M/y H:m:s.SSS');
tbl = readtable(fnm, opt)

tbl = 816×5 table
            Time             LTE(Adv)-Q_PCell DL QPSK Rate    LTE(Adv)-Q_PCell DL 16QAM Rate    LTE(Adv)-Q_PCell DL 64QAM Rate    LTE(Adv)-Q_PCell DL 256QAM Rate
    _____________________    _____________________________    ______________________________    ______________________________    _______________________________

    2/5/2022 11:35:21.000                 NaN                              NaN                                NaN                                NaN             
    2/5/2022 11:35:22.000                   0                                0                              15.22                              84.78             
    2/5/2022 11:35:23.000                   0                                0                              69.57                              30.43             
    2/5/2022 11:35:24.000                   0                                0                               4.35                              95.65             
    2/5/2022 11:35:25.000                   0                                0                                  0                                100             
    2/5/2022 11:35:26.000                   0                                0                                  0                                100             
    2/5/2022 11:35:27.000                   0                                0                                  0                                100             
    2/5/2022 11:35:28.000                   0                                0                               2.17                              97.83             
    2/5/2022 11:35:29.000                   0                                0                              23.91                              76.09             
    2/5/2022 11:35:30.000                   0                                0                              95.65                               4.35             
    2/5/2022 11:35:31.000                   0                                0                              19.57                              80.43             
    2/5/2022 11:35:32.000                   0                                0                                  0                                100             
    2/5/2022 11:35:33.000                   0                                0                              45.65                              54.35             
    2/5/2022 11:35:34.000                   0                                0                              10.87                              89.13             
    2/5/2022 11:35:35.000                   0                                0                              30.43                              69.57             
    2/5/2022 11:35:36.000                   0                                0                                  0                                100             

[Karel Starý]

Thanks for the answer!

I understand the issue now but I am struggling with the implementation of the code you provided.

I have tried to implement this without the setvaropts, but it's loading all the info into one column.

I know this is heading away from the initial question but I would appreciate one more hint :)

Thanks!

[Stephen23]

@Karel Starý: please upload the file "lte log.csv" by clicking the paperclip button. Clearly it does not have the same file format as the first file that you uploaded.

Have you opened & saved any of these files using MS Excel?

[Karel Starý]

• Have you opened & saved any of these files using MS Excel?

Nope, I am using SW for extranting the data from DRM to CSV. I am usually not using any office SW to wokr on them.

[Stephen23]

The file "lte log.csv" uses a completey different file format: specifically it uses a comma delimiter (unlike your "data.csv" file which used tab delimiter) and also attempts to use the comma as the decimal separator. However it fails at this because in order to use a decimal comma it has to use double quotes around every single number with a decimal fraction, turning them all into quoted text.

That is an awful file format to work with.

Original file "data.csv" subset (tab delimiter):

New file "lte log.csv" subset (comma delimiter, and check out those double quoted numbers):

"I am using SW for extranting the data from DRM to CSV."

What kind of SW is that? Does it use your OS locale settings?

Well, in any case, you have so far uploaded two different files with two different formats: while it is possible to write code to handle multple file formats, doing so significantly complicates the code. I strongly recommend that you streamline your data pipeline to ensure that it only returns one file format.

[Karel Starý]

Thanks. I really apprechiated this help.