How do I parse and erase from a string while importing CSV with tableread?

I have found extractBefore, but how do I succinctly apply it to each opts.VariableNames element? Should I use cellfun? I will try it instead of a for loop, but I don't see how to use cellfun for a function requiring an input argument (namely, '_').

Why does MATLAB not allow for the following 'intuitive' syntax?

>> opts.VariableNames(:) = extractBefore(opts.VariableNames(:),'_')

Error using matlab.io.ImportOptions/set.VariableNames (line 185)
Expected a cell array of valid variable names.

>> opts.VariableNames = extractBefore(opts.VariableNames,'_')
Error using matlab.io.ImportOptions/set.VariableNames (line 185)
Expected a cell array of valid variable names.

The following code is what I am seeking to improve with this task, by automatically truncating the latter column names.

filepath = 'easy.csv';
opts = detectImportOptions(filepath, 'NumHeaderLines', 1);
opts = setvartype(opts,opts.VariableNames, 'double');
opts.VariableNames(1)={'Dose'};
DVH = readtable(filepath,opts)

[Guillaume]

Why does MATLAB not allow for the following 'intuitive' syntax?

>> opts.VariableNames(:) = extractBefore(opts.VariableNames(:),'_')

Because

extractBefore(opts.VariableNames{1},'_')

returns an empty char array (since VariableNames{1} doesn't have a _) which is not a valid variable name.

opts.VariableNames(2:end) = extractBefore(opts.VariableNames(2:end),'_')

would have worked.

By the way, note that the (:) in VariableNames(:) was pointless in your original code. The only thing that it did is transpose the cell array from a row vector to a column vector.

Answer 1

[Daniel Bridges]

The following code generates the table as desired.

function data = ReadMIMDVH(filepath)
  opts = detectImportOptions(filepath, 'NumHeaderLines', 1);
  opts = setvartype(opts,opts.VariableNames, 'double');
  opts.VariableNames(1)={'Dose_'};
  opts.VariableNames = extractBefore(opts.VariableNames,'_');
  data = readtable(filepath,opts);
end

However, I hope someone will teach me how to accomplish this result in one line (via cellfun?) instead of four lines with a function.

[Walter Roberson]

opts.VariableNames = extractBefore(opts.VariableNames, '_')

You choose to use a function for separation of functionality. Sure you could have used regexp instead like

opts.VariableNames = regexprep(opts.VariableNames, '_.*', '');

but there is no inherent reason why you would put that "in-line" but not put the extractBefore "in-line".

[Guillaume]

Well as Walter says, the StripExtraStuff function with just one line is a bit pointless. Since you've added a _ to the first variable name, you could just have

opts.VariableNames{1} = 'Dose_';
opts.VariableNames = extractBefore(opts.VariableNames, '_');

If you use regexprep, then you don't need the extra _ for the first variable.

opts.VariableNames{1} = 'Dose';
opts.VariableNames = regexprep(opts.VariableNames, '_.*', '');  %replace _ followed by any number of any characters by empty

Both methods are just as efficient.

[Daniel Bridges]

I just reduced it to one line as Walter suggested and it works; I've edited my answer to simplify accordingly.

I think I made it into a function because the extractBefore documentation says a string is expected, and I neglected to check the Input Arguments section to see that a cell array of character vectors was also permitted. Sorry for the confusion; thanks for the clarity.

Answer 2

[Peter Perkins]

Fixing the names in the importOptions setting is one choice, but an alternative might have been to patch up the names after reading the file with readtable (you'd still have to skip the header line). Perhaps something like

DVH.Properties.VariableNames{1} = 'Dose'; DVH.Properties.VariableNames(2:end) = extractBefore(DVH.Properties.VariableNames(2:end), '_')

[Daniel Bridges]

How do you determine which is preferable or more efficient? Thinking of mechanically copying and erasing, it seems better to me to 'write it correctly the first time', to get the computer to imitate the person who sees the text but only copies prior to the underscore: Are you not suggesting to copy everything and then erase after copying?

[Guillaume]

If you were just using detectImportOptions so as to be able to override the variable names then changing them after the fact and not bothering with the import options may be simpler (*)

Since you've got to do the import options to override the type then you may has well override the names.

(*) Except that, frustratingly, in some cases, readtable without import options and readtable with default import options will read the table differently.

[Peter Perkins]

Daniel, I'm usually an advocate of fixing things at their source. My suggestion was more about simplicity. But as Guillaume says, if you're using detectimportoptions anyway ...

Guillaume, I hear you about frustrating, but it's a tension between backwards compatibility, and better new behavior. We try to walk a fine line between providing better behavior by default, and not breaking code that relied on the older behavior.