How to force textscan to include the custom EOL character

2 views (last 30 days)
Hi Folks,
I am loading multiline chunks of texts from very big files each starting from '@' character into separate cells. Here is my code to do that:
y=textscan(x,'%s',1e7,'EndOfLine','@'); y=y{1};
The result is good and fast except I wanted to also include the opening '@'. I am running it against massive TB files in blocks of 1e7 chunks. Obviously I know I can do y=strcat('@',y) afterwards, but this postfix takes longer than the original textscan itself. Is there a way to force textscan to also include the specified EOL character, or any other faster solution for that?
Here is a testing line where I create a big multiline string to simulate the file:
x=repmat(['@abc:1:abc:1:2:3:4\ndef:1:abc:1:2:3:4\n'],1,1e7); tic;
y=textscan(x,'%s','EndOfLine','@'); y=y{1};
t(1)=toc;
y=strcat('@',y);
t(2)=toc
Note: I want to retain the capability to rapidly put filtered string/file back together by x=[y{:}];
Help much appreciated

Accepted Answer

Walter Roberson
Walter Roberson on 20 Sep 2022
No, textscan() will always eat the EndOfLine delimiters.
One approach:
Keep a buffer of unprocessed text, initially []
while ~feof(fid)
buffer = [buffer, fread(fid, '*uchar', CHUNKSIZE))];
if isempty(buffer) %end of file
break
elseif buffer(1) ~= '@'
%something is wrong with the input stream, we expected a @ at the
%beginning
else
parts = regexp(buffer, '@[^@]*', 'match');
buffer = parts{end};
parts(end) = [];
%now process the chunks in cell array parts
end
end
At this point, buffer should be non-empty and should hold the last chunk. Since the last chunk is not followed by @ then at the time we read it we cannot know that it is a complete chunk: we can only know that any particular character is the end of a chunk by peeking ahead to see a @ next character or by detecting that we reached end of file. So expect buffer to have the last chunk in it after the loop.
  1 Comment
dymitr ruta
dymitr ruta on 21 Sep 2022
Edited: dymitr ruta on 21 Sep 2022
Fantastic, slashed the time by half, but your code does not work with '*uchar' as regexp does not accept uint8s that you get as output from fread(fid, CHUNKSIZE, '*uchar'). It works when you do: fread(fid, CHUNKSIZE, '*char'), but then the read chunk occupies twice more space. There is also a limitation of the chunksize in Matlab set at 2^29 (as the error message informs) but I managed to get chunksize=1e9 working, with my data it given me only 2.9m parts at the end, so not that much compared to 1e7 I was working with but given the speedup it is worth it. Thanks

Sign in to comment.

More Answers (1)

dpb
dpb on 20 Sep 2022
x=repmat(['@abc:1:abc:1:2:3:4\ndef:1:abc:1:2:3:4\n'],1,1e7); tic;
y=textscan(x,'%s','EndOfLine','@'); y=y{1};
t(1)=toc;
y=strcat('@',y);
t(2)=toc; disp([t sum(t)])
2.5350 21.9892 24.5242
tic;
y=textscan(x,'%s','EndOfLine','@'); y=string(y{1});
t(1)=toc;
y="@"+y;
t(2)=toc; disp([t sum(t)])
3.7476 5.2454 8.9931
Trades the very expensive strcat function for direction addition with the newer string class -- takes a second longer to convert to string, but save 15 or so in the catenation operation.
The direct catenation of the cellstr array with the cellufn variant was even slower than strcat without more effort than I had time to give at the moment, but the above may lead to some other ideas on direct memory manipulation -- presuming the character strings in the real application aren't of uniform length, the conversion to a straight char() array is probably not the way to go so I didn't even look at that variant.
  2 Comments
dymitr ruta
dymitr ruta on 22 Sep 2022
After some experimentation I got your string solution to work the fastest in combination with the split function, even beating Walter's regexp:
y='@'+split(string(x),'@'); y(1)=[];
Fantastic, many thanks for the lead

Sign in to comment.

Categories

Find more on Data Import and Export in Help Center and File Exchange

Products


Release

R2022a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!