Vectorization approach to find matching elements in a matrix.
3 views (last 30 days)
Show older comments
Khoa Tran on 27 Mar 2023
Commented: Khoa Tran on 29 Mar 2023
Hi experts, I am new to Matlab and I need your help with vectorization approach to improve speed of my code.
I have a big 10,000,000 x 15 matrix A containing integers in the range of 1-255, each row has 15 different integers. The integers in different rows can be the same, but each row is unique. I also have a smaller matrix B of 240,000 x 15 containing integers in the same range (i.e., 1-255), with the same conditions as of matrix A.
I need to find the rows in matrix A that has at least 2/3 of matching elements to the elements in each row of matrix B. If a row in matrix B has rows in matrix with matched conditions, then store those rows as below, or else just skip it.
- C = rows from matrix A
- D = rows from matrix B
Currently I am using for-loop to do this, but this takes a lot of time and could not even finish.
C = ;
D = ;
for i = 1:1:size(matrix_B,1)
C = [C; A(sum(ismember(matrix_A,matrix_B(i,:)),2)>=10,:)];
D = [D; B(i,:)];
Is there a vectorization approach that I can do to solve this problem? Or should I re-define my problem and change the entire solution? I also heard about Parallel Computing which would be a big help to this problem, but I do not know how to use it.
John D'Errico on 28 Mar 2023
Edited: John D'Errico on 28 Mar 2023
The part of your code that is a serious problem is where you build the arrays by growing them at each step. This is a really bad thing to do in MATLAB, and is why your code will be so abysmally slow. Your code forces MATLAB to reallocate the memory for those arrays millions of times, each time for a slightly larger amount of memeory. Then it is forced to copy over EVERYTHING from the old version of the array to the new one. UGH. Don't do things that way!
How would I solve this? I would probably define the distance between two rows as the number of elements in row 1 that are NOT in row 2. You already do something like that, using ismember.
But now you should be able to use rangesearch, to locate rows in one array that are "close" to rows in the other array, and rangesearch is the perfect tool to use a distance like this on a user provided function.
More Answers (0)
Find more on Resizing and Reshaping Matrices in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!