Test equivalence between mex and (.m) matlab functions

What is the best way to create a test that compares outputs from a .mex and .m function? It has become a common workflow to write a processing pipeline in matlab then, if needed, find bottlenecks and try to rewrite them in a lower language with mex. I would like to confirm that my mex functions and matlab function produce equivalent (or nearly equivalent) results. Is there a builtin for running a matlab file even when a mex file of the same name exists? Being that matlab fallsback to a .m file if no mex file is compilled, it should be best practice to have both, matlab and mex, versions of the function and, therefore, it should be a common need to write tests that ensure the two are the same.

If there is no function for running the matlab file, how should I go about writing this test? I could write the functions in different file names (i.e. foo_matlab.m foo_C.c) then have a wrapper function to call the desired function (i.e. default to C if it exists but have a 'useC' option). But this is clearly not the matlab way about this as it skirts around the default behavior and then rewrites similar logic. The other option I can think of would be to delete the mex file (if it exists) in the test set up, run the function, then compile within the same test, and rerun the function in the same way, and compare the results. This has the benefit of ensuring the compiled function is up to date but it requires recompiling for every test or using an ugly method to run all tests with the matlab function, storing results somewhere, compiling the mex function, then reruning tests and comparing. But that doesn't follow normal testing convention where both parts should be run in the same test function.

Edit

Forgot to mention originally but, I do not have ground truth values for many of my test cases so it is not enough for both versions to pass the tests (see my response to Jan's answers for more details). Secondly, the solution should be able to work with packages (where functions are called in a different way than stray functions in matlab's path) since any code I test is written as a package.

Answer 1

[Jan]

I store the compiled MEX functions in a specific folder. This is useful also to support different Matlab version, so each version has its own Folder for the mex files.

Then including the MEX folder in the path tests the MEX function, and without this folder, the M-files in their original folders are tested. If both functions pass the known answer test and random tests, it can be assumed, that their output is equivalent.

[David Connell]

That's a good solution except I don't know how this would work when using package namespaces, which most of what I write is in. In this case, I would either have to put the mex files in a subpackage, which requires I call it differently (i.e. package.matlab.foo vs package.mex_r2021b.foo), or put it into a private subfolder, which tests don't have access to. I believe, for packages, either the whole package is in the path or non is so I don't believe there is a way to add a folder within a package to the path and still get normal namespace behavior.

Secondly, I meant to add to the original question but forgot, I often don't have known answers. For example, I'm currently writing some algorithms that test network modularity, which is a relative score. The value is dependent on size of network, number of edges, and the partitioning. There is not a definitive score that a given network should get, instead I check that the score decreases as I reduce a network's modularity by starting with a perfectly modular network then randomly switching edges. Passing this test doesn't mean the scores are the same, and in fact there are different modularity scoring algorithms that all have this behavior but don't give the same results. There are some edge cases which I know should have a score of 0, but again just because it gets the edge cases right doesn't mean it's the same algorithm.

And the exact values are important, not just that it orders my test correctly because my test is intentionally simplified so there's a definitive correct order. In practice different algorithms could order different partitioning of the same network differently so I need to be able to say that this partition got a higher score using this specific modularity scoring algorithm. While none of my tests gurentee the algorithm is correctly impelmented, both implementation giving the same score is still a good check and I have more confidence in the results being consistent if some networks end up getting run with the matlab files and others with the mex files (considering some smaller networks have already been run with matlab, we didn't know the need for the mex versions until we started on larger networks). Sorry I was not clearer in the question.

[Jan]

Yes, packages need a different management.

If there is a "small" number of MEX functions and they are called only "a few times", while the most computational time is spent inside the MEX (or corresponding M-function), a manual switch can be used:

function main
param.useMex = true;
data   = rand(1, 1e8);
result = sub1(data, param)
resultToPublication(result);
end

function result = sub1(data, param)
if param.useMex
   s = sin_mex(data);
else
   s = sin(data);
end
result = sum(s);
end

But if a function is called frequently, this has a severe overhead:

s = 0;
for k = 1:1e8
   s = s + sub1(rand, param);
end

An alternative can be to provide the function handles:

function main(useMex)
if useMex
    param.fcn = @sin;
else
    param.fcn = @sin_Mex;
end

data   = rand(1, 1e8);
result = sub1(data, param)
resultToPublication(result);
end

function result = sub1(data, param)
fcn    = param.fcn;
s      = fcn(data);
result = sum(s);
end

I had a comparable problem: I've written a program in Matlab used for clinical decision making. Because the results are critical, the customers stayed at a fixed Matlab version. This was Matlab 6.5 for 8 years. Then the labs switched to R2009a, which offers much better debugging tools and more powerful graphic features. The program has some 100'000 lines of code and using the modern features allowed to improve many functions. But how can I prove that the results are the same? The next step was to an upgrade to R2018b. Now even the graphics have changed substantially. The PDF files of the measurement curves look a little bit different now due to the modern graphics.

I've solved this by testing the elementary functions using unit test. You still find all my FEX submissions accompanied by a uTest_<tool> function, which does not need the moden Unittest framework of Matlab, because they are not available in ancient Matlab versions. If the unit testing is successful, a list of integration tests is performed, e.g. the data of 40 persons are processed and the PDF output are compared visually (this takes 12 hours) and automatically. Then a set of 1000 persons is chosen from a data base and the average and deviations are calculated and compared between the different Matlab versions.

But, of course, the results are slightly different, because some of the elementary Matlab functions have changed the algorithm and rounding effects can be observed. This is useful to check, if the applied methods are susceptible for such effects.

Then the actual work starts: Detect the cause of the small differences and check, if they are bugs or unavoidable deviations. A complete test took about 2 weeks of work for 3 persons. We detect about 5 to 10 undocumented changes of Matlab functions during the tests, e.g. the vanishing of the vax-d format in fopen or the repeated changes of the result of strncmp(a, b, 0).

There is a general problem with maintaining an M and a MEX version of the algorithm: As soon as you fix a bug, there will be a confusion, which of the versions reply the "correct" answer. Of course, in some cases you can fix the bug in both versions successfully, but as usual fixing one bug can change the behavior of other parts of the code.

The best solution is to split the program into controllable small parts and to proove by unit testing, that they do, what they are expected to do, e.g. by a comparison with an M-function. If the MEX functions pass the tests, you can trust them in the integration test also.

In the real world we are working in brown-field projects, where we find large blocks of untested codes inherited from former members of the lab. Then the reliable way remains a refactoring - in other words: rewriting everything from scratch including an exhaustive unit testing, version history, bug reporting system, automatic integration tests.

Your actual question, how to test a MEX versus an M implementation, is important and opens a window. I could be a hint, that the best way is to introduce a new programming strategy, e.g. a test-driven Kanban method.

[David Connell]

Thanks for the experienced response. I intend on using the if statement and a logical parameter based on your top example. Using input parser, add an optional parameter defaulting to mex if a MEX file exists but still allowing me to test both versions using a TestParameter property for consistency.

I'm intentionally trying to move enough logic to the MEX function to only need a single call. For the M file, the number of calls to some matrix operations increases with network size, so I'm hoping moving everything to MEX will reduce time by reducing number of calls to lower level functions, especially since those calls are passed large matrices. (There is a way to vectorize the whole thing but memory requirements didn't scale up well.)

I've been pleased with how feature-rich and flexible matlab's testing framework has been so I'm a surprised there is not a built-in method for comparing MEX and M functions. The workflow of writing in matlab then moving slow parts to MEX as needed would be the norm for working with MEX and the benefits to comparing the two methods seem obvious. Even if they are both written correctly, as you mention, various minor differences such as those rounding errors can lead to slight variances between the results and that's needed information for interpretation.

Based on your response, I still have much to learn about testing projects, but so far the benefits of unit testing alone have already proven to far exceed the overhead and the workflow even results in a high-level map to guide development and help determine where to break out functions. Thank you again for your advice.