parfeval with background pool fails when using Java classes

I'm trying to either use an existing database connection, based on com.mysql.cj.jdbc.Driver or to get that created in a parfeval call using the background worker.

Minimal examples using all the workarounds with and without function handles, parallel.pool.Constant(), external functions, etc.

db = com.mysql.cj.jdbc.Driver
fun1 = @()db.acceptsURL("asdf")
fun2 = @(db)db.acceptsURL("asdf")
fun3 = @(c)c.Value.acceptsURL("asdf")

C1 = parallel.pool.Constant(db);

getDriver = @()com.mysql.cj.jdbc.Driver;
C2 = parallel.pool.Constant(@getDriver);

C3 = parallel.pool.Constant(@getDriverExt);

fun1() % no error, direct call, outputs false
fun2(db) % no error, direct call, outputs false
fetchOutputs(parfeval(backgroundPool, fun1, 1)) % error serializing
fetchOutputs(parfeval(backgroundPool, fun2, 1, db)) % error serializing
fetchOutputs(parfeval(backgroundPool, fun3, 1, C1)) % error serializing, The 'Value' field cannot be retrieved because the parallel.pool.Constant is invalid. 
fetchOutputs(parfeval(backgroundPool, fun3, 1, C2)) % Unrecognized function or variable 'getDriver'. 
fetchOutputs(parfeval(backgroundPool, fun3, 1, C3)) % Unable to resolve the name 'com.mysql.cj.jdbc.Driver'. 

getDriverExt.m

function conn = getDriverExt()
    conn = com.mysql.cj.jdbc.Driver;
end

Here are the results of all runs:

>>                 fun1() % no error, direct call, outputs false
ans =
  logical
   0
   
>>                 fun2(db) % no error, direct call, outputs false
ans =
  logical
   0
    
>>                 fetchOutputs(parfeval(backgroundPool, fun1, 1)) % error serializing
Warning: com.mysql.cj.jdbc.Driver@3662c887 is not serializable 
Error using parallel.BackgroundPool/parfeval
Internal error occurred when passing the input argument of type "function_handle" to the thread-based worker:
MATLAB:Serialize
Error during serialization 

>>                 fetchOutputs(parfeval(backgroundPool, fun2, 1, db)) % error serializing
Warning: com.mysql.cj.jdbc.Driver@3662c887 is not serializable 
Error using parallel.BackgroundPool/parfeval
Internal error occurred when passing the input argument of type "com.mysql.cj.jdbc.Driver" to the thread-based worker:
MATLAB:Serialize
Error during serialization 

>>                 fetchOutputs(parfeval(backgroundPool, fun3, 1, C1)) % error serializing, The 'Value' field cannot be retrieved because the parallel.pool.Constant is invalid. 
Error using parallel.Future/fetchOutputs
One or more futures resulted in an error.
Caused by:
    Error using parallel.pool.Constant/get.Value (line 191)
    The 'Value' field cannot be retrieved because the parallel.pool.Constant is invalid. 
    
>>                 fetchOutputs(parfeval(backgroundPool, fun3, 1, C2)) % Unrecognized function or variable 'getDriver'. 
Error using parallel.Future/fetchOutputs
One or more futures resulted in an error.
Caused by:
    Error using parallel.pool.Constant/get.Value (line 196)
    Error occurred while creating parallel.pool.Constant on the workers.
        Unrecognized function or variable 'getDriver'. 
        
>>                 fetchOutputs(parfeval(backgroundPool, fun3, 1, C3)) % Unable to resolve the name 'com.mysql.cj.jdbc.Driver'. 
Error using parallel.Future/fetchOutputs
One or more futures resulted in an error.
Caused by:
    Error using parallel.pool.Constant/get.Value (line 196)
    Error occurred while creating parallel.pool.Constant on the workers.
        Unable to resolve the name 'com.mysql.cj.jdbc.Driver'. 

Is it impossible to use Java classes in the workers? If no, how to do it properly?

Thanks!

Answer 1

[Harald]

Hi Jan,

for functionality to work in a background pool, it needs to be explicitly supported. For more information, please see here:

https://www.mathworks.com/help/matlab/matlab_prog/run-functions-on-threads.html#mw_1cce412a-b5cb-42a4-b0b3-335654833576

Have you tried the desired functionality in a process-based parallel pool, and would that be an alternative for your application?

p = parpool("Processes");

Best wishes,

Harald

[Jan Kappen]

I unfortunately need to use thread-based parallel computing. It's just to run a database read in background while keeping the UI responsive.

I can't find any listed problems with thread-based stuff in the list https://www.mathworks.com/help/matlab/matlab_prog/run-functions-on-threads.html#mw_1cce412a-b5cb-42a4-b0b3-335654833576

[Harald]

Hi,

Have you tried the desired functionality in a process-based parallel pool to rule out other issues? For example, Unrecognized function or variable 'getDriver' does not sound like a problem specific to thread-based pools.

I suspect the problematic part is External Language Interfaces, which includes Java.

While I understand that thread-based pools would be preferable, I do not yet understand why process-based pools are not a suitable alternative. You'd admittedly have to live with the initial time it takes to open the pool.

It may be a long shot, but could timers be an alternative?

Best wishes,

Harald

[Jan Kappen]

Hi Harald,

> While I understand that thread-based pools would be preferable, I do not yet understand why process-based pools are not a suitable alternative. You'd admittedly have to live with the initial time it takes to open the pool.

Not all users have parallel computing toolbox. backgroundPool was introduced for these reasons if I understand correctly.

> I suspect the problematic part is External Language Interfaces, which includes Java.

yes... but there's an example that uses python (an external language) which works fine via constant:

pass Python object as argument to function in "parfeval" - MATLAB Answers - MATLAB Central (mathworks.com)

(edit: this seems to use processes, not threads :()

Two cross links:

Is it possible to use parfeval and backgroundPool to write to a Mongo database in a background thread? - MATLAB Answers - MATLAB Central (mathworks.com)

(Why) Is Java not available in a thread-based environment? - MATLAB Answers - MATLAB Central (mathworks.com)

> It may be a long shot, but could timers be an alternative?

yes, maybe.

Is a Matlab AppDesigner GUI running on a different thread per default? I just realized, that GUI elements remain responsive (or at least animations still work). How does that work?