Do worker nodes in a parallel pool need direct connectivity to the client?

I'm trying to start a parallel pool on a HPC cluster using a windows client. I've installed the third-party SLURM tool add-on and configured a cluster. All validation checks pass except the last one for parpool:

it says:

Error Report: Failed to initialize the interactive session.

Caused by:
    Error using parallel.internal.pool.AbstractInteractiveClient>iThrowIfBadParallelJobStatus (line 423)
    The interactive communicating job errored with the following message: MatlabPoolPeerInstance{fLabIndex=1, fNumberOfLabs=2, fUuid=0c40d093-9791-4944-8fe5-30feb46c98d5} was unable to find the host for workstation:27370 due to a JVM UnknownHostException: null

"workstation" being the hostname of my local machine.

It looks like the worker nodes are trying to reach my local machine, which of course is not possible, as they are isolated. I feel they should be maybe contacting the ClusterHost to which MATLAB connets via SSH and from which the jobs are submitted?

Answer 1

[Raymond Norris]

Our scheduler, MJS, supports interactive pools running from your desktop machine because the traffic is routed via the scheduler on the cluster. For other schedulers, like Slurm, we initiate the server socket on the desktop machine. As noted it's less likely this will work (since it's probably not reachable), but you have a couple of options.

• For starters, if your desktop machine is reachable via IP address, you can configure it as such

pctconfig('hostname','a.b.c.d');

This requires being called each time you start MATLAB (can be placed in your startup.m file) and before you call any parallel code (e.g., parpool).

• If you can't resolve your public IP address, if you VPN into the university/company network, try using the IP address provided by your VPN.

[max.s]

Thanks! This was really helpful and I got the connection set up.

[Raymond Norris]

Great to hear. Can you tell us what step(s) you took? Was it just changing the hostname or did you need to VPN in as well?