Parallel computing corruption problem (Windows)

Hello,

I'm having a problem with parallel computing that is driving me nuts as I feel like I've exhausted all plausible causes. When I first installed matlab, parpool worked and I was able to run a parallel for loop. After that it didn't work and keeps failing with a long chain of errors.

Warning: Failed to remove the following invalid children under the profiles node: local > In NamedNodesCollection>NamedNodesCollection.refreshCache at 302 In NamedNodesCollection>NamedNodesCollection.refreshCacheIfRequired at 267 In NamedNodesCollection>NamedNodesCollection.getAllNames at 115 In Settings.Settings>iReplaceBadDefaultProfileWithLocal at 674 In Settings.Settings>Settings.hGetProperty at 148 In Settings.Settings>Settings.buildProperties at 629 In Settings.Settings>Settings.Settings at 324 In argumentsParser>iGetClusterAndNumWorkers at 112 In argumentsParser at 43 In parpool at 91 Warning: Failed to remove the following invalid children under the profiles node: local > In NamedNodesCollection>NamedNodesCollection.refreshCache at 302 In NamedNodesCollection>NamedNodesCollection.refreshCacheIfRequired at 267 In NamedNodesCollection>NamedNodesCollection.getAll at 59 In NamedNodesCollection>NamedNodesCollection.getAllNames at 116 In Settings.Settings>iReplaceBadDefaultProfileWithLocal at 674 In Settings.Settings>Settings.hGetProperty at 148 In Settings.Settings>Settings.buildProperties at 629 In Settings.Settings>Settings.Settings at 324 In argumentsParser>iGetClusterAndNumWorkers at 112 In argumentsParser at 43 In parpool at 91 Warning: Failed to remove the following invalid children under the profiles node: local > In NamedNodesCollection>NamedNodesCollection.refreshCache at 302 In NamedNodesCollection>NamedNodesCollection.refreshCacheIfRequired at 267 In NamedNodesCollection>NamedNodesCollection.nameExists at 107 In Settings.Settings>iReplaceBadDefaultProfileWithLocal at 695 In Settings.Settings>Settings.hGetProperty at 148 In Settings.Settings>Settings.buildProperties at 629 In Settings.Settings>Settings.Settings at 324 In argumentsParser>iGetClusterAndNumWorkers at 112 In argumentsParser at 43 In parpool at 91 Warning: The local profile cannot be found. Your installation might be corrupt. Error using parpool (line 99) The operation must modify one or more Settings files for these levels: factory, user, but you do not have write permission on these levels: user.

I reinstalled matlab, and then it worked again the first time, but not again. Since then it hasn't worked even after reinstalling. Trying to open the cluster manager fails with error:

com.mathworks.jmi.MatlabException: The operation must modify one or more Settings files for these levels: factory, user, but you do not have write permission on these levels: user. at com.mathworks.jmi.NativeMatlab.SendMatlabMessage(Native Method) at com.mathworks.jmi.NativeMatlab.sendMatlabMessage(NativeMatlab.java:265) at com.mathworks.jmi.MatlabLooper.sendMatlabMessage(MatlabLooper.java:120) at com.mathworks.jmi.Matlab.mtFeval(Matlab.java:1541) at com.mathworks.jmi.MatlabWorker.feval(MatlabWorker.java:197) at com.mathworks.toolbox.distcomp.ui.profile.model.MatlabProfileManager$1.runOnMatlabThread(MatlabProfileManager.java:72) at com.mathworks.jmi.MatlabWorker$2.run(MatlabWorker.java:79) at com.mathworks.jmi.NativeMatlab.dispatchMTRequests(NativeMatlab.java:440)

I have full access to all of these places and matlab is not being very clear about what any of this means. I deleted the files that it generates in AppData/Roaming/Mathworks, but to no effect (I followed another post). All matlab-related folders have write permissions. I can't find any mention of this error while searching on Google so I'm completely stuck.

Does anyone have any idea what's going on?

Thanks!

[José-Luis]

Have you tried to run Matlab as an administrator?

[Bogdan]

Yup, I've tried that as well!

Answer 1

[Elwin Chan]

Hi Bogdan,

It looks like you've got corrupted settings somehow. Can you try closing MATLAB and then deleting the parallel.settings file in your preferences directory? You can find out the location of your preferences directory using this command in MATLAB

prefdir

Also, please can you check that this file exists: matlabroot \toolbox\distcomp\parallel.settings, and perhaps paste its contents or attach it so that I can check it is correct?

Finally, do you have write access to your preferences directory from inside MATLAB?

Thanks,

Elwin

[Bogdan]

Hey Elwin, thanks for your help! I've already tried all of these things based on other people's suggestions. None of them have helped. I deleted parallel.settings in the preferences directory, but it had no effect (and Matlab didn't generate it again). I have full write access to all of these places according to Windows security settings. The really annoying thing is that I gave up and installed Linux on a VM hoping that it would work on that however I am having the EXACT same problem, even running with sudo.

Here are the contents of paralle.settings in my the toolbox folder:

<?xml version="1.0" encoding="UTF-8"?>
<!--Copyright 2011-2013 The MathWorks, Inc. -->
<settings name="parallel" visible="false" xmlns="urn:mathworks.matlab.settings" xsi:schemaLocation="urn:mathworks.matlab.settings settings.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <settings name="client" visible="false">
        <key name="DefaultProfile" visible="false">
            <string>
                <value>local</value>
            </string>
        </key>
        <key name="ConfigurationsLastUpgradeTime" visible="false">
            <!-- Note that the factory value of ConfigurationsLastUpgradeTime is never used, but
            we must have a value for it in the factory settings file in order to ensure that the
            key always exists. -->
            <double>
                <value>0</value>
            </double>
        </key>
        <settings name="pool" visible="false">
            <key name="AutoCreate" visible="false">
                <bool>
                    <value>1</value>
                </bool>
            </key>
            <key name="IdleTimeout" visible="false">
                <!-- Note that this timeout is in minutes. -->
                <double>
                    <value>30.0</value>
                </double>
            </key>
            <key name="PreferredNumWorkers" visible="false">
                <double>
                    <value>12.0</value>
                </double>
            </key>
            <key name="RestartOnClusterChange" visible="false">
                <bool>
                    <value>1</value>
                </bool>
            </key>
            <key name="RestartOnPreferredNumWorkersChange" visible="false">
                <bool>
                    <value>1</value>
                </bool>
            </key>
        </settings>
    </settings>
    <settings name="profiles" visible="false">
        <settings name="local" visible="true">
            <key name="Description" visible="true">
                <string>
                    <value>The local cluster</value>
                </string>
            </key>
            <key name="SchedulerComponent" visible="true">
                <string>
                    <value>localSchedulerComponent</value>
                </string>
            </key>
        </settings>
    </settings>
    <settings name="internal" visible="false">
        <settings name="profilesWithReadOnlySched" visible="false">
            <key name="local" visible="true">
                <bool>
                    <value>1</value>
                </bool>
            </key>
        </settings>
    </settings>
    <settings name="projectcomponents" visible="false">
    </settings>
    <settings name="schedulercomponents" visible="false">
        <settings name="Cloud" visible="true">
        </settings>
        <settings name="Generic" visible="true">
        </settings>
        <settings name="HPCServer" visible="true">
        </settings>
        <settings name="LSF" visible="true">
        </settings>
        <settings name="Local" visible="true">
            <settings name="localSchedulerComponent" visible="true">
                <!-- TODO:later Remove this nasty hack.
                Add a dummy key to the local scheduler component to ensure it 
                that the localSchedulerComponent can't be removed.  It looks like nodes 
                in the factory file that don't have keys under them can be removed.
                This dummy key will not affect the correct behavior of Profiles.-->
                <key name="DummyKey" visible="true">
                    <string>
                        <value>dummy</value>
                    </string>
                </key>
            </settings>
        </settings>
        <settings name="MJS" visible="true">
        </settings>
        <settings name="MJSComputeCloud" visible="true">
        </settings>
        <settings name="PBSPro" visible="true">
        </settings>
        <settings name="Torque" visible="true">
        </settings>
    </settings>
</settings>

[Elwin Chan]

Hi Bogdan,

That file all looks fine, so it's not a corruption problem. It looks like you might not be able to write to your user preferences to store your parallel settings, since the parallel.settings file should be created in there once you used a PCT command such as parpool.

Please can you try the following in MATLAB and let me know what happens:

 s = Settings;
 s.parallel
 s.parallel.addNode('test')
 s.parallel.test.addKey('testKey')
 s.parallel.test.isSet('testKey')
 s.parallel.removeNode('test')

Also, it would be good to find out where your preferences directory is located in both the Windows and Linux case. Finally, can you please confirm which release of MATLAB you are using?

Regards, Elwin

[Bogdan]

Hey Elwin.

Here's a picture of the security settings for the parallel.settings file in my prefdir folder. The folder is in User(my user name)/AppData/Roaming/MathWorks/MATLAB.

Here is the matlab output for running those commands:

>> s = Settings;
>> s.parallel
ans =
  Settings with no properties.
>> s.parallel.addNode('test')
>> s.parallel.test.addKey('testKey')
>> s.parallel.test.isSet('testKey')
ans =
     0
>> s.parallel.removeNode('test')

I am using Matlab2014a 32bit, however I've tried 2013 as well, both 64 and 32 bit.

The strange thing is that Matlab clearly creates parallel.settings when it first runs, as I can run all the parallel computing stuff fine. If I go and delete everything in that folder, the contents are recreated, EXCEPT for parallel.settings.

At the moment I am having to completely reinstall Matlab every time I want to run it again. :(

[Elwin Chan]

Hi Bogdan,

Unfortunately, I missed out a line of code from my previous comment. Apologies for that. Please can you try this:

 s = Settings;
 s.parallel
 s.parallel.addNode('test')
 s.parallel.test.addKey('testKey')
 s.parallel.test.testKey = {}
 s.parallel.test.isSet('testKey')
 s.parallel.removeNode('test')

We are actually trying to write to the parallel.settings file in your preferences directory - it certainly seems as if you have the permissions to do so, but for some reason we aren't able to.

I'm hoping that this new code will throw some light on things.

Thanks, Elwin

Answer 2

[Matlab2010]

Spoke to Matlab tech support about this. The answer is as follows. Type

   which -all assert

In my case, the issue was caused by the built in ASSERT function being shadowed by something else (Kevin Murphys toolbox). Hence rename the function of concern. Solved:)

Answer 3

[Bogdan]

Unfortunately after going through every folder on my PC and setting full access for all users, I reinstalled Matlab and got parpool to work once, and since than I am getting the same error. Pretty close to giving up on this. I can't load the profile manager or parallel computing options, all of which crash with the same error saying I don't have permission on the 'user' level. I doesn't say which file it wants to modify or what to do.

[Matlab2010]

I am getting this error as well. It has started all of a sudden in some code that was running parfor for ages without any problem. Prior to the start of the prblem, no changes were made to any settings on my PC. Any solutions? Matlab?

Warning: Failed to remove the following invalid children under the profiles node:
  local 
> In NamedNodesCollection>NamedNodesCollection.refreshCache at 302
  In NamedNodesCollection>NamedNodesCollection.refreshCacheIfRequired at 267
  In NamedNodesCollection>NamedNodesCollection.getAllNames at 115
  In Settings.Settings>iReplaceBadDefaultProfileWithLocal at 678
  In Settings.Settings>Settings.hGetProperty at 148
  In Settings.Settings>Settings.buildProperties at 630
  In Settings.Settings>Settings.Settings at 325
  In PoolArrayManager>iShouldAutoCreate at 152
  In PoolArrayManager>PoolArrayManager.getOrAutoCreateWithCleanup at 42
  In pctTryCreatePoolIfNecessary at 23
  In remoteparfor>remoteparfor.tryRemoteParfor at 27
  In parallel_function at 433
  In myFun at 16
  In st_NIKE_03>@(x)myFun(x,LPM{k}.Data) at 30
  In st_NIKE_parameterSweep at 56
  In st_NIKE_03 at 31

and my ver

K>> ver
----------------------------------------------------------------------------------------------------
MATLAB Version: 8.3.0.532 (R2014a)
MATLAB License Number: 
Operating System: Microsoft Windows 7 Professional  Version 6.1 (Build 7601: Service Pack 1)
Java Version: Java 1.7.0_11-b21 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
----------------------------------------------------------------------------------------------------
MATLAB                                                Version 8.3        (R2014a)
Simulink                                              Version 8.3        (R2014a)
Bioinformatics Toolbox                                Version 4.4        (R2014a)
Control System Toolbox                                Version 9.7        (R2014a)
Curve Fitting Toolbox                                 Version 3.4.1      (R2014a)
DSP System Toolbox                                    Version 8.6        (R2014a)
Data Acquisition Toolbox                              Version 3.5        (R2014a)
Financial Toolbox                                     Version 5.3        (R2014a)
Image Processing Toolbox                              Version 9.0        (R2014a)
Instrument Control Toolbox                            Version 3.5        (R2014a)
Optimization Toolbox                                  Version 7.0        (R2014a)
Parallel Computing Toolbox                            Version 6.4        (R2014a)
Signal Processing Toolbox                             Version 6.21       (R2014a)
SimMechanics                                          Version 4.4        (R2014a)
Simscape                                              Version 3.11       (R2014a)
Simulink Control Design                               Version 4.0        (R2014a)
Stateflow                                             Version 8.3        (R2014a)
Statistics Toolbox                                    Version 9.0        (R2014a)
Symbolic Math Toolbox                                 Version 6.0        (R2014a)
Wavelet Toolbox                                       Version 4.13       (R2014a)