Segmention Violation Detected on Trainnet

Hello. I'm training a Convolutional Neural Network using either trainnet or train, and using both cpu or gpu, and I keep getting the following error no matter the configuration. I have tried to simplify the nn and reduce to minibatch size to 32, but I keep getting the error. I was previously using an older version of MATLAB, this is my first code with MATLAB R2024a. I have already deleted and reinstalled MathWorks Service Host.

------------------------------------------------------------------------

Segmentation violation detected at Sat Jun 15 22:39:27 2024

------------------------------------------------------------------------

Architecture : glnxa64

Processor ID : Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz

Host Name : -

Locale : es_ES

Abnormal termination:

Segmentation violation

Current Thread: 'FTP_2_1' id 140126648170240

Register State (from fault):

RAX = 65532f736b726f57 RBX = 00007f71c71664f0

RCX = 0000000000000000 RDX = 0000000000000009

RSP = 00007f71c71664b0 RBP = 00007f71c7166710

RSI = 0000000000000002 RDI = 0000000000000002

R8 = 00007f71c71664f8 R9 = 0000000000000052

R10 = 00007f71ac000082 R11 = 00007f71c7166468

R12 = 00007f71c7166538 R13 = 00007f71e825bea0

R14 = 00007f71e825a8c0 R15 = 00007f71c7166550

RIP = 00007f7210c62b16 EFL = 0000000000010246

CS = 0033 FS = 0000 GS = 0000

Stack Trace (from fault):

[ 0] 0x00007f7210c62b16 /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/spf/serviceproviders/login/spf/matlabloginserviceprovider/libmwmatlabloginserviceprovider.so+00371478

[ 1] 0x00007f721053746b /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/mathworksservicehost/rcf/matlabconnector/serviceprocess/rcf/client-v1/../../../../../../../../bin/glnxa64/libmwagentspfcore.so+00152683 _ZNK5agent3spf4core16ExportedFunctionclEON10foundation10concurrent10threadpool10stop_tokenEONS1_14GenericPayloadE+00000347

[ 2] 0x00007f720340a5f6 /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/spf/servicepublishers/agent/spf/legacyservicepublisher/../../../../../../../bin/glnxa64/libmwagentspflegacyservicewrapper.so+00128502

[ 3] 0x00007f720340a681 /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/spf/servicepublishers/agent/spf/legacyservicepublisher/../../../../../../../bin/glnxa64/libmwagentspflegacyservicewrapper.so+00128641

[ 4] 0x00007f7203371060 /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/spf/servicepublishers/agent/spf/legacyservicepublisher/../../../../../../../bin/glnxa64/libmwagentnanomsgtransport.so+00082016

[ 5] 0x00007f7215a32c34 /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/libmwfoundation_threadpool.so+00146484

[ 6] 0x00007f7215a37928 /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/libmwfoundation_threadpool.so+00166184

[ 7] 0x00007f7215a37d54 /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/libmwfoundation_threadpool.so+00167252

[ 8] 0x00007f72163fc8c3 /home/fran/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6+00903363

[ 9] 0x00007f721654d609 /lib/x86_64-linux-gnu/libpthread.so.0+00034313

[ 10] 0x00007f721622a353 /lib/x86_64-linux-gnu/libc.so.6+01176403 clone+00000067

[Francisco José]

After some experiments, I have seen that this error only appears when I run multiple MATLAB process simultanously, but not when a single MATLAB process is running. However, with the previous MATLAB version I ran more than 20 process at the same time without error.

[Umar]

Hi Francisco,

The segmentation violation error in MATLAB R2024a indicates a memory access violation, which can occur due to various reasons such as accessing an invalid memory location or exceeding memory limits. Here are a few possible solutions to address this issue:

Check System Requirements: Ensure that your system meets the minimum requirements for MATLAB R2024a, including sufficient RAM and GPU capabilities if using GPU acceleration.

Update MATLAB: Check for any available updates or patches for MATLAB R2024a. MathWorks often releases updates to address known issues and improve stability.

Check GPU Drivers: If you are using GPU acceleration, make sure you have the latest drivers installed for your GPU. Outdated or incompatible GPU drivers can cause memory access issues.

Reduce Memory Usage: If your neural network model or dataset is memory-intensive, consider reducing the memory usage by decreasing the model size, using data augmentation techniques, or loading data in smaller batches.

Increase System Memory: If possible, try increasing the amount of RAM in your system. Insufficient memory can lead to memory access violations, especially when running multiple MATLAB processes simultaneously.

Run MATLAB Processes Sequentially: As a temporary workaround, you can try running the MATLAB processes sequentially instead of simultaneously. This may help identify if the issue is related to running multiple processes concurrently.

Contact MathWorks Support: If the issue persists, consider reaching out to MathWorks support for further assistance. They can provide specific guidance based on your system configuration and the details of the error.

[Francisco José]

I installed MATLAB R2024a last week, so it is the last version. It si running in a server with 256 GB of RAM and 8 GPUs, so it is not a problem of capacity. Regarding the GPU drivers, I'm not entirely sure about them, but the GPUs are NVIDIA Tesla M10 with drivers 545.23.08 and CUDA 12.3.

Answer 1

[Francisco José]

I contacted MathWorks Support and it was a problem with MathWorks Service Host librarys. I installed libfort.so.0.4.2 and libfort.so.0.4.2 and created symbolic links, and it is now properly working.

$> cd $HOME/.MathWorks/ServiceHost/-mw_shared_installs/v2024.6.0.6/bin/glnxa64

$> ln -s libfort.so.0.4.2 libfort.so

$> ln -s libfort.so.0.4.2 libfort.so.0.4

[Peter]

Hi,

I apper to have the same issue. How did you install libfort?

Thanks.

[Umar]

Hi Peter,

To install libfort in Matlab, you can follow these steps:

Download the libfort library from a trusted source. Extract the downloaded files to a location on your computer. In Matlab, navigate to the folder where you extracted the libfort files using the cd command. Use the mex command in Matlab to compile the libfort source files.

For example: mex -v -I/path/to/libfort/include -L/path/to/libfort/lib -lfort your_file.c

Once the compilation is successful, you can use the libfort functions in your Matlab scripts.

Hope this will help you now to install libfort.

For more information regarding libfort, please refer to

https://pkgs.alpinelinux.org/contents?branch=edge&name=libfort&arch=x86&repo=testing

[Kevin]

I just resolved this same issue - required libfort files provided by support and multiple reinstalls. Make sure you killall -9 MATLAB and killall -9 MathWorksServiceHost, then remove everything in /usr/local/MATLAB.... and ~/.MathWorks, and any traces you find in /tmp. Then reinstall, place the libfort files, then launch MATLAB and login.

If you have this issue, I suggest contacting support and referencing Case 07117312 and this comment. MathWorks should really make a knowledgebase post, or at least push a fixed installer that doesn't have this issue.

[Umar]

Hi Kevin,

Sometimes the problem is resolved as a team working together. I am glad that you have shared your knowledge with us because this is the purpose of Mathworks to share your knowledge.

Happy 4th of July to all of you.