How to add epochs to a previously trained CNN

Hi, I am have constructed a CNN to classify images. I have a dataset of 226,330 images and want to train my network using 100 epochs. MATLAB crashes and throws an out of memory error after the 9th epoch. So I decided to train the network sequentially. Where I train on 5 epochs, save the trained network's weights and biases for all convolutional and fully connected layers, then use those weights and biases to initialize all the weights and biases of the layers in the following 5 epochs. Does this make sense? The problem is that after the first 5 epochs are done, the network reaches an accuracy of almost 93% as shown below.

However, the next 5 epochs accuracy start from 48%, as shown below

. I was expecting it to be around 80-90%. I would appreciate any help. Thanks