LSTM Layer Architecture: LSTM units and sequence length

I am a researcher of human motion analysis. In my work, I want to build a LSTM neural network. After some trying, my net can work. But I want to check some things about the function, "LSTMlayer(outputSize)".

My data is a sequence that have 1000 frames. And I think that LSTM is not appropriate for being longer than 500, so I set the outputSize as 200-400. But I am not sure whether the "outputSize" is the same as "time step" in matlab. I had search lots of page and information. But I did not find any information about it.

Also, if the "time step" is shorter than the length of sequence input, the LSTM Architecture is like a window and slide through all data? Or the all sequence will input into the LSTM units (one LSTM unit receive one more frame)?

Thanks for any answers or discussion!!

Answer 1

[Maria Duarte Rosa]

Hi Otto,

The outputSize of a LSTM layer is not directly related to a time window that slides through the data. The entire sequence runs through the LSTM unit. The outputSize is more like a complexity parameter, where a larger outputSize will allow the network to learn more complex recurrent patterns from the data, while being more prone to overfitting. A smaller outputSize will not be able to learn very complex patterns but will be less prone to overfit. There isn't a hard rule to set the outputSize parameter as this is highly problem specific. Please see more information on how to use LSTM networks in MATLAB in the following links:

https://www.mathworks.com/help/nnet/ug/long-short-term-memory-networks.html

https://uk.mathworks.com/help/nnet/examples/classify-sequence-data-using-lstm-networks.html ml

[Otto Mo]

Hi, Rosa. Firstly thank you for the answer.

But I am still confused by this picture (attachment: lstmflow.png).

It seems that the length of time steps is the same as the length of X (so time step in my data is 1000?). And one time step corresponds to one LSTM unit, who corresponds to one output Yi.

I am not sure the outputSize means the size of the Y-sequence or just to the size of another hidden layer.

For all, I want to check that the relation between "InputSize", "OutputSize", and "Time Steps."

I had tried to read all the document from matlab (web and local doc). I still could not understand the setting of LSTMLayer in MATLAB. I really want to figure out it!

Thank you again and if possible, please help me.

Answer 2

[Y Q]

I have the same confusion. My understanding is the outputSize is dimensions of the output unit and the cell state. for example, if the input sequences have the dimension of 12*50 (50 is the time steps), outputSize is set to be 10, then the dimensions of the hidden unit and the cell state are 10*1, which don't have anything to do with the dimension of the input sequence. I have studied all the LiveScript example and found out the above conclusion. Is it correct?

What I still confusing is the parameter 'OutputMode: sequence/last'. if lstm layer is followed by a fully connected (FC) layer, the number of the input neurons in FC is equal to the outputSize set in the lstm layer. The change of the parameter 'OutputMode: sequence/last' doesn't change the dimensions of the hidden unit and the cell state, and also doesn't change the number of the weights in the FC layer. My question is when 'OutputMode' is set to be 'sequence' what is exactly passed from the last layer to the FC layer, the whole sequence of the hidden unit, with the dimension of 10*50(from the example I listed above)? if so, how can the FC layer handel this 10*50 data with a 10 input neurons?

[Ning Wang]

I have exactly the same question. Would you please give me some inspiration if you have solved this question.

[Asvin Kumar]

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.fullyconnectedlayer.html#mw_5c678e5e-5bd0-491b-9272-b6b72f65e417