Too many parameters (fc layers) in both cnn encoder and rnn decoder, causing dramatic overfitting!

[mashijie1028]

There are so many fc layers in both CNN encoder and RNN decoder, only one is enough. When I implement the CRNN training, I got over 70% test acc with only one fc layer in both CNN and LSTM (However, there is still a huge overfitting). When the num_fc_layers increases, the performance degrades.

Plus, BatchNorm probably contradicts with dropout, because dropout could affect the statistics of BN, BN is already a regularizer. Maybe no dropout is better.

[mashijie1028]

I was wondering how you could get 85.68% test acc in ResNet-152 + LSTM, could you please tell me the hyper-parameters? Thanks!
@HHTseng

[mashijie1028]

I use ResNet-18(pretrained) + LSTM and get over 80% test acc, but only 40% test acc when training ResNet-18 + LSTM from scratch. It seems that pretraining ResNet CNN encoder on ImageNet is essential.