Analysis of pre-training and initialization methods of neural networks

B. V. Bobyl; V. M. Tereschenko

doi:10.17721/1812-5409.2018/4.12

Authors

B. V. Bobyl Taras Shevchenko National University of Kyiv
V. M. Tereschenko Taras Shevchenko National University of Kyiv

DOI:

https://doi.org/10.17721/1812-5409.2018/4.12

Abstract

In this paper we investigate main pre-training and initialization methods of parameter values of neural networks such as pre-training using restricted Boltzmann machines, deep autoencoders, Glorot and He initialization of parameters, transfer learning and domain adaptation. Given methods are useful for finding of appropriate parameter values and initial initialization of neural network, what is necessary condition for further efficient training of deep models, because it give a possibility during training to reduce negative effects such as vanishing or explosion of gradient, overfitting, stucking in one of local minimums of loss function, etc. These methods belong to group of unsupervised training algorithms and do not need any labeling for data which will be used later for model’s training after parameters initialization. Firstly, in this paper, we analyze all these methods and describe advantages and disadvantages of each of them. Secondly, we describe results of our experiments applying these methods for solving of classification task of MNIST dataset and introduce ideas for further development and improvement of these algorithms.

Key words: neural network, pre-training, restricted Boltzmann machines, contrastive divergence algorithm, autoencoder, Glorot initialization, He initialization, domain adaptation, Jaccard index.

Pages of the article in the issue: 84-91

Language of the article: Ukrainian

References

VAN DER MAATEN, L. J. P. (2008) Visualizing High-Dimensional Data Using t-SNE. [Electronic resource] Journal of Machine Learning Research. Access: http://www.jmlr.org/papers/v9/vandermaaten08a.html.

GOODFELLOW, I., BENGIO, Y., COURVILLE, A., (2016) Deep Learning. Cambridge, MA: The MIT Press. – 800 p. – (Adaptive Computation and Machine Learning series).

BISHOP, C. M. (1998) Neural networks and machine learning. Berlin: Springer. – 353 p. – (Nato ASI Subseries F). 4. BISHOP, C. M. (2006) Pattern recognition and machine learning. New York: Springer-Verlag. – 738 p. – (Information Science and Statistics).

GLOROT, X., BENGIO, Y. (2010) Understanding the difficulty of training deep feedforward neural networks. [Electronic resource] Journal of Machine Learning Research. Access: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf.

HE, K., ZHANG, X., REN, S., SUN, S. (2015) Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. 2015 IEEE International Conference on Computer Vision (ICCV). – p. 1026–1034.

HINTON, G. E., SALAKHUTDINOV, R. R. (2016) Reducing the Dimensionality of Data with Neural Networks. New York: Science. – 313. – p. 504–507.

SALAKHUTDINOV, R. R., MNIH, A., HINTON, G. E. (2007) Restricted Boltzmann machines for collaborative filtering. ACM International Conference Proceeding Series. – 227. – p. 791–798.

CARREIRA-PERPINAN, M. A., HINTON, G. E. (2005) On Contrastive Divergence Learning. AISTATS 10th Int. Workshop on Artificial Intelligence and Statistics. – p. 59–66.

PRATT, L. Y., HANSON, S. J., GILES, C. L., COWAN, J. D., (1993) Discriminability-Based Transfer between Neural Networks. Advances in Neural Information Processing Systems. – 5. – p. 204–211.

VINCENT, P., LAROCHELE, H., LAJOIE, I., BENGIO, Y. (2010) Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. [Electronic resource] Journal of Machine Learning Research. Access: http://www.jmlr.org/papers/volume11/vincent10a/vincent10a.pdf.

HE, K., ZHANG, X., REN, S., SUN, S. (2016) Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV: IEEE, 2016. – p. 770–778.