Tying of embeddings for improving regularization in neural networks for named entity recognition task
DOI:
https://doi.org/10.17721/1812-5409.2018/3.8Abstract
We analyze neural network architectures that yield state of the art results on named entity recognition task and propose a new architecture for improving results even further. We have analyzed a number of ideas and approaches that researchers have used to achieve state of the art results in a variety of NLP tasks. In this work, we present a few of them which we consider to be most likely to improve existing state of the art solutions for named entity recognition task. The architecture is inspired by recent developments in language modeling task. The suggested solution is based on a multi-task learning approach. We incorporate part of speech tags as input for the network. Part of speech tags to be yielded by some state of the art tagger and also ask the network to produce those tags in addition to the main named entity recognition tags. This way knowledge distillation from a strong part of speech tagger to our smaller network is happening. We hypothesize that designing neural network architecture in this way improves the generalizability of the system and provide arguments to support this statement.
Key words: neural networks, named entity recognition, regularization, generalizability, multi-task learning.
Pages of the article in the issue: 59 - 64
Language of the article: Ukrainian
References
MA, X. & HOVY, E. (2016) End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . pp. 1064-1074.
HOCHREITER, S. & SCHMIDHUBER, J. (1997). Long short-term memory. Neural Computation. 9 : pp. 1735–1780.
LECUNN, Y., BENGIO, Y. (1995). Convolutional Networks for Image, Speech, and Time-Series. In The handbook of brain theory and neural networks. pp. 255-258.
WALLACH, H. (2004). Conditional Random Fields: An Introduction. In Foundations and Trends in Machine Learning archive . Volume 4 Issue 4, pp. 267-273
LAMPLE, G. & BALLESTEROS, M. & SUBRAMANIAN, S. & KAWAKAMI, K. & DYER, C. (2016). Neural Architectures for Named Entity Recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . pp. 260–270.
MELIS, G. & DYER, C. & BLUNSOM, P. (2017). On the state of the art of evaluation in neural language models. In International Conference on Learning Representations . pp. 214-216.
LAFFERTY, J., MCCALLUM, A., PEREIRA, F. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Proc. 18th International Conf. on Machine Learning. Morgan Kaufmann. pp. 282–289.
Downloads
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
