Hostname: page-component-77c89778f8-m42fx Total loading time: 0 Render date: 2024-07-21T08:04:19.889Z Has data issue: false hasContentIssue false
  • English
  • Français

Towards improving the robustness of sequential labeling models against typographical adversarial examples using triplet loss

Published online by Cambridge University Press:  04 February 2022

Can Udomcharoenchaikit Open the ORCID record for Can Udomcharoenchaikit [Opens in a new window] ,
Prachya Boonkwan  and
Peerapon Vateekul Open the ORCID record for Peerapon Vateekul [Opens in a new window]
Can Udomcharoenchaikit
Affiliation:
Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
Prachya Boonkwan
Affiliation:
Language and Semantic Technology Lab (LST), NECTEC, Pathumthani, Thailand
Peerapon Vateekul*
Affiliation:
Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Bangkok, Thailand
*
*Corresponding author. E-mail: peerapon.v@chula.ac.th
Get access Rights & Permissions [Opens in a new window]

Abstract

Many fundamental tasks in natural language processing (NLP) such as part-of-speech tagging, text chunking, and named-entity recognition can be formulated as sequence labeling problems. Although neural sequence labeling models have shown excellent results on standard test sets, they are very brittle when presented with misspelled texts. In this paper, we introduce an adversarial training framework that enhances the robustness against typographical adversarial examples. We evaluate the robustness of sequence labeling models with an adversarial evaluation scheme that includes typographical adversarial examples. We generate two types of adversarial examples without access (black-box) or with full access (white-box) to the target model’s parameters. We conducted a series of extensive experiments on three languages (English, Thai, and German) across three sequence labeling tasks. Experiments show that the proposed adversarial training framework provides better resistance against adversarial examples on all tasks. We found that we can further improve the model’s robustness on the chunking task by including a triplet loss constraint.

Keywords

TaggingEvaluationPart-of-speech taggingInformation extraction
Type
Article
Information
Natural Language Engineering , Volume 29 , Issue 2 , March 2023 , pp. 287 - 315
Check for updates
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Akbik, A., Bergmann, T. and Vollgraf, R. (2019). Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 724–728.CrossRefGoogle Scholar
Akbik, A., Blythe, D. and Vollgraf, R. (2018). Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 1638–1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.Google Scholar
Belinkov, Y. and Bisk, Y. (2018). Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations.Google Scholar
Bilmes, L. (1995). The grammaticalization of thai’come’and’go’. In Annual Meeting of the Berkeley Linguistics Society, volume 21, pp. 33–46.CrossRefGoogle Scholar
Bodapati, S., Yun, H. and Al-Onaizan, Y. (2019). Robustness to capitalization errors in named entity recognition. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), Hong Kong, China. Association for Computational Linguistics, pp. 237–242.CrossRefGoogle Scholar
Bojanowski, P., Grave, E., Joulin, A. and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, 135–146.CrossRefGoogle Scholar
Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 4171–4186.Google Scholar
Ebrahimi, J., Rao, A., Lowd, D. and Dou, D. (2018). Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),pp. 31–36.CrossRefGoogle Scholar
El Boukkouri, H., Ferret, O., Lavergne, T., Noji, H., Zweigenbaum, P. and Tsujii, J. (2020). CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online). International Committee on Computational Linguistics, pp. 6903–6915.CrossRefGoogle Scholar
Gao, J., Lanchantin, J., Soffa, M.L. and Qi, Y. (2018). Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 50–56.CrossRefGoogle Scholar
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020). Evaluating nlp models via contrast sets. arXiv preprint arXiv:2004.02709.Google Scholar
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N.F., Peters, M., Schmitz, M. and Zettlemoyer, L.S. (2017). Allennlp: A deep semantic natural language processing platform.CrossRefGoogle Scholar
Goodfellow, I., Shlens, J. and Szegedy, C. (2015). Explaining and harnessing adversarial examples. In International Conference on Learning Representations.Google Scholar
Heigold, G., Varanasi, S., Neumann, G. and van Genabith, J. (2018). How robust are character-based word embeddings in tagging and MT against wrod scramlbing or randdm nouse? In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), Boston, MA. Association for Machine Translation in the Americas, pp. 68–80.Google Scholar
Heinzerling, B. and Strube, M. (2019). Sequence tagging with contextual and non-contextual subword representations: A multilingual evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics, pp. 273–291.CrossRefGoogle Scholar
Huang, Z., Xu, W. and Yu, K. (2015). Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991.Google Scholar
Jayanthi, S.M., Pruthi, D. and Neubig, G. (2020). NeuSpell: A neural spelling correction toolkit. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 158–164, Online. Association for Computational Linguistics.Google Scholar
Jia, R. and Liang, P. (2017). Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark. Association for Computational Linguistics, pp. 2021–2031.CrossRefGoogle Scholar
Jiang, Y., Hu, C., Xiao, T., Zhang, C. and Zhu, J. (2019). Improved differentiable architecture search for language modeling and named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics, pp. 3585–3590.CrossRefGoogle Scholar
Karpukhin, V., Levy, O., Eisenstein, J. and Ghazvininejad, M. (2019). Training on synthetic noise improves robustness to natural noise in machine translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), Hong Kong, China. Association for Computational Linguistics, pp. 42–47.CrossRefGoogle Scholar
Kumar, S., Garg, S., Mehta, K. and Rasiwasia, N. (2019). Improving answer selection and answer triggering using hard negatives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics, pp. 5911–5917.CrossRefGoogle Scholar
Lafferty, J.D., McCallum, A. and Pereira, F.C.N. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc, 282–289.Google Scholar
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K. and Dyer, C. (2016). Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, California. Association for Computational Linguistics, pp. 260–270.CrossRefGoogle Scholar
Ling, W., Dyer, C., Black, A.W., Trancoso, I., Fermandez, R., Amir, S., Marujo, L. and Lus, T. (2015). Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal. Association for Computational Linguistics,pp. 1520–1530.CrossRefGoogle Scholar
Liu, H., Zhang, Y., Wang, Y., Lin, Z. and Chen, Y. (2020). Joint character-level word embedding and adversarial stability training to defend adversarial text. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7–12, 2020. AAAI Press, pp. 8384–8391.CrossRefGoogle Scholar
Liu, Z., Tang, B., Wang, X. and Chen, Q. (2017). De-identification of clinical notes via recurrent neural network and conditional random field. Journal of Biomedical Informatics, 75, S34–S42. Supplement: A Natural Language Processing Challenge for Clinical Records: Research Domains Criteria (RDoC) for Psychiatry. Google Scholar
Ma, X. and Xia, F. (2014). Unsupervised dependency parsing with transferring distribution via parallel guidance and entropy regularization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, Maryland. Association for Computational Linguistics, pp. 1337–1348.CrossRefGoogle Scholar
Michel, P., Li, X., Neubig, G. and Pino, J. (2019). On evaluation of adversarial perturbations for sequence-to-sequence models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 3103–3114.CrossRefGoogle Scholar
Minegishi, M. (2011). Description of thai as an isolating language. Social Science Information 50(1), 62–80.CrossRefGoogle Scholar
Miyato, T., Dai, A.M. and Goodfellow, I.J. (2017). Adversarial training methods for semi-supervised text classification. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.Google Scholar
Nebhi, K., Bontcheva, K. and Gorrell, G. (2015). Restoring capitalization in #tweets. In Proceedings of the 24th International Conference on World Wide Web, WWW 2015 Companion, New York, NY, USA. Association for Computing Machinery, 1111–1115.CrossRefGoogle Scholar
Nguyen, N. and Guo, Y. (2007). Comparisons of sequence labeling algorithms and extensions. In Proceedings of the 24th International Conference on Machine Learning, ICML 2007, 681–688, New York, NY, USA. Association for Computing Machinery.CrossRefGoogle Scholar
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K. and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana. Association for Computational Linguistics, pp. 2227–2237.CrossRefGoogle Scholar
Petrov, S., Das, D. and McDonald, R. (2012). A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA), pp. 2089–2096.Google Scholar
Piktus, A., Edizel, N.B., Bojanowski, P., Grave, E., Ferreira, R. and Silvestri, F. (2019). Misspelling oblivious word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 3226–3234.CrossRefGoogle Scholar
Pruthi, D., Dhingra, B. and Lipton, Z.C. (2019). Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics, pp. 5582–5591.CrossRefGoogle Scholar
Ratinov, L. and Roth, D. (2009). Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), Boulder, Colorado. Association for Computational Linguistics, pp. 147–155.CrossRefGoogle Scholar
Schroff, F., Kalenichenko, D. and Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823.CrossRefGoogle Scholar
Sennrich, R., Haddow, B. and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany. Association for Computational Linguistics, pp. 1715–1725.CrossRefGoogle Scholar
Sornlertlamvanich, V., Takahashi, N. and Isahara, H. (1998). Thai part-of-speech tagged corpus: Orchid. In Proceedings of the Oriental COCOSDA Workshop, pp. 131–138.Google Scholar
Straková, J., Straka, M. and Hajic, J. (2019). Neural architectures for nested NER through linearization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5326–5331, Florence, Italy. Association for Computational Linguistics.CrossRefGoogle Scholar
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. and Fergus, R. (2014). Intriguing properties of neural networks. In International Conference on Learning Representations.Google Scholar
Telljohann, H., Hinrichs, E., KÜbler, S. and KÜbler, R. (2004). The tÜba-d/z treebank: Annotating german with a context-free backbone. In In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004). Citeseer.Google Scholar
Telljohann, H., Hinrichs, E.W., KÜbler, S., Zinsmeister, H. and Beck, K. (2005). Stylebook for the tÜbingen treebank of written german (tÜba-d/z). In Seminar fÜr Sprachwissenschaft, Universität Tübingen, Germany. Google Scholar
Tirasaroj, N. and Aroonmanakun, W. (2009). Thai named entity recognition based on conditional random fields. In 2009 Eighth International Symposium on Natural Language Processing, pp. 216–220.CrossRefGoogle Scholar
Tjong Kim Sang, E.F. and De Meulder, F. (2003). Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003,pp. 142–147.CrossRefGoogle Scholar
Uzuner, Z., Luo, Y. and Szolovits, P. (2007). Evaluating the State-of-the-Art in Automatic De-identification. Journal of the American Medical Informatics Association 14(5), 550–563.CrossRefGoogle ScholarPubMed
Wallace, E., Feng, S., Kandpal, N., Gardner, M. and Singh, S. (2019). Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics, pp. 2153–2162.CrossRefGoogle Scholar
Xin, Y., Hart, E., Mahajan, V. and Ruvini, J.-D. (2018). Learning better internal structure of words for sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics, pp. 2584–2593.CrossRefGoogle Scholar
Yasunaga, M., Kasai, J. and Radev, D. (2018). Robust multilingual part-of-speech tagging via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana. Association for Computational Linguistics, pp. 976–986.CrossRefGoogle Scholar
Zhou, J.T., Zhang, H., Jin, D., Peng, X., Xiao, Y. and Cao, Z. (2019). Roseq: Robust sequence labeling. IEEE Transactions on Neural Networks and Learning Systems.CrossRefGoogle Scholar