Handling issues with unknown words in POS Tagging

Abstract

Appearance of unknown words is one of the frequently occurring problems facing in part of speech tagging process, i.e., the words that appear in sentences, but are not contained within the lexicon. New words are continually coined to the language, and people will often use words that a parsing system may not expect. This problem get worse when NLP systems are used for more and more on-line computer applications. Acronyms and proper names are created very often and new nouns and verbs are adding to the language in a surprising rate. So it is impossible to train the tagger for every possible word in the language. So unknown words are non-negligible in part of speech tagging and such unknown words should be handled by further processing with exceptional mechanism, since the statistical information or rules for those words are unknown. Therefore, in order to build a complete tagger, tagger must be incurred with some knowledge of suggesting the tag for an unknown word.