Japanese Input Processors Chain
Introduction
An Input Processor (IP) pre-processes inputs for the Teneo Engine to be able to perform different processes on them, such as normalization and tokenization of inputs or spelling correction. Each language supported by the Teneo Platform has a chain of Input Processors which know how to process that particular language.
Japanese was the first language of the Teneo Platform to fully take advantage of the possibility to customize user input processing according to the Teneo Engine architecture and while more and more language take advantage of this approach, the Japenese Input Processors keep evolving.
The Japanese Input Processors chain implemented in Teneo 5.1.1 is backwards incompatible with the Japanese Teneo Lexical Resource and solutions of former Teneo versions due to different tokenization.
IP Chain Setup
The below graph displays the default setup of the Japanese Input Processors chain.
The default Input Processors are listed below with a short description of the input processor's functionality; the following sections and sub-sections also provide further details.
- Japanese Tokenizer: performs sentence segmentation and tokenization.
- Japanese Annotator: creates annotations related to lemma, Part-of-Speech (POS), morphosyntactic information and named entities.
- Japanese Number Recognizer: recognizes the following types of number expressions: Arabic numbers, formal and colloquial Kanji numbers, Hiragana numbers, numbers with counters not split from the actual numeric expression, numbers with factors both larger and smaller than zero, decimal numbers and fractions.
- System Annotation: sets a number of annotations based on properties of the user input text.
- Language Detector: identifies the language of the input sentence provided and annotates it with the predicted language together with a confidence score.
- Predict: classifies user inputs based on a machine learning model and annotates the user input with the predicted top intent classes and a confidence score; note that as of Teneo 7.3, deferred intent classification is applied.
- Date Time Recognizer: distinguishes and annotates various date and/or time expressions which are used by Language Objects to support the date and time interpretation.
Japanese Simplifier
The Japanese Simplifier is a special kind of processors that is used to normalize the user input by:
- converting full width Latin letters and Arabic digits into their half width version, and
- lowercasing the uppercase Latin letters.
This Simplifier is special because it is not run as part of the Input Processors chain, but rather by the Tokenizer when it puts the tokens generated by Kuromoji into a Teneo data structure. Additionally, the Simplifier is also run by the condition parser inside the Teneo Engine, which normalizes the Language Object syntax words before adding them to the internal Engine dictionary.
Input Processors
Japanese Tokenizer
The Japanese Tokenizer, which performs sentence segmentation and tokenization, runs Kuromoji (Japanese tokenizer) on raw input strings and then processes the tokens returned by Kuromoji into words and sentences for Teneo. Since the tokenization of Kuromoji is too aggressive for the purpose of Teneo, the processing of the tokens involves a set of hard-coded rules that concatenate some of the tokens into bigger units.
Please note the distinction in the terminology:
A token is a object returned by Kuromoji and an element of the original presentation.
In the Teneo Platform, the words that are used in the Platform are referred to as words, which might be the concatenation of several Kuromoji tokens.
In the exceptional case of the Japanese interpunct symbol “·”, the Japanese Tokenizer also splits tokens from Kuromoji.
The concatenation is done by a separate helper class Japanese Concatenator which is instantiated for each input. The functionality of concatenation and sentence segmentation is exclusively implemented in that class. The concatenation rules are hard-coded and process a sequence from left to right, deciding whether the observed tokens should be concatenated and if a new sentence should be started.
When a word object is created, the features from the Kuromoji tokens that form part of that word are passed into the property map. Those features are then retrieved from the property map of each word by the Japanese Annotator.
The concatenation of the Japanese Tokenizer can be overridden by introducing entries into the solution dictionary (i.e. in a language object) that follow the below pattern:
DICTEXT_tok_lemma
In other words, entries that have the prefix DICTEXT will be considered by the concatenator and the token tok will never be concatenated, i.e. will be a standalone word that will have the lemma annotation lemma.
Splitting of Tokens with Numbers
As of Teneo 6, in order to cater for date and time recognition in Japanese, the Japanese Tokenizer splits tokens that contain slashes, dashes, tildes, colons, commas, dots and interpuncts in certain contexts, as detailed below:
- Splits numbers from slashes and keep the slash as a separate token if there are at least two slashes in-between numbers, e.g. 25/04/2020, but not when there is a single slash between numbers as that is a fraction that the Number Recognizer should recognize, e.g. 2/3.
- Splits numbers from dashes, tildes and interpuncts and keep them as separate tokens, e.g. 25 - 04 - 2020, 25 ~ 04 ~ 2020, 25 ・ 04 ・ 2020.
- Splits numbers from dots and keep the dots as separate tokens, e.g. 25 . 04 . 2020; but not when there is a single dot between numbers as in that case the dot could be a decimal marker for numbers, e.g. 1.5.
- Splits numbers from comma and keep the comma as a separate token, e.g. 2,3; but not when there are three digits after the comma as in that case the comma could be a thousands separator for numbers, e.g. 1,000.
- Splits numbers from colons and keep the colon as a separate token, e.g. 10 : 30
- Splits special number tokens that Kuromoji doesn't split, e.g. ２人 or ２，３.
Configuration Properties
Non Word Token
| Name | Type | Required | Default |
|---|---|---|---|
| nonWordToken | String | no | " “ ” \r 『』 ' 「」( ) [ ] { }（）〔〕［］｛｝〈〉《》 ! ！? ？… , ，、. ．。｡・〚〛〘〙〖〗【】｟｠＇＂ |
List of characters that will not be output as a word.
Sentence Delimiters
| Name | Type | Required | Default |
|---|---|---|---|
| sentenceDelimiters | String | no | ! ！? ？ . ．。｡ … |
List of characters that delimit a sentence.
Decimal Points
| Name | Type | Required | Default |
|---|---|---|---|
| decimalPoints | String | no | ． . . 点 ・ |
List of characters that is used as a decimal point in numbers.
No Separation Regex
| Name | Type | Required | Default |
|---|---|---|---|
| noSeparationRegEx | String | no | (?:\w+(?:[-+.']\w+)*@\w+(?:[-.]\w+)*\.\w+(?:[-.]\w+)*)| (?iuU:(?:(?:(?:https?|ftp|file)://)|www\.)[-\p{IsLatin}0-9+&@#/%?=~_|!:,.;]*[-\p{IsLatin}0-9+&@#/%=~_|])* |
Regex for sequences that will not be split if the regex convers the full surface forms of multiple Kuromoji tokens. The default covers most URLs and emails.
Emojis
| Name | Type | Required | Default |
|---|---|---|---|
| Emojis | String | no | [\x{1f000}-\x{1f7ff}\x{2600}-\x{27ff}\x{1f900}-\x{1f9e6}\] |
Regex to detect emojis that will be tokenized into individual words.
Japanese Annotator
The Japanese Annotator processes each word one by one and for each Kuromoji token that forms part of a word, i.e., that form part of the concatenation, it goes through the list of annotation rules defined in the property JapaneseAnnotator.properties and produces the annotations if the rules match on the context of the current token.
The context of the current token can be the features themselves returned by Kuromoji for that token, such as the POS tag or the lemma, or they can be the annotations that were assigned to the previous token in the same word, if applicable. Note that the context of the first token contains NULL as previous annotation.
The Japanese Annotator creates annotations of the following types:
- Lemma (the lemma of a word is provided as an annotation if available)
- Part-of-Speech (POS) information
- Morphosyntactic information
- Named-entity information.
Configuration Properties
The Japanese Annotations can be configured by two types of properties: annotation and annotationRegex properties.
Annotation
The value of annotation properties for the Japanese Annotator need to follow the below syntax:
<feature 1>,<feature 2>,<... feature N>=<annotation 1>-<annotation 2>-<... annotation M>|<remove 1>-<remove 2>-<... remove M>
This means that a property key is defined by a list of features separated by a comma. The order of the features is irrelevant and a feature can be any of the features provided by Kuromoji, such as POS tags or lemmas, or any of the annotations of the previous token of the same word.
The right-hand side of the property is separated by the equals (=) character and split into two parts, separated by a vertical line (|) character. The left part indicates which annotations should be produced if the property key applies to the current token. The right part is a list of annotations that will be removed for the current word if they were previously produced for the same word by a previous rule or in a previous token. Note that therefore the ordering of the properties matters.
The following is an example of two annotation properties:
annotation.33 = 助詞,接続助詞=FW.POS-PARTICLE.POS
annotation.34 = 助動詞,助動詞-ダ,連体形-一般=FW.POS-PARTICLE.POS|COPULA.POS-VB.POS
The first property (33) in the above example produces the annotations FW.POS and PARTICLE.POS. The second property (34) produces the same annotations and at the same time removes the annotations COPULA.POS and VB.POS from the current word if they exist.
Annotation Regex
The annotation regex properties map entire words to annotations and their values need to follow the below syntax:
<regex>=<annotation 1>-<annotation 2>-<... annotation M>
Note that the regex needs to match on the entire word. The annotation regex rules will be executed before the annotation properties. If the regex matches, its corresponding annotations are assigned to the word. No other annotations will be assigned after a regex matches. Therefore, the order of the annotation regex is relevant.
The following is an example of an annotation regex property that matches on e-mails and produces two annotations: EMAIL.NER and NN.POS.
annotationRegex.2 = \\w+([-+.']\\w+)*@\\w+([-.]\\w+)*\\.\\w+([-.]\\w+)*=EMAIL.NER-NN.POS
Annotations Available in Teneo Studio
The following Part-of-Speech (POS), morphosyntactic and named-entity annotations may be generated by the Japanese Annotator according to the default properties.
A token can have multiple annotations and each annotation carry one of the following suffixes in order to distinguish their type: POS (Part-of-Speech), MST (Morphosyntactic) or NER (Named Entity).
For information related to the ANNOT Language Objects available in the Japanese Lexical Resource, please see the lists of POS/MST ANNOT language objects and/or NER ANNOT language objects for Japanese.
| Annotation | Type |
|---|---|
| ADJ.POS | POS |
| ADV.POS | POS |
| CARDINAL.POS | POS |
| CONJ.POS | POS |
| COPULA.POS | POS |
| COUNTER.POS | POS |
| DET.POS | POS |
| FW.POS | POS |
| INTERJ.POS | POS |
| NN.POS | POS |
| PARTICLE.POS | POS |
| PREFIX.POS | POS |
| PREP.POS | POS |
| PRON.POS | POS |
| PROPER.POS | POS |
| SUFFIX.POS | POS |
| SYM.POS | POS |
| VB.POS | POS |
| ALMOST.MST | MST |
| ASSUMPTION.MST | MST |
| CAUSATIVE.MST | MST |
| DASU.MST | MST |
| DESIRE.MST | MST |
| EXCESS.MST | MST |
| FORMAL.MST | MST |
| GARU.MST | MST |
| GERUND.MST | MST |
| HAJIMERU.MST | MST |
| IMPERATIVE.MST | MST |
| ITERATIVE.MST | MST |
| KANERU.MST | MST |
| KIRU.MST | MST |
| NEGATION.MST | MST |
| OERU.MST | MST |
| OWARU.MST | MST |
| PASSIVE.MST | MST |
| PAST.MST | MST |
| PROGRESSIVE.MST | MST |
| RENYOKEI.MST | MST |
| TEAGERU.MST | MST |
| TEIKU.MST | MST |
| TEITADAKU.MST | MST |
| TEKUDASARU.MST | MST |
| TEKURERU.MST | MST |
| TEMIRU.MST | MST |
| TEMORAU.MST | MST |
| TEOKU.MST | MST |
| TESHIMAU.MST | MST |
| TEYARU.MST | MST |
| VOLITION.MST | MST |
| YAGARU.MST | MST |
| EMAIL.NER | NER |
| LOCATION.NER | NER |
| PERSON.NER | NER |
| URL.NER | NER |
Japanese Number Recognizer
The Japanese Number Recognizer is capable of recognizing the following types of number expressions:
- Arabic numbers
- Formal and colloquial Kanji numbers
- Hiragana numbers
- Numbers with counters not split from the actual numeric expression
- Numbers with factors both larger and smaller than zero
- Decimal numbers
- Fractions.
When a number expression is detected in a user input, the following annotation is created with a variable which holds the found number.
| Annotation | Variable | Description |
|---|---|---|
| NUMBER | numericValue | Annotation created for identified numbers |
System Annotation
The System Annotation Input Processor performs simple analysis of the sentence texts to set some annotations and the decision algorithms are configurable by various properties.
Further customization is possible by sub-classing this Input Processor and overriding one or more of the following methods: decideBinary, decideBrackets, decideEmpty, decideExclamation, decideNonsense, decideQuestion, decideQuote.
This Input Processor works on the sentences passed in, but does not modify them.
Several configuration properties are available for the System Annotation Input Processor; please see the details here.
Other Considerations
Extra request parameters read by this input processor: (none)
Processing options read by this input processor: (none)
Annotations this input processor may generate:
| Annotation | Description |
|---|---|
| _BINARY | The input consists only of characters specified by the properties binaryCharacters (at least one of them) and binaryIgnoredCharacters (zero or more of them) |
| _BRACKETPAIR | The input contains at least one matching pair of the bracket characters specified under the property bracketPairCharacters |
| _EXCLAMATION | The input contains at least one of the characters specified for the property exclamationMarkCharacters |
| _EM3 | The input contains three or more characters in a row of the characters specified with exclamationMarkCharacters |
| _EMPTY | The input contains no text / the sentence text is empty |
| _NONSENSE | The input probably contains nonsense text as configured with the properties consonants, nonsenseThreshold.absolute and nonsenseThreshold.relative |
| _QUESTION | The input contains at least one of the characters specified in property questionMarkCharacters |
| _QT3 | The input contains three or more characters specified in property questionMarkCharacters |
| _QUOTE | The input contains at least one of the characters specified with property quoteCharacters |
| _DBLQUOTE | The input contains at least one of the characters specified with property doubleQuoteCharacters |
Special System Annotations
The following two, special annotations are set by the Teneo Engine. These special system annotations are not related to individual inputs but rather to whole dialogues and are dependent on the session state.
| Annotation | Description |
|---|---|
| _INIT | Indicates session start, i.e., the first input in a dialogue |
| _TIMEOUT | Indicates the continuation of a previously timed-out session/dialogue |
Language Detector
The Language Detector uses a machine learning model to predict the language of a given user input and adds an annotation, as seen in below table, to the input together with a confidence score of the prediction.
| Annotation | Variable | Description |
|---|---|---|
| <language label>.LANG, e.g., %$DA.LANG | Confidence | Annotation created for the predicted language |
The Language Detector can predict the following 45 languages; the language label used to create the annotation name is in brackets:
Arabic (AR), Bulgarian (BG), Bengali (BN), Catalan (CA), Czech (CS), Danish (DA), German (DE), Greek (EL), English (EN), Esperanto (EO), Spanish (ES), Estonian (ET), Basque (EU), Persian (FA), Finnish (FI), French (FR), Hebrew (HE), Hindi (HI), Hungarian (HU), Indonesian-Malay (ID_MS), Icelandic (IS), Italian (IT), Japanese (JA), Korean (KO), Lithuanian (LT), Latvian (LV), Macedonian (MK), Dutch (NL), Norwegian (NO), Polish (PL), Portuguese (PT), Romanian (RO), Russian (RU), Slovak (SK), Slovenian (SL), Serbian-Croatian-Bosnian (SR_HR), Swedish (SV), Tamil (TA), Telugu (TE), Thai (TH), Tagalog (TL), Turkish (TR), Urdu (UR), Vietnamese (VI) and Chinese (ZH).
Serbian, Bosnian and Croatian are treated as one language under the label SR_HR, and Indonesian and Malay are treated as one language under the label ID_MS
A number of regexes are also in use by the Input Processor, helping the model to not predict a language for fully numerical inputs, URLs or other type of nonsense inputs.
The Language Detector will provide an annotation when the confidence prediction threshold is above 0.2 for the languages, but for the following listed languages, language annotations are always created (even for predictions below 0.2) since the Language Detector is mostly accurate when predicting them: Arabic, Bengali, Greek, Hebrew, Hindi, Japanese, Korean, Tamil, Telugu, Thai, Chinese, Vietnamese, Persian and Urdu.
Predict
The Predict Input Processor makes use of an intent model generated when classes are available in a Teneo Studio solution to annotate user inputs with the defined classes; intent models can be generated either with Teneo Learn or CLU. Note that as of Teneo 7.3, deferred intent classification is applied and annotations are only created by Predict if references to class annotations are found during the input matching process.
When Predict receives a user input, confidence scores are calculated for each class based on the model and annotations created for the most confident class and for each other class that matches the following criteria:
- the confidence is above the minimum confidence (defaults to 0.01)
- the confidence is higher than 0.5 times the confidence value of the top class.
For each selected class, an annotation with the scheme <CLASS_NAME>.INTENT is created, with the value of the model's confidence in the class as well as an annotation variable specifying the used classifier (i.e., Learn, CLU or LearnFallback) and an Order variable defining the order of the selected classes (i.e., 0 for the class with the highest confidence score and 4 for the selected class with the lowest confidence score).
A special annotation <CLASS_NAME>.TOP_INTENT is created for the class with the highest confidence score.
| Annotation | Variable | Variable | Variable | Description |
|---|---|---|---|---|
| <CLASS_NAME>.TOP_INTENT | classifier | confidence | Annotation created for the class with the highest confidence score | |
| <CLASS_NAME>.INTENT | classifier | confidence | Order | Annotation given to each selected class with a maximum of five top classes |
The Predict Input Processor creates a maximum of 5 annotations, regardless of how many classes match the criteria.
Configuration Properties
Min Confidence Similarity Distance
| Name | Type | Required | Default |
|---|---|---|---|
| minConfidenceSimilarityDistance | float | no | 0.5 |
Confidence percentage of the top score confidence a class must have in order to be considered, e.g., if the top confidence class has a confidence of 0.7, classes with confidence lower than 0.5 x 0.7 = 0.35 will be discarded.
Max Number of Annotations
| Name | Type | Required | Default |
|---|---|---|---|
| maxNumberOfAnnotations | int | no | 5 |
The maximum number of class annotations created for each user input.
Min Confidence Threshold
| Name | Type | Required | Default |
|---|---|---|---|
| minConfidenceThreshold | float | no | 0.01 |
The minimum value of confidence a model must have for a class in order to add it as one of the candidate annotations.
Intent Model File Name
| Name | Type | Required | Default |
|---|---|---|---|
| ìntent.model.file.name | string (filename) | no | empty |
Name of the file containing the machine learning model; it is usually set automatically by Teneo Studio so no configuration is required.
Date Time Recognizer
The Date Time Recognizer available in the Japanese input processing chain recognizes and annotates various date and/or time expressions which are then used by language objects to support the date and time interpretation; the following annotations - with the listed variables - are created by the Date Time Recognizer.
| Annotation | Variables | Description | Examples |
|---|---|---|---|
| DATE.DATETIME | dmy (map), mdy (map), ymd (map), all with keys (int) day_of_month, month and year | Annotation for collated date expressions | "140219", "14.02.19", "3.4.2018", "03.04.2018", "20180403" |
| TIME.DATETIME | hour (int), minute (int), second (int), meridiem (string) | Annotation for collated time expressions | "15h30", "11pm", "30 sec" |
To read more about how to use the native understanding and interpretation of date and time expressions in the Teneo Platform, please see here.