Skip to main content
`Ctrl`+`K`

# 7.8.13. `TokenMecab`

## 7.8.13.1. Summary

`TokenMecab` is a tokenizer based on MeCab part-of-speech and morphological analyzer.

MeCab doesn’t depend on Japanese. You can use MeCab for other languages by creating dictionary for the languages. You can use NAIST Japanese Dictionary for Japanese.

You need to install an additional package to using TokenMecab. For more detail of how to installing an additional package, see Install .

`TokenMecab` is good for precision rather than recall. You can find `東京都` and `京都` texts by `京都` query with TokenBigram but `東京都` isn’t expected. You can find only `京都` text by `京都` query with `TokenMecab`.

If you want to support neologisms, you need to keep updating your MeCab dictionary. It needs maintain cost. (TokenBigram doesn’t require dictionary maintenance because TokenBigram doesn’t use dictionary.) mecab-ipadic-NEologd : Neologism dictionary for MeCab may help you.

## 7.8.13.2. Syntax

`TokenMecab` has optional parameter.

No options:

    TokenMecab

Specify option:

    TokenMecab("include_class", true)

    TokenMecab("target_class", "a_part_of_speech")

    TokenMecab("include_reading", true)

    TokenMecab("include_form", true)

    TokenMecab("use_reading", true)

Specify multiple options:

    TokenMecab("target_class", "名詞", "include_reading", true)

`TokenMecab` also specify multiple options as above. You can also specify mingle multiple options except above example.

## 7.8.13.3. Usage

## 7.8.13.4. Simple usage

Here is an example of `TokenMeCab`. `東京都` is tokenized to `東京` and `都`. They don’t include `京都`:

Execution example:

    tokenize TokenMecab "東京都"
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "東京",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "都",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

`TokenMecab` can also specify options. `TokenMecab` has `target_class` option, `include_class` option, `include_reading` option, `include_form` option and `use_reading` option.

`target_class` option searches a token of specifying a part-of-speech. For example, you can search only a noun as below.

Execution example:

    tokenize 'TokenMecab("target_class", "名詞")' '彼の名前は山田さんのはずです。'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "彼",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "名前",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "山田",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "さん",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "はず",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

`include_class` option outputs class and subclass in MeCab’s metadata as below.

Execution example:

    tokenize 'TokenMecab("include_class", true)' '彼の名前は山田さんのはずです。'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "彼",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "名詞",
    #         "subclass0": "代名詞",
    #         "subclass1": "一般"
    #       }
    #     },
    #     {
    #       "value": "の",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "助詞",
    #         "subclass0": "連体化"
    #       }
    #     },
    #     {
    #       "value": "名前",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "名詞",
    #         "subclass0": "一般"
    #       }
    #     },
    #     {
    #       "value": "は",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "助詞",
    #         "subclass0": "係助詞"
    #       }
    #     },
    #     {
    #       "value": "山田",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "名詞",
    #         "subclass0": "固有名詞",
    #         "subclass1": "人名",
    #         "subclass2": "姓"
    #       }
    #     },
    #     {
    #       "value": "さん",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "名詞",
    #         "subclass0": "接尾",
    #         "subclass1": "人名"
    #       }
    #     },
    #     {
    #       "value": "の",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "助詞",
    #         "subclass0": "連体化"
    #       }
    #     },
    #     {
    #       "value": "はず",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "名詞",
    #         "subclass0": "非自立",
    #         "subclass1": "一般"
    #       }
    #     },
    #     {
    #       "value": "です",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "助動詞"
    #       }
    #     },
    #     {
    #       "value": "。",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "class": "記号",
    #         "subclass0": "句点"
    #       }
    #     }
    #   ]
    # ]

You can exclude needless token with `target_class` and class and sub class of this option outputs.

`include_reading` outputs reading in MeCab’s metadata as below.

Execution example:

    tokenize 'TokenMecab("include_reading", true)' '彼の名前は山田さんのはずです。'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "彼",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "カレ"
    #       }
    #     },
    #     {
    #       "value": "の",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ノ"
    #       }
    #     },
    #     {
    #       "value": "名前",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ナマエ"
    #       }
    #     },
    #     {
    #       "value": "は",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ハ"
    #       }
    #     },
    #     {
    #       "value": "山田",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ヤマダ"
    #       }
    #     },
    #     {
    #       "value": "さん",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "サン"
    #       }
    #     },
    #     {
    #       "value": "の",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ノ"
    #       }
    #     },
    #     {
    #       "value": "はず",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ハズ"
    #       }
    #     },
    #     {
    #       "value": "です",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "デス"
    #       }
    #     },
    #     {
    #       "value": "。",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "。"
    #       }
    #     }
    #   ]
    # ]

You can get reading of a token with this option.

`include_form` outputs inflected_type, inflected_form and base_form in MeCab’s metadata as below.

Execution example:

    tokenize 'TokenMecab("include_form", true)' '彼の名前は山田さんのはずです。'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "彼",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "彼"
    #       }
    #     },
    #     {
    #       "value": "の",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "の"
    #       }
    #     },
    #     {
    #       "value": "名前",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "名前"
    #       }
    #     },
    #     {
    #       "value": "は",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "は"
    #       }
    #     },
    #     {
    #       "value": "山田",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "山田"
    #       }
    #     },
    #     {
    #       "value": "さん",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "さん"
    #       }
    #     },
    #     {
    #       "value": "の",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "の"
    #       }
    #     },
    #     {
    #       "value": "はず",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "はず"
    #       }
    #     },
    #     {
    #       "value": "です",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "inflected_type": "特殊・デス",
    #         "inflected_form": "基本形",
    #         "base_form": "です"
    #       }
    #     },
    #     {
    #       "value": "。",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "base_form": "。"
    #       }
    #     }
    #   ]
    # ]

`use_reading` supports a search by kana. This option is useful for countermeasure of orthographical variants because it searches with kana.

Execution example:

    tokenize 'TokenMecab("use_reading", true)' '彼の名前は山田さんのはずです。'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "カレ",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ノ",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ナマエ",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ハ",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ヤマダ",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "サン",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ノ",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ハズ",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "デス",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "。",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

## 7.8.13.5. Advanced usage

`target_class` option can also specify subclasses and exclude or add specific part-of-speech of specific using + or -. So, you can also search a noun with excluding non-independent word and suffix of person name as below.

In this way you can search exclude the noise of token.

Execution example:

    tokenize 'TokenMecab("target_class", "-名詞/非自立", "target_class", "-名詞/接尾/人名", "target_class", "名詞")' '彼の名前は山田さんのはずです。'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "彼",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "名前",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "山田",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

In addition, you can get reading of a token that exclude the noise with `include_reading` option as below.

Execution example:

    tokenize 'TokenMecab("target_class", "-名詞/非自立", "target_class", "-名詞/接尾/人名", "target_class", "名詞", "include_reading", true)' '彼の名前は山田さんのはずです。'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "彼",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "カレ"
    #       }
    #     },
    #     {
    #       "value": "名前",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ナマエ"
    #       }
    #     },
    #     {
    #       "value": "山田",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ヤマダ"
    #       }
    #     }
    #   ]
    # ]

## 7.8.13.6. Parameters

### 7.8.13.6.1. Optional parameter

There are four optional parameters `include_class` , `target_class` , `include_form` and `use_reading` .

#### 7.8.13.6.1.1. `include_class`

Outputs class and subclass in MeCab’s metadata.

#### 7.8.13.6.1.2. `target_class`

Outputs a token of specifying a part-of-speech.

#### 7.8.13.6.1.3. `include_reading`

Outputs reading in MeCab’s metadata.

#### 7.8.13.6.1.4. `include_form`

Outputs inflected_type, inflected_form and base_form in MeCab’s metadata.

#### 7.8.13.6.1.5. `use_reading`

Outputs reading of token.

## 7.8.13.7. See also

- tokenize