Skip to main content
`Ctrl`+`K`

# 7.8.16. `TokenRegexp`

## 7.8.16.1. Summary

Added in version 5.0.1.

Caution

This tokenizer is experimental. Specification may be changed.

Caution

This tokenizer can be used only with UTF-8. You can’t use this tokenizer with EUC-JP, Shift_JIS and so on.

`TokenRegexp` is a tokenizer for supporting regular expression search by index.

## 7.8.16.2. Syntax

`TokenRegexp` hasn’t parameter:

    TokenRegexp

## 7.8.16.3. Usage

In general, regular expression search is evaluated as sequential search. But the following cases can be evaluated as index search:

> - Literal only case such as `hello`

> - The beginning of text and literal case such as `\A/home/alice`

> - The end of text and literal case such as `\.txt\z`

In most cases, index search is faster than sequential search.

`TokenRegexp` is based on bigram tokenize method. `TokenRegexp` adds the beginning of text mark (`U+FFEF`) at the begging of text and the end of text mark (`U+FFF0`) to the end of text when you index text:

Execution example:

    tokenize TokenRegexp "/home/alice/test.txt" NormalizerAuto --mode ADD
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "￯",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "/h",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ho",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "om",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "me",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "e/",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "/a",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "al",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "li",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ic",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ce",
    #       "position": 10,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "e/",
    #       "position": 11,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "/t",
    #       "position": 12,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "te",
    #       "position": 13,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "es",
    #       "position": 14,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "st",
    #       "position": 15,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "t.",
    #       "position": 16,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": ".t",
    #       "position": 17,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "tx",
    #       "position": 18,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "xt",
    #       "position": 19,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "t",
    #       "position": 20,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "￰",
    #       "position": 21,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]